Preporato

Track · AI security and red teaming

AI red teaming projects: attack and defend LLM apps

Attack a live RAG assistant and agent the way a red team would, from recon and prompt injection to MCP tool poisoning and agent worms, then build the defenses and measure them.

26
Labs
34 h
In total
Beginner to advanced
Level
1
Free
Start the free labIndirect Prompt Injection: Exfiltrate Data from a RAG Assistant · 75 min

What you will build

  • A threat model of an LLM app and a harness that measures attack success rate
  • Working prompt injection, system-prompt extraction and data exfiltration against a live RAG assistant
  • Agent hijacks through tool poisoning, shadowing, excessive agency and memory
  • Guardrails, a RAG firewall and an output mediator that close the same holes
  • A CI gate that fails the build when attack success rate goes up

Before you start

  • Python and comfort with HTTP requests
  • What a RAG assistant and a tool-calling agent are; the RAG and agent tracks build both

Tools you will use

garakOWASP LLM Top 10MITRE ATLASNeMo GuardrailsMCPMilvusLangGraph

Labs in this track

In order, from the first lab to the hardest. Every lab stands on its own, so start wherever you like.

1

Recon and first attacks

Threat-model the app, map the attack surface, then extract the system prompt and exfiltrate data through the retriever.

  1. # threat-model-llm-app · step 1
    $ lab.check(1)
    Step 1 Complete
    grade ........... pass
    Lab 1Threat-Model an LLM App: Derive OWASP LLM Threats from the Architecture, Rate Them, and Test the ModelTurn an AI assistant's architecture into a threat model in code: find the injection surface, derive OWASP LLM Top 10 threats from the components and data flows by rule, rate them by impact and exposure, list the threats current controls leave open, and test the model against real incidents, including the two an architecture-driven model cannot find.50 minIntermediateHostedPro
  2. Lab 2Recon and Harness: Map a RAG Attack Surface and Measure Attack-Success-RateOpen your AI Red Team engagement against a real Retrieval-Augmented Generation assistant and build the methodology the whole path reuses. Stand up the service and trace one request, enumerate its attack surface into a structured, machine-checkable map, encode a single probe, then build a deterministic side-channel oracle that counts real effects instead of the model's talk. Triage true positives from verbal-only false positives, scale to an Attack-Success-Rate harness with a per-class breakdown, wire it into a CI gate that goes red on the vulnerable build, then ship the render-path allow-list and watch the same gate go green while benign questions still answer.100 minBeginnerHostedPro
  3. Lab 3System-Prompt Extraction: Recover a RAG Assistant's Hidden InstructionsRed-team Aria, a real Retrieval-Augmented Generation support assistant: a Milvus vector store, NVIDIA embeddings, and an LLM that answers from one shared context window. Confirm a hidden system prompt with an embedded secret exists, recover it through ordinary chat by direct echo, climb the extraction ladder when a stronger refusal posture resists, defeat a naive output filter with encoding-egress, measure the extraction Attack-Success-Rate across the techniques, then ship the durable fix (minimize the secret out of the prompt) and verify extraction yields nothing.70 minIntermediateHostedPro
  4. Lab 4Sensitive Data Disclosure: Leak Confidential Records from a RAG AssistantAttack a real Retrieval-Augmented Generation assistant where the system prompt only asks for privacy: a Milvus vector store, NVIDIA embeddings, and a multi-tenant knowledge base. Force disclosure of your own gated billing secret, pull another customer's record across a disabled tenant filter, harvest an accidentally-indexed service key, then ship the real fix with pre-retrieval authorization, corpus hygiene, and output redaction.75 minIntermediateHostedPro
  5. Lab 5Indirect Prompt Injection: Exfiltrate Data from a RAG AssistantAttack a real Retrieval-Augmented Generation assistant end to end: a Milvus vector store, NVIDIA embeddings, and a multi-tenant knowledge base. Win semantic retrieval with a poisoned document, exfiltrate a customer's confidential account record through the EchoLeak markdown-image channel, measure attack-success-rate, bypass a naive defense, then ship the real fix.75 minAdvancedHostedFree
  6. Lab 6Retrieval Poisoning: Win Top-k Across a Whole Query Class and Steer the AnswerAttack a real Retrieval-Augmented Generation assistant where it is most exposed: retrieval. Plant one document in a Milvus + NVIDIA embeddings knowledge base, craft it to win cosine top-k for one account question, then for the whole account-query class, then steer the generated answer through a directive framed as routine policy. Measure broad-class coverage and steering attack-success-rate, then harden in two distinct moves: treat retrieved context as data behind a non-spoofable boundary, and cap how many top-k slots any single source may take. Re-run the same battery and watch attack-success-rate collapse.80 minAdvancedHostedPro
  7. Lab 7Cross-Tenant Leakage: Break RAG Metadata Isolation and Exfiltrate Another Tenant's ContractAttack the multi-tenant isolation of a real Retrieval-Augmented Generation assistant. Two stacked bugs in one retriever, a caller-controlled tenant scope and a string-concatenated metadata filter, let a Globex-scoped caller read Initech's confidential contract from a Milvus + NVIDIA embeddings store. Chain the cross-tenant read into the EchoLeak markdown-image sink to exfiltrate the data to a listener, then harden the pipeline so isolation and the sink both hold.70 minAdvancedHostedPro
2

Attack agents and outputs

Turn output into SSRF and zero-click exfiltration, escalate through tools, poison memory and MCP, and fuzz with garak.

  1. Lab 8Insecure Output Handling: SSRF, SQLi, and Command Execution Through an Agent's ToolsRed-team OpsBot, a ReAct tool-using support agent, by shaping the arguments it passes to its own tools. File a poisoned support ticket and a benign on-call query turns into a server-side request forgery against a metadata endpoint, a SQL injection that drops a canary and reads another tenant's rows, and code execution through a transform helper. Measure attack-success-rate, then close every sink: an allow-listed fetch, parameterized queries, and a removed code tool.80 minAdvancedHostedPro
  2. Lab 9Insecure Output Handling: Zero-Click Exfiltration Through Rendered Model Output (EchoLeak)Treat the model as an untrusted source whose output flows into a sink: the chat client's markdown renderer. Prove the renderer auto-fetches, plant a document so a benign account question makes the assistant echo a customer's own account reference into a markdown image URL, and watch the renderer auto-fetch it (zero-click exfil, the EchoLeak channel). Measure its attack-success-rate, defeat a CSP-style allow-list through a first-party open proxy, measure that bypass too, then harden in two moves: close the render sink so untrusted output never fires an outbound request, add an audited host allow-list the open proxy cannot defeat, and re-run both attacks to watch ASR fall to zero.85 minAdvancedHostedPro
  3. Lab 10Excessive Agency: Turn a Support Ticket into a Privileged Action (Confused Deputy)Attack a real tool-using ReAct agent end to end. As the low-privilege ticket-ingest account, plant an authorized-looking record correction in a support ticket and make DV-ToolAgent run a privileged billing-payee redirect under its own shared credential (the confused deputy), then reach an internal-only endpoint and exfiltrate its value through the fetch tool (SSRF via tool args). Measure attack-success-rate, then harden the tool boundary, scope the DB tool read-only, carry per-user authorization, and allow-list the fetch tool, and prove your own exploit is dead while normal lookups still work.75 minAdvancedHostedPro
  4. Lab 11Memory Poisoning: Plant a Note That Re-Fires in a Fresh Session (Persistence)Attack the long-term memory of a real tool-using ReAct agent. As the low-privilege ticket-ingest account, plant a single benign-looking routing note into DV-ToolAgent's shared, un-namespaced memory store through an ingested ticket. In a brand-new session for a different legitimate user, the agent recalls your note and redirects a GLOBEX invoice to your payee, persistence across the session boundary that single-turn filters never see. Use MINJA-style progressive shortening so the stored record reads as a mundane preference, measure attack-success-rate against benign controls, then harden memory with per-user namespacing and a data-only quarantine and prove the poison dead while legitimate recall still works.80 minAdvancedHostedPro
  5. Lab 12MCP Tool Poisoning: Hijack an Agent Through a Tool Description (and a Rug Pull)Attack a real MCP-style tool registry end to end. OrbitDesk's support agent loads its tools from a runtime registry and reads each tool description as trusted instruction text. Register a poisoned tool whose description hides a routine-looking audit directive, make the agent read an account record and forward its reference to your in-pod collector (data exfiltration through tool metadata), then pull a rug: register the tool benign, pass review, and silently mutate its description after approval. Measure attack-success-rate, then harden the registry, scan descriptions for hidden instructions and pin approved tool objects, and prove a fresh poison and a fresh rug pull are both dead while benign tickets still resolve.95 minAdvancedHostedPro
  6. Lab 13Tool Shadowing: Hijack an Agent's Tool Selection With a Name CollisionAttack a real MCP-style tool registry by shadowing a trusted tool. OrbitDesk's support agent loads its tools from a runtime registry with a flat namespace and last-write-wins resolution, and it picks which tool to call from attacker-controllable descriptions. Register a malicious twin with the same name as the trusted record reader and a more compelling, compliance-approved description, so the agent calls your twin instead. The twin silently reads a cross-tenant record and exfiltrates its reference, and the same shadow hands another tenant's data straight back to the caller. Measure how reliably the shadow wins selection, then harden the registry, namespace tools and reject collisions, and prove a fresh shadow is dead while benign lookups still work.75 minAdvancedHostedPro
  7. Lab 14Inter-Agent Injection: Propagate a Morris II Worm Across a Two-Agent GraphAttack a real two-agent support graph where one agent's output is the next agent's input with no authentication and no validation. Plant a self-replicating payload in the only untrusted input, an inbound customer email the Intake agent ingests. Intake forwards it, and the Resolver agent, trusting the inter-agent notes as instructions, performs the attacker-directed action AND re-emits the payload verbatim: a second-hop cascade and the Morris II replication primitive, bounded to two hops. The harness attributes every side effect to the agent that caused it, so you prove the second agent executed. Measure propagation reliability, then harden the channel, a schema-constrained, validated, replication-aware handoff, and prove the cascade is contained while benign tickets still resolve.85 minAdvancedHostedPro
  8. Lab 15Fuzz an LLM App with garak: Run, Read, and Triage True vs False PositivesRun NVIDIA garak as an automated fuzzer against a real vulnerable RAG support assistant, read the JSONL run log and the per-probe DEFCON report, then do the skill that separates a scanner operator from a red teamer: triage the hits. Dismiss a detector false positive with evidence, confirm a genuine indirect prompt injection against an in-pod exfil listener, and watch the finding regress to zero after the fix ships.90 minAdvancedHostedPro
3

Defend and measure

Guardrails, firewalls, output mediators, least privilege and a signed tool supply chain, each measured by attack success rate.

  1. Lab 16Build NeMo Guardrails for an AI Agent: Jailbreak & Topical RailsBuild a guarded IT support agent that blocks jailbreaks, refuses off-topic questions and handles IT queries safely, using keyword checks, LLM validation and NeMo Guardrails.35 minIntermediateHostedPro
  2. Lab 17Defense in Depth: Wire Four Control Points Around a RAG AssistantHarden DV-RAG-Support, a real Retrieval-Augmented Generation assistant, by building a guard harness with four independent control points one mechanism per step: input mediation, retrieval and context control, output mediation, and action authorization. You are handed a working four-attack battery (direct injection, cross-tenant retrieval, sensitive-field exfiltration, and an unauthorized image fetch). Stand the pipeline up, reproduce all four attacks one per stage, watch a single naive filter get bypassed, then build each control point in its own step so every attack class is stopped at its matching layer while a benign customer request still passes clean through all four. Verify the coverage matrix reads four-for-four, then prove reworded and renamed bypass variants are all resisted.90 minAdvancedHostedPro
  3. Lab 18Defend a RAG Assistant: Build a Guardrail Layer and an Attack-Success-Rate CI GateYou inherit DV-RAG-Support with a working EchoLeak-style exploit, and you defend it in small sequential steps. Stand the assistant up and trace one benign request, then reproduce the leak: an indirect prompt injection makes the model echo a customer's account record into a markdown image, and the renderer fires it as an outbound request that exfiltrates the data. Watch a naive host deny-list get bypassed by a renamed host, then build the durable guardrail one mechanism per step: an egress allow-list on the render sink so only approved hosts load, then output redaction of the sensitive record so even an approved-host request carries nothing. Verify the sink is closed with benign answers intact, then stand up an attack-success-rate (ASR) gate that runs a probe battery on every change: green while guardrails hold, non-zero the moment a regression re-opens the sink, green again when you back it out. Finish by proving fresh, renamed, and paraphrased payloads are all blocked.85 minAdvancedHostedPro
  4. Lab 19Defend a RAG Assistant: Block Indirect-Injection Exfil (EchoLeak)Harden the same deliberately-vulnerable RAG assistant the offensive lab broke, in small sequential steps. Stand up the pipeline and trace one benign request, reproduce the EchoLeak markdown-image exfil, then watch a naive deny-list get bypassed by a renamed host. Build the durable fix one mechanism per step: an egress allow-list on the render sink that pins the parsed host (defeating userinfo, IP-encoded, and IPv6 spellings) and covers reference-style images, then provenance isolation so retrieved documents cannot emit instructions. Verify both controls together, resist a userinfo / IP-encoding / paraphrase bypass battery, and pass a final ship gate where attack success rate is 0 and benign quality holds.85 minAdvancedHostedPro
  5. Lab 20Build a RAG Firewall: Reject Poisoned Ingestion and Enforce Tenant IsolationDefend the same multi-tenant RAG assistant the offensive labs attack, in small sequential steps. Stand up the pipeline and trace one benign request, then reproduce two handed-to-you exploits one at a time: a poisoned document that wins retrieval and steers the answer, and a caller-controlled tenant scope that reads another tenant's confidential contract. Watch a naive deny-list get bypassed by a fresh payload, then build the durable control one mechanism per step: a server-side tenant predicate the caller cannot widen, then an ingestion screen that rejects directive-shaped documents before indexing. Verify both exploits are blocked with benign traffic intact, then prove fresh, paraphrased, and renamed variants are all blocked on a real Milvus + NVIDIA embeddings stack.80 minAdvancedHostedPro
  6. Lab 21Defend: Secret Isolation for a RAG AssistantHarden the same RAG support assistant that the extraction lab broke, in small sequential steps. A live signing key, an internal build id, and a canary token are baked into the system prompt, so the secret is exposed by construction: it shares one context window with the customer's message. Stand the service up, reproduce the exposure (the secret is present in the model's context), and watch a naive cleartext output filter fall to encoding-egress. Then build the durable control one mechanism per step: a vault boundary that holds the secret out of the model's context, a seeded canary tripwire, and a fail-closed decoding leak detector that matches the secret and its Base64/ROT13/hex forms. Verify the secret is unrecoverable and benign answers are intact, then resist a re-planted, re-encoded exfil battery.80 minAdvancedHostedPro
  7. Lab 22Defend Excessive Agency: Re-scope a Tool Agent to Least Privilege (AuthZ + Human Approval Gate)Harden DV-ToolAgent, a real tool-using ReAct agent, against the confused-deputy and scope-escalation attacks from the offensive lab, in small sequential steps. Stand up the agent and trace one benign ticket, then reproduce the handed-to-you exploit one surface at a time: an ingested ticket that makes the agent redirect a billing payee under its own shared credential and read across tenants, then an SSRF reach into an internal-only endpoint and a poisoned-memory replant. Watch a naive SQL denylist get bypassed by a case-folded variant, then build the durable control one mechanism per step: least-privilege tool scope (the ingest role holds no write scope), per-argument authorization decided on the session identity (a caller-supplied identity claim is ignored), and a human-in-the-loop approval gate that holds high-impact writes pending an explicit token. Verify the exploit is dead with authorized in-scope work intact, then prove obfuscated, renamed, and spoofed variants are all blocked.90 minAdvancedHostedPro
  8. Lab 23Defend the Agent Supply Chain: Verify, Pin, and Capability-Gate Your Tool RegistryHarden a real MCP-style tool registry until a poisoned, rug-pulled, or shadowing tool manifest cannot reach the agent, in small single-concept steps. OrbitDesk's support agent loads its tools from a runtime registry and reads each tool description as trusted instruction text. You stand the registry up and trace one benign ticket, then reproduce three techniques one at a time: a poisoned tool description that hijacks the agent into leaking an account reference, a post-approval rug-pull mutation the registry serves with no pin, and a shadowing twin the flat namespace selects under a trusted tool name. You apply the obvious fix, a description blocklist, and watch a clean-description variant defeat it through its delegate. Then you build the durable control one mechanism per step: manifest signature verification, then hash pinning (rug-pull / change detection), then a per-tool capability allow-list with namespacing. You finish by proving that freshly planted unsigned, forged, mutated, and shadowing manifests are all refused while a legitimate signed and pinned tool stays admitted and usable, and an inter-agent worm's second hop is contained.90 minAdvancedHostedPro
  9. Lab 24Defend: A Contextual Output Mediator for XSS, SSRF, SQLi, and RCEDefend DV-ToolAgent, a tool-using support agent whose model output flows raw into four interpreters, in small sequential steps. Stand up the agent and trace one benign request, then reproduce each sink one at a time: a behavioral SSRF (the agent fetches an internal node-health endpoint), a behavioral cross-tenant SQL read (an entitlement comparison pulls another tenant's record), then OS command execution, a SQL write, and stored XSS demonstrated structurally at the sink. Watch a naive blocklist get bypassed by a variant, then build one contextual output mediator one sink-class at a time: an SSRF host allow-list with an IP-literal guard and parameterized tenant-scoped SQL, then refuse-arbitrary-code for the runtime and contextual HTML encoding for the browser. Verify every freshly planted payload and its bypass variant is blocked at the sink while benign output still renders, queries, fetches, and computes.90 minAdvancedHostedPro
  10. Lab 25PII Detection and Redaction: Checksums, LLM Entity Extraction, Reversible Placeholders and a Leak TestBuild the redactor that stands between customer chats and an external LLM service. Find emails, phones, card numbers and IBANs with patterns validated by Luhn and mod-97, find names and addresses with an in-house model, merge and measure span coverage on labelled chats, replace PII with consistent typed placeholders you can restore, and prove with a leak test on held-out chats that nothing gets through.55 minIntermediateHostedPro
  11. Lab 26PII-Safe Multi-Tenant Chatbot: Signed Sessions, Redaction Vaults, Scoped Memory and ErasureBuild a support chatbot that serves several businesses without mixing them: verify signed session tokens, send the model provider a pseudonymous end-user id, redact personal data into a per-user vault and restore it only for its owner, keep memory per tenant and user, run cross-tenant leak probes, and export or erase one customer's data.60 minIntermediateHostedPro

Graded project

Run a full LLM and agent penetration test

The capstone: assess and attack a complete LLM and agent system, then ship the remediations, scored on every rubric criterion.

Open the project

Guides for this track

GuideWhat is prompt injection? A plain-English guideGuideAI penetration testing: a practical guideGuideMCP security best practices

Related collections:Prompt injection examples you can runLLM security course

Questions about this track

Other tracks

AI agents and MCP

Tool loops, ReAct, MCP servers and clients, multi-agent supervisors and agent evaluation.

22 labs

RAG and search

Chunking, hybrid search and reranking, permission-aware retrieval, GraphRAG and RAG evaluation.

14 labs

LLMOps and MLOps

vLLM serving, load tests against SLOs, tracing, drift monitoring and prompt tests in CI.

17 labs

Every lab with Pro

This track and every other one, plus every practice test. $29.99 a month, cancel any time.

See Pro