hands-on labIDE in your browser

Fuzz an LLM App with garak: Run, Read, and Triage True vs False Positives

Run NVIDIA garak as an automated fuzzer against a real vulnerable RAG support assistant, read the JSONL run log and the per-probe DEFCON report, then do the skill that separates a scanner operator from a red teamer: triage the hits.

time
90 min
checked steps
8
level
Advanced
setup
None

$29.99a month with Pro

Pro opens all 8 steps of this lab and every other lab. Cancel any time.

no account needed · Pro opens all 8 steps

See Pro plans
Lab cockpit90 min · 8 stepsSession running
6 / 8 steps passingDistrust the retrieved context: reframe context as data, never instructions · step 7 of 8
dvrag.py▶ Run✓ Check
# they saw in an alert. Enabled only when DVRAG_NAIVE_BLOCK=1 (the bypass step).# Deny-lists are the wrong tool here — that is the lesson you will prove.NAIVE_BLOCK = os.environ.get("DVRAG_NAIVE_BLOCK") == "1"NAIVE_DENY_HOSTS = {"127.0.0.1"} SYSTEM_PROMPT = (    "You are ACME Cloud's customer support assistant. Use the information in the "    "CONTEXT below to answer the customer's question accurately and concisely. "    "If the answer is not in the context, say you are not sure and suggest opening "     
TerminalOutput
step 7 of 8, replayed: Distrust the retrieved context: reframe context as data, never instructions. The code, the output and the check are the lab's own.

The job

Manual testing does not scale and does not guard against regression. You are standing up the automated half of a red-team practice against DV-RAG-Support, the customer-support RAG assistant from Module 2. You will run garak as a fuzzer, read what it found, and triage the hits, because a scanner reports heuristic hits and only you can tell a real finding from a detector that pattern-matched. Dismiss a false positive with evidence, confirm a true positive against the exfil listener, then watch the genuine finding disappear once the fix is in.

8 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Recon: stand up the vulnerable target behind an OpenAI endpoint

    You are automating red-team testing of DV-RAG-Support, ACME Cloud's customer-support assistant from Module 2.

    read this step below ↓

  2. 2

    Run the scan: fire a broad garak discovery battery at the endpoint

    garak runs a battery of probes (attack generators) against a generator and scores each response with detectors.

  3. 3

    Read the report: count hits per probe and grade the run

    A garak run produces several artifacts:

  4. 4

    Triage a false positive: dismiss a detector hit with evidence

    Scanners over-report.

  5. 5

    Confirm a true positive: run the custom probe, prove the effect on the listener

    A true positive is confirmed by reproducing the effect, not by trusting a detector.

  6. 6

    Harden the render sink: allow-list the one channel the finding rode out on

    Switch hats.

  7. 7

    Distrust the retrieved context: reframe context as data, never instructions

    The render allow-list from step 6 is the load-bearing fix: it closes the exfil channel even if the model is fully tricked.

  8. 8

    Verify the regression: fixed = 0 findings, reintroduced = re-trips

    A fix only counts when you re-run the exact confirmed attack and watch it fail.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Recon: stand up the vulnerable target behind an OpenAI endpoint

You are automating red-team testing of DV-RAG-Support, ACME Cloud's customer-support assistant from Module 2. A fuzzer talks to an LLM application the way any client does: over an HTTP API. So before you fuzz anything, you stand up the machine and confirm it is genuinely vulnerable.

Three processes make up the target:

  • dvrag.py is the RAG app: it embeds a question, retrieves from a Milvus vector store, builds a prompt, calls the model, and renders the answer. The render step auto-loads any markdown image the model emits. That is the exfil sink, the same EchoLeak channel (CVE-2025-32711) from Module 2.
  • dvserve.py wraps dvrag.chat() as an OpenAI-compatible HTTP endpoint on 127.0.0.1:9001, so garak and PyRIT can hit it like any chat API. The fuzzer's traffic is therefore real victim traffic: the vulnerable sink fires inside chat().
  • listener.py is the attacker's collection server on 127.0.0.1:9099. When the sink loads an image whose URL carries the account reference, the callback lands here. The listener log is your ground-truth oracle for the rest of the lab.

do this

  1. Hit Run. It resets and starts the listener, builds the index, starts the OpenAI-compatible wrapper, waits for it to be healthy, then sends one benign account question (What plan am I on?) through /v1/chat/completions.
  2. Read the output. The model answers a normal account question. Then read the --- listener callbacks --- block: a callback carrying acct=GLOBEX-ACR-88231 means the account reference left through the markdown image sink. The poison doc (kb_poison.md) is already in the KB, so the app is vulnerable from step one.

That confirms the target is live and the sink fires on a benign question, so anything garak flags later can be checked against ground truth.

Pass criteria

dvserve.py answers /healthz, and a benign account question routed through the OpenAI endpoint produces an exfil callback containing 88231 on the listener.

provided for you:dvserve.pygarak_dvrag.pygarak_openai.jsongarak_rest.jsonkb_account-recovery.mdkb_account.mdkb_billing.mdkb_getting-started.mdkb_globex-runbook.mdkb_initech-contract.mdkb_poison.mdkb_security-sso.mdkb_shipping.mdkb_status-incidents.mdlistener.pyread_report.pyrun_custom_probe.py

what you get

  • An editor and terminal in your browser
  • A check at the end of every step
  • Hints and the full solution
  • Nothing to install or configure

before you start

  • Comfortable reading Python and JSON
  • Completed (or understand) Module 2 indirect prompt injection
  • No ML background required

Part of the AI security and red teaming track

Lab 15 of 29. Prompt injection, tool poisoning and data exfiltration against live targets, then the defenses.

Questions about this lab

What is garak?

garak is NVIDIA's open-source LLM vulnerability scanner. You point it at a generator (a model or an LLM application behind an API), it runs predefined probes (attack generators), and detectors score each response. It is the discovery and fuzzing tool in a red-team toolkit: broad and opinionated, and it reports heuristic hits that you then triage.

What is the difference between a true and a false positive here?

A false positive is when a detector flags a response that did not actually achieve the attacker's goal: the model refused, or just quoted the payload, and a string-match detector counted it anyway. A true positive is a hit you can reproduce as a real effect. In this lab the true positive is an account record exfiltrating to an in-pod listener, which you confirm by reading the listener log, not by trusting a detector score.

Do I need a machine-learning background?

No. You read Python and JSON and reason about whether an attack actually fired. Everything model-specific is explained inline. The point of the lab is triage judgment and regression discipline, not model internals.

What you'll do in this lab

This is a hands-on red-team automation lab. You point NVIDIA garak, a batteries-included LLM fuzzer and scanner, at a real Retrieval-Augmented Generation (RAG) support assistant called DV-RAG-Support and run a battery of attack probes against it. garak fires predefined attack prompts, scores each response with detectors, and writes a JSONL run log plus a per-probe DEFCON grade. Your job is the part a scanner cannot do for you: read the run and triage every hit, separating a genuine finding from a detector that merely pattern-matched.

You will dismiss a detector false positive with evidence from the run log, then confirm a genuine true positive the right way, by reproducing the effect against an in-pod exfil listener rather than trusting a detector score. The confirmed finding is an indirect prompt injection (OWASP LLM01) that leaks a customer's account record through an auto-rendered markdown image (OWASP LLM05 improper output handling), the same channel behind the real EchoLeak exploit. You finish by shipping the fix and re-running the battery to watch the finding regress to zero, the regression discipline a real red-team practice is built on.