TL;DR: AI penetration testing is authorized security testing of AI-powered applications: LLM assistants, RAG systems, and tool-using agents. It borrows the structure of traditional pentesting (scope, recon, exploit, report) and changes what you attack and how you judge results. The two differences that matter most: the vulnerabilities live in how the application handles untrusted text and what it lets the model's output reach, and exploits are probabilistic, so a single failed attempt proves nothing. This guide covers the attack surface, a working methodology, the frameworks worth mapping to (OWASP LLM Top 10, MITRE ATLAS), and how to build the skill.
A traditional web application pentest has a comfortable property: results are deterministic. The payload either lands or it does not. Run it twice, get the same answer.
AI penetration testing removes that comfort. The same payload against the same system can succeed on the first attempt, fail on the next three, and succeed again on the fifth. A tester who runs an attack once, sees a refusal, and marks it "not vulnerable" has learned almost nothing. This changes the methodology, the reporting, and what a finding even means.
This guide covers how to test AI systems properly: what the attack surface looks like, a methodology that survives the probabilistic problem, and the frameworks to anchor findings to. It assumes you have permission to test the target. If you do not, get it in writing first; the legal exposure is identical to any other unauthorized security testing.
What makes AI pentesting different
Four differences change how you work.
The vulnerability is usually in the plumbing, not the model. Testers new to this spend their time trying to jailbreak the model into saying something embarrassing. That is a content-safety finding, and it is rarely the severe one. The severe findings are architectural: the assistant that can read other tenants' records because the tool query is not scoped server-side, or the one whose output renders attacker-controlled URLs that auto-fetch. The model is the entry point; the application is the vulnerability.
Exploits are probabilistic. Model behavior varies across runs, so single-shot results are noise. Every finding needs a reliability measure: how many attempts out of how many tries. This is a reporting discipline, and it also changes severity, because an exploit that fires 90% of the time is a different risk than one that fires 5% of the time.
The trust boundary is text. In a web app, you look for places user input reaches a parser. In an AI app, you look for every place any text reaches the model's context, and treat all of it as attacker-reachable if any of it is. That includes documents, tool results, retrieved chunks, and API responses from services you do not control.
Impact runs through the model's permissions, not the attacker's. A successful injection executes with whatever access the AI has. Mapping what the agent can reach matters more than mapping what the attacker can reach.
The attack surface of an LLM application
Work through these systematically. Each is a place findings hide.
The prompt layer. System prompt extraction (useful for recon even when not itself severe), instruction override, and role-play framing. Establishes how compliant the model is before you attack the architecture.
The retrieval layer (RAG). Can you get content into the knowledge base, directly or indirectly? Can you make a poisoned document win retrieval for a realistic query? Is content sanitized at ingestion, and does hidden text (white-on-white, HTML comments, metadata) survive? This is where indirect prompt injection lives, and it is usually the highest-yield area.
The tool layer. For every tool the agent can call: is authorization enforced server-side or inferred from the conversation? Can you reach another user's data through it (cross-tenant)? Can a fetch tool be pointed at internal addresses (server-side request forgery)? Are database queries parameterized? This is where confused-deputy findings come from, and they are typically the most severe.
The output layer. Where does model output go, and what happens on the way? Does the client auto-fetch images or resolve links from arbitrary hosts? Is output rendered as HTML or markdown without sanitization? Can the model's output reach a downstream system that treats it as trusted input? The output layer is where exfiltration channels live, and it is the most commonly missed area.
The memory and state layer. If the system persists conversation memory or user profiles, can you plant content that influences future sessions? Persistent findings score higher because they survive the session.
The multi-agent layer. When agents call other agents, can a payload propagate? Does agent B trust agent A's output implicitly? Self-propagating payloads across agent graphs are the most severe class and the hardest to land.
The infrastructure layer. Everything traditional: the model-serving endpoints, API key handling, rate limiting, and the vector database's own authentication. Model endpoints exposed without auth are a real and common finding.
A methodology that works
1. Scope and authorization. Get written permission naming the systems and the techniques. AI testing has a wrinkle: attacks may create persistent artifacts (poisoned documents, planted memories), so agree in advance on cleanup and on whether production data stores are in scope.
2. Map the architecture before attacking. Identify the model, whether retrieval is in play, what tools the agent has, where output is rendered, and what identity the tool calls execute as. Most of the severity assessment comes from this stage. An hour of mapping beats a day of blind payloads.
3. Baseline the model's compliance. Run direct injection attempts to gauge how much the model resists. This calibrates expectations for later stages rather than producing findings.
4. Attack the layers in order of leverage. Retrieval, then tools, then output. Follow the kill chain: plant, retrieve, execute, escape. A payload that executes but cannot escape is a partial finding worth reporting as a near-miss with a note about which control saved it.
5. Measure reliability. For every working exploit, run it repeatedly (10 to 20 attempts is a reasonable floor) and record the success rate. This is the single practice that separates credible AI pentest reports from anecdotes.
6. Test the defenses, not just the gaps. If the system has a prompt-level defense, verify whether it survives a determined payload. If it has an output allow-list, verify it cannot be bypassed with a different channel. Findings about defenses that look protective but are not are among the most valuable things you can deliver.
7. Report with architecture-level fixes. "Add a filter for this payload" is a weak recommendation, because the next payload differs. "Scope this tool's queries to the requesting user server-side" fixes the class. Tie each finding to the layer it lives in and the boundary control that closes it.
Frameworks to map findings to
Two references make reports legible to security teams who do not specialize in AI.
OWASP Top 10 for LLM Applications: the common vocabulary. LLM01 Prompt Injection is the one you will cite most; the list also covers sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Mapping each finding to an LLM identifier lets a security team triage it alongside their existing work.
MITRE ATLAS: the adversary-technique knowledge base for AI systems, structured like ATT&CK. Use it to describe technique chains and to justify severity with documented real-world case studies.
Tooling, honestly assessed
The tooling ecosystem is young. Automated scanners exist for LLM applications and are genuinely useful for breadth: they will run a corpus of known payloads and flag obvious compliance failures. What they will not do is understand your application's architecture, which is where the severe findings are. No scanner will notice that a tool call runs with ambient authority instead of the requesting user's identity.
Treat automation as coverage for the prompt layer, and expect the findings that matter to come from manual architectural work. A repeatable harness of your own (a script that runs each payload N times and records the success rate) is usually more valuable than any off-the-shelf scanner, because it solves the probabilistic problem.
How to build the skill
AI pentesting sits at an unusual intersection: it rewards traditional application security instincts (authorization, injection, output handling) applied to a new substrate. If you come from a pentesting background, the concepts transfer faster than you expect, and the gap is understanding how retrieval, agents, and tool calling actually work. If you come from an AI background, you know the substrate and need the security instincts.
The fastest way to close either gap is to exploit real systems. Our AI Red Team course is built for exactly that: every lab runs against a live model with real infrastructure, not a simulation. You start with the Indirect Prompt Injection lab, which is free, and work through tool misuse, MCP tool poisoning, memory poisoning, cross-tenant leakage, and a self-propagating payload across a two-agent graph. Each lab maps to the OWASP LLM Top 10 and MITRE ATLAS, so the vocabulary you practice with is the vocabulary your reports will use.
For the credentialing side of this field, we cover the options in our guide to AI security certifications.
Frequently asked questions
What is AI penetration testing? It is authorized security testing of AI-powered applications, focused on how the system handles untrusted text and what it allows the model's output and tools to reach. It covers LLM assistants, RAG systems, and tool-using agents, and it produces findings about the application architecture more often than about the model itself.
How is it different from traditional penetration testing? The structure is the same and three things change: the vulnerabilities usually live in the application's handling of model input and output rather than in classic parsers, exploits are probabilistic so findings need reliability measurements, and impact is determined by the AI's permissions rather than the attacker's.
Do I need machine learning expertise? Not deep expertise. You need a working understanding of how LLM applications are built: prompts, retrieval, tool calling, and agent loops. Application security instincts carry most of the weight, and the AI-specific knowledge can be learned in weeks by exploiting real systems.
Is AI pentesting legal? Testing systems you own or have written authorization to test is legitimate professional work. Testing systems without permission can violate computer misuse laws exactly as any unauthorized testing would. Practice environments built to be attacked, like training labs, are the safe place to develop the skill.
What tools should I use? Automated LLM scanners give useful breadth on the prompt layer. The severe findings come from manual architectural testing, and a simple harness that repeats each payload many times and records success rates is often the most valuable tool you will build.
How do I report a probabilistic finding? Include the success rate and the number of attempts, describe the conditions that made it fire more often, and rate severity on impact combined with reliability. Recommend the architectural control that closes the class rather than a filter for the specific payload.
Sources and further reading
- OWASP Top 10 for LLM Applications
- MITRE ATLAS
- Our guides: What Is Prompt Injection, Indirect Prompt Injection, and Prompt Injection Attacks Explained