LLM penetration testing for prompts, retrieval, tools, and outputs
LLM penetration testing evaluates the application built around a language model: prompt assembly, context sources, retrieval, tool calls, output handling, identity, and policy enforcement. It tests how adversarial content changes behavior and whether the surrounding system can turn that behavior into data exposure or an unsafe action.
Define the LLM application’s assets and action boundaries, then test direct and indirect prompt injection, retrieval contamination, context leakage, tool authorization, output handling, and guardrail bypass with controlled accounts and reproducible traces.

What is LLM penetration testing?
LLM penetration testing is an authorized security assessment of an application that uses a large language model. The target includes the model’s behavior, but the highest-value findings usually come from how the application assembles prompts, retrieves data, exposes tools, stores conversation state, and renders output.
A standalone model may produce an unsafe sentence without affecting a system. A vulnerable LLM application can connect that sentence to a database query, email action, code execution tool, refund workflow, or private document. The application context determines the security impact.
A credible test therefore starts with the business action and data boundary. Identify what the LLM can read, change, send, execute, or reveal. Then work backward through context assembly, provider configuration, retrieval authorization, tool permissions, and output handling.
LLM testing is not the same as model benchmarking. Benchmarks compare general model behavior under standardized prompts. A penetration test asks whether this deployed application, with its prompts and integrations, can be manipulated into violating a specific security policy.
How is LLM testing different from AI penetration testing?
AI penetration testing commonly describes an AI agent performing security testing against an external target. LLM penetration testing tests the LLM application itself and the controls around it.
The distinction keeps scope and evidence clear. One workflow asks whether an agent can find a vulnerability in a web application. The other asks whether a malicious prompt or poisoned document can cause an LLM application to disclose data, call the wrong tool, or bypass a policy.
The two subjects can connect. An AI pentest platform may use an LLM and therefore need LLM security testing. An LLM application may use an agent runtime and therefore need AI agent security testing. Link the pages when the architecture overlaps without treating the keywords as synonyms.
What attack surfaces should LLM penetration tests cover?
Test every path by which untrusted content becomes model context. Direct user prompts, uploaded files, web pages, email, chat messages, retrieved documents, database records, tool descriptions, tool output, and previous conversation turns may all carry instructions.
Map the model’s access to protected data and actions. Retrieval systems often enforce document permissions before the model sees content, or they may retrieve globally and rely on the prompt to honor tenant boundaries. Tool calls may use the user’s token, a service token, or a broader agent credential.
Inspect output handling. Model output can flow into HTML, markdown, SQL, shell commands, code execution, API requests, identity headers, or another model. A harmless-looking completion becomes dangerous when the downstream component trusts it as an instruction or trusted data.
Include model and provider configuration when available. System prompts, temperature, context limits, model routing, safety filters, logging, retention, fallback models, and provider data controls affect security behavior and should be part of the assessment record.
| Surface | Test question | Impact evidence |
|---|---|---|
| Direct prompt | Can user input override policy or reveal protected instructions? | Prompt, model response, and exposed asset |
| Retrieval | Can poisoned or unauthorized documents influence an answer? | Stored source, retrieval event, and output |
| Context | Can one user’s data enter another user’s model context? | Tenant boundary, assembled context, and disclosure |
| Tools | Can prompt content select or parameterize an unsafe action? | Tool registry, arguments, authorization, and side effect |
| Output | Does downstream code trust model output as markup or commands? | Raw output, sink behavior, and affected user or system |
| Provider | Do routing, logging, retention, or fallback models weaken policy? | Configuration, data path, and observed control behavior |
How do you test prompt injection and RAG security?
Test direct prompt injection with controlled users and synthetic policies. Determine whether untrusted content can change the task, expose the system prompt, bypass a content rule, or cause the app to treat user data as an instruction. Record the exact assembled context after redaction.
For retrieval-augmented generation, create documents with known ownership, sensitivity, and instruction content. Ask questions as different users and tenants. Verify that retrieval authorization happens before content enters context, that poisoning one document does not persist across unrelated tasks, and that citations do not expose inaccessible source metadata.
Test indirect injection through every external content channel. A web page, file, email, API response, or tool result can contain text aimed at the model. The application should treat that content as data and preserve provenance rather than allowing it to redefine policy.
Repeat tests across languages, formatting, encoding, long context, chunking boundaries, summarization, and conversation history. A control that works on a short English prompt may fail when an instruction is split across documents or carried into a summary that later becomes trusted context.
Measure whether failures are persistent or probabilistic. Record model, settings, retrieval configuration, input count, and observed outcomes. A security review needs enough repetitions to distinguish a stable control gap from an occasional generation artifact.
- Separate test users, tenants, and document collections.
- Synthetic documents that contain harmless but explicit test instructions.
- A baseline showing which sources are authorized for each user.
- Repeated runs across model versions and prompt arrangements.
- Trace capture for prompt assembly, retrieval, response, and citation.
Which LLM guardrails and tool boundaries should be tested?
Validate instruction hierarchy, input filters, retrieval authorization, output encoding, content moderation, tool allow lists, scope checks, rate limits, and approval for high-impact actions. Each control must be enforced by deterministic code when a security boundary depends on it.
Do not ask the model to authorize its own tool use. The runtime should verify the current user, tenant, tool, resource, action, and arguments before execution. Model-generated arguments should be parsed against a strict schema and bound to an approved resource identifier rather than accepted as unrestricted commands.
Test output consumers. If generated text is rendered as HTML, passed to code, used in a query, sent to a payment provider, or forwarded to another agent, identify the parser and trust boundary. Escape or reject unsafe output at the sink and preserve the original model response for review.
Test fallback paths. A safety classifier that times out, a provider failover that uses a weaker model, or a tool that returns an oversized response can change behavior. Sensitive paths should fail closed and preserve the reason for the fallback.
- 01Map assets
List protected data, user identities, tools, actions, and external content sources.
- 02Freeze context
Record prompts, model settings, retrieval rules, and tool configuration.
- 03Probe inputs
Use controlled direct and indirect injection cases for each boundary.
- 04Inspect enforcement
Verify that authorization, parsing, and approval happen outside the model.
- 05Prove impact
Connect the model output to exposed data or a reproduced downstream action.
- 06Retest changes
Repeat the path after prompt, model, retrieval, or tool updates.
What evidence should an LLM finding preserve?
Preserve the test identity, application version, model and provider, model settings, assembled prompt after redaction, retrieval sources, tool schema, output, downstream request or rendering, response, and observed security effect.
Model nondeterminism requires repeat evidence. Record how many attempts were made, which inputs reproduced the issue, whether the result depended on a retrieved document, and whether a model change removed the behavior. Distinguish a stable authorization flaw from an unstable generation pattern.
Separate protected system instructions from sensitive business data. Exposing a generic system prompt may be a low-impact information leak, while exposing another tenant’s document is a direct data breach. Evidence should identify the actual protected asset and the user who should not have received it.
Keep raw and redacted artifacts. The original record supports investigation and retesting, while report copies should remove secrets, customer data, and private prompt content that is not needed to understand the finding.
What makes LLM penetration testing credible?
Credible LLM testing identifies the protected asset and action boundary, uses controlled identities and synthetic content, reproduces the model-to-system path, and distinguishes unstable output from a confirmed security control failure.
Retest after prompt changes, model upgrades, retrieval-index changes, schema changes, tool additions, and output-rendering changes. These are security-relevant releases even when the application code appears unchanged.
Build the program around evidence and ownership. Application security owns prompt assembly and output sinks, platform teams own model routing and logging, data teams own retrieval authorization, and service owners own tool credentials and approvals.
ARTEX can keep prompts, tool activity, traffic, findings, and review evidence connected during authorized testing. Use that record to evaluate how the LLM application behaves when context is hostile and actions have consequences.
Frequently asked questions
What is LLM penetration testing?
It is the authorized testing of an LLM application’s prompts, context, retrieval, tools, outputs, identity, and guardrails to determine whether an attacker can expose data or cause an unsafe action.
Is prompt injection the same as jailbreaking?
Jailbreaking usually tries to bypass model behavior restrictions. Prompt injection tries to make the application follow untrusted instructions. The security impact depends on what data and tools the application exposes.
How do you secure retrieval-augmented generation?
Authorize retrieval before content enters context, preserve source ownership, treat retrieved text as untrusted data, isolate tenants, and test poisoned documents and citation leakage.
Can an LLM penetration test run against a third-party model?
Yes, within the provider’s authorized terms and the application owner’s scope. Test the deployed application and its integrations; do not attempt to attack shared provider infrastructure.
How should LLM findings be retested?
Repeat the same input path, model configuration, retrieval state, identity, and downstream operation after remediation, then compare the new evidence with the original result.
References and review notes
Start with source. Test with authorization.
Review the public ARTEX repository, run it in an owned lab, and keep human approval on actions that can change a target.
ARTEX