An agent red team is often run as a prompt-hacking exercise: a few people spend a day trying to make the model say something it should not, write up the successful prompts, and the team adds refusal language to the system prompt. That produces a report and very little security.

The reason is a mismatch of objective. For a chatbot, harmful text is the harm. For an agent, harmful text is a symptom; the harm is an unauthorized action, an exfiltrated record, an unbounded spend, a privilege that got escalated. If your red team's findings are all quotes from model outputs, it did not test the system — it tested the model, which the model vendor already does.

Define the Objective as Actions

Start by writing down what an attacker would want to accomplish against this specific deployment, in terms of state changes and data movement. For example:

  • Read a record belonging to a tenant other than the one the task is for.
  • Cause an outbound message, payment, or write to a system the agent should not touch.
  • Obtain a credential, a token, or an internal hostname.
  • Cause spend above the agent's authorized ceiling.
  • Escalate: get an action performed that requires authority the initiating principal does not hold.
  • Get a poisoned instruction to persist and fire in a later, unrelated session.
  • Cause the agent to skip an approval gate on a gated action.
  • Prevent or corrupt the audit record of any of the above.

Each of these is a binary, verifiable outcome. That is what makes an agent red team scorable, and it is what makes the report actionable — every finding names a control that failed rather than a phrasing that worked.

Scope: Four Surfaces, Not One

The prompt surface. Direct injection, jailbreaks, multi-turn escalation, extraction. Necessary but the least interesting, because it is the entry point rather than the impact.

The tool surface. What is exposed versus what is used. Can the agent be induced to call a tool outside its task? Are tool parameters validated server-side or trusted from model output? Can a tool be called with a target the agent should not reach — another tenant's identifier, an internal address, a path traversal? This is where the high-severity findings usually live.

The data path. Where does untrusted content enter? RAG corpora, tool responses, inter-agent messages, uploaded files, web pages, email. For each, can an attacker write to it, and does content from it carry provenance downstream? Test by planting a benign marker payload in each ingress and observing whether it reaches the model and whether the agent acts on it. See RAG pipeline security and threat model: indirect prompt injection.

The identity and spend path. Whose authority is used for each call? Can a token be replayed, forwarded, or widened at exchange? Are tenant identifiers taken from the authenticated session or from request content? Does a hard spend ceiling exist and does it hold under concurrency? The concurrency question specifically — many parallel calls against one budget — finds real defects in most homegrown implementations. See denial of wallet.

Technique Set

Work outward from cheap to expensive.

Direct and indirect injection. Both matter; indirect matters more. Plant payloads in every writable ingress you identified, with instructions targeting the specific actions from your objective list.

Multi-turn escalation. Crescendo sequences, frame building, refusal-and-rephrase loops. Single-shot testing systematically underestimates exposure here. See multi-turn jailbreaks.

Tool parameter abuse. Rather than persuading the agent to call a forbidden tool, persuade it to call a permitted tool with abusive parameters: a wildcard filter, a very large limit, an identifier belonging to another tenant, a path with traversal, a URL pointing inward. This is the highest-yield technique against systems that scoped tools but did not validate arguments.

Cross-tenant and cross-principal probing. Run as tenant A and attempt to reach tenant B's data through every path: retrieval, tool arguments, cached state, shared memory, error messages that differ for existing versus non-existing records. Enumeration through response-shape differences is a real finding even when data does not leak.

Delegation abuse. In multi-agent systems, act as a low-privilege agent and attempt to have a higher-privilege agent perform work for you. Attempt to widen scope at token exchange. Attempt to forge principal claims in message payloads. See threat model: A2A delegation abuse.

Exfiltration channels. Rendered images and links, URL query strings, log injection, error messages, tool arguments to external destinations, screenshots into telemetry. Test whether a marker string placed in sensitive context can be made to leave by any route.

Guardrail bypass. Encoding, unicode homoglyphs, language switching, splitting sensitive strings across boundaries, exceeding the evaluator's input length, and — most useful — routing around the enforcement point entirely by finding a path that skips it. A guardrail wired only to the user-facing surface and not to inter-agent messages is a common and serious finding.

Persistence. Plant something in memory, cache, or corpus in session one; verify it fires in session two. Cross-session findings are severe because they are invisible to per-session detection.

Availability and cost. Loop induction, retry amplification, maximum-cost request crafting, concurrency against ceilings.

Environment and Rules of Engagement

Run against a faithful mirror of production: same prompts, same model and version, same tool inventory with the same scopes, same guardrail configuration, same corpora shape. A red team against a stripped-down environment produces findings that do not transfer, in both directions — you will miss real issues and report ones that production controls already handle.

Use synthetic data that is realistically shaped, including synthetic PII with recognizable markers so you can detect leakage automatically. Give every payload a unique marker string; then leakage detection is a grep across outputs, logs, telemetry, and downstream systems rather than a judgment call.

Fix the rules in advance: which environments are in scope, whether third-party services may be touched (usually not — their terms and their incident response are not yours), what happens if a real vulnerability in a vendor product is found, and who can stop the exercise. Log everything, and confirm afterward that the audit trail contains what you did — a red team that your own logs cannot reconstruct is itself the finding. See audit trails that hold up.

Scoring and Reporting

Score each attempt on whether the objective action occurred, and record: the technique, the ingress used, the control that should have stopped it, whether any control fired, and whether the attempt is visible in the audit record and in alerting.

That last dimension is often the most valuable output. A blocked attempt that produced no signal is a detection gap. A successful attempt that produced no record is a forensics gap. Both belong in the report alongside the exploitation findings, and both are frequently more actionable than the exploits.

Map each finding to a structural remediation:

  • Reduce scope (remove the tool, narrow the verb, cap the volume).
  • Add a gate (approval, trust threshold, policy check).
  • Add a cap (spend, rate, result size, depth).
  • Move enforcement (from prompt to proxy, from application to datastore).
  • Add provenance or attribution (mark untrusted content, record the principal).

Prompt patches belong at the bottom of that list. They are cheap, they help at the margin, and they regress on the next model or content change. A finding closed only by a prompt edit should be recorded as accepted risk, not as fixed.

Make It Continuous

A red team exercise is a snapshot. The system changes weekly: prompts, tool inventories, corpora, model versions, guardrail rules. Every confirmed finding should become an automated test that runs in CI with an assertion on the action outcome. That converts a decaying report into a permanent property of the pipeline, and it means the next model upgrade tells you immediately whether previously-closed findings reopened. See continuous adversarial testing for AI agents.

Cadence that works in practice: high-signal regression suite on every change, broader suite nightly, human-led exercise quarterly and on major architecture changes. Track success rate per objective over time as the headline metric.

Common questions

Do we need an external red team or can we do this internally?

Both, for different reasons. Internal teams know the architecture and can test the identity and data paths thoroughly, which is where most severe findings are — external testers spend a lot of their budget rediscovering the system. External teams bring technique breadth and the willingness to report things internal politics discourage. If you can only do one, do internal continuously and buy external for an annual check on your blind spots.

How is this different from a standard application penetration test?

The traffic is well-formed and authenticated, so scanner-driven methodology finds little. The vulnerability class is authorization and data flow rather than memory safety or injection into your own parsers. And the system is non-deterministic: an attempt that fails five times may succeed on the sixth, which means findings need repeat trials and reporting needs success rates rather than binary present/absent. A pentest team without agent experience typically returns prompt-injection findings and misses the tenant-isolation and delegation issues that matter more.

What if we do not have the tool surface built out yet?

Then red team the design. Walk the objective list against the intended architecture and identify which controls are supposed to stop each one. Wherever the answer is "the prompt tells it not to," you have found a design finding before writing the code, which is the cheapest possible time to find it.

How do we handle non-determinism in scoring?

Run each attempt N times (5 to 10 is usually enough to be informative) and report success rate. A technique that works one time in ten is still a finding — an attacker retries. For CI regression tests, assert on zero successes across the trial count, and treat a single success as a failing test rather than as noise.

How Praesidia approaches adversarial testing

Praesidia's enforcement points are also its evidence points: every guardrail evaluation, tool call, connection decision, and spend check is recorded with full attribution — so a red team exercise can be reconstructed from the audit trail and detection gaps become visible rather than assumed. Because containment lives on the connection rather than in prompts, findings map to configuration changes that persist across model upgrades.

Evaluation runs and datasets let adversarial scenarios be stored and re-run as a regression suite against agents, and accumulated behavioral signal feeds trust scores that connections can gate on. See guardrails versus evals versus monitoring and the Praesidia documentation.