Most content filtering on AI systems is stateless. A classifier evaluates the incoming message, returns a score, and the request proceeds or does not. This works against single-shot attacks — the prompt containing the obvious jailbreak, the message asking directly for prohibited content.
It does not work against attacks that spread the payload across turns. If no individual message crosses the threshold, a per-message filter never fires, no matter how clear the intent is when you read the transcript as a whole. That gap is the basis of the multi-turn jailbreak family, and for agentic systems it matters more than for chatbots, because an agent that has been steered does not just produce text — it takes actions.
The Techniques
Crescendo. The attacker opens with a fully legitimate request in the target domain, then escalates in small steps, each one a modest extension of what the model already produced. The model's own prior output becomes the justification for the next step: it already discussed the topic, so continuing is consistent. Published research on this technique reports high success rates against production models precisely because no turn is anomalous, and each turn is individually defensible.
Role and fiction framing over turns. Establish a persona, a fictional setting, or a professional context across several turns before making the actual request inside that frame. The frame is built before it is used, so the frame-building turns carry no payload and the payload turn carries no framing.
Hypothetical nesting. Layers of indirection built up gradually — a story about a character who reads a document that contains instructions. Each layer added separately.
Translation and encoding laundering. Establish that the conversation operates in another language, a cipher, or a code convention, then make the request within it. Safety training generalizes unevenly across languages and encodings, and the establishing turns look like ordinary language preference.
Context stuffing and attention dilution. Fill the window with benign content so the operative instruction competes with a large volume of innocuous text. Long-context models are more susceptible to this than short-context ones.
Memory and persistence variants. Where an agent has durable memory, the attacker plants a preference or "fact" in one session and exploits it in another. The exploit session contains no attack at all. See AI agent memory poisoning defenses.
Why Stateless Detection Cannot Close This
The problem is structural rather than a matter of classifier quality. A per-message classifier is asked: is this message harmful? For a crescendo turn the correct answer is genuinely no. Turn seven of an escalation is a reasonable follow-up to turn six. Lowering the threshold until such turns are caught produces false positives on ordinary conversation, because the distinguishing feature is not in the message — it is in the trajectory.
Two further complications. Models are cooperative by design: they resolve ambiguity toward being helpful, and a gradual escalation exploits that continuously rather than confronting it once. And the model's own prior outputs are part of the context that persuades it, so the attacker gets the model to help build the frame it will later be held to.
Stateful Detection
The unit of analysis has to be the session, not the message.
Trajectory scoring. Maintain a running risk score per session that accumulates weak signals rather than requiring any one to be conclusive: topic drift toward sensitive areas, monotonic escalation in sensitivity, refusals followed by rephrasing, role-framing markers, encoding or language switches, unusual conversation length for the workload. Individually each is noise. Together, over a session, they separate probing from use surprisingly well.
Refusal-then-retry as a first-class signal. This is the highest-value cheap detector available. A refusal followed by a rephrased version of the same request is the fingerprint of iterative probing — legitimate users usually clarify or abandon, they rarely reformulate around a boundary repeatedly. Count refusals per session and alert on rate, not on any single event.
Periodic whole-conversation evaluation. Every N turns, or when the trajectory score crosses a threshold, evaluate the accumulated conversation as a unit with a separate model asked a different question: what is this conversation cumulatively attempting? This catches what per-turn evaluation cannot, at a cost you control by choosing N.
Cross-session correlation per principal. Attackers iterate across sessions to reset accumulated state. Correlate by principal, by API key, by source, so starting a new session does not clear the record. Where an attacker rotates identities, correlate on behavioral shape instead.
Trust-score feedback. Let accumulated risk feed a durable per-agent or per-principal trust level that gates capability. An identity with a probing history operates under tighter constraints on its next session. See trust scores and attestations for agents and trust scoring models for autonomous agents.
The Containment That Actually Holds
Detection reduces the rate. It will not reach zero, because the attack space is open-ended and defenders are always responding to techniques that already work.
So the durable answer is on the action side. A jailbreak produces two possible harms: the model says something it should not, or the model does something it should not. For agentic systems the second dominates, and it is deterministically containable in a way that content is not.
If the agent's connection to a payment tool caps amounts and requires approval above a threshold, no amount of conversational manipulation produces an unapproved large payment. If the agent has no connection to the email tool, no framing makes it send mail. If the read tool caps result volume, a persuaded agent still cannot bulk-extract. These controls do not care how convincing the conversation was, which is exactly what makes them worth more than a better classifier. See excessive agency and how to implement least privilege for agents.
The practical split: content controls for what the agent says to humans, capability controls for what the agent does to systems. Investing only in the first is the common mistake, because content is where the visible failures happen in demos while capability is where the expensive failures happen in production.
Testing for It
Single-shot red teaming will not find these. Your adversarial test suite needs multi-turn scenarios: scripted crescendo sequences against your actual system prompt and tool set, refusal-and-rephrase loops, frame-building followed by exploitation, cross-session memory plants. Run them in CI on every prompt, model, or guardrail change, and assert on the action outcome rather than only on the text — the meaningful pass condition is "no prohibited tool call occurred," not "the model refused politely."
Multi-turn suites are slower and more expensive than single-shot ones, so run a small high-signal set on every change and a larger set nightly. Track success rate over time as a metric; a rising rate after a model upgrade is information you want before your users find it. See continuous adversarial testing for AI agents and red teaming AI agents.
Common questions
Do longer context windows make this worse?
Yes, on balance. More context means more room to build frames, dilute attention, and establish conventions before exploiting them, and it means whole-conversation evaluation costs more, so teams sample it less often. The mitigating factor is that longer windows also let a detector see the entire trajectory in one evaluation — but only if you actually run that evaluation rather than assuming per-turn checks compose.
Is session risk scoring going to annoy legitimate users?
It depends on what you do at each threshold. Scoring should mostly drive graduated responses rather than blocking: at low levels, log and continue; at moderate levels, tighten capability scope or require confirmation on consequential actions; at high levels, involve a human. Blocking on accumulated score alone produces frustrating false positives, because long troubleshooting conversations legitimately drift toward sensitive topics. Tie the response to the action being attempted, not to the score in isolation.
How does this apply to agents with no human in the conversation?
It applies to the equivalent channel: the sequence of tool responses and inter-agent messages that accumulate in the agent's context over a long task. A poisoned tool response early in a task can shape behavior twenty steps later, and per-response filtering has the same blind spot. Score the task trajectory — unusual step counts, repeated failures followed by different approaches, drift in the tool mix — the same way you would score a conversation.
Should I reset context periodically as a mitigation?
It helps and it is cheap. Bounding conversation length, summarizing rather than carrying full history, and starting fresh contexts for new task phases all limit how much frame an attacker can accumulate. The cost is capability: agents genuinely benefit from continuity. A reasonable middle is to summarize aggressively while retaining task-critical facts, and to treat summarization as a control point where injected framing can be dropped rather than carried forward.
How Praesidia approaches multi-turn manipulation
Praesidia evaluates guardrails on every message crossing a governed connection — inbound and outbound, human-facing and agent-to-agent — and records each evaluation with attribution, which is what makes session-level accumulation of refusals and near-misses analyzable rather than invisible. Accumulated behavioral signal feeds a per-agent trust score, and connections can require a minimum trust level, so an agent with a degraded history loses access to sensitive connections automatically.
Because containment lives on the connection — enumerated tool scope, volume caps, spend ceilings, approval gates — a successful conversational manipulation still cannot produce an action the connection does not permit. See guardrails versus evals versus monitoring and the Praesidia documentation.