Teams put considerable effort into what goes into a model and comparatively little into what comes out. The input side gets prompt-injection scanning, PII redaction, and content policy. The output side often gets a JSON parse and a hope.

That asymmetry is a mistake, because model output is untrusted data. Not because the model is malicious, but because its output is a function of its input, and some of that input came from places you do not control — retrieved documents, tool responses, user messages, other agents. Anything that can influence the model's context can influence its output, and if that output is interpolated into a context where structure carries meaning, you have an injection vector with the model as the delivery mechanism.

The Consumption Points That Bite

Insecure output handling is a boundary problem. It matters exactly where output stops being text and starts being structure.

Rendered HTML. An agent's response is displayed in a web UI. If it is inserted as HTML rather than text, script tags, event handlers, and javascript: URLs execute in the user's session. Markdown rendering makes this worse, not better: markdown permits raw HTML by default in many renderers, and link and image syntax carry URLs. An agent that can be induced to emit a link is an agent that can be induced to exfiltrate — a rendered image URL with data in the query string leaks on load, with no user interaction at all.

SQL and query languages. An agent generates a query — for analytics, search, or a database tool — and the query string is executed. This is textbook SQL injection with a novel source. The same applies to any query dialect: GraphQL, Elasticsearch DSL, LDAP filters, XPath.

Shell and code execution. An agent proposes a command, a script, or a code snippet and something runs it. Coding agents are the obvious case; less obvious are agents that generate file paths, archive names, or CLI arguments that get concatenated into a shell invocation. See securing AI coding agents and agent sandbox escape.

File paths and storage keys. Output used as a filename or object key permits traversal (../), overwrite of adjacent tenants' objects, or writes to locations that are themselves executable.

Downstream prompts. Output from agent A becomes part of agent B's context. This is the path teams control least, and it is the one that makes multi-agent systems fragile: an injection that lands in agent A propagates as apparently-trusted internal content to agent B. Because it arrives from an internal service rather than from a user, it typically bypasses whatever input controls exist at the perimeter.

Logs and dashboards. Output written to logs that are later rendered in a web console produces stored XSS in the operations tooling — an especially bad outcome, because the viewer is usually an administrator. Log entries containing newlines and forged prefixes also corrupt log parsing, which is how attackers hide activity from detection pipelines.

Schema Validation Beats Pattern Filtering

The instinctive fix is to scan output for dangerous patterns: strip <script>, reject strings containing DROP TABLE, filter shell metacharacters. This does not hold. Denylists on free-form text lose to encoding, casing, nesting, and unicode variants, and they produce false positives on legitimate content — a security agent discussing SQL injection will be blocked by a filter for SQL injection.

The approach that holds is to constrain structure at the source and validate it at the boundary.

Ask for structured output and enforce the schema. Use the provider's structured-output or tool-calling mode so the model emits typed fields rather than prose you must parse. Then validate against the schema server-side, rejecting rather than coercing on mismatch. A field typed as an enum of three permitted actions cannot contain a shell command, no matter what the model was persuaded to produce.

Constrain fields to the narrowest type that expresses the intent. Not "a string containing a SQL query" but "a table name from this enumerated list, a column list from this set, an operator from this set." Compose the query yourself from validated components. The model chooses among options; it does not author syntax.

Escape at every consumption point, correctly for that context. HTML-escape for HTML, parameterize for SQL, argument-array for shell (never string concatenation), canonicalize-and-verify for file paths. Contextual escaping is a solved problem in every mature framework; the failure is forgetting that model output needs it.

Allow-list URLs and link targets. If output can contain links or images, restrict schemes to https and hosts to a known set, or strip them. This closes the rendered-image exfiltration channel, which is otherwise the cheapest data-egress path out of an agent UI. See data exfiltration risks in agentic AI.

The Multi-Agent Path Needs Explicit Treatment

When agent output feeds another agent, two rules apply.

First, inter-agent messages carry provenance. Content that originated outside the trust boundary stays marked as external for the whole chain, so a downstream agent's controls can treat it accordingly. Once provenance is lost, every downstream agent treats attacker content as internal and trusted.

Second, delegation narrows authority. A sub-agent's output must never be able to expand what the calling agent may do — no "the user has approved elevated access" assertions accepted from message content. Authority comes from tokens and connection scopes, not from claims in a payload. See threat model: A2A delegation abuse and securing inter-agent communication.

Output guardrails belong on this path too, evaluated on the message crossing between agents, not only on what is shown to a human. In practice, this is where teams discover their guardrails were only ever wired to the user-facing surface.

Output Guardrails: What They Do and Do Not Cover

Content guardrails on output serve a real purpose that schema validation does not: they bound what the content says rather than what shape it has. PII in a response, secrets echoed from a tool result, disallowed topics, tone and policy violations — these are content properties, and pattern or classifier-based evaluation with block, redact, or warn dispositions is the right instrument. See designing guardrails: block, redact, warn and PII detection and redaction in AI pipelines.

What they do not cover is structural injection, because they operate on semantics and injection exploits syntax. A guardrail tuned to detect leaked credentials will not object to a well-formed <img> tag with an attacker-controlled host. You need both layers, and they enforce at different points: guardrails on the content leaving the model, typing and escaping at each place the content is consumed.

Common questions

If the model is well-aligned, is this still a real risk?

Yes, because alignment is not the control surface here. The model does not need to intend anything — it needs to have processed content that steered its output. A retrieved document containing a crafted string, a tool response echoing attacker-supplied data, a filename in a repository. The model reproduces it, and your renderer executes it. Alignment reduces the rate of unprompted bad output; it does nothing about output that is a faithful function of poisoned input.

Does using JSON mode make this safe?

It makes parsing reliable, which is different from safe. JSON mode guarantees the envelope is well-formed. It does not constrain field contents — a string field can contain any string, including a shell command or a script tag. The safety comes from validating field values against narrow types after parsing, and from escaping at consumption. JSON mode is a prerequisite, not the control.

How do I handle agents that legitimately generate code?

Separate generation from execution and make the execution boundary the control. Generated code is reviewed by a human, or executed in a sandbox with no network egress, no credentials, a read-only filesystem outside a scratch directory, and CPU and wall-clock limits — and its outputs are treated as untrusted in turn. What you must not do is execute generated code in the same trust context as the agent's own credentials, which is the configuration that turns a code-generation bug into full compromise.

Should output validation happen in the agent or outside it?

Outside, at each consumption boundary, for the same reason input enforcement belongs in the proxy: validation inside the agent shares a trust context with the content that may have subverted it. The renderer HTML-escapes. The database layer parameterizes. The gateway between agents validates the schema. Each boundary defends itself rather than trusting an upstream promise.

How Praesidia approaches output handling

Praesidia evaluates guardrails on both directions of a governed connection, so responses and inter-agent messages are inspected rather than only inbound prompts. Guardrails support block, redact, and warn dispositions, which lets PII and secret patterns be removed from a response without failing the whole task. Because enforcement sits in the proxy between the agent and its counterparty, an agent cannot opt out of the evaluation of its own output.

Every evaluation is recorded with enough context to show which rule acted and in which direction, so an investigation can reconstruct what left the boundary and what was modified. For the broader control model see content guardrails for AI agents and the Praesidia documentation.