Agent loops resend context. On every step, the accumulated conversation, tool definitions, system instructions, and retrieved documents go back to the model along with the incremental new information. A ten-step task sends the stable portion of its context ten times.
Prompt caching addresses exactly this: providers cache the computed state for a prompt prefix and charge substantially less for cached input tokens than for fresh ones. For agent workloads the savings are large — often the single highest-leverage cost optimization available, and considerably easier to adopt than model routing because it requires no quality trade-off.
It is also easy to lose accidentally, which is the part worth understanding.
Caching Is Prefix-Based
The mechanism is a shared prefix. The provider caches the computed state up to some point in the prompt, and a subsequent request whose prompt begins with the identical byte sequence reuses it. The match must be exact and it must be from the start.
The consequence dominates everything else about prompt design: anything variable placed early in the prompt destroys the cache for everything after it. A timestamp in the system message, a request identifier, a user name, a randomized instruction ordering — any of these near the top means every request is a cache miss regardless of how much stable content follows.
This is the most common reason teams see poor cache performance. The content is cacheable; the layout is not.
Structuring Prompts for Hits
Order from most stable to most variable:
- System instructions. Static per agent version. The most valuable thing to have in the cached prefix.
- Tool definitions. Static per agent configuration, and often large — tool schemas with descriptions can be thousands of tokens.
- Stable reference context. Policy documents, product catalogues, schema descriptions, few-shot examples. Anything the agent needs on every request.
- Semi-stable context. Tenant or user-specific reference data that is stable within a session. Cacheable per session or per tenant.
- Conversation history. Grows monotonically, so earlier turns remain a stable prefix as the conversation extends — which is why append-only history caches well and history rewriting does not.
- The current request. Always variable, always last.
Specific things to remove from the early portion:
- Timestamps and dates. If the agent needs the current date, put it late in the prompt, or resolve it through a tool call instead.
- Request and trace identifiers. These belong in metadata, not in the prompt.
- Randomized example or tool ordering. Sometimes done deliberately to counter position bias; it is expensive and usually not worth the trade.
- Dynamically assembled tool lists. If the tool set varies per request, the definitions block is not cacheable. Prefer a stable per-agent tool set — which is also better scope hygiene.
What Invalidates the Cache
Any prefix edit. Changing a word in the system prompt invalidates every cached entry for that agent. This is correct behavior and it means prompt changes have an immediate cost effect until the cache re-warms.
TTL expiry. Provider caches are short-lived — typically minutes. Low-traffic agents may miss consistently because entries expire between requests. High-traffic agents keep prefixes warm naturally.
Tool definition changes. Adding a tool, editing a description, or changing a schema invalidates. Tool descriptions get edited casually; the cost consequence is not obvious.
Context reordering. A refactor that moves a block earlier or later invalidates everything from the change point.
Retrieved content in the prefix. If retrieved documents are placed before the conversation history, every retrieval variation invalidates. Put retrieval results after the stable blocks.
The practical implication: cache hit rate is a metric to monitor, and a drop is a cost regression that deserves an alert. A one-word prompt edit that lands during a deploy can measurably increase your inference bill, and without a hit-rate metric nobody will connect the two. See anomaly detection on agent spend.
Cache Scope and Isolation
Caching interacts with multi-tenancy and it needs deliberate handling.
Never share a cached prefix across tenants if it contains tenant data. A prefix containing one tenant's reference documents must not be reusable by another tenant's request. Provider caches are generally scoped to your account and matched on exact content, so cross-tenant reuse requires identical content — but if your prompt assembly places tenant-specific content in the prefix, you should be scoping the cache key by tenant explicitly rather than relying on content inequality.
Shared prefixes should contain only shared content. System instructions, tool definitions, and general policy documents are safe to share across all tenants. Tenant-specific reference data goes in a per-tenant cache segment.
Be careful with caching as a side channel. Where cache hits are observable through latency, an attacker could in principle infer whether specific content was recently processed. This is a low-severity concern for most deployments and worth noting for high-sensitivity ones — the mitigation is not to place secret-dependent content in a cacheable prefix.
Attribution of savings. When one tenant's traffic warms a cache that another tenant benefits from, the cost saving is not cleanly attributable. Pick a rule — usually charging cached tokens at the cached rate to whoever used them — and publish it. See spend attribution and showback.
Beyond Provider Caching
Provider prompt caching is one layer. Two others are worth having.
Response caching for identical requests. If the same question with the same context arrives twice, serving the stored response costs nothing. This works for narrow, deterministic-ish workloads — classification, extraction from identical documents, FAQ answering — and needs care about staleness and about whether the underlying data changed. Scope the cache key to include the tenant and the relevant data version, or you will serve one tenant's answer to another or serve a stale answer after an update.
Semantic caching. Matching near-identical requests by embedding similarity rather than exact match. Higher hit rates and a real correctness risk: two questions that embed similarly can have different correct answers, and the failure is silent. Use a high similarity threshold, restrict it to narrow domains where near-duplicates genuinely have the same answer, and treat it as an optimization requiring evaluation rather than a free win.
Context trimming as the complementary lever. Caching makes resending cheap; trimming makes there be less to resend. Summarize old conversation turns rather than carrying them verbatim, cap retrieved chunk counts, and drop tool results once they have been used. Trimming and caching interact — trimming rewrites history and therefore invalidates the prefix — so summarize at defined boundaries rather than continuously, and accept the invalidation at those points.
Common questions
How much does prompt caching save in practice?
It depends on the ratio of stable prefix to variable content and on how many steps a typical task takes. Agents with large tool definitions and long system prompts running multi-step loops see the biggest effect; single-turn agents with short prompts see little. The way to find out is to measure your prefix-to-total token ratio, which is a straightforward calculation from data you already have if you record token counts per call.
Do I need to change my code to use it?
Depends on the provider — some cache automatically above a minimum prefix length, others require explicit cache markers in the request. Either way, the work is in prompt layout rather than in code: ordering blocks from stable to variable, and removing variable content from the top. That is usually a small change with a disproportionate effect.
Does caching affect output quality?
No. The cache stores computed state for a prefix that is byte-identical to what would otherwise be computed, so the model sees the same input. This is what makes it unusual among cost optimizations — there is no quality trade-off to evaluate, unlike model routing or context trimming.
Should I cache across agent versions?
You cannot meaningfully — a version change that alters the system prompt or tool definitions invalidates the prefix by construction. Plan for a cost bump around prompt deployments, and consider that as one more reason to batch prompt changes rather than shipping them continuously. Where a deployment causes a large cache-miss period, staged rollout limits the cost as well as the risk. See staged guardrail rollout and shadow mode.
How Praesidia approaches caching and context cost
Praesidia meters cached and uncached input tokens separately per governed call, so cache hit rate is a first-class metric per agent and per connection rather than something inferred from a bill. A drop in hit rate after a prompt or tool-definition change shows up as a cost signal with the change correlated, which is what makes the regression attributable.
Because prompts, tool definitions, and LLM configurations are versioned platform configuration rather than code embedded in each agent, prefix stability is something you can manage deliberately — and per-tenant configuration keeps tenant-specific reference content out of shared prefixes. See cost control for LLM applications and the Praesidia documentation.