Hard spend ceilings are the control that prevents a catastrophic bill. They are deliberately blunt: when the cap is hit, the call is blocked. That is the right behavior for a runaway loop or an abuse attempt.
What caps do not catch is drift. Cost per task creeps up 15% after a model update. An agent that used to take four steps now takes seven. A retry path that fires occasionally starts firing often. None of these breach a cap. All of them are expensive, and the total effect over a quarter usually exceeds what any single blocked incident would have cost.
That is the gap anomaly detection fills, and agent workloads are unusually amenable to it.
Why Agents Baseline Well
Anomaly detection on human behavior is hard because humans are erratic — they travel, change roles, work odd hours, and have legitimate one-off needs. Agents are the opposite:
Repetitive. The same agent runs the same task type with similar inputs, so cost per task has a narrow distribution.
Scheduled or event-driven. Volume follows a predictable pattern with clear daily and weekly seasonality.
Narrow in scope. A given agent uses a small, stable set of tools and models.
High volume. Enough observations to establish a distribution quickly, often within days.
That means simple methods work well and you do not need anything sophisticated. Rolling percentiles, week-over-week comparison at the same hour, and standard-deviation bands on per-task cost catch the great majority of real anomalies. Reaching for machine learning here is usually premature and produces a detector nobody can explain when it fires.
The Signals Worth Watching
Cost per task. The single highest-signal metric. It isolates efficiency from volume — if total spend rises because you are doing more work, that is business as usual; if cost per task rises, something changed. A step increase in cost per task with no deployment is almost always a provider model change.
Tokens per task, split input and output. More diagnostic than cost, because it distinguishes causes. Rising input tokens means context growth — longer histories, more retrieved chunks, a bigger prompt. Rising output tokens means the model became more verbose. Different fixes.
Steps per task. Trajectory length. An increase means the agent is working harder for the same outcome: worse tool selection, more retries, more clarification loops. Cost grows roughly quadratically with step count in most agent loops because context accumulates, so this is an early warning for a large cost move.
Spend rate, compared against the same hour last week. Volume anomalies with seasonality handled. Comparing against yesterday catches weekday-weekend artifacts; same-hour-last-week is the more reliable baseline.
Retry and failure rate. Retries are pure cost with no output. A rising retry rate is both a cost and a reliability signal, and it often precedes a user-visible problem.
Cache hit rate, where prompt caching is used. A drop means a prompt-prefix change invalidated the cache, and the cost effect can be large and immediate. See prompt caching to cut inference cost.
Share of spend by principal. One user or tenant taking a rising share is either a growing account or an abuse pattern — and either way you want to know.
Model mix. Traffic shifting toward more expensive models. Usually a routing change or a fallback firing more often than expected. See model routing and cost-quality tradeoffs.
First-seen dimensions. A new model, a new destination, a new agent appearing in spend data. Not necessarily an anomaly, always worth a notification.
Ratios and Rates, Not Totals
Alerting on absolute totals produces alerts on normal growth, which trains people to ignore the channel. Alert on:
Rates of change. Cost per task up 20% week over week. Steps per task up 30%.
Ratios. Retry cost as a fraction of total. Cache misses as a fraction of eligible requests. One agent's share of organization spend.
Deviation from a baseline band. Current hour's spend outside the p5–p95 band for that hour of that weekday over the last four weeks.
Forecast breach. Projected month-end spend against budget, based on run rate. This is the one that gives you time to act rather than telling you afterward. See token usage forecasting and capacity planning.
Use multiple time windows. A short window (minutes) catches runaway loops; a medium window (hours) catches abuse; a long window (days to weeks) catches drift. A single window misses two of the three, and most implementations only build the short one because that is the scary case.
Making Alerts Actionable
An alert channel that fires often and vaguely gets muted within a month, and then you have neither detection nor caps that catch drift.
Every alert should carry:
The subject, attributed. Which agent, which connection, which principal, which tenant. "Spend anomaly detected" is unactionable. "Agent invoice-parser cost per task up 34% since 14:00, affecting tenant X" is actionable. This requires attribution captured at call time — see spend attribution and showback.
The magnitude and the money. Percentage change and projected additional cost if it continues. The dollar figure determines whether anyone cares.
The probable cause, where inferable. Correlate the change with recent events: a deployment, a prompt version change, a model version change, a corpus reindex, a traffic source change. Most anomalies have a proximate cause visible in your own change log, and surfacing it removes the first twenty minutes of investigation.
A suggested action. Investigate the agent, check the model version, review the routing rule, or contact the tenant.
Route by severity and audience. A projected budget breach goes to the owning team; a suspected abuse pattern goes to security; a hard-cap breach pages. Not everything belongs in the same channel. See Slack and multi-channel alerting and in-app notifications.
Responding Automatically, Carefully
Some responses can be automated safely, and the safe ones share a property: they are reversible and they degrade rather than deny.
Safe to automate: tightening a rate limit, routing to a cheaper model for non-critical task classes, reducing retrieval breadth, lowering the retry allowance, and requiring approval for the affected agent's consequential actions.
Not safe to automate: disabling an agent that serves customer-facing traffic, blocking a tenant, or changing a hard cap. These need a human, because the cost of a false positive is a production outage.
The middle path most teams settle on is automated tightening plus notification, with a documented manual escalation for anything that would deny service. And the automation itself needs a ceiling — an automated response that can be triggered repeatedly by a flapping signal should have a cooldown.
Common questions
How long before I have a usable baseline?
For a high-volume agent, days. For per-task metrics like cost per task, a few hundred tasks is usually enough for a stable distribution. Low-volume agents — a handful of runs per day — will not baseline well, and for those, static thresholds set from observed maxima are more practical than statistical detection.
Won't a model upgrade trigger alerts constantly?
It should, once, and that is the point — a provider model change altering your cost per task by 20% is exactly what you want to hear about. The fix is not to suppress the alert but to re-baseline after a deliberate change: mark the change in your event log, expect the alert, verify the new level is acceptable, and reset the baseline. Alerts that fire on unexplained changes and stay quiet on explained ones require that you record the explained ones.
Should I detect anomalies per agent or per organization?
Both, at different sensitivities. Per-agent detection catches specific regressions and is where most actionable findings are. Organization-level detection catches aggregate drift that no individual agent shows — a hundred agents each 5% worse is invisible per agent and material in total. Per-tenant detection matters additionally if you bill customers, since a tenant's cost anomaly may be a margin problem before it is an engineering problem.
How does this relate to hard caps?
Complementary and both necessary. Caps are deterministic, enforced pre-dispatch, and prevent catastrophe — they are the control. Anomaly detection is statistical, evaluated after the fact, and catches the expensive-but-not-catastrophic. A system with caps and no detection bleeds money slowly; a system with detection and no caps eventually gets a very large invoice. See budgets versus rate limits and budget policies and hard spend caps.
How Praesidia approaches spend anomalies
Praesidia meters every governed call with per-agent, per-connection, per-principal, and per-tenant attribution, and exposes spend, token volume, step counts, latency, and per-task cost as time series — which is the substrate baselining needs without separate instrumentation. Because guardrail outcomes, model versions, and retry paths are recorded on the same records, a cost change can be correlated with the configuration change that caused it.
Hard ceilings enforce pre-dispatch at agent, connection, team, and organization level, so detection sits alongside a control rather than substituting for one, and alerts route through Slack, email, web push, and in-app channels with the affected subject attributed. See advanced analytics for AI operations and the Praesidia documentation.