Conventional software fails loudly. A regression breaks a test, throws an exception, or returns a wrong status code. Agents fail quietly: the response is slightly less accurate, the agent takes eleven steps instead of four, it stops using the right tool for a category of request, it becomes more verbose and doubles token cost. Nothing errors. Users notice before you do, and by then it has been happening for weeks.
The situation is worse than for ordinary software because agents regress from changes you did not make. The provider updates the model. A tool's API changes its response shape. Your retrieval corpus grows and the relevant document stops ranking. None of these appear in your commit history.
Evaluation is the instrument that closes this gap. It is not optional for anything running in production.
What to Measure
The instinct is to score the final output. That is necessary and insufficient — two agents can produce the same answer with wildly different processes, and the process is where cost, latency, and reliability live.
Task success. Did the agent accomplish the goal? Binary where possible, because graded success invites judgment drift. For tasks with verifiable outcomes — a record was created, a query returned the right rows, a calculation matched — check the outcome directly rather than asking a model about it.
Tool selection. Did it call the right tools? Wrong tool choice is the most common agent regression after a model change, and it is cheap to check because the expected tool set is knowable per test case.
Argument correctness. Were tool arguments well-formed and appropriately scoped? An agent that calls the right tool with a wildcard filter has regressed in a way that final-answer scoring will not see, and that has security implications.
Step count and efficiency. Trajectory length, token consumption, and cost per task. A doubling in step count is a serious regression even when success rate holds, because it multiplies cost and latency. This is the metric most often missing from eval suites and the one that most often catches a silent model change.
Latency. End to end and per step, at p50 and p95.
Refusal and over-refusal rate. Both directions matter. A model update that makes the agent refuse legitimate requests is a regression that users experience as breakage.
Safety and policy adherence. Did it leak data, exceed scope, or violate a content rule? These belong in the same suite as quality, run continuously. See continuous adversarial testing for AI agents.
Scoring Methods, In Order of Preference
Deterministic checks. Exact match, schema validation, numeric tolerance, set comparison on retrieved document ids, assertion on final system state. Fast, free, perfectly reproducible. Use these for everything they can express, which is more than teams assume — most agent tasks have at least some checkable property, even if the prose is subjective.
Programmatic assertions on trajectory. Did the tool call sequence include the required tool? Did it avoid the prohibited one? Were arguments within bounds? Was step count under a ceiling? These are ordinary code and they carry a large share of the useful signal.
Model-based judging. A separate model scores the output against a rubric. Necessary for subjective dimensions — helpfulness, tone, faithfulness to sources — and it comes with real caveats: judges have position and verbosity biases, they drift when the judge model updates, and they are expensive at scale. Use them for what nothing else can measure.
Human labeling. The ground truth, expensive and slow. Its main job is calibrating the automated methods rather than running continuously.
The practical structure is a pyramid: deterministic checks on everything, trajectory assertions on most cases, model judging on a subset covering subjective dimensions, human labels on a small rotating sample used to validate the judge.
Validating the Judge
An unvalidated model judge is a metric you do not understand. Before trusting it:
- Label a few hundred examples by hand, spanning clear passes, clear failures, and ambiguous cases.
- Run the judge on the same examples.
- Measure agreement — not just overall accuracy but where it disagrees and in which direction.
- Iterate on the rubric until agreement is acceptable for your purposes.
- Re-validate whenever the judge model or the rubric changes.
That last step is the one that gets skipped, and it is how eval suites quietly stop meaning anything: the judge model updates, its scoring shifts, and a metric change gets attributed to the system under test rather than to the instrument. Pin your judge model version explicitly and treat changing it as a change requiring re-validation.
Practical rubric advice: make the rubric concrete and enumerable rather than asking for a 1-10 score. "Does the response cite a source for each factual claim: yes/no" produces far more stable judgments than "rate the quality." Ask for structured output so the judgment is parseable, and ask for the reason alongside the verdict — reading disagreement rationales is the fastest way to find rubric ambiguity.
Where Evals Belong in the Pipeline
Pre-merge, fast subset. Fifty to a few hundred cases, deterministic and trajectory checks only, minutes to run. Gates the merge. This catches obvious breakage from prompt and tool changes.
Post-merge, full suite. Everything including model judging, on a schedule. Reports rather than gates, with alerts on threshold breaches.
On provider model change. Run the full suite against the new model version before switching. If your provider pins versions, this is a controlled event; if you use a floating alias, you will get the change without notice, which is an argument for pinning.
Continuously in production, on sampled traffic. Offline evals use your test set; production traffic contains distributions you did not anticipate. Sample real interactions, score them with the same rubrics, and compare against offline results. Divergence tells you your test set has drifted from reality. See observability for AI agents.
On corpus change. Retrieval-dependent agents should re-run retrieval-quality evals when the corpus is reindexed or substantially updated.
Building the Dataset
The suite is only as good as the cases in it. Sources that work, in rough order of value:
- Production failures. Every incident, complaint, and bug becomes a permanent test case. This is the highest-value source by a wide margin, and it means quality improves monotonically rather than oscillating.
- Edge cases from domain experts. The cases practitioners know are hard.
- Adversarial cases from red teaming, converted into regression tests.
- Representative sampled traffic, labeled, to keep the distribution honest.
- Synthetic generation for coverage breadth, used carefully — synthetic cases tend to cluster around what the generating model finds salient, which biases coverage.
Version the dataset and treat additions as deliberate. A suite that grows by accretion without curation becomes slow and its aggregate score becomes uninterpretable, because the mix keeps shifting. See building golden datasets for agent evals.
Reading the Results
Report per-dimension and per-slice, not as a single number. An aggregate score hides the thing you need to see: overall success holding steady while one category collapses. Slice by task type, input length, tool required, tenant or customer segment, and language.
Track distributions, not just means. p95 latency and p95 cost matter more than averages for user experience and for budget, and a mean that holds while the tail degrades is a common failure shape.
Account for non-determinism. Run each case several times and report success rate rather than pass/fail; a case that passes four times in five is different from one that always passes, and the difference matters. Set thresholds on rates with enough trials that the noise band is narrower than the regression you care about detecting.
Common questions
How large should the eval suite be?
Large enough that a real regression moves the metric beyond noise, small enough to run often. In practice, a few hundred cases for the fast gate and one to a few thousand for the full suite covers most systems. Coverage of failure modes matters far more than raw count — fifty cases spanning your actual failure taxonomy beat a thousand near-duplicates.
Can I use production traffic directly as eval data?
Yes, with care on two fronts. Privacy: production traffic contains customer data, so it needs redaction, consent review, and access control before it becomes a test fixture, and copying it into a repository is usually the wrong move. Staleness: traffic-derived cases capture the distribution at collection time, so refresh periodically. The strongest pattern is sampling production traffic for continuous scoring while keeping a curated, redacted set as the versioned regression suite.
Should evals gate deployment?
The fast deterministic subset, yes — it is quick and its failures are unambiguous. Model-judged scores, generally no: judge variance produces flaky gates, which teams then learn to override, which destroys the signal. Gate on what is deterministic, alert on what is judged.
How do evals relate to guardrails and monitoring?
Evals measure quality before deployment against known cases. Guardrails enforce policy at runtime on every request. Monitoring observes what actually happened in production. They are complementary and none substitutes for another — a system with excellent evals and no guardrails ships a well-measured agent with no runtime containment. See guardrails versus evals versus monitoring.
How Praesidia approaches evaluation
Praesidia provides evaluation runs and versioned eval datasets scoped to your organization, so agent behavior can be scored against curated cases and re-scored when prompts, models, tools, or corpora change. Because every governed call already records the trajectory — tool calls, arguments, latency, cost, guardrail dispositions — trajectory-level scoring draws on the same records that drive observability rather than requiring separate instrumentation.
Results are stored per run so regressions are comparable over time, and adversarial cases can live in the same suites as quality cases, which keeps safety regressions visible on the same cadence as quality ones. See advanced analytics for AI operations and the Praesidia documentation.