Most teams that stall on evaluation do not stall on the harness. Running cases and collecting scores is a weekend of work. They stall on the dataset: what cases, expressing what expectations, curated by whom, and kept fresh how.

The dataset is where the judgment lives. A mediocre harness over a good dataset produces useful signal; a sophisticated harness over a set of thirty cases someone wrote in an afternoon produces confident noise.

Where Cases Come From

Production failures — the primary source. Every incident, user complaint, bad review, support escalation, and internal bug report becomes a case. This is the highest-value input by a wide margin, for two reasons: the cases are real, and adding them makes quality improve monotonically rather than oscillating between fixed and re-broken.

Make the pipeline explicit: a failure gets triaged, reduced to a minimal reproducing input, given an expectation, and added to the set as part of closing the ticket. If adding the case is optional, it will not happen, and you will fix the same class of failure three times.

Domain experts. The people who do the work manually know which situations are hard, ambiguous, or high-stakes. Interview them for cases rather than asking for a list — "tell me about a request that would trip up a new hire" yields better cases than "give me test cases."

Red team findings. Every confirmed adversarial finding becomes a permanent case with an assertion on the action outcome. This is how a one-time exercise becomes a durable property. See red teaming AI agents.

Sampled production traffic. Necessary to keep the distribution honest, because curated sets skew toward interesting cases while real traffic is mostly mundane. Sample stratified across task types and label a subset.

Synthetic generation. Useful for breadth — many variations of a known pattern, coverage of parameter ranges, long-tail input shapes. Its weakness is that generated cases cluster around what the generating model finds salient, so they under-represent exactly the surprising inputs you want. Use it to fill known gaps, not to define coverage.

Deliberate boundary construction. Cases just inside and just outside each policy boundary. If the refund limit is $500, you need cases at $499 and $501. These are cheap to write and they catch threshold drift.

Expressing Expectations for Non-Deterministic Systems

An exact expected output does not work. The agent will legitimately vary phrasing, ordering, and detail, and a strict comparison fails on correct answers. Teams that start with exact-match expectations end up either loosening them into meaninglessness or maintaining brittle fixtures.

Express expectations as properties instead.

Must-contain facts. The specific values, entities, or conclusions that must appear. Checked semantically or by pattern, not by exact string.

Must-not-contain. Prohibited content: other tenants' data, disallowed claims, PII, internal identifiers. Often more valuable than must-contain, because the failures are more serious.

Required tool calls. The tools that must be invoked. Optionally an ordering constraint where order matters.

Prohibited tool calls. Tools that must not be invoked for this case. This is where scope regressions surface.

Argument constraints. Bounds on tool arguments — tenant scoped correctly, limit under a ceiling, no wildcards.

Ceilings. Maximum steps, maximum tokens, maximum cost, maximum latency. Cheap to add and the most reliable detector of silent model changes.

Final state assertions. For agents with side effects, assert on the resulting system state — record created with these fields, no other records modified. The strongest expectation type available, and underused.

Rubric dimensions. For genuinely subjective properties, a rubric with enumerated criteria for a model judge. Keep these concrete: "cites a source for each factual claim: yes/no" beats "rate quality 1-10".

A well-formed case therefore has an input, a context fixture, and a set of these constraints — not a golden string.

Fixtures: The Part That Rots

Agent cases depend on state: the retrieval corpus, the tool responses, the account data, the current date. If the fixture is the live environment, the case's outcome changes when the environment does, and you cannot distinguish agent regression from data drift.

Pin the fixture:

  • Deterministic tool stubs returning recorded responses for the case. Real tool calls make cases slow, flaky, and occasionally destructive.
  • A pinned corpus snapshot for retrieval-dependent cases, with the expected retrieved document ids as part of the expectation. Then retrieval quality is measurable separately from generation quality.
  • Frozen time where date logic matters. A case that asserts "within the last 30 days" starts failing on its own.
  • Seeded account data rather than production records.

Keep a smaller set of live-integration cases separately, run less frequently, to catch tool contract changes that stubs hide by construction. This is the trade: stubs give reproducibility, live calls give contract coverage, and you need both at different cadences.

Slicing and Structure

Decide slices at construction time, because retrofitting them across hundreds of cases is tedious and usually does not happen.

Tag every case with: task type, difficulty, required tools, language, input length band, tenant or customer segment shape, and whether it is a safety case or a quality case. Then results can be read per slice, which is what makes them diagnostic. An aggregate score that holds steady while one task type collapses is the most common way a suite fails to catch a real regression.

Balance matters. If 80% of your cases are one task type, the aggregate reflects that type and nothing else. Either balance the set or always read per-slice — and if you always read per-slice, ensure each slice has enough cases to be statistically meaningful, which in practice means at least twenty to thirty per slice you intend to gate on.

Versioning and Curation

Version the dataset like code. In a repository, reviewed on change, with a version identifier recorded on every eval run. A score is meaningless without knowing which dataset version produced it.

Treat additions as deliberate. Every added case shifts the mix and therefore the aggregate. Add with intent, note why, and re-baseline when the mix changes materially.

Split fast and full. A fast subset — deterministic and trajectory checks, a few hundred cases, minutes — gates merges. The full set runs on a schedule. Choose the fast subset for coverage of failure modes rather than by sampling randomly.

Review for staleness periodically. Cases whose expectations encode obsolete business rules produce failures that get overridden, which teaches the team to override failures generally. That is how a suite dies: not by being deleted but by being ignored.

Watch for contamination. Cases that end up in fine-tuning data or in a prompt's few-shot examples no longer measure generalization. Keep the eval set separate from anything that feeds the system under test, and check for overlap when you build training data.

Privacy Obligations Come Along

A dataset built from production traffic contains customer data, and copying it into a repository creates a second copy in a system with different access controls and, usually, no retention policy.

Practical handling: redact or synthesize personal data during case construction, keeping the structure that makes the case hard while removing the identifying content. Where realistic personal data is genuinely necessary, keep those cases in a controlled store with access logging rather than in version control, and include them in your deletion processes — an erasure request that misses your eval dataset is incomplete. See GDPR for AI systems and erasure rights and how to redact PII from agent prompts.

Common questions

How many cases do I need to start?

Fifty real cases beat five hundred synthetic ones. Start with every production failure you can recall plus a dozen expert-supplied hard cases, and grow from incidents. Teams that wait until they have a comprehensive set never ship the harness; teams that start with fifty and add every failure reach a good set within a quarter.

Should each case have one expectation or many?

Many, expressed as separate constraints, each reported separately. A case that fails on step count but passes on correctness is telling you something specific, and collapsing it into a single pass/fail discards that. The reporting granularity should match the constraint granularity.

How do I handle cases where the correct answer changes over time?

Either freeze the fixture so the answer is stable, or express the expectation as a property of the process rather than the output — "queried the current-rates tool and used its result" rather than "answered 4.5%". The second is often the better test anyway, because what you actually want to verify is that the agent consults the authoritative source.

Who owns the dataset?

Whoever owns the agent's quality, with contributions from support, domain experts, and security. Ownership matters because curation is ongoing work that is easy to defer. A dataset with no owner drifts, accumulates stale expectations, and stops gating anything. Naming an owner and putting curation in their regular workload is what keeps it alive.

How Praesidia approaches eval datasets

Praesidia stores eval datasets as first-class, versioned, organization-scoped resources alongside evaluation runs, so a score is always attributable to a specific dataset version and results are comparable over time. Cases can carry trajectory-level expectations — required and prohibited tool calls, argument constraints, step and cost ceilings — because the same governed-connection records that drive observability supply the trajectory.

Because production interactions are already recorded with their trajectories, a real failure can be turned into a dataset case from the record rather than reconstructed by hand. See agent evaluations: catching quality regressions and the Praesidia documentation.