Evaluation Datasets
An evaluation dataset is a named, organization-scoped set of test cases attached to an evaluation. Cases are added manually or sampled from recent production agent traffic, and two runs over the same dataset can be compared case-by-case to see what regressed.
Before you start
- You need an existing evaluation to attach dataset cases to — a dataset supplies cases, the evaluation runs the LLM-as-Judge scoring pass over them.
- Dataset and case management requires the permission to update agents (
agents.update) for most operations. Sampling cases from production additionally requirescompliance.manage, because that route reads redacted production task input and is treated as a compliance-adjacent data export, not an ordinary agent-config edit. - The feature gate on evaluations must be enabled for your organization.
1. Create a dataset
In the product, open Assurance → Datasets and select New Dataset. Give it a name (2–255 characters) and an optional description. You can optionally scope a dataset to a single agent — when scoped, sampling from production only pulls that agent's tasks.
A dataset tracks how it was populated: manual (cases added one at a time) or sampled-from-prod (bulk-imported from recent completed agent tasks). It also keeps a running count of its non-deleted cases.
POST /organizations/{orgId}/evaluations/datasets
{ "name": "Password-reset regression suite", "description": "...", "agentId": "..." }
2. Add cases
Each case has an input payload (the object sent to the agent or prompt version — shape is yours to define), an optional expectedOutput reference string the judge compares against, and up to 50 free-form tags for grouping and filtering. A case must reference the evaluation it belongs to, so it can be picked up by an evaluation run.
POST /organizations/{orgId}/evaluations/datasets/{datasetId}/cases
{ "evaluationId": "...", "input": { "userMessage": "How do I reset my password?" }, "expectedOutput": "...", "tags": ["regression", "golden"] }
List, and remove, cases with GET and DELETE on the same cases path.
3. Sample cases from production
Instead of writing cases by hand, sample them from recently completed agent tasks:
POST /organizations/{orgId}/evaluations/datasets/{datasetId}/sample-from-prod
{ "evaluationId": "...", "limit": 50 }
limit accepts 1–200 (default 50). Sampled task input is redacted on copy before it becomes a case; review sampled cases before relying on them, and treat the dataset itself as sensitive if your production traffic contains customer data — see PII redaction in agent prompts.
4. Compare two runs
To see whether a change regressed the dataset, compare two evaluation runs over the same set of cases:
GET /organizations/{orgId}/evaluations/datasets/{datasetId}/compare?evalId=...&v1RunId=...&v2RunId=...
The response lists which case IDs regressed, improved, or stayed unchanged between the baseline run (v1RunId) and the candidate run (v2RunId), plus an aggregate regression rate. Read per-slice using your tags rather than only the aggregate — see building golden datasets for agent evals for why an aggregate score can hide a collapsed slice.
In the product
The Assurance page has a Datasets tab alongside the Evaluations tab. It lists your datasets with case counts, and lets you create a dataset, open one to add or remove cases, and trigger a production sample — the same operations as the API above, from the UI. For what else the Assurance surface covers, see Assurance.
Continue with guardrails to gate what a scored agent can do, or model routing and usage for the evaluation itself.