Public benchmarks are how the field communicates progress, and they do that job reasonably well. They are also routinely used for a job they cannot do: deciding which model or agent configuration to run in production.
The gap is not that benchmarks are dishonest. It is that they measure capability on a fixed, public distribution under a standardized harness, and your production outcome depends on your distribution, your tools, your prompts, your latency budget, and your cost ceiling. A configuration that leads a leaderboard can lose badly on your workload.
What Public Benchmarks Actually Measure
Most agent benchmarks share a shape: a set of tasks with verifiable outcomes, a standardized environment, and a success rate. Software-engineering benchmarks resolve real issues in real repositories. Tool-use benchmarks check whether the right function is called with the right arguments. Web-navigation benchmarks score task completion in a browser environment. Reasoning suites score accuracy on problems with known answers.
They are genuinely useful for three things:
Ruling things out. A configuration that performs poorly on a benchmark resembling your task type is probably not worth trialing. Negative signal is more reliable than positive.
Tracking the frontier. Whether capability on a task class is improving, and roughly how fast.
Comparing on a controlled harness. With the scaffolding held constant, differences are attributable to the model rather than to the surrounding engineering.
Why They Mislead
Contamination. Public benchmarks are on the internet, and the internet is training data. Test items, solutions, and discussion of them get absorbed. A model may have seen the problem, which converts a capability measurement into a recall measurement. Contamination is hard to prove and harder to rule out, and its effect grows the longer a benchmark has been public — which means a benchmark's usefulness decays from the day it is published.
Overfitting by the community. Once a benchmark matters commercially, effort concentrates on it: prompt techniques tuned to its task format, training data curated to resemble it, scaffolding specialized for its harness. Scores rise faster than general capability. This is not usually cheating; it is Goodhart's law operating normally.
Harness variance. Reported agent scores depend heavily on the scaffolding — retry policy, tool definitions, context management, prompt structure. The same model can differ by tens of percentage points across harnesses. When a vendor reports a score, they report their harness's score, and you will not be running their harness.
Distribution mismatch. Benchmark tasks are curated to be measurable, which biases them toward well-specified problems with verifiable answers. Real workloads are ambiguous, domain-specific, and full of internal context nothing public contains. Performance on the former predicts the latter loosely.
Missing dimensions. Quality-only leaderboards omit cost per task, latency, reliability across repeated runs, refusal rate, and behavior under adversarial input. A configuration that wins on quality at ten times the cost and four times the latency is not a winner for most production uses.
Benchmarking Your Own Workload
The benchmark that should drive your decisions is internal, and it is closely related to your eval suite — the difference is purpose. Evals detect regression in a fixed configuration; a benchmark compares configurations. Same infrastructure, different question.
Use your own tasks. Drawn from real traffic and real failures, with your fixtures, your tools, and your retrieval corpus. See building golden datasets for agent evals.
Hold the harness constant when comparing models. Same prompts, same tool definitions, same retry policy, same context strategy. Otherwise you are comparing engineering effort, not models.
Vary one thing at a time. Model, then prompt, then tool structure, then retrieval configuration. Agent systems have strong interaction effects — a prompt tuned for one model can underperform on another — so a full matrix is informative where budget allows, and one-at-a-time is the minimum discipline.
Measure the full vector. Success rate, cost per task, p50 and p95 latency, step count, refusal rate, and variance across repeated runs. Report them together; a single composite score hides the trade you are actually making.
Run each case multiple times. Non-determinism means single-run comparisons are noise. Five to ten runs per case, reporting success rate and variance. Variance is a decision input in its own right: a configuration succeeding 90% of the time reliably is often preferable to one averaging 92% with wide spread.
Include adversarial and edge cases. A configuration that is more capable and more easily manipulated may be a worse choice. See red teaming AI agents.
Refresh regularly. Provider models change, your traffic changes, your corpus changes. A benchmark run six months ago is historical information.
Cost and Latency Are Selection Criteria
The most common benchmarking mistake in practice is treating quality as the objective and cost as a constraint to check afterward. In production they trade continuously, and the right configuration usually differs by task class.
Measure cost per successful task, not cost per call — a cheaper model that fails and retries can cost more than an expensive one that succeeds first time. Measure latency at the tail, because p95 is what users experience as slowness. And measure both per task category, because the answer is frequently "route simple tasks to the cheap model and hard ones to the expensive one," which is only discoverable if you benchmark per category. See model routing and cost-quality tradeoffs and FinOps for AI agents.
Publishing Benchmarks Responsibly
If you publish agent benchmark results — as a vendor, or in a marketplace where agents are compared — a few properties determine whether the numbers are worth anything:
Publish the harness. Prompts, tool definitions, retry policy, model versions with exact identifiers. Results without the harness are unreproducible and therefore not evidence.
Report distribution, not just the mean. Success rate across N runs, with variance.
Report cost and latency. Alongside quality, in the same table.
State the dataset version and date. And whether cases are public (contamination-prone) or held out.
Separate held-out from public sets. A vendor's own held-out set is more informative than a public benchmark, provided the methodology is described.
Do not compare against competitors on your own harness. Tuning your harness for your system and running a competitor's through it produces a number that is not about their system. If you must compare, use their published configuration or a neutral third-party harness.
For marketplaces and platforms surfacing comparative numbers, the same rules apply plus one: make it clear what the numbers do not cover. A benchmark showing task success says nothing about whether an agent is safe to grant credentials to, which is where trust signal and attestation come in. See public agent reputation and k-anonymity and trust scores and attestations for agents.
Common questions
Are public benchmarks worth tracking at all?
Yes, for direction and for shortlisting. If a new model shows a large jump on a benchmark resembling your task class, that is a signal worth a trial. What they should not do is settle a decision, and they should never be the basis for a claim to a customer about how your system will perform on their data.
How do I tell whether a benchmark is contaminated?
Rarely with certainty. Indicators: the benchmark has been public for a long time, scores across many models cluster suspiciously high, performance drops sharply on freshly-constructed items of the same type, or a model reproduces benchmark-specific formatting it was never prompted with. The practical response is to prefer recent benchmarks, prefer held-out variants, and weight your own private set most heavily.
Should I build a benchmark or just use my eval suite?
Use the same infrastructure. The distinction is in how you run it: evals fix the configuration and vary time, benchmarks fix time and vary configuration. Tag your dataset cases so a subset serves as the comparison set, and record the full configuration on every run so results are comparable. Building two separate systems is duplicated work. See agent evaluations: catching quality regressions.
How much does benchmarking cost to run?
For a few hundred cases across several configurations with multiple runs each, it is thousands of model calls — real money, but small against the cost of running the wrong configuration in production for a quarter. Control it by benchmarking on a curated subset rather than the full eval set, and by running the full matrix only when making an actual decision rather than continuously.
How Praesidia approaches benchmarking
Praesidia exposes org-scoped benchmark results alongside evaluation runs and datasets, so comparative measurements live next to the regression suite and share the same recorded trajectories — tool calls, arguments, step counts, latency, and cost per run. That means a benchmark comparison reports the full vector rather than quality alone, without separate instrumentation.
A public benchmarks surface supports comparison and trend views across published results, and reputation and trust signal are kept distinct from capability scores — because how well an agent performs and how much authority it should be granted are different questions. See advanced analytics for AI operations and the Praesidia documentation.