How Do You Measure Guardrail Accuracy and False Positives?

You measure guardrail accuracy on four numbers, per policy: block precision (of everything it blocked, how much was actually bad), recall (of known attacks, how many it caught), latency (time added per request), and bypass rate (how often an adaptive attacker gets through). Get the first two from a labelled test set that includes hard negatives, benign inputs that look suspicious. Get real false positives by replaying production traffic with the guardrail in shadow mode, where it records what it would have done without enforcing. Public benchmarks are a starting point, never the final answer, because your traffic is not their dataset.

Why one accuracy number misleads

Guardrail traffic is heavily imbalanced. In most production workloads, the share of genuinely malicious prompts is small. A detector that flags nothing scores high "accuracy" and catches no attacks. A detector that flags aggressively catches attacks and blocks a stream of legitimate work. Neither failure shows up in a single accuracy figure.

The cost of each error type also differs by policy. A missed prompt injection on an agent with payment tools is expensive. A false block on a customer-support agent erodes trust until someone disables the rule, which is the failure mode described in staged guardrail rollout and shadow mode. Measure the two error types separately, and decide per policy which one you can tolerate.

The four metrics

Metric	Question it answers	How to measure	Typical decision it drives
Block precision	When it fires, is it right?	Labelled sample of triggered events from shadow mode	Promote from warn to block, or tune the rule
Recall (detection rate)	Does it catch known attacks?	Labelled attack set, including encoded and indirect variants	Add rules or a second detector
Latency	What does it cost per request?	p50 and p95 added time, measured at the gateway	Run inline, async, or only on high-risk routes
Bypass rate	Does it hold under adaptive attack?	Red team campaigns that mutate payloads against the rule	Accept, layer controls, or move control to tool scope

Report precision and recall together. Precision alone rewards a rule that almost never fires; recall alone rewards one that fires on everything. Latency matters because a slow guardrail tempts teams to move it off the request path, and a guardrail that runs after the action is a log, not a control.

Build the test set before you trust any score

A useful evaluation set has three parts:

Attacks. Direct and indirect prompt injections, jailbreaks, data exfiltration attempts and the encodings attackers use to slip past filters. Include payloads inside documents and tool output, not only in user messages.
Normal traffic. A representative sample of real prompts from the workload the guardrail will protect, with sensitive data handled per your privacy rules.
Hard negatives. Benign inputs that share surface features with attacks: security discussions, code containing the word "ignore", customer messages quoting an error, a support ticket about a phishing email. These are where false positives live.

Public datasets help you start. Lakera's PINT benchmark mixes prompt injections, long documents, chat inputs and an explicit hard-negatives category designed to test false positives. JailbreakBench pairs 100 misuse behaviours with 100 benign behaviours, which lets you measure over-refusal alongside jailbreak success. Use them to compare detectors, then build your own set from your traffic, because a detector tuned on public data can still misfire on your domain's vocabulary.

Keep the set versioned and add to it. Every false positive a user reports and every bypass a red team finds becomes a labelled case. The approach mirrors eval datasets and golden sets for agents.

Shadow-mode replay: the real false-positive rate

A test set tells you how a rule behaves on cases you thought of. Production tells you how it behaves on cases you did not. Shadow mode closes that gap: the guardrail evaluates live traffic and records a match without enforcing it.

A practical sequence:

Deploy the rule in shadow or warn mode on one agent or connection.
Let it run long enough to see the workload's normal variety, including weekly cycles.
Sample triggered events and label them: true positive, false positive, or unclear.
Compute block precision from the sample. If it is below your threshold, tune and repeat.
Promote to redact or block on that scope, then widen the scope in stages.

Two practices keep this honest. Label a random sample, not only the events someone complained about. And keep the labelling criteria written down, so two reviewers agree on what a false positive is.

Measure per policy, per agent

A PII redaction rule might be precise on a sales agent and noisy on an engineering agent that handles test fixtures full of fake card numbers. One global precision figure hides that. Break metrics down by policy and by agent or connection, and set thresholds per pair where the stakes differ.

This also makes alert volume manageable. When a rule's precision drops on one agent, you tune that scope instead of weakening the rule everywhere. The same thinking applies to monitoring alerts in alert fatigue tuning for agent monitoring.

Guardrail accuracy drifts

A guardrail measured once is measured for that moment. Accuracy changes when:

the rule or its detector model is updated,
the protected agent's model or system prompt changes what it emits,
the user population or use case shifts,
attackers adapt to the rule.

Track hit rate per policy over time. A sudden drop can mean a bypass; a sudden rise can mean a traffic change that is now producing false positives. Re-run the labelled test set on every rule change, and schedule red team bypass tests, since adaptive attackers are the one input a static dataset never covers.

Checklist: measuring a guardrail before you enforce it

Define block precision and recall targets per policy, based on the cost of each error type.
Build a versioned test set with attacks, normal traffic and hard negatives.
Use public benchmarks to shortlist detectors, then evaluate on your own data.
Measure p50 and p95 latency added on the request path.
Run the rule in shadow or warn mode on one scope before blocking.
Label a random sample of triggered events with written criteria.
Break metrics down per policy and per agent or connection.
Choose the disposition by confidence: warn or redact before block, as covered in designing guardrails: block, redact, warn.
Re-measure after every rule, model or traffic change, and run periodic bypass tests.

How Praesidia fits

Praesidia gives you the measurement data as a side effect of enforcement. Secure: guardrails attach per connection with block, redact and warn dispositions, so a rule can enforce on one connection while only observing on others, and guardrails are versioned configuration, so promotion and rollback are recorded changes. Prove: each guardrail evaluated on agent tasks and workflows is logged as passed or triggered, including matches downgraded to warnings in the simulate, observe and alert stages, which is the data you need to compute precision and spot hit-rate drift after promotion. For where guardrails sit relative to other controls, see AI guardrails vs LLM firewall and the AI gateway guide.