Putting a control plane in the request path is what makes enforcement real: policy that is evaluated before the call cannot be bypassed by a persuaded agent. It also makes the control plane a dependency for every agent call, which means its availability is now your agents' availability.

That trade is worth making, and it obliges you to design the failure modes deliberately. The worst outcome is not an outage — it is an outage whose behavior nobody decided, where some components fail open and some fail closed according to whichever exception handler was written first.

Fail-Closed and Fail-Open, Decided Per Component

Authorization: fail closed. If the system cannot determine whether a call is permitted, the call does not proceed. Failing open on authorization means an outage becomes an unbounded-permission window, which is precisely when an attacker would want it. There is no configuration in which this is the right trade.

Spend enforcement: fail closed. A cap that stops applying during an outage is not a cap. If the reservation cannot be recorded, the call does not go out.

Content guardrails: fail closed for blocking rules, degrade for advisory ones. A PII redaction rule that cannot evaluate must not let unredacted content through. A warn-level rule that cannot evaluate can be skipped and recorded as skipped. The distinction is the disposition: anything whose purpose is to prevent must fail closed. See designing guardrails: block, redact, warn.

Audit writing: durable queue, not fail-closed. This is the one genuinely arguable case. Refusing to serve traffic because a log write failed is a poor trade for most organizations, and dropping the record silently is unacceptable. The resolution is a durable local queue: the record is persisted locally, the call proceeds, and the queue drains when the audit store recovers — with queue depth alarmed and a bounded capacity beyond which you do fail closed. Then the failure mode is "audit is delayed" rather than "audit is missing." For deployments where audit is a hard regulatory requirement, fail-closed is defensible and should be a configuration choice rather than an assumption.

Analytics, dashboards, reporting: degrade freely. These are not in the enforcement path and should never affect it. A common architectural mistake is a synchronous metrics write in the request path, which makes the analytics store an availability dependency for agent traffic.

The containment path: highest availability in the system. Kill switch, credential revocation, connection disable. These must work when everything else is degraded, because that is when you need them. Which means they should depend on as little as possible — ideally a single write to a store that enforcement points poll, rather than a chain of services. See rogue AI agents and kill switches.

Local Policy Cache With Bounded Staleness

Fail-closed authorization plus a control-plane outage equals all agent traffic stopped. That is safe and it is a bad availability posture if it happens on every transient blip.

The standard mitigation is a local cache at the enforcement point: the compiled policy, connection scopes, and guardrail rules are cached with a TTL, so a brief control-plane unavailability does not stop traffic. Requests continue to be evaluated against the cached policy.

The design parameters that matter:

TTL bounds the staleness. A short TTL — seconds to a low number of minutes — means a policy tightening propagates quickly. A long TTL survives longer outages. Pick based on how fast you need tightening to take effect, which is usually the binding constraint.

Revocation must not wait for TTL. A kill switch or credential revocation needs a push or a short-poll path that bypasses the cache TTL. If revocation takes five minutes because that is the cache lifetime, your containment control has a five-minute lag exactly when it matters.

Serve-stale has a hard limit. Past a maximum staleness — meaningfully longer than the TTL but still bounded — the enforcement point fails closed rather than serving indefinitely from an old policy. A cache that never expires under sustained outage is an unbounded-permission window with extra steps.

Cache the deny side too. Caching only allow decisions and re-querying denies means an outage turns denies into hangs.

Where Redundancy Goes

The enforcement path should be stateless and horizontally scaled. Multiple instances, multiple availability zones, no instance holding unique state. Then instance failure is a load-balancer event.

Datastores need managed replication and a tested failover. The word "tested" is the load-bearing part — an untested failover is a hypothesis. Practise it.

Queues need redundancy proportional to what they carry. An audit queue losing messages is worse than a metrics queue losing messages.

Provider dependencies need fallbacks. Model providers have outages. A routing layer with configured fallback keeps agents working, and it needs to be exercised — a fallback whose prompts have never been evaluated may be worse than an outage. See model routing and cost-quality tradeoffs.

External signing substrates need a decision. If audit entries are signed by a KMS and the KMS is unreachable, you cannot sign. Queue and sign on recovery, with the queue bounded and alarmed. See BYO KMS and per-tenant signing keys.

Multi-Region Is Two Different Problems

Multi-region gets conflated between availability and residency, and they pull in opposite directions.

For availability, you want data replicated across regions so a regional failure is survivable. Active-active if your consistency requirements allow; active-passive with a tested promotion path otherwise.

For residency, you want data confined to a region — which means you cannot replicate it elsewhere, so a regional failure is not survivable by failover for that tenant's data. See data residency for AI agents.

The reconciliation is per-tenant regional placement: each tenant's data lives in its designated region, with in-region redundancy across availability zones providing availability, and no cross-region replication of tenant data. The control plane itself — configuration, policy, routing — can be globally replicated because it is not tenant content, which lets enforcement work everywhere while data stays put.

Be explicit about the consequence: a tenant pinned to one region has that region's availability, not global availability. That belongs in the contract, not in a footnote discovered during an outage.

Test the Failure Modes

Every degradation path described above is a hypothesis until exercised.

Worth testing deliberately, in a non-production environment and ideally as scheduled exercises:

  • Control plane unreachable: does traffic continue on cached policy, and does it stop at the staleness limit?
  • Audit store unreachable: does the queue absorb it, is depth alarmed, does it drain and preserve chain continuity?
  • KMS unreachable: what happens to signing?
  • Provider outage: does the fallback engage, and is output quality acceptable?
  • Datastore failover: how long, and what is lost?
  • Revocation during an outage: does the kill switch still work?
  • Cache poisoned or stale beyond the limit: does it fail closed?

The revocation-during-outage test is the one most often skipped and most important, because it validates the assumption that your containment control is independent of the components most likely to be broken during an incident. See health probes and readiness for AI infrastructure and AI incident readiness checklist.

Common questions

Does a control plane in the request path add unacceptable latency?

The evaluation itself is sub-millisecond to low single-digit milliseconds — policy lookup against a local cache, a signature verification, an atomic reservation. Against model inference measured in hundreds of milliseconds to seconds, it is not material. The latency risk is not the evaluation; it is a synchronous dependency on something slow, like a remote metrics write or an unbatched audit write to a distant store. Keep the hot path local and push everything else asynchronous.

Should enforcement be a sidecar or a central service?

A sidecar or in-process library gives you lower latency and better failure isolation — the enforcement point fails with its own workload rather than globally. A central service gives you simpler deployment and consistent policy without propagation concerns. The common shape is a central control plane distributing policy to local enforcement points, which gets you the availability properties of the sidecar with the management properties of the central service.

How do I keep audit chain integrity through an outage?

The chain is a sequence, so entries must be appended in order. A local durable queue preserving order, drained sequentially on recovery, maintains the chain. What breaks it is parallel drains from multiple queues into one chain, or a gap-filling strategy that inserts entries out of order. Design the drain as ordered and single-writer per chain, and verify the chain after a recovery rather than assuming.

What SLO should a control plane target?

Higher than the services depending on it, because its unavailability multiplies across them — if every agent call depends on it, its availability is a ceiling on theirs. In practice that means designing for graceful degradation rather than chasing additional nines: cached policy, asynchronous audit, and independent containment give you a system whose worst case is bounded, which is worth more than a higher headline number with a cliff-edge failure. See SLOs for AI services.

How Praesidia approaches availability

Praesidia separates the enforcement path from the reporting path: authorization and spend checks are evaluated inline and fail closed, audit writes go through a durable path so a storage outage delays rather than drops records, and analytics never sit in the request path. Guardrail dispositions determine failure behavior — blocking rules fail closed, advisory rules degrade and record the skip.

Containment — killing an agent, revoking credentials, disabling a connection — takes effect on in-flight work and is deliberately independent of the components most likely to be degraded during an incident. Tenant data placement is per organization for residency, with in-region redundancy for availability, and the audit chain is appended in order so integrity survives a recovery. See health probes and readiness for AI infrastructure and the Praesidia documentation.