Agent workflows run on queues, and queues deliver at least once. A worker that crashes after acting but before acknowledging will have its message redelivered. A visibility timeout that expires while a long step is still running produces a second concurrent execution. A client that times out on an ambiguous response retries.

For read-only work this is harmless. For agents that send messages, issue payments, create records, or spend budget, it produces duplicates — and the duplicates are usually discovered by the recipient rather than by monitoring.

Idempotency Keys, Derived Not Generated

An idempotency key identifies a logical operation so a repeat can be recognized. The mistake that defeats it is generating the key at attempt time: a fresh UUID per attempt means every retry presents a new key and every retry executes.

The key must be derived from the operation's identity — something stable across attempts. Practical constructions:

  • The task identifier combined with the step it belongs to, where a step runs at most once per task.
  • The task identifier combined with a hash of the operation's semantic content, where a step may legitimately repeat with different inputs.
  • For looping work, whatever combination uniquely names the execution, the step, and the iteration.

The test is simple: if the process dies and the message is redelivered, does the handler compute the same key? If not, the key is wrong.

Then enforce it where it cannot race:

A uniqueness constraint at the datastore. Insert the operation record with the key as a unique column. A duplicate insert fails at the database, atomically, regardless of concurrency. An application-level "check if the key exists, then act" is a time-of-check-to-time-of-use race that fails precisely under the concurrent redelivery you are defending against.

Return the original result on duplicate. The caller should be unable to distinguish the first call from the second. Erroring on duplicate is better than double-executing and worse than returning the stored result, because the caller then has to handle an error that is not an error.

Retain keys long enough. Cover the longest retry horizon of any component that might retry — which with a long visibility timeout plus a multi-attempt retry policy can be hours. Keys expired too early allow a late redelivery to execute.

Where Side Effects Actually Live

Enumerate them, because the list is longer than it first appears:

External communication. Emails, messages, webhooks, notifications. Duplicates here are visible to customers, which makes them the most reputationally expensive.

Financial operations. Charges, refunds, credit grants, escrow releases, budget reservations. See credit wallets for agent payments and agent task marketplaces and escrow.

Record creation and mutation. In your systems and in third-party systems, where you may have no visibility into the duplicate.

Model calls. Duplicated inference is pure cost with no output, and it is the easiest duplicate to overlook because nothing breaks.

Tool calls with effects. Any MCP tool that writes.

Sub-agent dispatch. A duplicated dispatch multiplies an entire subtree of work. This is the most expensive duplication shape in orchestrated systems.

For each, decide the strategy: natural idempotency (a PUT to a known key), an idempotency key passed to the downstream API (most payment and messaging APIs support one — use it), or a local record of completion checked before acting.

Retries Are a Cost Multiplier

Retry on transient failure is correct and it needs a budget, because under adversarial or malformed input the retry path becomes a cost amplifier the caller controls.

The mechanism: a model produces malformed output, validation rejects it, the step retries, the model produces malformed output again. With unbounded retries and a large context, that loop can consume a substantial amount of money quickly. Content crafted to reliably produce malformed output turns this into an attack. See denial of wallet.

Controls:

A finite per-task retry allowance, drawn from a shared pool rather than counted per step. Otherwise a workflow with twenty steps each allowing three retries has sixty retries available.

Exponential backoff with jitter. Prevents synchronized retry storms across a fleet, which is how a downstream hiccup becomes an outage.

Retry only what is retryable. A validation failure on model output may be worth one retry with a corrective prompt; it is not worth five. A 400 from an API is not transient — retrying it is pure waste. Classify errors and retry only the transient class.

Total task budget and wall-clock ceiling that retries count against, so the worst case is bounded even when the retry logic is wrong.

Metered retry cost. Track spend attributable to retries as a distinct figure. A rising retry share is both a reliability and a cost signal, and it is invisible if retry spend is aggregated with primary spend. See anomaly detection on agent spend.

Replaying Non-Deterministic Steps

Workflow engines that recover by replaying history assume steps are deterministic: replay the recorded inputs, get the same outputs, skip to where you left off. Model calls violate that assumption — the same prompt can produce a different result.

If you replay a model call, the agent may take a different branch than it took originally, and the workflow diverges from its own history. Side effects already performed under the first branch are now orphaned.

The resolution is to treat each model call as a recorded decision rather than a reproducible computation:

  • Execute the model call once, record the output as part of the execution history.
  • On replay, return the recorded output instead of calling the model again.
  • The workflow then follows the same path deterministically, because the non-determinism was resolved once and persisted.

This is the standard durable-execution pattern and it applies to anything non-deterministic: model calls, random choices, current time, external API results that vary. Record the result, replay the record. It also has a cost benefit — replay after a crash does not re-pay for inference.

Where you genuinely want a fresh attempt rather than a replay, that is a new step with a new idempotency key, not a replay of the old one. Keeping those two operations distinct in the engine is what prevents accidental re-execution.

Making Failure Legible

Retries that eventually succeed should still be visible. A workflow that succeeds on the fourth attempt is degraded, and if only the final success is recorded, the degradation is invisible until it becomes a failure.

Record per attempt: the attempt number, the error, the latency, and the cost. Then surface retry rate per step as a metric. The step with the highest retry rate is usually where a real problem is — a flaky tool, a prompt that produces malformed output on a subset of inputs, a downstream service near its limit.

When retries exhaust, the work needs somewhere to go rather than disappearing. See dead-letter queues for failed agent jobs.

Common questions

Can I just make everything idempotent and stop worrying about retries?

Idempotency handles the correctness half — no duplicate side effects. It does not handle the cost half, because a retried model call costs money whether or not the result is discarded, and it does not handle the reliability half, because a step retrying constantly is a problem even when it is safe. You need both idempotency and retry budgets.

What about steps where duplicate execution is genuinely harmless?

Then you do not need a key for that step, and skipping it is a legitimate optimization. Be careful about what counts as harmless: a duplicated read is harmless for correctness and not for cost or for exfiltration volume, and a duplicated log write is harmless until it corrupts a count someone bills on. The bar for "harmless" is that a reviewer can state why, not that nobody has noticed a problem.

How do I handle a third-party API with no idempotency support?

Record locally before and after the call, so on redelivery you can determine whether the call was already made. The window between the call and the local record is the vulnerable one, and you narrow it by writing the intent record before the call and the completion record after — then a redelivery finding an intent record without a completion record knows the outcome is uncertain and can query the third party's state before deciding. It is not perfect, and it is much better than blind retry.

Does this apply to interactive agents, or only to queued workflows?

Both. An interactive agent whose client retries on timeout produces the same duplicate, and users retry more aggressively than queues do. The same keys apply, derived from the conversation and turn identifier rather than from a task and step.

How Praesidia approaches retries and duplicates

Praesidia treats value-moving and state-changing operations as safe against duplicate delivery, with uniqueness enforced at the datastore rather than by an application-level check, so a redelivered message cannot double-execute a reservation, a capture, or a dispatch. spend ceilings hold under concurrent load, and retries draw from bounded per-task allowances rather than being unlimited.

Every attempt is recorded with attribution, error, and cost, so retry rate and retry-attributable spend are visible per step rather than aggregated away — and work that exhausts its retries lands in a dead-letter path rather than disappearing. See executing and monitoring workflow runs and the Praesidia documentation.