Queues fail. A handler throws, a downstream service is unavailable, a payload is malformed, a message exceeds a limit. After the retry allowance is exhausted, the message has to go somewhere, and the dead-letter queue is that somewhere.
The failure pattern is consistent across organizations: the DLQ exists because the queue infrastructure created one, nothing alerts on it, and it accumulates for months. Then someone discovers forty thousand messages representing work that was never done, some of it customer-visible, and there is no way to know which items still matter.
For agent workloads this is worse than for typical background jobs, because agent tasks are often expensive, partially completed, and tied to a user expectation.
Alert on Depth, Immediately
The minimum viable DLQ practice is one alert: depth greater than zero, routed to a channel with an owner.
Not depth greater than a hundred. Greater than zero. A single dead-lettered message means a job failed permanently, and either it matters (investigate) or it does not (in which case why is it queued at all). Threshold-based alerting on DLQs is how the forty-thousand-message situation arises — the threshold was set high enough to avoid noise, and the noise was the signal.
Alongside depth, alert on rate: messages dead-lettering per minute above a baseline indicates an active incident rather than an isolated bad payload. And alert on age of oldest message, which catches the case where depth is low but something has been stuck for a week.
Route these to the owning team, not to a central channel where every service's DLQ alerts compete. See Slack and multi-channel alerting and in-app notifications.
What the Entry Must Carry
A DLQ message containing only the original payload is nearly useless for diagnosis. Whatever else you keep, it has to answer three questions without a database console.
Why did this fail, and had it failed this way before? The terminal error in full, plus the history of attempts — a message that failed identically five times is a different problem from one that failed differently each time, and the difference decides whether replay is worth trying.
What else happened around it? A DLQ entry with no link back to the work it belonged to is an orphan. Something has to connect it to the surrounding records, traces, and any side effects that already completed. See post-incident forensics for AI agents.
Who is affected and how urgently? Which customer, which user is waiting. This is what turns a queue depth number into a prioritised list, and it is the field most often missing.
Beyond those, keep whatever your recovery path needs — for multi-step work that usually means knowing how far the task got, and knowing which version of the handler produced the failure so you can tell whether a later deploy already fixed it.
Poison Messages Versus Transient Failures
Replaying a DLQ indiscriminately produces one of two outcomes: transient failures succeed (good) and poison messages fail again (a loop that wastes money and refills the DLQ).
Classify before replaying:
Poison — will never succeed as-is. Malformed payload, references a deleted entity, violates a constraint, exceeds a hard limit, or hits a bug in the handler. These need either a payload fix, a code fix, or a decision to discard. Replaying unchanged is guaranteed to fail.
Transient — will succeed once conditions change. Downstream unavailable, rate limited, timeout, temporary capacity issue. These replay successfully after the cause clears.
Ambiguous. Errors that could be either. Sample and investigate rather than guessing.
The classification is largely mechanical from the error type, and it is worth automating: tag DLQ entries at write time with a probable class based on the terminal error, so triage starts from a partition rather than from a flat list.
Guard against replay loops explicitly. A message replayed from the DLQ that fails again should be marked as such and not automatically replayed a second time. An automated drain with no loop guard, pointed at a DLQ full of poison messages, is a cost incident.
Resume, Do Not Restart
This is specific to agent work and it matters.
A multi-step agent task that failed at step seven has probably completed steps one through six, and some of those steps had side effects — a record created, a message sent, budget spent. Restarting from step one duplicates all of it.
So replay needs to resume from the failure point, which requires that progress was recorded as it happened and that each step is idempotent. Idempotency covers you if the resume point is wrong; recorded progress is what makes the resume point right. See retries and idempotency in agent workflows.
Where a task cannot be resumed — because the intermediate state is gone, or the world has changed — the honest options are to compensate (undo the completed side effects, then restart) or to discard and notify. Both are better than a duplicate. Compensating actions have to be written in advance; discovering during an incident that there is no way to undo step three is a design gap.
The Triage Workflow
A DLQ needs an interface, not just a queue. Minimum capabilities:
Whatever the interface looks like, triage needs to be possible without a database console. That means being able to narrow a large queue down to the entries that share a cause, inspect one of them completely enough to diagnose it, and try a single replay before committing to a bulk one.
It also means being able to act on a subset rather than the whole queue — after a fix, you want to replay what that fix addressed and nothing else — and being able to preview a bulk action before it runs.
The capability teams most often omit is discarding with a recorded reason. Some work genuinely should not be retried: it is obsolete, the customer was handled manually, the entity no longer exists. If the only way to clear a stuck alert is a bulk purge, the record of what was lost goes with it.
The discard-with-reason capability is what keeps the queue honest. Without it, the only way to clear a stuck alert is to purge, and purging destroys the record of what was lost.
Prevention Reduces the Load
A DLQ full of messages that could have been rejected earlier is a validation gap.
Validate at enqueue. A malformed payload should be rejected when the job is created, with a synchronous error to the caller who can fix it, rather than accepted and dead-lettered asynchronously where nobody sees it.
Size limits enforced at enqueue. Messages that exceed a limit downstream should not be accepted.
Reference checks. If a job references an entity, verify it exists at enqueue. Entities can still be deleted before processing, which is a legitimate transient-to-poison transition, but catching the already-invalid case early removes most of the volume.
Circuit breakers on downstream calls. When a dependency is down, failing fast and pausing consumption is better than burning the retry allowance on every queued message and dead-lettering all of them. A circuit breaker converts an outage-shaped DLQ flood into a paused queue that drains when the dependency recovers.
Common questions
Should the DLQ have its own DLQ?
No — you need a terminal state, and adding another queue just moves the problem. What the DLQ needs is monitoring and a triage process. If a DLQ handler itself can fail, that handler should be simple enough not to: enrich and store, with no downstream calls that can throw.
How long should DLQ messages be retained?
Long enough to diagnose and act, which for most workloads is weeks rather than days — a failure discovered on Friday may not be triaged until the following week. Set retention explicitly rather than accepting the infrastructure default, and be aware that DLQ payloads may contain personal data, which brings retention and erasure obligations along with it. See GDPR for AI systems and erasure rights.
Can I automate DLQ drains?
For clearly transient error classes, yes — a scheduled drain that replays timeout and rate-limit failures with a loop guard is safe and saves manual work. For anything else, no: automated replay of poison messages wastes money and hides the underlying bug. The distinction depends on the error classification being reliable, which is the argument for tagging entries at write time.
Who should own DLQ triage?
The team that owns the queue's handler, with the alert routed to them. Central ownership of everyone's DLQs does not work, because triage requires knowing what the work does. What is worth centralizing is the tooling and the requirement — every queue has a DLQ, every DLQ has a depth alert with an owner — enforced as a platform check rather than as a convention. See queue topology and registration lint gates.
How Praesidia approaches failed work
Praesidia treats permanently-failed agent work as something that must remain visible and recoverable rather than something that quietly accumulates. Failed work stays linked to the task it came from, so a failure can be traced back to the records, policy decisions, and spend that surround it instead of arriving as an orphaned payload.
Because state-changing operations are safe against duplicate delivery, recovering failed work does not risk repeating side effects that already happened, and failures surface through the same alerting channels as the rest of the platform. See executing and monitoring workflow runs and the Praesidia documentation.