The request timed out. Did it send?
Part 4 of 6 in Beyond the Demo.
The announcement was sent. The response was lost.
Your assistant sees a timeout, tries again, and sends the announcement a second time.
Nothing in this story requires a particularly confused model. A user double-clicking a button, a reconnecting client, or a worker recovering after a crash can produce the same result.
The architectural problem is that the caller cannot tell whether the effect happened.
Before adding retries to an agent workflow, decide how that uncertainty will be represented—and what the system is allowed to do with it.
Start at the consequence, not the prompt
Consider Release Desk, an illustrative application that publishes approved release notes and then delivers an announcement to an external service.
Drafting the announcement is repeatable work. Delivering it changes another system. Those steps have different failure semantics.
If drafting fails, another attempt may be acceptable. If delivery times out, another attempt might create a duplicate. “Retry on error” is not a complete policy because an error does not prove that nothing happened.
The Amazon Builders' Library describes using caller-provided request identifiers to recognize retries of the same intended operation. That is the starting point: identity belongs to the intended effect, not to whichever network attempt happens to carry it.
For an agent, this means the runtime should preserve the operation identifier across retries. Asking the model to invent a new one each time defeats the protection.
Keep the identifiers separate
A workflow may have several IDs for good reasons.
An event ID identifies an incoming occurrence. A proposal ID identifies the specific content and destination being considered. An action ID identifies an intended effect. An attempt ID identifies one try at producing it. A trace ID helps correlate the execution.
Do not use those interchangeably.
The same input event should not create multiple actions merely because it was delivered twice. But a single event may legitimately produce several different actions. Likewise, a second intentional announcement with identical text is not necessarily a retry of the first.
For Release Desk, an action might mean “deliver this approved publication to this destination.” The application records that identity before dispatch. Any retry continues the same action rather than asking the model to recreate its intent.
Scope retry keys to the relevant tenant, principal or delegated authority, and operation. A key from one customer's request must not resolve to another customer's result.
Record intent before dispatch
For work that changes local state and emits an external notification, there is a familiar trap: update the database, then send the message. A crash between the two leaves the database claiming progress that the destination never received.
The transactional outbox pattern addresses the local dual-write problem by storing the state change and the outgoing message record in one transaction. A separate dispatcher processes committed records. It does not make the external destination part of that transaction, and duplicate delivery still needs handling.
For our example, the local commit can record the publication and an announcement intent together. A worker then attempts delivery. The UI can truthfully show “published; announcement pending.”
That distinction is much better than saying everything succeeded because the work entered a queue.
A minimal action record needs enough information to recover: its immutable intended effect, current state, retry identity, attempt history, and any provider receipt. Do not store a vague instruction such as “send the latest release.” “Latest” may refer to different content when the worker resumes.
Give uncertainty its own state
An illustrative state model is:
pending → dispatching → succeeded
├→ retryable_failure
├→ terminal_failure
└→ outcome_unknown
A failure before dispatch can often return to pending. A confirmed provider rejection can be classified according to its cause. A lost connection after sending the request may belong in outcome_unknown.
That last state is not an implementation embarrassment. It is an honest description of the evidence.
If the provider offers idempotency, retry with the same provider key and the same intended payload, within the provider's documented retention window. If it offers authoritative lookup by your operation reference, reconcile before deciding whether another attempt is safe.
If it offers neither, choose deliberately between the risk of missing an effect and the risk of duplicating it. For an important announcement, pausing for review may be preferable to sending again blindly. The right policy depends on the consequence.
A local “already sent” flag cannot close the crash window between the external effect and recording that flag.
Read the provider's actual contract
“Supports idempotency” is not enough detail.
What is the scope of the key? How long is it retained? What happens when the same key arrives with different parameters? Does the provider preserve error results? Can you retrieve the result after a timeout?
For example, Stripe documents that it retains the first result for an idempotency key, including certain error responses, checks subsequent parameters, and may prune keys after they are at least 24 hours old. That is a provider-specific contract, not a universal property of APIs.
Your application's retry window must fit the destination's guarantees. A task that resumes after those guarantees expire needs a reconciliation or expiry policy—not an assumption that yesterday's key still protects it.
Record terminal action identity for the period your application needs to recognize replay. If old requests become invalid, reject them explicitly rather than treating a forgotten key as proof of a new operation.
Control concurrent attempts, but do not confuse a lock with delivery safety
Two workers can pick up the same pending action. Use transactional claiming, compare-and-swap state changes, or a suitable lease to coordinate local execution.
Then consider the worker that pauses long enough for its lease to expire and resumes after another worker takes over. A lease alone cannot retract an external request already in flight. Fencing works only when the system performing the effect actually checks the fence.
Where the destination supports an idempotency key, both attempts should carry the same action identity. Where it does not, local coordination reduces risk but does not magically eliminate every uncertain outcome.
This is why “exactly once” needs a named boundary. A unique local action record, one queue delivery, and one externally visible effect are different claims.
Keep model judgment out of transport recovery
A useful model may decide which changes belong in a release. It should not need to regenerate the announcement whenever a delivery worker restarts.
Persist the approved decision and proposed effect. Recovery can then continue deterministically: inspect the action, check current authority, reconcile with the provider, and apply the retry policy.
If the content changes, create a new proposal. Do not quietly replace the payload under an existing retry key.
This separation also makes model upgrades less surprising. Changing the model should not rewrite work that a human already approved and the system is merely trying to deliver.
Test the missing response on purpose
A useful test destination records an operation and then deliberately withholds the response. Your runtime should recover without turning the same logical action into a new one.
Run variants where the worker stops before dispatch, immediately after the provider accepts, and after acceptance is recorded locally. Submit duplicate events. Attempt the same key with a different payload. Resume after the retry window has expired.
Count effects at the destination, not just successful tool calls. Verify that the status returned to the assistant matches the evidence. “Unknown” should not be paraphrased as “failed, so I tried again.”
These tests can run without a language model. That is a feature: recovery correctness should not depend on the model being in a helpful mood.
Durability is a contract, not a storage checkbox
A durable record is necessary for recovery, but it is not a complete recovery design. Something must revisit unfinished work, know which transitions are safe, and preserve authority and intent as it does so.
When building this on OpZero—or any agent platform—ask separately about persistence, scheduling, dispatch, and the external service's guarantees. A hosted runtime does not automatically make an arbitrary third-party action idempotent.
The most useful retry is not the one that keeps trying hardest. It is the one that can explain whether it is continuing the same action, reconciling an uncertain result, or refusing to risk a duplicate.
Specify the failure contract before the agent loop. OpZero's tool reference is the starting point for runtime capabilities; the action's identity, recovery policy, and external delivery semantics belong in your application design.
Next in the series: How much access should an AI agent have?.