Orchestration, fault tolerance and when NOT to use an agent

Choose orchestration patterns and a tool/execution layer, operate an agent with causal observability, make it fault-tolerant (idempotency, retries, exactly-once as an end-to-end intent, commit gates, sagas), recognise the recurring patterns and antipatterns, and exercise the engineering discipline of knowing when a deterministic workflow beats an agent — "the best agent is sometimes no agent".

Track

Engineering

Module

10 · Agentic architectures

Prerequisites

Multi-agent systems: communication, coordination and holonic organization

Orchestrating well is deciding where the order of the steps lives. It can live in a component that commands the participants (orchestration), or be spread across the reactions each participant has to events (choreography). It can be written as an acyclic graph, as a hierarchical state machine, or as durable code — or it can be decided by a model at every step. The whole discipline is separating those decisions, and the sharpest one of all is whether to use an agent at all. A workflow is a system where the model and tools are orchestrated along predefined code paths; an agent is a system where the model dynamically directs its own process. The default advice is the simplest thing that works, which is frequently not an agentic system.

This is the same boundary the multi-agent lesson drew: what counts is who chooses the next step, not how many times a model is called. A graph with fixed edges is a workflow; the same graph with one conditional edge a model decides each turn is an agent's loop. The graph does not tell you which it is — the person choosing the edge does. Getting this right up front is the single highest-leverage architectural decision, because it determines what you can verify before you run and how you reason about failure.

There are three natural shapes for a process with effects. A directed acyclic graph (DAG) fits a batch: extract, check in parallel, decide, execute, no going back — acyclicity is checkable before you run, parallelism is readable in the graph levels, and resuming after a crash means not repeating nodes whose result is recorded. A statechart fits a lifecycle: an entity moving through states at the pace of events it does not control — its state space is finite and explicit, so properties like "never paid without being approved" are checkable by exhaustive exploration, and it acts as a filter that rejects any event not permitted in the current state. Durable execution writes the flow as ordinary code with loops and long waits, and the engine survives crashes by replaying it against a history — gaining expressiveness, losing the exhaustive verifiability of a statechart.

Orchestration

A component owns the instance state and sends the participants commands. Asking where invoice F-17 is is one query; changing the flow is one component; compensation runs in reverse from one place. The cost is a central component that must be durable, and coupling to it.

Choreography

Each service reacts to events and publishes others; the flow lives in the subscriptions and no component owns the instance state. Gains decoupling, loses visibility (state is scattered), cheap change (inserting a step touches neighbours), and — most dangerously — invites implicit loops.

Workflow vs. agent

A workflow fixes the next-step function before execution (even if a step calls a model). An agent computes that function at run time from goals, beliefs and perception. A hybrid is a workflow with bounded nodes in which an agent decides.

For processes with real effects, compensation and deadlines, the default is the orchestrated saga on a durable engine — you want to know where each instance is, and to compensate from one place. Choreography earns its keep at the boundaries between domains with different owners, where no orchestrator has authority over all, and for broadcasting facts that several consumers use for their own ends. Its most dangerous failure is the implicit loop: two rules, each reasonable alone — "on budget released, announce it" and "on budget available, retry the payment" — form a cycle that no single team sees, because each team knows only its own rule. The event graph makes that cycle detectable statically.

A tool is the interface through which an agent produces effects, so its design decides which of the model's errors reach the world and which stop at the boundary. Treat a tool as a contract: an input schema and an output schema, declared precondition and postcondition, a declared effect class, retry semantics, and a version. A gateway mediates every call — it checks the input validates and the preconditions hold against the system of record, applies the effect class, executes, and checks the output. The arguments come from a stochastic component, the model, so they are unverified evidence: the schema checks the shape, but that a payment amount matches the invoice is a precondition only the holder of the record can check. The gateway is not an agent; it decides whether a call meets its contract, nothing more.

Idempotency is the load-bearing property, and it must live in the interface, not in the caller. An agent adds a source of repeats that distributed systems do not have: the model may re-propose an action already executed if its result never reached the model's context. The intent key must identify the intention (pay invoice F-000101 in this run of the flow), not the attempt — a fresh key per attempt is the same as having no key, and it is a frequent bug when the key is generated inside the HTTP client. Derive the key from stable flow identifiers and record it in the durable history before the first attempt.

synthetic — idempotency in the interface: intent key + content fingerprint
# the intent key identifies the INTENTION, not the attempt:
key = fingerprint(run_id, step_id, invoice_id)   # stable across retries; NOT a per-call uuid

def pay(args, key):                               # gateway, one atomic operation at the store
    seen = ledger.get(key)
    if seen and seen.fingerprint == hash(args):
        return seen.result                        # idempotent replay: return stored result
    if seen and seen.fingerprint != hash(args):
        raise Conflict(409, key)                  # same key, different args -> refuse, don't
                                                  #   silently return a result for other args
    if seen and seen.in_progress:
        raise Retry(409, "same key in flight, retry later")
    result = effect(args)                         # the real effect
    ledger.put(key, hash(args), result)           # record effect + key ATOMICALLY with it
    return result
# consequence: exactly-once of the EFFECT = at-least-once delivery (retries)
#   composed with an idempotent receiver. "Exactly-once" is never a channel property.

Effect class decides the rest of the architecture. A read tool can be retried and simulated freely; an idempotent tool can be retried; a non-idempotent tool needs a key; a tool that is part of a long transaction needs compensation (a saga). Keep the effect class in the agent's own tool registry, maintained by whoever operates it — do not infer it from what a server says about itself. And write errors the caller can act on: an error that says where, what was expected, what arrived and what to do ("/amount_cents: expected integer, received '1200.00'") costs fewer retries than an opaque "call failed", which forces the model to guess.

An agentic system fails in three ways that classical engineering treated separately. It fails like any distributed system (processes crash, messages are lost, responses are slow). It fails like a statistical component (the model returns a well-formed, plausible, wrong output that nothing distinguishes from a correct one — an unbounded but non-adversarial fault). And it fails like an organization (agents drift from their role, ignore each other, or treat as verified what is not). The worst are the silent ones, because each downstream component takes them as data and propagates them, and the effect fires far from where the error was born.

Timeout = suspicion, not detection

A slow process and a dead one are indistinguishable in an asynchronous system. A timeout turns absence of response into a suspicion; the design goal is to make a false suspicion harmless (idempotent retry, server-side cancellation, a suspected holder that cannot write).

Zombie + fencing

A worker whose lease expired but keeps executing. The defence: increment an epoch (a fencing token) so the store rejects its stale writes, inventory its partial effects by idempotency key, then decide to go forward or compensate.

Commit gate

A decision correct on its snapshot can be wrong at the instant of the effect (a TOCTOU race). The gate re-checks the declared preconditions and the holder's epoch in the same atomic operation that produces the effect, and rejects otherwise.

Exactly-once (end to end)

Not a property of any one component: it is at-least-once delivery (lease, redelivery, retry) composed with an idempotent receiver keyed by a business key, epoch fencing for zombies, and durable compensation for what cannot complete.

Containment and recovery follow a few disciplines. For infrastructure failures: bulkheads give each class of work its own quota so a slow dependency cannot starve others; a circuit breaker stops sending work that would fail and lets the dependency drain; a retry budget caps the feedback so offered load cannot multiply by the number of attempts (the fix for the retry-storm metastable failure). For silent semantic failures, stacking model-based validators has a floor: a model that checks another model tends to find plausible what the first found plausible, so a second validator with the same blind spot buys little — an independent signal (a deterministic rule, a different source, a person) is worth more than another validator of the same kind. Then degrade deliberately down a declared ladder.

1. 1Detect by suspicion, and make the false suspicion inert: idempotent retry, cancel the abandoned work at the server, and ensure the suspected holder cannot write after its epoch is fenced.
2. 2Gate the effect: re-check preconditions and epoch in the same atomic operation that commits, so a stale decision is rejected rather than executed.
3. 3Contain by resource: bulkhead per class, circuit breaker per dependency, retry budget — none of which sees a semantic error, because the call succeeded.
4. 4Verify by independent signal, not by more validators of the same kind, because the common blind spot is a fixed floor no amount of stacking beats.
5. 5Degrade down a declared ladder (autonomous -> assisted -> shadow -> deterministic fallback) triggered by an explicit signal, not by the model's own judgement — and exercise the degraded path, or it fails the day you need it.

An agent in production is a distributed system with a stochastic component that changes without warning: the provider updates the model behind the same name, someone edits a prompt, a tool changes version, or the input distribution drifts. Operating it means being able to reconstruct why it did what it did, knowing whether it meets its service objectives, deploying changes without exposing everyone, and detecting degradation before users do. The call tree answers "who called whom"; it does not answer "why did it do this". For that you need evidence links — a decision span linked to the tool results it consumed and the effect span linked to the decision — plus the version of model, prompt, policy and tools on every span, with the heavy content stored apart and referenced.

Cost and latency per episode have heavy tails by construction: the number of reasoning-loop iterations varies, and context grows each iteration, so a few runaway episodes dominate the total. The mean is a poor summary — report high percentiles, enforce a per-episode budget, and cap the steps so a loop becomes an escalation. Deploy gradually through a ladder of modes (shadow: decides on real traffic with no effects; assisted: effects need approval; autonomous: acts under policy), and canary a new version against a parallel control, decided by a sequential test — because peeking repeatedly at a fixed-horizon test inflates false positives. Reverting or degrading should be a write to the agent registry, not a code deployment.

Across a whole system the same solutions recur, and a pattern is only worth the name if it states the invariant it protects and the hypotheses under which it protects it — otherwise it is a slogan. "Validate the model's outputs" is a slogan. "A plan is validated against an action model before it reaches the executor; invariant: no plan invalid with respect to the model reaches the executor; hypothesis: the action model describes the world" is a pattern — and note that when the hypothesis fails, the pattern still keeps its invariant and the system fails anyway. The symmetric figure is the antipattern: a recurring solution that looks reasonable and whose consequences are worse than the problem, described with the refactoring that fixes it.

• ▸Agent where a flow suffices: known, enumerable variants solved by an expensive, unauditable model. Refactor to a deterministic flow with a bounded agent node only for the tail.
• ▸Multi-agent by default: several agents that reduce to one, paying in calls and conflicts. Refactor by applying the non-reducibility test and using a flow with an agent node.
• ▸Blind retry: duplicated effects after a timeout. Refactor to an idempotent receiver with a business key, behind a tool gateway.
• ▸Key per attempt: the gateway deduplicates but each retry brings a fresh key, so nothing is deduplicated. Refactor to a key derived from the task and step.
• ▸Double write: state written and command lost (or the reverse) because they are not atomic. Refactor to a transactional outbox, then an idempotent receiver.
• ▸Check-then-act (TOCTOU): effects on state that changed between check and use. Refactor to a commit gate that re-checks in the committing operation.
• ▸Judge of the same model: reviews that confirm what is plausible, with a fixed blind-spot floor. Refactor to a signal with other information.
• ▸Wildcard tool (run_sql, opaque errors, no effect class): refactor to a contract with schemas, effect class, keys and actionable errors.

synthetic — antipattern contrast: blind retry vs. idempotent receiver + commit gate
# ANTIPATTERN: blind retry (AP10) + check-then-act (AP14)
if order.refunded == False:        # checked against a stale snapshot
    ...                            # ... decision takes seconds, world moves ...
retry_until_response(refund(order))  # timeout -> retry -> refund runs again
# result: double refund on a lost response, and a refund even if another
#         channel refunded during the think. Two silent duplicate effects.

# PATTERN: idempotent receiver with business key (P21) + commit gate (P27)
key = fingerprint(run_id, step_id, order.id)
def refund(args, key):             # one atomic operation at the destination
    if ledger.has(key): return ledger.get(key)          # dedup the retry
    if store.refunded(args.order): raise Conflict("already refunded")  # gate: re-check NOW
    r = do_refund(args); store.mark_refunded(args.order); ledger.put(key, r)  # atomic
    return r
# NOTE: the gate stops the TOCTOU duplicate ONLY if the effect is self-falsifying
#   (marking refunded makes the precondition false). For an effect that doesn't
#   change its own precondition, a stable key is what prevents the duplicate.

Four criteria decide. Coverage: if most cases belong to known, enumerable variants, a workflow solves them better, cheaper and with readable traces, and the agent only adds value in the tail of unforeseen variants. Cost and reversibility of the error: a draft reviewed by a person tolerates errors; a payment does not — the more expensive and irreversible the effect, the higher the accuracy an agent must reach to beat escalating to a person. Auditability: a workflow is explained by its definition; an agent's trajectory only by its log. Budget: latency and cost per task. The answer is usually not binary — the hybrid puts the agent inside the flow, in bounded nodes (classify an exception, propose a hard reconciliation, draft), with a typed contract and later validation. It beats the inverse — the flow inside the agent — where the model decides when to invoke tools that hide subprocesses.

A simple cost model makes the threshold explicit. Let the workflow cover a fraction of tasks, each miss escalating to a person of cost H; let the agent on an uncovered task succeed with some probability and, on a miss, either be caught and escalated (cost H) or produce an erroneous effect of cost E. The hybrid beats the workflow on the uncovered tasks only if the agent's expected cost there is below the person's — a threshold that rises sharply with E and falls as the chance of catching the agent's error rises. The honest reading: coverage decides how much the difference weighs, not its sign. With an irreversible, expensive error, an agent that succeeds on most uncovered cases should still not be used there — escalating is cheaper; with a cheap, reversible error, almost any agent beats the person.

Expected direction: Extraction is a bounded agent/model node with a typed contract and recorded output; reconciliation can be deterministic (or an agent node only for invoices matching no order); the pay decision is deterministic policy over typed evidence; reserve/pay/post form an orchestrated saga on a durable engine. Safety: orchestrated saga; an idempotency key derived from stable flow identifiers (run, step, invoice) recorded in the durable history before the first attempt — NOT a per-attempt uuid; the commit gate re-checks that the invoice is approved and the amount matches, in the same atomic operation as the effect. The learner should explain that recording a payment does not change its own precondition (not self-falsifying), so the gate would let a retry through — only the stable business key prevents the duplicate; a refund marked "refunded" IS self-falsifying, so there the gate also blocks the duplicate.

What distinguishes a workflow from an agent, and why is that the decision to make first?

A workflow fixes the next-step function before execution (even if a step calls a model); an agent computes it at run time from goals and perception. What counts is who chooses the next step, not how many model calls there are. It is first because it determines what you can verify before running and how you reason about failure; a fixed-edge graph is a workflow, the same graph with a model-chosen edge is an agent loop.

Why is "exactly-once" not a property of the channel, and how is the end-to-end effect achieved?

No finite number of retries guarantees delivery on a lossy channel, and retrying without an idempotent receiver duplicates effects even on a reliable one. Exactly-once of the EFFECT is achieved by composing at-least-once delivery (lease, redelivery, retry) with an idempotent receiver keyed by a business key, epoch fencing against zombie writers, and durable compensation for what cannot complete.

Why does stacking model-based validators hit a floor, and what beats it?

A model that checks another model tends to find plausible what the first found plausible, so there is a common blind spot — a fixed fraction of errors every same-kind validator misses. Adding another validator of the same kind has diminishing returns toward that floor. Lowering the floor needs an independent signal: a deterministic rule, a different source, or a person.

Using the cost model, when should a hybrid keep escalating an uncovered case to a person instead of letting the agent act?

When the expected cost of the agent acting exceeds the person's cost — which happens when the error is expensive and irreversible (large E) and the chance of catching the agent's error is low. Then even an agent that succeeds on most uncovered cases should not act there, because the rare irreversible error dominates; coverage scales how much this weighs but not its sign.