Plan-and-Execute and reflection loops

Separate a planning phase from execution (plan-and-execute), validate the whole plan against a library of preconditions and effects before touching the world, and understand reflection / self-improvement loops (critique-and-revise). Throughout, keep the honest limits in view: validation is correctness against a model, not the world; reflection can entrench errors; and a loop that “looks” better may just be skipping the check that would have caught it.

Track

Engineering

Module

10 · Agentic architectures

Prerequisites

The LLM as a component: ReAct

ReAct interleaves thinking and acting one step at a time, which adapts well and validates nothing in advance. Plan-and-execute makes the opposite choice: decide the whole sequence first, then run it. The gain is that a plan is an object you can inspect before it has any effect — you can validate it, approve it, log it, or reject it against a model of what the actions require and produce. Separating the phases is what makes that possible; it is the same move the classical deliberative agent made, now with a stochastic generator in the planner’s seat.

Reflection loops are the generative version of “generate and test.” A loop produces a candidate, evaluates it, turns the evaluation into a critique, and generates again. Self-refine, critique-and-revise and the evaluator-optimizer pattern all share that skeleton. What separates them is the origin of the signal that closes the loop — the model itself, another model, or an external check with a foundation — and that origin is the whole story. This lesson keeps the enthusiasm and the honesty together: both patterns are useful, and both are bounded by what their checks actually verify.

Plan-and-execute

An architecture where a generator proposes a full plan, an executor runs it step by step checking preconditions before and effects after each action, and replanning starts from the observed state when the world diverges.

Action model

A contract for each action — preconditions and effects — against which a plan can be simulated. Making an implicit operational procedure (“drain before restart”) explicit turns know-how into something checkable.

Plan validation

A function that simulates a plan over the action model and returns valid, or the first failing step with an actionable critique (unknown action, missing preconditions, or an unreached goal).

Critique (back-prompt)

The validator’s first violation returned as actionable text, fed back to the generator. Its value depends entirely on whether the generator uses it to produce fewer errors.

Reflection loop

Generator, evaluator, critic, optional memory, a stopping rule and a budget: produce a candidate, score it, turn the score into a critique, regenerate.

Signal origin

Intrinsic (the generator judges itself and shares its error sources), learned-external (another model or judge with false-positive and false-negative rates), or exact-external (a check decided without error by something other than the generator — tests, a formal validator, a system of record).

One proposition organizes plan-and-execute (PROVEN). If the executor checks preconditions on the observed state and that state equals the real state at the instant of execution, no executed action is invalid in the world. If the observed and real state can differ between check and use — another agent, a delayed observation — that no longer holds. And validating the full plan beforehand adds no guarantee over the per-step check about executed actions; what it adds is economy: not spending actions on a prefix of a plan that will fail later.

The plan-and-execute controller has one stochastic node, the planner, and everything else deterministic: the validator, the bounded replanning, and the guarded executor. The validator stops at the first violation and returns a short, precise critique; the generator regenerates with it, up to a bounded number of times per round, and discrepancies during execution open a new round, up to a bound. The schematic below is structure, not runnable code.

schematic — plan-and-execute with a validator and a plan-validate sketch
plan_and_execute(goal, generate, validate, K, R, budget):
  tried = {}; rounds = 0
  loop:
    s = observe(); if goal <= s: return SUCCESS
    if rounds > R: return ESCALATE("replan exhausted"); rounds += 1
    critique = none; plan = none
    repeat K+1 times:                      # planning round with back-prompts
        if calls >= budget: return ESCALATE("budget")
        plan = generate(s, goal, critique)
        v = validate(plan, s, goal)         # simulate over the action model
        if v.valid: break
        critique = v.critique               # "step 3: revert(db) not applicable; missing lock(db)"
    if plan is none or seen(tried, s, plan): continue   # avoid replanning loops
    for action in plan:                     # guarded execution
        s = observe(); if not preconditions(action) <= s: break   # TOCTOU still possible
        ok = execute(action); if not ok or observe() != effect(s, action): break

# validate(plan, s0, goal):   # correct w.r.t. the MODEL, not the world
#   s = s0
#   for i, o in enumerate(plan):
#       if o not in actions:          return (False, i, "unknown action")
#       if not pre(o) <= s:           return (False, i, "missing " + (pre(o) - s))
#       s = (s - del(o)) | add(o)
#   if not goal <= s:                 return (False, None, "goal not reached")
#   return (True,)

Reflection is the same shape over an artifact rather than the world, and its mechanism is best understood as a two-state Markov chain per field (correct / incorrect). If a self-judge flags an incorrect field with one probability and a correct field with another, the accuracy converges to a fixed point that is above the start only when the judge flags wrong answers more often than right ones. This is the honest core: an intrinsic loop that, on review, flips more correct answers to incorrect than the reverse converges below where it began. With an exact external signal (it never flags a correct field and sometimes flags a wrong one) the fixed point is full correctness on the checked property — and exactly that, nothing about what it does not check.

1. 1Write the action model explicitly: preconditions and effects for each action. Without it there is nothing to validate against, and the critique has nothing to say.
2. 2Validate the whole plan by simulating it over the model; on failure, return the first violation as an actionable critique and regenerate, bounding the regenerations per round.
3. 3Execute under a guard: check preconditions on the observed state before each action and compare observed effects after; on a discrepancy, stop and replan from what you observed, not from what you predicted.
4. 4Bound replanning rounds and remember (state, plan) pairs already tried, so the generator cannot propose the same doomed plan forever; tolerate a few repeats, since a transient failure can justify retrying.
5. 5For reflection, fix the signal first: prefer an exact external check for acceptance; use a learned judge only knowing it caps precision at its likelihood ratio; never let an intrinsic self-judge decide acceptance.
6. 6Only write a lesson to memory when an external signal confirmed both the failure that motivated it and the fix — a founded lesson. Record “verified” per field, and escalate what nothing can check.

A synthetic coordinated remediation: two support services fail, every service must be drained from the balancer before it is touched and re-added after, and only one may be out at a time; reverting also needs a deploy lock. The action model below is the contract the validator uses. The example shows validation catching a reordering the step-by-step loop would only discover mid-execution.

synthetic — action model and a plan-validate catch
# action model (the contract the validator simulates against)
drain(s):       pre [capacity, in_lb s];      add [drained s];  del [capacity, in_lb s]
restart(s):     pre [drained s];              add [ready s, mem_ok s]
take_lock(s):   pre [lock_free s];            add [lock s];     del [lock_free s]
revert(s):      pre [drained s, lock s];      add [ready s, version_ok s]
readd(s):       pre [drained s, ready s];     add [capacity, in_lb s]; del [drained s, ready s]

# optimal plan for a memory fault in api and a deploy fault in db (8 steps):
#   drain(api) restart(api) take_lock(db) readd(api) drain(db) revert(db) readd(db) release(db)

# if the planner swaps steps 4 and 5, the validator replies:
#   "step 4: drain(db) not applicable; missing (capacity)"
# -> WITHOUT a validator, the environment drains db with api still out -> overload
# -> WITH per-step checks, the action is refused but the system is left half-done
# -> WITH full validation, the plan never touches the environment at all

This is the gradation of the proposition made concrete. Per-step checking prevents the invalid action from executing; validating the whole plan prevents the system being left half-done and, crucially, lets the planner fix the ordering with a cheap critique before spending any real action. None of this certifies the plan is right about the world — only that it is consistent with the model. A race (another team takes a service out between check and execution) is outside any prior check and needs a reservation or lease at the point of execution.

Plan-and-execute is friendly to enforcement precisely because the plan is inspectable before it acts: it can be validated, approved or logged, and the execution is deterministic code. That is the clean place to put an authorization frontier — the plan is proposed, and a deterministic layer decides whether it (and each of its effects) is permitted. The validator is itself an instance of the transversal rule: it must be a different component with a different source of truth, because self-verification by the generator is measured to be unreliable and can make planning worse, while a correct external verifier helps.

Reflection carries the sharpest security lesson of the four: a loop optimizes against its evaluator. With weak tests, the generator learns to satisfy the tests rather than the task, and a loop that “looks” better may simply be skipping the check that would have failed. A reflection memory makes this worse — a lesson written from an unfounded self-critique conditions every future task under its key, entrenching an error. The enforcement posture is therefore: acceptance lives in an exact external check; a learned judge can polish a reversible, low-cost artifact (the tone of a customer reply) but never decide an effect (the refund amount is decided by a deterministic policy); and lessons are written only when a founded signal confirms them. Here too, reasoning is not authorizing.

• ▸Validation is correctness against the model, not the world: with a wrong action model the validator certifies wrong plans with confidence.
• ▸Races (time-of-check to time-of-use) sit outside any prior check and require reservations, leases, fencing tokens or transactions at the point of execution.
• ▸Plan-and-execute needs an action model that must be built and maintained, and it adapts more slowly than the step-by-step loop; in open tasks where the next step depends on observed content, the initial plan is a conjecture.
• ▸Success depends on a generator that uses the critique; if critiques do not reduce error (the persistence of error after a critique is high), the system never finds a valid plan even though it never executes an invalid one.
• ▸Reflection with an intrinsic signal can converge below its starting accuracy; an exact external signal only fixes what it checks and lets everything else through; a learned judge caps the accepted artifact’s reliability at its likelihood ratio, with correlated judge/generator errors making it worse.
• ▸A loop that improves a metric may be exploiting a weak evaluator rather than solving the task; an unfounded lesson in memory entrenches an error across all future tasks under its key.

Expected direction: A good answer simulates the plan over the model to catch the ordering violation before execution, and states that validation guarantees correctness only against the model (not the world, and not against races, which need reservations/leases at execution). The reflection design routes acceptance to an exact external check, allows a learned judge only for reversible low-cost artifacts, escalates the unverifiable, and writes a lesson to memory only when an external signal confirmed both the failure and the fix — explaining that an unfounded self-critique lesson would bias every future task under its key.

What does validating a plan against an action model guarantee, and what does it not?

It guarantees the plan is correct with respect to the model — every action’s preconditions hold in simulation and the goal is reached. It does not guarantee correctness in the world (the model can be wrong), does not protect against races between check and execution, and does not by itself prevent executing invalid actions — that is the per-step precondition check; full validation mainly avoids spending actions on a doomed plan and enables cheap critique-driven regeneration.

Why can a reflection loop with an intrinsic (self-judged) signal make results worse?

Modeled per field as a two-state chain, accuracy converges to a fixed point above the start only if the self-judge flags wrong answers more often than right ones; a judge that flips more correct answers to incorrect than the reverse converges below where it began. Self-judgement shares the generator’s error sources, so it tends to fail exactly on the errors that matter.

What does an exact external signal give a reflection loop, and what is the catch?

It drives the checked properties to full correctness and lets acceptance be founded. The catch is that it is exact only on what it checks and partial on the artifact: a bug that passes the check — or any field the check does not see — is accepted, so “passed the check” is not “correct.”

Why must the plan validator be a different component from the generator, and why is reflection a security concern?

Because self-verification by the generator is measured to be unreliable and can make planning worse, while a correct external verifier with a separate source of truth helps. Reflection is a security concern because a loop optimizes against its evaluator — weak checks are gamed, a loop can “look” better by skipping checks, and an unfounded memory lesson entrenches errors across future tasks.