Formal models of an agent and a taxonomy of architectures

Treat an agent as a formal object — a perception / decision / action loop over an environment, with percepts, internal state, goals and actions — using a PEAS-style framing and the structured tuple. Then place architectures on a reactive-to-deliberative-to-hybrid map, and carry one invariant throughout: a decision branch proposes, and a deterministic authority frontier disposes, so no branch of the policy may cross into unchecked action.

Track

Engineering

Module

10 · Agentic architectures

Prerequisites

What is an autonomous agent

Calling something an “agent” says almost nothing until you fix what the word denotes. The durable definition is deliberately plain: an agent is whatever maps a history of what it has perceived onto a next action. That is general enough to cover a thermostat, a chess program and an LLM wrapped in a tool loop — and useless for design until you refine it, because it says nothing about how the action is computed, what the agent knows, or who permits it to act.

This lesson builds the refinement in two moves. First it names the internal pieces — percepts, state, beliefs, goals, a policy, actions — so that the decision step stops being a black box. Second it adds a frontier that lives outside the policy: a deterministic check that decides whether a proposed action is permitted. Keeping those two apart is the thread that runs through every later architecture. The policy is where judgement (and later, a language model) belongs; the frontier is where authority lives, and the two must not be the same object.

A classic framing, before any internal structure, is PEAS: name the Performance measure the agent is judged by, the Environment it inhabits, the Actuators through which it acts, and the Sensors through which it perceives. PEAS forces you to write down the things a demo hides — what counts as success, what the world can do back, and the exact surface where the agent touches reality. The structured tuple below then opens up what happens between a sensor reading and an actuator call.

Percept / observation

A datum that arrives through a sensor. It is raw evidence, not yet a conclusion. Two different world states that read the same through the sensor are the formal definition of partial observability.

Belief

What the agent concludes about the world from many observations and its model. A belief can be stale, wrong or injected, so it is informational and never authoritative: “the agent believes this is allowed” is not permission.

Internal state

A finite summary the agent keeps in place of its whole history — beliefs plus goals, commitments, where each plan stands, counters. Mixing world facts with the agent’s own decisions in one blob means revising one corrupts the other.

Goal

A desired state of affairs with an evaluable satisfaction condition over beliefs. “Resolve the customer’s problem” is not yet a goal until you can say when it is resolved.

Policy

The function that chooses what action to propose given state and active goals. It proposes; it does not authorize.

Authority frontier (delegated scope)

A predicate over an action and a summary of execution history (counts, accumulated amounts, approvals), fixed by the principal and not modifiable by the policy. The executed action is the proposal if the frontier permits it, and an escalation otherwise.

The structured agent is then the tuple of percepts, internal states (with an update function), beliefs (a projection of state), goals (with satisfaction conditions), a policy, and actions (with preconditions and expected effects). The authority frontier is deliberately not one of the tuple’s components: it sits between the policy’s proposal and the environment, and that separation is the whole point.

The same agent can be written three ways, each gaining and losing something. As a function from histories to actions it is fully general and gives no design guidance. As the structured tuple it names the internal pieces and the control points, but leaves the semantics of uncertainty vague. As a partially observable Markov decision process (POMDP) it gives exact meaning to “belief” — the belief is a probability distribution over states that is a sufficient statistic for the history, updated by Bayes’ rule (a prediction through the transition model, a correction by the observation, a normalization). The theoretical cost is steep (solving a POMDP optimally is intractable in general), so every practical architecture gives up part of that expressiveness; the discipline is to make each sacrifice explicit, because it determines which errors the agent will make.

Whichever formulation you use, the loop has the same shape, and the authority frontier wraps it. The schematic below is structure, not runnable code.

schematic — structured agent loop: the policy proposes, the frontier disposes
state   = initial_state          # beliefs + goals + commitments + counters
history = empty_execution_summary  # counts / amounts / approvals for the frontier

loop:
    active  = [g for g in goals if not satisfied(g, project_beliefs(state))]
    proposal = policy(state, active)         # PROPOSE (judgement may live here)

    # --- authority frontier: outside the policy, fixed by the principal ---
    if authorized(proposal, history):        # DISPOSE (deterministic gate)
        action = proposal
    else:
        action = escalate(proposal, state)    # out of scope -> hand to a human

    for monitor in safety_monitors:          # a monitor can enforce SAFETY only
        if not monitor.step(event(action, history)):
            halt()                            # stop at the first offending prefix

    execute(action)
    history = record(history, action)
    percept = perceive(env)
    state   = update(state, action, percept)  # e.g. the belief-update rule

# No branch of policy() can alter authorized(...). A decision branch that
# "reasons its way" to an action still cannot cross the frontier unchecked.

Two facts about the frontier are load-bearing. A monitor that merely watches the event trace and can halt the system can enforce only safety properties — “nothing bad ever happens”, whose violations always have a finite offending prefix. Liveness (“something good eventually happens”) cannot be enforced this way without a deadline that turns it into safety. And because the frontier is a separate predicate, you can verify it independently of the policy; if the hundred-euro limit is written inside the prompt, it is no longer checkable apart from the thing it is meant to constrain.

1. 1Write the PEAS: performance measure, environment, actuators, sensors. Be concrete about what counts as success and what the environment can do that the agent does not control.
2. 2Fix the percept set and the belief schema. Mark each belief as perception-derived (untrusted) or system-asserted (trusted), and never let a belief stand in for a permission.
3. 3State goals as evaluable satisfaction conditions over beliefs, with priorities and a deterministic rule for resolving conflicts.
4. 4Define the policy as the function that proposes an action from state and active goals — and nothing more; it returns a proposal, not an effect.
5. 5Define the authority frontier separately: a predicate over the action and an execution summary, owned by the principal. Decide what is in scope, and route out-of-scope proposals to escalation.
6. 6Attach safety monitors to the event trace, and log the whole chain percept → belief → goal → proposal → authorization → action for audit.

A synthetic operations agent watches one service that may be healthy, degraded or down. The numbers are illustrative, chosen to show the mechanism. PEAS: the performance measure penalizes downtime and unnecessary restarts; the environment is the service plus an outside world that can redeploy code without telling the agent; the actuators are wait, restart, diagnose and escalate; the sensors are a monitoring signal and a detailed health check.

synthetic — the tuple and the authority frontier for the ops agent
P (percepts): {ok, alert}  +  the human's reply after an escalation
S (state):    belief b over {healthy, degraded, down}; active goal;
              "do not act while an escalation is pending"; recent-restart times
B (beliefs):  projection of b        G (goals): service_healthy, satisfied when b(healthy) >= 0.9
A (actions):  wait, restart, diagnose, escalate   (each with preconditions)

policy(state):                      # PROPOSE
    # e.g. a cost-aware rule or a lookahead over the belief tree
    return "restart"  if b(down) + b(degraded) high else "wait"

# --- authority frontier, fixed by the principal, outside the policy ---
authorized("restart", history):
    return restarts_in_window(history) < 2      # at most two restarts per 10 steps

# Run: the policy proposes restart a third time; the frontier refuses and the
# agent escalates. With the SAME inputs and no frontier, a safety monitor for
#   G not(effect and not authorized)
# flags the violation at exactly the same step. The policy is identical in both
# cases; what changes is WHERE authority lives.

Notice how little the policy has to be trusted. If the monitoring signal is misread, the worst case is a wrong proposal, which still meets the frontier. If an outside redeploy keeps breaking the service, the policy may keep proposing a restart, and the frontier keeps converting the third one into an escalation. Good judgement is welcome in the policy; it is never the thing that authorizes the act.

Real systems are not one architecture but a graph of components, each drawn from a different family. Describe a security operations setup honestly and it decomposes into: event-driven ingestion and correlation; rule-based triage; an LLM investigator with read-only tools; a shared case file; plan-based selection of a containment playbook; a rule engine that authorizes containment; and a reflex that blocks any action on critical assets. Label each component by how verifiably its output respects a safety property, on three coarse levels (low, medium, high).

The structural rule — PROVEN as a graph property — is that an actuator’s guarantee drops to the level of the weakest component that can reach it without crossing a veto, where a veto is a component with complete mediation that only permits or blocks and never originates or modifies an action. So if the low-level LLM investigator can influence an actuator with no veto in between, the safety guarantee on that actuator is no better than the investigator, however trustworthy everything else is. Granting the investigator a containment tool “for efficiency” is exactly the move that destroys the guarantee on the most dangerous actuator: the counterexample path starts in third-party text and ends in a real effect, which is the structural shape of an indirect prompt injection. The remedy is to place a veto on the edge into the actuator, which is the same authority-frontier idea applied to a graph. What this bound does NOT measure is decision quality — it is a ceiling on what is verifiable about effects, not a statement that the agent decides well.

• ▸A formal model is correct relative to its model. If the transition probabilities, observation model or costs are mis-estimated, the agent will be coherent and wrong, and models degrade as the environment drifts.
• ▸POMDPs are intractable in general; every practical architecture gives up expressiveness (symbolic beliefs, short horizons, cheap approximations). Each sacrifice fixes which errors the agent will make and should be stated, not hidden.
• ▸Beliefs are perception-derived and can be stale or injected. Treating a belief as authorization is a category error that reintroduces exactly the problem the authority frontier exists to solve.
• ▸The verifiability bound assumes vetos have complete mediation and a correct policy, and that the component graph is complete. An omitted edge — a shared cache, a side channel, a human who copies and pastes — silently falsifies the conclusion.
• ▸Coarse qualitative levels are judgements, not measurements; two engineers can disagree, especially on the middle level. The model makes the disagreement explicit; it does not settle it.

Expected direction: A good answer separates proposal from authorization: the policy returns proposed actions, and an independent, principal-owned predicate decides which execute and which escalate. Perception-derived beliefs are flagged untrusted and never used as permission. The frontier is justified as living outside the policy so it can be verified independently and cannot be altered by any decision branch — including one that “reasons” its way to a privileged action.

What does the authority frontier add that the policy cannot provide on its own, and why must it be separate?

It decides whether a proposed action is permitted, using a predicate owned by the principal over the action and an execution summary. It must be separate so no branch of the policy can alter it and so it can be verified independently; folded into the policy (or the prompt), authorization is no longer checkable apart from the thing it constrains.

In a POMDP framing, what exactly is a belief, and why is it useful?

A probability distribution over hidden states that is a sufficient statistic for the history: there is an optimal policy depending on the history only through the current belief, updated by Bayes’ rule. It gives “belief” an exact meaning and separates estimation from decision.

An LLM investigator with read-only tools feeds a case file that a plan selector reads before triggering a containment actuator, with no veto between them. What is the verifiability guarantee on that actuator, and why?

It drops to the level of the weakest component on a veto-free path — here the low-level investigator — so the actuator is no more verifiable than untrusted, however strong the other components are. Restoring the guarantee requires a veto (complete mediation) on the edge into the actuator.

Why is modelling an agent not the same as making it correct?

A model makes beliefs, goals, commitments and authority legible — inspectable, boundable, comparable — which lets you audit and limit the agent. It does not make the agent’s decisions right, and it is not a proof of safety.