Cognitive architectures and BDI

Understand the Belief–Desire–Intention model and hierarchical task network (HTN) planning as ways to structure an agent's reasoning, see how an LLM fits as a perception/proposal component rather than the whole mind, and learn why a structured architecture gives bounded autonomy.

Track

Fundamentals

Module

1 · Foundations of agents & security

Prerequisites

What is an autonomous agent · Deterministic vs probabilistic agents

The first lesson described the agent loop but left the decision step as a black box. A cognitive architecture is a principled way to open that box: it specifies the internal data the agent maintains and how that data flows into decisions. The point is not philosophical realism about "minds" but engineering — explicit internal state that you can inspect, test and constrain, instead of a monolithic policy whose reasoning is invisible.

The most durable such architecture is BDI — Belief, Desire, Intention — formalised in the agent-theory literature and inspired by a philosophical account of practical reasoning. Its appeal is that it separates three things that a naive agent collapses together: what the agent takes to be true (beliefs), what outcomes it would like (desires/goals), and what it has actually committed to pursuing right now (intentions). Keeping intentions distinct from desires is what stops an agent from thrashing — abandoning a plan the instant any new desire appears.

Beliefs

The agent's model of the current state of the world, built from perception. Beliefs can be stale or wrong, so they are informational, never authoritative — a belief that an action is permitted is not permission.

Desires

The states of affairs the agent would prefer; its goals. Desires can conflict and need not all be achievable.

Intentions

The desires the agent has committed to and is actively pursuing with a plan. Commitment gives stability and focus; the agent drops an intention only on explicit reconsideration.

HTN planning

Hierarchical Task Network planning: decompose a high-level task into a tree of sub-tasks down to primitive, directly executable actions, using methods that specify how a task may be broken down.

The BDI control cycle reads perception into beliefs, generates options (candidate desires) from beliefs, filters them into intentions via a deliberation step, and then executes the plan for the current intentions — reconsidering only when beliefs change enough to matter. HTN complements this by giving intentions structure: an intention like "resolve ticket" decomposes into "diagnose", "propose fix", "apply fix", each of which decomposes further until every leaf is a concrete tool call. Because the decomposition is explicit, each leaf can be permission-checked independently.

The modern pattern uses an LLM where the classical BDI loop had hand-written symbolic reasoning — but critically, only in the roles where a probabilistic component belongs: turning raw observations into structured beliefs, and proposing candidate options and decompositions. Deliberation (which intentions to commit to) and execution (whether a leaf action is permitted) stay in deterministic, auditable code, per the previous lesson.

BDI control cycle with bounded LLM roles (schematic)
beliefs = {}                      # world model, updated from perception
intentions = []                   # committed plans (stable)

while running:
    percept  = perceive(env)
    beliefs  = llm_interpret(percept, beliefs)     # LLM as perception (fallible)
    options  = llm_propose(beliefs, goals)         # LLM proposes desires/plans
    intentions = deliberate(options, intentions)   # DETERMINISTIC: what to commit to
    plan     = htn_decompose(intentions, beliefs)  # goal -> sub-tasks -> leaves
    for leaf in plan.next_primitive_actions():
        if authorized(leaf, ctx):                  # DETERMINISTIC gate per leaf
            execute(leaf)
        else:
            reconsider(intentions, leaf)           # drop/replace, log

Two boundaries are load-bearing. First, beliefs are derived by a fallible component, so nothing may treat a belief as authority — "the model believes this is allowed" is not a permission. Second, every HTN leaf passes an independent authorization check; decomposing a goal never grants the right to perform its parts. The LLM widens what the agent can understand and propose; the deterministic frame decides what it may commit to and do.

1. 1Define the belief schema: the explicit facts the agent tracks. Mark each as derived-from-perception (untrusted) or asserted-by-the-system (trusted).
2. 2State the desires/goals and their priority. Decide, deterministically, how conflicts are resolved.
3. 3Write the deliberation rule that turns options into committed intentions — including when an intention may be dropped, so the agent is stable but not stubborn.
4. 4Author HTN methods: for each task, the legal decompositions into sub-tasks, bottoming out in primitive tool calls.
5. 5Attach an independent authorization check to every primitive leaf, and log the full chain belief→intention→plan→leaf→decision for audit.

A synthetic on-call "incident" agent. Perception (log lines, alerts) is interpreted by an LLM into beliefs: "service checkout returns 500s", "error rate 12%". Its standing desire is "keep error rate under 1%". Deliberation commits to the intention "mitigate checkout outage" (not to every possible desire). HTN decomposes that into diagnose → identify bad deploy → roll back, bottoming out in the primitive rollback(service, version). The rollback leaf hits an authorization check: production rollbacks require an on-call human token.

synthetic — incident agent, bounded decomposition
beliefs    = llm_interpret(alerts)         # {"checkout": "5xx", "rate": 0.12}
intention  = deliberate(beliefs)           # commit: "mitigate checkout outage"
plan       = htn("mitigate outage")        # -> diagnose -> find_deploy -> rollback
leaf       = rollback("checkout", "v1.4.2")
if authorized(leaf, ctx):                  # prod rollback needs human token
    execute(leaf)
else:
    request_human_approval(leaf)           # bounded autonomy: stop at the boundary
# A wrong belief ("rate 0.12" was a parse error) at worst produces a proposal;
# the human-token rule on the rollback leaf keeps it from acting unilaterally.

Notice how structure contains failure. If the LLM misreads the logs, the damage is a wrong belief, which yields a wrong proposal, which still must pass the per-leaf gate. If the goal is genuine, the architecture also prevents thrash: a new minor alert mid-rollback does not make the agent abandon its committed intention, because only significant belief change triggers reconsideration. Legible state plus per-leaf authority is what makes the autonomy bounded.

Bounded autonomy is the whole reason to pay the cost of an explicit architecture. A monolithic LLM-as-everything agent is hard to audit because its beliefs, goals and commitments are tangled in free-form text; you cannot point to "the moment it decided" or "the fact it relied on". BDI externalises those as inspectable state, and HTN externalises the plan as a tree whose every leaf is an authorization decision point. Together they give you the hooks that identity, authorization and trust-boundary enforcement (the remaining lessons) plug into.

This also clarifies where the LLM genuinely helps. Classical BDI/HTN systems were brittle because humans had to hand-code belief extraction and every decomposition method for a messy world. An LLM absorbs that messiness — interpreting ambiguous input into beliefs, proposing decompositions for situations no one pre-wrote. The architecture keeps that flexibility from becoming unaccountable by confining it to the perception/proposal roles and keeping commitment and authority deterministic.

• ▸Beliefs are derived from perception and can be wrong, stale or injected; treating a belief as authorization is a category error that reintroduces the problem the architecture was meant to solve.
• ▸An HTN method set that is too permissive lets the planner reach dangerous leaves; decomposition must never imply authorization — gate every leaf independently.
• ▸Over-eager reconsideration makes the agent thrash (abandoning intentions constantly); under-eager reconsideration makes it stubbornly pursue a goal the world has moved past. The reconsideration policy is a real design parameter.
• ▸An architecture adds legibility, not correctness. Explicit beliefs and plans make failures findable; they do not make the agent right, and they are not a formal proof of safety.
• ▸Complexity has a cost: more moving parts and state to maintain, test and secure. Use structure where auditability and bounded authority justify it, not reflexively.

Expected direction: A correct answer separates beliefs/desires/intentions explicitly, flags perception-derived beliefs as untrusted (never authority), gives a legal decomposition of a goal into primitive leaves, and identifies the privileged leaves (those touching the outside world) as requiring an authorization gate independent of the decomposition.

What does separating intentions from desires buy an agent?

Stability: an intention is a desire the agent has committed to and keeps pursuing until explicit reconsideration, so the agent does not thrash by abandoning its plan every time a new desire appears.

In an LLM-augmented BDI agent, which roles should the LLM play and which must stay deterministic?

The LLM interprets perception into beliefs and proposes options/decompositions (judgement); deliberation (which intentions to commit to) and per-leaf authorization (what may execute) must stay deterministic and auditable.

Why must every HTN leaf be authorization-checked even though the goal was already approved?

Because decomposing a goal does not grant the right to perform its parts; a legitimate-looking plan can bottom out in a privileged action, so each primitive leaf is checked independently against permissions.