Insecure Output Handling

Domain

AI, LLM & agents

ROOTVIEL support

Partial — Core is built; parts are not yet wired end-to-end.

Analysis family

FAM-GENAI-STATIC

CWE

CWE-20

OWASP

LLM05

Insecure output handling is the failure to treat a language model’s output as untrusted input to whatever consumes it. The model produces text; the application forwards that text into a component that acts on it — a system shell, a database query, an HTML page, a code evaluator, a downstream service — without the validation, sanitisation, or encoding that component requires. The model is effectively an untrusted user of the downstream system, but the code treats it as a trusted source.

This is the bridge between the AI layer and classic application security. The vulnerability class in the consuming component is ordinary — command injection, SQL injection, cross-site scripting, SSRF — but the source of the malicious string is the model, which may have been steered by a prompt injection upstream. The defect lives in the handling, not the model: output is consumed at a privileged sink with no intervening validator.

This appears wherever model output flows onward instead of stopping at a human reader: agents that execute generated code or commands, assistants that build and run database queries, systems that render model output as HTML, and pipelines where one model’s output becomes another component’s input. The more “agentic” the system — the more it acts on its own output — the more exposed it is.

• ▸Agents that run model-generated shell commands or code.
• ▸Natural-language-to-SQL features that execute the generated query.
• ▸Applications that render model output as HTML or Markdown in a browser.
• ▸Pipelines where model output is passed to a downstream API as parameters.

The clean way to see this is as a dataflow: the model’s output is a taint source, and any privileged component that interprets a string as a command, a query, or markup is a sink. A vulnerability exists when there is a path from that source to a sink with no validator on the path that neutralises the string for that sink’s grammar. This is identical to how one reasons about injection generally; the only novelty is that the untrusted source is a model.

Crucially, this is a structural property of the design, visible in code, not something that needs a live model to exercise. If output reaches `exec`, a raw SQL string, `innerHTML`, or a request builder with no encoding or allow-listing in between, the weakness is present regardless of what the model happens to emit today. That makes it amenable to static analysis: you look for the source-to-sink path and the missing validator, not for a specific malicious output.

Privileged sink

A component that interprets a string as an instruction — a shell, SQL engine, HTML renderer, code evaluator, or request builder.

Validator

A control on the path that neutralises the string for the sink’s grammar: encoding, parameterisation, allow-listing, or schema checks.

Structural signal

Evidence the weakness exists from the shape of the code — a source-to-sink path with no validator — rather than from a runtime exploit.

The defect is a missing trust boundary at the moment model output becomes input to another component. The developer mentally classifies model output as “our system’s result” and forwards it with the implicit trust that carries, when it should be classified as untrusted input requiring the same validation any external string would. The absence of a validator on a model-output→privileged-sink path is the root cause, and it is a property of the application’s wiring.

1. 1The attacker influences the model’s output — often through a prompt injection upstream — so it contains a payload for the downstream sink.
2. 2The application takes the model’s output and passes it to a privileged component without validation or encoding.
3. 3The component interprets the payload according to its own grammar: the shell runs a command, the database runs injected SQL, the browser executes injected script.
4. 4The effect executes with the privileges of the consuming component, not the model.
5. 5Because there was no validator on the path, nothing stood between the model’s string and the component’s execution.

A synthetic, illustrative agent that executes a shell command built from model output with no validation:

synthetic — do not use; illustrates the flaw
# UNSAFE: model output flows straight into a privileged sink (the shell).
cmd = model(user_task)                 # model may be steered to emit a payload
os.system(cmd)                         # sink: interprets the string as a command

# same shape with other sinks:
db.execute("SELECT * FROM t WHERE k = '" + model(q) + "'")  # SQL sink, no parameterisation
page.innerHTML = model(q)                                   # HTML sink, no encoding

The fix is to put a validator on the path: never let model output be the command/query/markup itself. Constrain the model to choose from a bounded, code-defined set of actions, and have deterministic code construct the privileged call using the component’s safe API (parameterisation, encoding, allow-lists).

synthetic — validated action, code builds the privileged call
# SAFER: model proposes a bounded action; code validates and builds the call.
proposal = model(user_task)                    # model returns a structured choice, not a raw command
action = parse_to_enum(proposal, allowed=ALLOWED_ACTIONS)  # reject anything off the allow-list
if action is None:
    deny("action not permitted")
else:
    run_action(action)                         # code maps enum -> safe, parameterised operation

# sink-specific safe APIs instead of string building:
db.execute("SELECT * FROM t WHERE k = ?", [validated_key])  # parameterised
page.textContent = model_output                              # encoded, not interpreted as HTML

• ▸Code execution: model output reaching a shell or evaluator can run attacker-chosen commands.
• ▸Data integrity/confidentiality: unvalidated output in a query enables injection, exfiltration, or modification.
• ▸Client-side compromise: unencoded output rendered as HTML enables cross-site scripting.
• ▸Server-side request forgery and chaining: output used to build requests can reach internal systems; combined with upstream injection, the chain runs end to end.

Because the weakness is structural, the primary detection is static dataflow analysis: taint model output as a source, identify privileged sinks (shell, SQL, HTML, eval, request builders), and search for a path between them with no validator appropriate to the sink. A match is a structural signal that the handling is unsafe — it does not require demonstrating a live exploit. Runtime testing can confirm exploitability, but the finding stands on the missing validator in the code path.

Support level

Partial — a static design-graph detector exists for Python; it is a structural signal, not a runtime demonstration.

Family

GenAI (static): the analysis is over code and the design graph, not a live model.

Declared invariant

“A validator must exist on every model-output→privileged-sink path.”

The design-graph detector marks a missing validator on a model-output→privileged-sink path. It is a structural signal derived from Python static analysis: it reasons about the shape of the code — does output flow to a sink, and is there a validator in between — rather than demonstrating the exploit at runtime. A finding says “this path lacks an appropriate validator”; it does not claim to have executed an injection. Support is partial precisely because it covers the Python static surface and does not exercise the behaviour live.

Treat model output as untrusted input and interpose a sink-appropriate validator everywhere it flows onward.

• ▸Preventive: never pass raw model output to a shell, query, renderer, or evaluator; constrain the model to structured, allow-listed choices and let code build the privileged call with safe APIs (parameterisation, output encoding, allow-lists).
• ▸Preventive (design): minimise the number of privileged sinks reachable from model output; prefer structured outputs (enums, schemas) over free text at every boundary.
• ▸Detective: run static dataflow analysis to find model-output→sink paths missing a validator; keep it in the build so new paths are caught.
• ▸Corrective: when a path is found, add the correct validator for that sink and reduce the sink’s privileges so a future gap has a smaller effect.

Verify by re-running the static analysis and confirming no model-output→privileged-sink path remains without an appropriate validator, and by checking that each validator matches its sink’s grammar (parameterisation for SQL, encoding for HTML, allow-list for a shell). Treat a clean static result as UNKNOWN beyond the analysed Python surface, not as proof of safe handling everywhere.

• ▸Trusting model output because “it’s our system’s result”, rather than treating it as untrusted input to the next component.
• ▸Applying one generic sanitiser everywhere instead of the validator matching each specific sink’s grammar.
• ▸Letting the model emit the raw command/query/markup instead of a bounded, allow-listed action code can safely build from.
• ▸Reading a clean static scan as full safety, ignoring sinks the analysis could not resolve.

Insecure output handling is the downstream half of the injection story: a prompt injection steers the model, and insecure output handling lets the steered output reach a privileged sink. It overlaps with excessive agency — both concern what an agent’s output can cause — and it is the mechanism by which classic injection classes (command, SQL, XSS, SSRF) re-enter through an AI component.