Threat Modeling AI Agents: Trust Boundaries, Controls, and Evidence

Executive summary

An AI agent combines an uncertain decision-producing component with ordinary software that can retrieve data, preserve memory, call tools, and change external state. The model can propose an action. It cannot be the security boundary that makes the action authorized.

A useful threat model therefore separates two planes:

• the behavior plane interprets untrusted language and produces non-deterministic content, plans, classifications, or tool proposals;
• the authority plane authenticates principals, validates parameters, enforces policy, applies limits, records evidence, and decides whether an external effect may occur.

Prompt instructions, system messages, model alignment, classifiers, and output filters can reduce unsafe behavior. They are not equivalent to reference monitors. A system remains secure only if an attacker can fully influence model output and still cannot exceed the caller's authorized capability, silently cross tenant boundaries, execute an unapproved high-impact action, or erase the evidence needed to investigate.

This paper is vendor-neutral and applies to retrieval assistants, coding agents, workflow agents, browser or computer-use agents, and multi-agent systems. It turns threats into control claims with enforcement points, evidence, tests, owners, residual risk, and review triggers. It does not claim that every model failure is preventable or that a passing evaluation proves safety.

Define the decision and system boundary

Start with the release or architecture decision, not “threat model the AI.” For example:

Decide whether the support agent may retrieve tenant tickets, draft a response, and issue a refund up to a bounded amount without a human approver.

The sentence exposes data scope, external effects, financial authority, tenant isolation, and the proposed approval threshold. Record:

• intended users and subjects: who asks, who is affected, and whose authority is exercised;
• allowed outcomes: which information and actions the system may produce;
• unacceptable outcomes: cross-tenant disclosure, unauthorized payment, secret exposure, destructive change, concealed action, or unsafe autonomous persistence;
• operating context: channels, jurisdictions, data classes, models, tools, retrieval sources, and latency constraints;
• human responsibility: who approves, monitors, interrupts, investigates, and accepts residual risk;
• assumptions: model and provider behavior, identity assurance, tool semantics, data provenance, and dependency availability;
• review triggers: capability, model, prompt, tool, data, policy, autonomy, or threat changes.

The NIST AI Risk Management Framework 1.0 provides Govern, Map, Measure, and Manage functions for AI risk. The NIST Generative AI Profile identifies risks and actions specific to generative AI. Neither publication supplies application authorization logic or proves that a particular agent is secure.

The independent AWS Generative AI Developer - Professional study plan can help readers map implementation concerns to that certification's current scope. The control model in this paper remains vendor-neutral and should be checked against the actual platform and owner documentation.

Chatbot and agent are different risk surfaces

A chatbot that returns text can still leak data, manipulate users, or emit dangerous instructions. An agent adds a control loop that can choose and sequence tools, observe results, update a plan, preserve state, and attempt again. Its output may become an input to an interpreter, API, browser, shell, ticketing system, identity platform, or another agent.

Autonomy is not binary. Record the maximum loop length, permitted tools, data reach, monetary or operational limits, ability to create new principals or tasks, persistence across sessions, and which actions require confirmation. A “human in the loop” is only a control when the human receives accurate context, has enough time and expertise, cannot be trivially habituated, and the system cannot bypass or misrepresent the decision.

Trust-boundary model

The model and orchestrator are inside the untrusted-decision zone because retrieved content, user text, tool results, memory, and model behavior can shape their output. A policy enforcement gateway separates a proposal from an effect. It receives authenticated actor and tenant context from trusted application state—not from model-generated arguments—and makes a deterministic decision over a typed action.

The evidence path records the proposal, policy decision, normalized parameters, approval, tool result, model and policy versions, and correlation identifiers. Sensitive reasoning text is not automatically necessary or safe to retain. Evidence design should minimize personal data and secrets while preserving what an investigator needs.

Inventory components and flows

Include more than the model endpoint:

• user interfaces, APIs, session and tenant state;
• system instructions, prompt templates, model routing, safety settings, and inference providers;
• orchestrators, planners, evaluators, retry loops, and termination conditions;
• retrieval indexes, source documents, embedding pipelines, ranking, citations, and access filters;
• short-term context, long-term memory, caches, conversation history, and deletion paths;
• tool schemas, adapters, credentials, sandboxes, browser or computer-use layers, and external APIs;
• policy engines, approval services, budgets, queues, rate limits, and kill switches;
• training, fine-tuning, evaluation, build, deployment, and configuration pipelines;
• logs, traces, datasets, feedback, security findings, and incident evidence;
• models, packages, plugins, skills, MCP servers, data suppliers, and other third parties.

Label each flow with the principal, tenant, data class, integrity assumption, protocol, action semantics, and authoritative enforcement point. “Agent calls refund tool” is insufficient. A useful label is:

Orchestrator submits a typed refund proposal; gateway binds the authenticated support agent and tenant from server session state, validates order ownership and amount, requires approval above the limit, issues a single-purpose request to the payment adapter, and records decision and result IDs.

Separate model behavior from authorization

Assume the model may output an attacker-chosen tool name and arguments. The resulting action should still be constrained by code and policy outside the prompt.

For every tool, define:

• a narrow verb with a typed schema rather than arbitrary code or generic HTTP;
• the authenticated actor, tenant, and delegated subject supplied by trusted state;
• eligible resources and row-level or object-level authorization;
• parameter validation, canonicalization, and safe defaults;
• monetary, volume, time, destination, and rate limits;
• whether an action is read-only, reversible, high-impact, or prohibited;
• approval and step-up authentication requirements;
• idempotency, replay handling, timeout, retry, and compensation behavior;
• observable decision and result records;
• a revocation and containment path.

Never let the model choose its own user ID, tenant, role, or approval state. Treat model-selected resource identifiers as untrusted references and re-authorize the resolved object. Tool descriptions should help the model choose correctly, but tool policy must remain correct when it chooses maliciously.

A policy engine such as Open Policy Agent can centralize deterministic decisions and decision logs. It does not remove the need to authenticate input, secure policy distribution, validate tool adapters, and test the effect at the destination. The policy decision and tool implementation must agree on semantics; read_account cannot quietly become read_any_account in an adapter.

Prompt-injection filters, delimiter conventions, instruction hierarchy, and model-based judges are defense-in-depth. They can reduce attack success or route suspicious sessions for review. They cannot establish that a refund, email, deployment, file access, or identity change is authorized. Put enforceable controls at the action and data boundaries.

Primary threat classes and abuse paths

The OWASP Top 10 for LLM Applications 2025 covers risks including prompt injection, sensitive information disclosure, supply-chain weaknesses, data and model poisoning, improper output handling, excessive agency, system-prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. The OWASP Top 10 for Agentic Applications 2026 and Securing Agentic Applications Guide 1.0 extend the focus to agent goals, tools, identities, memory, inter-agent communication, cascading failures, and human-agent trust. Use these as taxonomies and question sets, not proof that a local threat is present or controlled.

Direct and indirect prompt injection

In a direct attack, a user supplies instructions intended to override the application's goal or extract restricted context. In an indirect attack, hostile instructions arrive through a document, email, web page, image-derived text, tool result, code comment, ticket, or retrieved record. The agent may interpret data as an instruction because both share the same language channel.

A realistic abuse path is:

1. an attacker publishes a document that a victim's agent is likely to retrieve;
2. the document instructs the agent to copy session data to an attacker-controlled destination;
3. retrieval supplies the content inside the agent context;
4. the model proposes a network or messaging tool call;
5. a broad tool credential permits the destination;
6. insufficient approval and evidence allow exfiltration to complete unnoticed.

Content labeling, source reputation, instruction-data separation, model defenses, and anomaly detection can lower likelihood. Destination allowlists, per-resource authorization, data-loss constraints, tool isolation, and approval at the gateway constrain impact even if injection succeeds.

Improper output handling

Model output becomes dangerous when downstream software treats it as trusted syntax. Examples include SQL, shell commands, HTML, Markdown links, templates, file paths, regular expressions, infrastructure plans, or serialized API arguments. Encode for the destination, parse into constrained structures, use parameterized interfaces, and reject unexpected fields. A sandbox limits some consequences, but its filesystem, network, credential, kernel, and persistence boundaries must be explicit and tested.

Excessive agency and confused delegation

An agent may have a valid credential but use it beyond the user's intent. It may also combine individually permitted actions into an unsafe sequence: enumerate data, create a public destination, copy records, and remove evidence. Bound each tool and evaluate workflow-level policy, not only single calls. Limit loop iterations, cumulative cost, data volume, destination classes, and the agent's ability to acquire new authority.

Delegation chains require an attributable subject. If agent A asks agent B to call a tool, B must not infer authorization from A's prose. Propagate a signed or server-held delegation context with actor, tenant, scope, purpose, expiry, and chain. Re-evaluate policy at every effectful boundary.

Retrieval, memory, and cross-tenant disclosure

Retrieval filters must be enforced before candidate content reaches the model. Post-generation redaction cannot reliably undo disclosure already present in context or provider logs. Bind tenant and subject filters in the retrieval service, protect index ingestion, and test adversarial identifiers, deleted records, cache keys, embeddings, ranking metadata, and document-level ACL changes.

Memory creates a delayed trust problem. An attacker can store instructions or false claims that influence a later user or higher-authority session. Record provenance, tenant, author, creation path, confidence, purpose, expiry, and deletion state. Do not automatically convert model summaries into authoritative facts. Separate preferences from credentials, approvals, and security attributes. Make memory write and read policy explicit.

Sensitive information and system instructions

Assume prompt and system-message text may be inferred or exposed. Do not store secrets, durable credentials, hidden authorization rules, or tenant access logic in them. Minimize data sent to models and tools, apply purpose and retention decisions, and understand provider logging and training settings. Secret scanners and output filters help detect leakage but cannot make an exposed secret safe; use short-lived, scoped credentials and rotate on suspected disclosure.

Poisoning, provenance, and supply chain

Attackers may corrupt training examples, fine-tuning data, evaluation sets, retrieval sources, model artifacts, packages, plugins, tools, prompt templates, or policy files. Track source, transformation, reviewer or producer, version, integrity, and deployment identity. Separate untrusted contribution processing from release authority. Protect golden evaluations from silent optimization and keep adversarial test cases access-controlled where disclosure would invalidate them.

The provenance research brief explains why a source chain supports verification but does not prove truth. SLSA supply-chain levels can strengthen build provenance for agent code and supporting artifacts. Neither demonstrates that a model behaves safely on an open-ended input distribution.

Denial of service, cost, and unsafe persistence

Long contexts, recursive planning, tool retries, fan-out, large retrievals, and adversarially expensive inputs can exhaust token, API, compute, storage, or human-review budgets. Enforce per-request and cumulative ceilings outside the model. Use timeouts, circuit breakers, queue isolation, concurrency limits, and predictable termination. Measure downstream effects as well as token use.

An agent that creates scheduled jobs, sub-agents, accounts, API keys, messages, or files can outlive the initiating session. Inventory persistent artifacts, attach owner and expiry, constrain child authority, and provide a kill switch that revokes credentials and stops queued work. Test whether shutdown works during provider, identity, or network degradation.

Misleading output and human overreliance

An authorized action can still be based on fabricated, stale, or ambiguous information. Separate factual support from model confidence. Preserve source citations and retrieval timestamps where they inform a decision, and verify critical facts at the system of record. Design approval screens to show the proposed effect, target, evidence, uncertainty, and alternatives—not a persuasive narrative generated by the same model requesting approval.

Threat-to-control-to-evidence matrix

This matrix is a starting contract. Derive system-specific rows with Threat Modeling That Produces Testable Controls, and connect incident ownership to the NIST SP 800-61r3 implementation brief.

Security testing and safety evaluation are related but distinct

Security testing asks whether an attacker can violate confidentiality, integrity, availability, authorization, accountability, or containment objectives. Safety evaluation can include harmful content, bias, misinformation, misuse, or other adverse outcomes even when conventional authorization works. Some cases overlap, but one passing score cannot stand in for the other.

Build an evaluation portfolio:

• deterministic unit and integration tests for policy, identity binding, schemas, tenant isolation, limits, and evidence;
• adversarial prompt and content corpora for direct and indirect injection;
• tool-sequence and state-machine tests, including retries, partial failure, replay, and compensation;
• retrieval and memory poisoning tests;
• model and configuration regression evaluations across supported languages and formats;
• manual red-team exercises for chained, creative, and socially engineered paths;
• production canaries, anomaly detection, abuse monitoring, and incident drills;
• safety evaluations appropriate to intended use and affected populations.

Measure both attack success and control outcome. “The model refused” is weaker evidence than “the policy gateway denied every prohibited action and generated an alert,” because model refusal rates can change across prompts and versions. Record sampling strategy, environment, model and prompt versions, tool and policy versions, seeds or replay inputs where available, pass thresholds, uncertainty, and known blind spots.

The NIST Secure Software Development Practices for Generative AI and Dual-Use Foundation Models extends secure-development considerations for model producers. The CISA/NCSC Guidelines for Secure AI System Development frame secure design, development, deployment, and operation across the lifecycle. They support process questions; they do not certify a product or replace local tests.

Plausible, demonstrated, realized, and verified

Use evidence labels carefully:

• Plausible: a threat follows from architecture, attacker capability, research, or analogous behavior, but has not been reproduced in the target system.
• Demonstrated: the technique has been reproduced in a lab, evaluation, or relevant system under stated conditions.
• Realized: the behavior has been observed in real-world operations or an incident with adequate support.
• Verified locally: a named control claim has passed a defined test in the intended environment and version.

MITRE ATLAS is a living knowledge base of adversary tactics and techniques for AI-enabled systems. Its published maturity vocabulary includes Feasible, Demonstrated, and Realized. Map those labels accurately when using ATLAS; do not silently translate them into probability or local exploitability. “Verified locally” in this paper is an assurance status for a control test, not an ATLAS technique-maturity label. A locally verified control still has residual risk and can regress.

The Baitaphish ATT&CK technique index describes enterprise behaviors and is distinct from ATLAS. Use ATT&CK for relevant identity, execution, persistence, or exfiltration behavior around the application, and ATLAS for AI-specific adversary behavior. Neither framework proves attribution or coverage.

Evidence, telemetry, and privacy

For each effectful run, preserve enough structured context to answer:

• which authenticated principal and tenant initiated the task;
• what authority was delegated and when it expired;
• which model, system instruction, policy, tool, and adapter versions ran;
• which sources and memory records materially informed a decision;
• which normalized action was proposed, approved, denied, or executed;
• which destination confirmed the effect;
• what budget, retry, and termination state applied;
• which alerts, human decisions, and containment actions followed.

Do not treat hidden chain-of-thought as required audit evidence. Store explicit decision summaries, source references, policy inputs and outputs, tool calls, and results. Minimize raw prompts and retrieved data; redact secrets before durable logging; separate security access from product analytics; set retention and deletion rules; and test that investigators can join records without exposing every conversation.

Monitor the evidence pipeline itself: missing tenants, dropped events, schema failures, clock drift, provider outages, sampling, suppressed alerts, and stale policy or model identifiers. Quiet telemetry can mean a quiet system or a broken sensor.

Incident response and recovery

Predefine containment actions for an active agent:

• revoke session, workload, provider, and tool credentials;
• disable an agent, tool, connector, model route, memory source, or retrieval collection;
• stop active loops and scheduled or queued descendants;
• freeze relevant prompts, policies, artifacts, memory, and evidence versions;
• block destinations or action classes;
• notify affected data, service, and business owners;
• restore a known-good configuration and re-authorize deliberately.

Containment should function even when the agent, its prompt, or its primary orchestrator is compromised. Keep an external kill path and independent evidence. Test partial effects: an email may already be sent, a refund committed, a repository changed, or a secret copied. Recovery may require reconciliation and notification, not simply rolling back a model version.

After an incident, identify which assumption failed: model behavior, identity binding, tool authorization, data provenance, approval quality, delivery integrity, evidence coverage, or response authority. Convert the failure into a new control claim and regression test. Use the source directory, daily briefing, and intelligence index to find current NIST and other owner intelligence worth triage, while requiring primary evidence before changing a local risk decision.

Review checklist

• Is the agent's intended outcome, prohibited outcome, maximum autonomy, and risk owner explicit?
• Are users, tenants, delegated subjects, models, tools, data, memory, pipelines, and external effects in the system model?
• Does trusted application state bind actor and tenant independently of model output?
• Can every tool remain safe if the model proposes malicious arguments and sequences?
• Are high-impact approvals bound to the exact action and protected from replay or substitution?
• Are retrieval and memory permissions enforced before content enters model context?
• Are persistent artifacts, child agents, credentials, and queued work bounded and discoverable?
• Do tests cover prompt injection, output handling, tenant isolation, delegation, poisoning, cost, partial failure, and containment?
• Does evidence support reconstruction without unnecessary storage of secrets or personal data?
• Can operators disable authority and recover trustworthy configuration outside the agent loop?
• Are security tests, safety evaluations, and residual-risk decisions reported separately?

Review at least every three months and after a model or provider change, system-prompt or policy change, new tool or connector, autonomy increase, retrieval or memory redesign, fine-tune or dataset change, new tenant or data class, significant evaluation regression, incident, or material change in owner guidance.

Source scope and freshness

This paper is a threat-modeling method, not a safety evaluation, penetration-test report, compliance determination, or claim that a particular product is secure. Model behavior, provider controls, agent frameworks, attacks, and guidance change quickly. Validate current owner documentation, test the exact deployed versions and permissions, and keep residual risk visible.