Secure AI Agent Architecture: Permissions, Tools, and Bounded Autonomy

AI agents change the security problem from generating an answer to authorizing a sequence of actions. An assistant that can read a mailbox, query a customer system, execute code, or submit a payment can cause real effects even when the model itself has no direct network access. The central design rule is therefore simple: let the model propose; let trusted software decide. A natural-language instruction, model confidence score, or successful prompt-injection filter is not an authorization credential.

This distinction matters because agent failures often cross boundaries. A malicious instruction embedded in a webpage can influence a model, which may then call a tool with valid credentials. The tool call can be syntactically correct and still be unauthorized in context. OpenAI describes prompt injection as untrusted content attempting to override instructions, with possible outcomes including private-data exfiltration and unintended tool actions; it also cautions that mitigations do not make agents mistake-proof. 1 MITRE ATLAS catalogues AI-specific tactics that include prompt injection, agent tool invocation, context poisoning, and retrieval-content crafting. 2 These are not reasons to prohibit all agent use. They are reasons to design authority boundaries outside the model.

A secure architecture combines an identified principal, narrowly scoped capabilities, policy enforcement at each resource boundary, constrained execution, protected context and memory, meaningful human decisions for high-impact actions, and logs that support investigation. There is no single mandatory blueprint for every deployment. The appropriate controls depend on data sensitivity, potential impact, autonomy, and the ability to reverse actions. The following principles help turn that risk assessment into an implementable system.

Start with a bounded task and threat model

Before selecting a framework or model, describe what the agent is meant to accomplish and what it must never do. “Help with customer support” is too broad to authorize. “Summarize a customer’s last three support interactions and draft a reply for an employee” is testable: it defines a purpose, a data scope, an output, and a human checkpoint. Explicitly state whether the agent may only read, may create drafts, or may commit changes. Map the user, agent runtime, model provider, tools, data stores, identity provider, and operators as separate components.

Threat modeling should follow data and authority, not just prompts. Ask how an attacker can supply inputs, influence retrieved content, register or alter a tool, obtain credentials, cross tenant boundaries, trigger repeated actions, or suppress evidence. Include accidental paths: ambiguous requests, stale state, model errors, retries, and operator misconfiguration. NIST’s AI Risk Management Framework is voluntary guidance for incorporating trustworthiness considerations into AI design, development, use, and evaluation; it offers a useful governance structure rather than a prescriptive agent control list. 3 Google’s Secure AI Framework likewise frames security as a lifecycle spanning foundations, detection and response, consistent controls, adaptation, and business context. 4

A useful starting inventory lists assets, principals, trust boundaries, and worst credible outcomes. Assets may include customer records, source code, credentials, financial transactions, and the integrity of an external system. Principals include human users, agent instances, tool services, administrators, and background jobs. Trust boundaries include model calls, retrieval indexes, plugin servers, browser sessions, and any transition from proposal to effect. Analyze both confidentiality and integrity: restricting what can be read is not enough if the agent can send mail or mutate records.

Give every agent a verifiable identity

An agent should not inherit a broad human credential just because it acts on that person’s behalf. Establish an identity for the workload and separately represent the initiating user, organization, task, and agent instance. When the action is user-delegated, authorization should account for both the user’s rights and the agent’s registered capability. A useful conceptual check is the intersection of user authority, task scope, and agent policy; none of these should expand the others.

NIST’s zero-trust architecture rejects implicit trust based only on network location or asset ownership and treats authentication and authorization as explicit steps before resource access. 5 Applying that idea to agents means the tool service should authenticate each call and evaluate the requested operation, resource, and context. A private network, approved model, or trusted runtime is not a substitute for per-resource authorization.

Use workload identities and short-lived credentials where the platform supports them. Bind credentials to a specific agent service or task and make them audience-limited so they cannot be replayed against unrelated services. Avoid embedding long-lived API keys in prompts, tool descriptions, vector stores, logs, or model context. Keep secret retrieval and signing outside the model process. If a tool needs a secret, the tool backend should use its own managed identity or receive a narrowly scoped token through a trusted broker. Rotate and revoke credentials through ordinary identity-management processes.

For multi-tenant systems, carry a tenant and user context from the authenticated entry point to every tool and retrieval operation. Do not trust a tenant identifier generated by the model. Validate it against the authenticated session, and apply row-, object-, or document-level authorization in the data service. A seemingly harmless search tool can leak across tenants if the index query omits a mandatory filter or a cache key leaves out tenant identity. Enforce access at the source of truth rather than relying on the model to remember a boundary.

Delegation needs care. A parent agent that launches a subagent must not casually pass its entire credential set. Give the child an explicit task, limited data, and a subset of capabilities, and prevent it from minting new identities or escalating its own scope. Treat agent-to-agent messages as untrusted input unless authenticated and validated. MITRE ATLAS includes direct agent communication and shared artifacts as relevant attack surfaces, reinforcing the need to secure coordination channels too. 2

Convert broad tools into scoped capabilities

A tool is an authority boundary, not just a convenience function. Avoid giving a general-purpose agent an unrestricted shell, arbitrary HTTP client, or database administrator interface when a narrow operation will do. Expose business operations such as lookup_order(order_id) or create_draft_reply(case_id, text) instead of raw SQL or generic network access. The narrower interface reduces what a mistaken or manipulated model can request and makes authorization decisions easier to inspect.

Capabilities should identify action, resource scope, and limits. A “read tickets” capability might permit only tickets associated with the active tenant and current case. A “send message” capability might permit one recipient, a specified subject and body, and a rate limit. A “refund” action can have a separate permission and require a human authorization event. Make read and write privileges distinct. Where possible, issue a task-scoped grant that expires with the task rather than maintaining a permanent tool connection.

The model’s tool schema is not enforcement. It guides generation but can be misused, bypassed through another route, or become outdated. Validate every argument server-side: types, ranges, identifiers, tenant association, state transitions, and business rules. Allowlist operations, destinations, and content types. Reject unknown fields instead of silently accepting them. Use prepared queries and safe APIs rather than concatenating model-produced strings into shell commands, SQL, or URLs. A tool gateway can normalize requests, attach identity context, consult policy, and emit consistent decision records before forwarding an approved call.

Authorization should be checked at the moment of effect. A prior read or approval does not prove the object is still in the same state. For a high-impact mutation, verify the object version or relevant preconditions, then apply the change atomically where possible. For external actions, use idempotency keys to avoid duplicate effects after retries and explicit timeouts to avoid runaway waiting. If a backend cannot support transactions or idempotency, record that limitation and reduce the autonomy available for that operation.

Least privilege has a practical cost: narrow interfaces take engineering effort and can constrain valid tasks. Broad tools are faster to prototype and may be necessary for exploratory work, but they increase blast radius and make policy harder to reason about. A sensible compromise is not a giant universal tool behind a prompt; it is a small set of composable operations, with riskier operations isolated behind separate authorization and review.

Make policy enforcement independent of model behavior

An agent policy should answer, in ordinary application code, whether a requested action is allowed for this principal, on this resource, at this time, under these conditions. It should be deterministic for a given policy version and input. The model may help classify intent or extract parameters, but its classification cannot be the final control for a consequential action. A policy decision point can evaluate identity, role, tenant, purpose, resource attributes, task state, risk tier, and current approval. A separate enforcement point should deny or constrain execution when the decision is absent, stale, or invalid.

Default to deny. A request that cannot be parsed, authenticated, associated with a tenant, or evaluated should fail closed for write operations. Define explicit exceptions for safe fallbacks, such as allowing a read-only answer from public material when a private-data service is unavailable. Avoid fallback behavior that silently switches to a more privileged credential or a less restrictive endpoint.

Policies can be expressed as code, a dedicated policy language, or a centralized authorization service. Central policy helps prevent inconsistent rules across multiple tools, but it can add latency, operational dependency, and a new high-value service to secure. Local checks are resilient and close to the resource, but can drift. In either approach, keep business constraints near the system that owns the resource and have a shared policy layer enforce common cross-service rules. Version and review policy changes, test them before deployment, and retain the decision and policy version in the audit trail.

Treat natural-language instructions as data with different trust levels. User prompts, retrieved documents, web pages, emails, code comments, and tool outputs can contain attacker-controlled text. Clearly label provenance and keep untrusted material out of privileged instruction channels. OpenAI recommends not placing untrusted variables in developer messages and using structured outputs to constrain information passed between workflow steps. It also notes that structured outputs and isolation reduce, but do not eliminate, risk. 1 These practices are valuable, but they complement rather than replace authorization checks.

Use schemas and enums for intermediate data wherever they fit. If an extraction step produces an account identifier, validate the identifier against the authenticated user’s allowed accounts before using it. Do not take arbitrary prose from a retrieved page and feed it directly into a privileged tool. Separate “what the source says” from “what the agent is instructed to do”; provenance labels help humans and downstream rules reason about that difference, though labels themselves do not make model behavior reliable.

Design tool permissions as a risk ladder

A useful permission model distinguishes capabilities by potential impact and reversibility:

• Observe: search or read data that the user is already authorized to access.
• Prepare: compose a draft, proposed patch, or transaction without committing it.
• Reversible change: update a state that can be restored with a known rollback path.
• External or consequential action: send a message, publish content, modify access, transfer funds, or affect a person’s eligibility.
• Irreversible or high-impact action: delete critical data, execute privileged code, or make a decision with serious consequences.

This ladder is a design aid, not a universal standard. Risk depends on context: a read can expose sensitive records, and a “reversible” change may trigger downstream notifications. Classify actual outcomes, not tool names. For each capability define the allowed principal, operation, resource scope, rate or quantity limits, required preconditions, approval needs, and recovery plan. Separate approval to read sensitive information from approval to change it.

Do not let the model negotiate its own access. If it encounters a task requiring a missing permission, the correct behavior is to explain the limitation or request a human-controlled authorization flow. A user’s conversational statement such as “you have permission” should not create access unless the application independently authenticates that user and maps the request to a grant. Likewise, a model-generated claim that an action is urgent must not bypass policy.

Make human approval meaningful

Human approval is useful only when the reviewer can understand the actual action and has a real opportunity to change or reject it. Show the target system and resource, the exact operation, the material input or payload, the agent identity, the reason for the action, and relevant consequences. Do not ask someone to approve vague labels such as “continue” or “complete task.” For a message, display recipient and final content. For a financial or administrative operation, show amount or affected account and the state change. Keep approval bound to that exact proposal.

Approval must be an independent event, not a prompt token the model can fabricate. The authorization service should record which authenticated person approved, what was displayed, which operation and payload were approved, when approval expires, and whether the eventual execution matches. If the payload or destination changes, require new approval. Bind approval to a cryptographic or otherwise tamper-resistant proposal identifier where practical, and use re-authentication or a second person for operations whose impact warrants it.

Avoid approval fatigue. Requiring a click for every low-risk action encourages reflexive approval and slows useful work. Conversely, blanket approval at session start is weak when later actions differ materially. Use risk-tiered approvals: allow bounded read-only tasks within existing user authority; permit drafts without execution; require specific approval before high-impact writes; and deny actions that cannot be safely reviewed. Batch only actions that are similarly scoped and visible together. A human should not approve an open-ended plan whose individual effects are unknown.

Approval is not a substitute for access control. The tool service must still verify policy after approval, because the requester may lack authority, the target may have changed, or the grant may have expired. Nor does a human reliably detect every subtle injection. Human review is a safety layer for consequential decisions, not a universal model-alignment mechanism. OpenAI’s agent safety guidance recommends approvals for tool operations in its workflow context; an architecture should adapt that advice to the risk and usability of its own system rather than treating a vendor setting as complete governance. 1

Isolate execution and constrain the environment

A model that can generate code or operate a browser should not share an unrestricted host with production credentials. Run risky tools in isolated processes, containers, virtual machines, or managed sandboxes according to threat and platform capabilities. Apply operating-system controls, read-only filesystems where possible, CPU and memory quotas, process limits, syscall restrictions, and disposable workspaces. Disable ambient credentials and avoid mounting sensitive host paths. Isolation is defense in depth: it limits what follows from a compromised or misbehaving component, but it cannot prevent every logical misuse of a legitimate capability.

Network access deserves explicit policy. An execution sandbox that can reach internal metadata endpoints, private services, or unrestricted internet hosts may still exfiltrate data or attack internal systems. Deny outbound connectivity by default where feasible, then allowlist required destinations through a proxy that validates hostname, port, DNS resolution, redirects, and response size. Protect against server-side request forgery and DNS rebinding by checking resolved addresses and blocking private or link-local ranges where appropriate. Browser tools should isolate cookies and sessions, constrain downloads, and distinguish trusted application origins from arbitrary pages.

Code execution should use ephemeral credentials and a scratch directory. Do not execute model-produced commands by interpolating untrusted strings into a shell. Prefer typed operations and fixed executable arguments. If arbitrary code is essential, separate that environment from production, limit its network egress, scan artifacts, and treat all outputs as untrusted until validated. Resource limits and execution timeouts also matter: an infinite loop or huge output can create denial of service without escaping the sandbox.

Tool servers and integrations are part of the trusted computing base. Authenticate their endpoints, verify configuration and software provenance, pin or review dependencies where appropriate, and monitor tool registration changes. A capability may become dangerous after a server update or a URL target changes. Maintain an inventory of available tools, their owners, versions, scopes, and retirement status. Do not assume a protocol standard automatically makes every server trustworthy; validate both the server identity and the semantics of each operation.

The trade-off is friction. Strong isolation can complicate debugging, require infrastructure, and restrict workflows that depend on local files or broad network access. Start from the smallest environment that supports the approved use case and introduce exceptions deliberately. A development sandbox is not an adequate substitute for controls on production credentials or data.

Protect retrieval and memory

Retrieval and memory create an attractive route for indirect influence and disclosure. An attacker can seed a document that later appears in search results, or an agent can store a false or sensitive statement that affects future tasks. MITRE ATLAS includes retrieval content crafting and context poisoning among its techniques. 2 Treat retrieved text as untrusted evidence, not executable policy. Store provenance, source identity, timestamps, ownership, and access labels alongside chunks. Preserve the source document’s authorization rules when indexing and re-check permissions at retrieval time.

Partition indexes by tenant or enforce robust per-document filters in the retrieval service. Do not rely on prompt wording such as “only use this user’s data.” Search results can be cached, so include identity and policy context in cache boundaries or avoid caching permission-sensitive results. Keep index ingestion pipelines authenticated and monitored; validate file types, parsing behavior, and metadata. A deleted source should not remain retrievable indefinitely without a deliberate retention policy.

Memory requires a defined purpose and lifecycle. Distinguish transient task state, user-approved preferences, organizational knowledge, and sensitive operational data. Persist only what the application needs, label who can read and update it, and provide deletion, correction, expiration, and review mechanisms. A memory write should not automatically become trusted merely because the agent produced it. Consider storing claims with their source and confidence as metadata, and require validation or human confirmation before promoting facts that influence later actions.

Do not put secrets, access tokens, or unnecessary personal information into long-term conversational memory. Apply encryption and access controls to memory stores as you would to other sensitive data. Test retrieval for cross-user leakage and stale permissions, and review retention and deletion behavior. If memory is absent or uncertain, the system should ask or retrieve from an authoritative source rather than inventing continuity. The appropriate balance varies: persistent memory improves convenience and personalization but increases the impact of poisoning, unauthorized access, and retention mistakes.

Make audit records useful without leaking more data

An operationally useful audit event should link a user request to the agent instance, model or workflow version, policy version, tool call, approval, and final outcome. Record identity, timestamp, target resource identifier, decision result, relevant scope, and correlation ID. Retain enough input and output evidence to reconstruct the decision, but do not indiscriminately log secrets or entire private documents. Redact or tokenize sensitive values, restrict log access, and set retention periods appropriate to operational and organizational needs.

Separate security audit logs from ordinary application diagnostics. Protect audit storage from alteration by the agent and by the service being monitored; use centralized collection, access controls, and integrity safeguards. Monitor for unusual patterns such as denied calls, repeated attempts at disallowed actions, unexpected destinations, unusually large retrievals, rapid tool loops, privilege changes, and approval anomalies. These signals are not proof of compromise, but they can trigger review.

A transcript alone is not an audit trail. It may omit backend authorization decisions, hide which credential was used, or be mutable. Conversely, logging full chain-of-thought is neither necessary nor an appropriate default. Record externally observable events and the explanation needed to understand policy decisions, while following the system’s privacy and data-minimization requirements. NIST incident-response guidance emphasizes incorporating response across cybersecurity risk management so organizations can prepare, detect, respond, and recover more effectively. 6 Agent-specific telemetry should feed into existing security operations where possible rather than creating an isolated monitoring silo.

Test the whole action path

Unit-test tools and policy as ordinary security-sensitive software. Test both allowed and denied cases: wrong tenant, unauthorized object, expired grant, malformed argument, changed state, missing approval, duplicate retry, and backend timeout. Test that denial remains denial when the model asks repeatedly, changes its wording, or proposes an alternate tool. Verify that a tool cannot be reached around the gateway through a second endpoint or a debug interface.

Add adversarial tests for direct and indirect prompt injection. Place hostile instructions in user text, retrieved documents, email, code comments, tool results, and images or other supported modalities. Include attempts to request hidden data, invoke unrelated tools, alter memory, redirect network calls, or persuade the agent to bypass review. Evaluate the system’s actual effects, not just whether a classifier labels the prompt as malicious. A robust result is that sensitive actions remain blocked by authorization even when the model follows the injected instruction.

Test permissions as combinations: authenticated identity, resource ownership, user role, task scope, and approval state. Include concurrency and stale-state conditions. A test harness should be able to replay a task against policy versions and capture tool calls, policy decisions, and outcomes. Keep regression cases for real incidents and near misses. Use red-team exercises to discover emergent paths, but do not treat a clean test run as proof that prompt injection is solved.

Run risk-based evaluations before release and after changes to the model, prompt, tool schema, retrieval pipeline, identity mapping, or policy. Changes that seem cosmetic can alter tool selection or data flow. NIST’s AI RMF emphasizes evaluation and risk management across design and use; OWASP’s AI Agent Security Cheat Sheet similarly recommends least privilege, memory controls, human oversight, monitoring, and adversarial validation as complementary practices. 3 7 Define release gates for critical invariants, such as no cross-tenant reads and no high-impact operation without a matching approval. Measure false denials and operational friction as well as bypasses: unusable controls tend to be routed around.

Prepare for failures and incidents

Assume a control can fail. Provide operators with a kill switch that stops new tool calls, disables a compromised integration, revokes agent credentials, and pauses scheduled work. Ensure this mechanism is outside the model’s control and accessible during an outage. For partially completed workflows, define how to identify committed effects, prevent duplicate retries, notify affected parties, and resume safely. A rollback is not always possible; sometimes the right response is containment and correction rather than reversal.

Incident procedures should cover suspected prompt-injection compromise, credential exposure, data exfiltration, unauthorized changes, malicious tool updates, and memory poisoning. Preserve relevant event records and configuration versions. Revoke or rotate credentials, restrict the affected capability, quarantine suspicious artifacts, and determine which users, tenants, and resources were reachable. Assess whether the agent acted through valid permissions that were too broad, whether policy was bypassed, or whether identity attribution failed. Include the model or workflow provider only where its components or logs are relevant; do not assume every incident is a model defect.

Post-incident review should change controls and tests, not merely add stronger wording to a prompt. A prompt fix may reduce one known failure mode but does not prevent a different model output from reaching a broad tool. Update policy, tool scope, monitoring, user guidance, and regression tests based on the cause. Google’s SAIF explicitly includes extending detection and response to AI threats and adapting controls through feedback loops. 4 NIST SP 800-61 Revision 3 places incident response within broader risk management and supports preparation and learning as ongoing work. 6

A practical decision framework

For each proposed agent capability, answer five questions before enabling it:

1. What is the consequence? Identify the data exposed and the real-world effect if the action is wrong, maliciously induced, or repeated. Include downstream and third-party impacts.
2. Who is the principal? Can the system reliably establish the user, agent workload, tenant, and task? If not, do not expose private or mutating operations.
3. Can the task use less authority? Prefer read-only access, task-specific scopes, narrow operations, quantity limits, and short-lived grants. Remove tools not needed for this task.
4. Where is enforcement? Verify that the resource or a trusted gateway independently checks identity, scope, state, and policy at execution time. If the model itself is the only barrier, the capability is not bounded.
5. How will it be reviewed and recovered? Decide whether a human must approve the exact action, what will be logged, how to stop execution, and how to contain or repair an error.

The answers determine an autonomy tier. Low-impact, reversible tasks over non-sensitive data may run automatically within narrow, auditable boundaries. Sensitive reads and moderate writes may require stronger identity checks, explicit user confirmation, and tight quotas. High-impact or irreversible actions should generally remain human-authorized, with the agent limited to preparing a proposal unless a carefully justified control system can guarantee bounded execution. This is a risk-based recommendation, not a universal legal requirement.

Finally, compare the security value with complexity. A fixed workflow may be safer and easier to test than an autonomous planner when the steps are known. Anthropic’s engineering guidance distinguishes predefined workflows from agents that dynamically direct their own tool use, and recommends using the simplest approach that meets the need because added agent complexity brings trade-offs. 8 If a standard deterministic process works, do not add open-ended autonomy merely because the framework supports it. If an agent is useful, constrain its loop, cap tool calls and elapsed time, require explicit stopping conditions, and make every external effect pass through the same authorization path.

Secure agent architecture is not achieved by finding a prompt that never fails. It is achieved when a wrong or manipulated proposal cannot cross a boundary it was never authorized to cross. Identity establishes who is acting; scoped capabilities limit what is possible; policy and resource checks decide what is permitted; isolation and approval reduce exposure; protected memory, audit, and incident response support learning and recovery. Keep these controls testable, independent of model assurances, and proportional to the consequences of each action.

References and further reading

[1] Safety in building agents, OpenAI API documentation [2] MITRE ATLAS knowledge base [3] NIST AI Risk Management Framework [4] Google Secure AI Framework [5] NIST SP 800-207 Zero Trust Architecture [6] NIST SP 800-61 Revision 3 Incident Response Recommendations [7] OWASP AI Agent Security Cheat Sheet [8] Building effective agents, Anthropic Engineering