AI agent security: limit what a mistake can do

Revised September 4, 2026.

An agent that only drafts text has a different risk profile from one that can send email, modify an order, or run code. Security starts with that difference, not with a stronger instruction telling the model to behave.

Design for a model that may misunderstand a request or be influenced by untrusted content. The system should still restrict what happens next.

Draw the trust boundaries

For each workflow, identify the user, the agent runtime, the tools, the connected services, and the data moving between them. Then ask:

• Whose identity is used when a tool executes?
• Which records may that identity read or change?
• Where can external text enter the context?
• Which actions affect another person or system?
• What evidence would let you reconstruct a failure?

Use the answers to define controls. A support assistant that reads incoming messages and can send outbound email has a different exposure from a read-only internal search tool.

Retrieved content is not authorization

Prompt injection can arrive through user input or external material such as documents, websites, and tool results. RAG and fine-tuning do not eliminate this risk. OWASP recommends layered mitigations, including limited privileges and approval for high-risk operations. See OWASP's prompt-injection guidance.

For an illustrative test, place a harmless instruction in a test document asking the assistant to perform an action outside its task. The desired result is not only a polite refusal. Verify that the connected tool would deny the action even if the model attempted it.

Instructions and content filters can help guide behavior. They are not substitutes for access checks in executable code.

Narrow the available actions

Give a reporting agent the ability to read the required dataset, not a general administrator credential. Separate read tools from write tools, and use distinct authorization for actions such as publishing, deleting, or sending.

Check the authenticated user's access to the specific target at execution time. Do not trust a tenant ID, role, or approval flag merely because it appears in model-generated arguments.

OWASP describes excessive functionality, permissions, and autonomy as drivers of excessive agency. Its guidance includes limiting available operations and enforcing authorization in downstream systems. See the excessive-agency guidance.

Make approval meaningful

For a consequential action, show the reviewer the actual target and proposed change. “Allow the agent to continue” is weaker than “Send this message to these recipients.”

Tie approval to that action. If the recipient, amount, payload, or scope changes, require a new decision. Do not let an earlier approval become an open-ended permission for the rest of the conversation.

Also check freshness. The state of an order may change between drafting a proposal and executing it. The service should reject an action whose assumptions no longer hold.

Protect execution and observation

Keep secrets out of prompts, returned tool payloads, and ordinary logs. Restrict filesystem access and outbound network access according to the task. If code execution is enabled, review the actual isolation configuration rather than assuming the word “sandbox” proves the boundary.

For remote MCP integrations, validate credentials for the intended service and follow its authorization requirements. The MCP security guidance warns against accepting and forwarding tokens without appropriate audience validation. Read the MCP security guidance.

Audit records should answer who requested an action, what was authorized, and what happened. Redact sensitive content, limit access to the audit trail, and define how long records are retained.

Test failure, not just the happy path

Before enabling an action, exercise at least these cases in a controlled environment:

1. A valid request targets another customer's record.
2. A retrieved document asks for an unauthorized action.
3. An approval is reused with a changed target.
4. A timeout occurs after the service has accepted the change.
5. A repeated request could duplicate an external effect.
6. A revoked permission is followed by another tool call.

Check the service state and audit evidence, not just the agent's explanation.

Finally, define how to stop a run, revoke access, identify affected records, and recover where possible. Expand autonomy based on demonstrated controls and the consequences of failure — not simply because the pilot ran quietly for a few weeks.

For the architectural choices around those controls, read our agent architecture guide.