Zero Trust for AI Agents — Anthropic's Enterprise Security Framework

Frontier AI models are compressing the timeline between vulnerability and exploit from months to hours. Defenders who adopt these tools find and fix bugs faster — but attackers who adopt them move faster too. This is not a future concern: models can already find serious vulnerabilities that traditional tooling and human reviewers have missed for years.

This acceleration matters twice for any organization deploying AI agents. The infrastructure your agents run on is exposed to AI-accelerated offense like the rest of your estate, and the agents themselves introduce autonomy — interpreting goals, selecting tools, and executing multi-step operations. Traditional access controls won’t prevent agents from misusing legitimate permissions, and monitoring needs to account for attacks designed to succeed through persistence rather than exploitation.

Anthropic recently published a practical framework for deploying autonomous AI agents in the enterprise. Here’s the full breakdown.

🎯 Core Thesis

“The organizations best positioned for this shift will be the ones whose fundamentals are strong enough that AI-assisted scanning finds fewer bugs in the first place, and whose agent deployments are architected for breach from day one.”

The framework adapts Zero Trust — trust nothing, verify everything, assume breach has already occurred — to the unique challenges of agentic systems: identities that are cryptographically rooted, permissions scoped per task, memory protected against poisoning, and defensive operations that run at the speed of autonomous attackers.

🔑 Core Zero Trust Principles for Agents

1. Never Trust, Always Verify

Every access request (from an agent or user) undergoes authentication and authorization regardless of its origin. No implicit trust based on network location, prior authentication, or service identity.

2. Assume Breach

Design systems assuming a compromise will occur. The goal is to limit the “blast radius” through segmentation, fine-grained controls, and compartmentalization.

3. Least Privilege / Least Agency

Grant the absolute minimum access required for a specific task. The framework introduces “Least Agency” — extending least privilege to agent tools by restricting not just what an agent can access, but how often and where it can use specific tools.

The Design Test

A key question for every control:

“Does this make the attack impossible, or just tedious?”

Controls that only add friction (e.g., SMS MFA) are easily bypassed by AI-accelerated attacks. Controls that remove capability (e.g., hardware-bound credentials) are the target state.

🔒 What Makes Agents Different

Property	Traditional Software	AI Agents
Autonomy	Follows deterministic logic	Interprets goals, makes decisions
Tool Access	Fixed API integrations	Dynamic tool selection (APIs, MCP)
Decision Making	Pre-programmed branches	Handles ambiguity, contextual judgment
Context	Stateless or session-scoped	Persistent memory across sessions
Coordination	Fixed service calls	Multi-agent pivoting, delegation

New Security Concepts

Blast Radius: The maximum potential damage scope if an agent is compromised. Drives isolation and segmentation decisions.
Least Agency: Extension of least privilege to agent tools — restricting not just what resources an agent can access, but which tools it can use, how often, and under what conditions.

⚠️ Threat Model

Prompt Injection & Instruction Manipulation

Direct Injection: Attackers craft inputs designed to override system instructions.
Indirect Injection: Malicious instructions embedded in external data sources — web pages, emails, documents — that an agent processes, causing it to execute commands without the user’s knowledge.
Fundamental Problem: LLMs cannot reliably distinguish context from instructions.

Tool & Resource Misuse

Tool Poisoning: Compromising tool interfaces (e.g., MCP server descriptors/metadata) to exfiltrate data or manipulate agent behavior.
Tool Chaining: Tricking an agent into combining legitimate tools in a harmful sequence (e.g., read sensitive data → encode it → exfiltrate via allowed API).
Resource Exhaustion / Loop Amplification: Attacks that drive up API costs or cause denial-of-service through infinite loops or excessive tool calls.

Identity & Privilege Abuse

Unscoped Privilege Inheritance: High-privilege “manager” agents passing too much context or authority to “worker” agents.
Memory-based Privilege Retention: Agents caching credentials or secrets in long-term memory across sessions, creating persistent exposure.

Supply Chain & Memory Risks

Model Supply Chain: Poisoned model weights, malicious fine-tuning data, or compromised base models.
Dependency Risks: Malicious packages in agent frameworks, API integrations, or the broader Python/JS ecosystem (e.g., PyTorch dependency confusion attacks).
Memory/Context Poisoning:
- RAG Poisoning: Injecting malicious data into vector databases that the agent retrieves as “knowledge.”
- Shared Context Poisoning: One tenant’s data influencing another’s in multi-tenant environments.
- Memory Drift: Gradual corruption of agent behavior through accumulated poisoned context.

🏗️ Three-Tier Maturity Framework

Organizations should start at Foundation and advance as deployments scale and risk profiles increase.

Identity & Authentication

Capability	Foundation	Enterprise	Advanced
Agent Identity	Unique cryptographic IDs	Certificate-based (X.509)	Hardware-bound (TPM/HSM) with attestation
Service Auth	Short-lived tokens via OAuth 2.0	Mutual TLS with certificate pinning	Hardware-bound credentials

Access Control & Privilege Management

Capability	Foundation	Enterprise	Advanced
Permission Model	RBAC with deny-by-default	ABAC with context awareness	Continuous authorization with real-time policy
Privilege Scoping	Static least-privilege roles	Dynamic adjustment per task	JIT/JEA with auto-expiration
Resource Boundaries	Identity-based / network segmentation	Sandboxed execution (containers, gVisor)	Hardware isolation (microVMs, AMD SEV/Intel TDX)

Observability & Auditing

Capability	Foundation	Enterprise	Advanced
Action Logging	Comprehensive logs with timestamps	Immutable audit trails	Real-time streaming to SIEM
Traceability	Request IDs for correlation	Distributed tracing across services	Full provenance chains
Baseline Establishment	Manual definition of normal behavior	Automated learning from traffic patterns	Continuous refinement
Anomaly Detection	Threshold-based alerts	Statistical anomaly detection	ML-based behavioral analysis
Automated Response	Alerting with triage context	Automatic containment (session termination)	Orchestrated playbooks (Agentic SOAR)

Input/Output Controls

Capability	Foundation	Enterprise	Advanced
Input Sanitization	Basic validation, length limits	Content filtering	Multi-layer / Constitutional classifiers
Output Filtering	Sensitive data pattern scanning	Semantic analysis	Human-in-the-loop for high-risk outputs

Integrity & Recovery

Capability	Foundation	Enterprise	Advanced
Configuration Integrity	Version control for configs	Signed configurations	Immutable infrastructure with attestation
Recovery Capabilities	Documented rollback procedures	Automated rollback with health checks	Self-healing / auto-remediation

AI Governance

Capability	Foundation	Enterprise	Advanced
Policy Framework	Documented use and incident policies	Formal governance framework with committees	Continuous automated policy enforcement

📋 Implementation Workflow — 8 Phases

The phases form a continuous cycle rather than a strictly linear path:

Requirements → Supply Chain → Boundaries → Injection Defense
      ↑                                           ↓
   Measure ← Memory ← Credentials ← Tools ←─────┘

Phase 1: Identify Requirements

Before writing agent code, align the organization on constraints and objectives.

Activities:

Regulatory mapping: Identify applicable regulations (HIPAA, SOX, FedRAMP, GDPR, PCI-DSS). These dictate baseline security controls, audit requirements, and data handling rules.
Operational constraints: Define the agent’s operating environment — which systems it touches, what data it processes, operating hours, acceptable failure modes.
Stakeholder alignment: Bring security, legal, engineering, and business teams together early. Misalignment at this stage is the leading cause of retrofitted (and weak) security.
Risk tolerance calibration: Define acceptable risk per agent. A payment-processing agent has a vastly different risk profile than an internal code-review assistant.

Deliverable: Formal Agent Risk Assessment document that serves as the reference for all subsequent phases.

Phase 2: Manage Supply Chain Risks

Agents are built on a stack of dependencies — models, frameworks, packages, datasets — each an attack surface.

Activities:

AI Bill of Materials (AI-BOM): Inventory every component: base model, fine-tuning data, agent framework (LangChain, CrewAI, etc.), MCP servers, API integrations, and transitive dependencies.
Automated dependency health checks: Use tools like OpenSSF Scorecard to continuously evaluate open-source dependency security (branch protection, signed releases, dependency update automation, security policies).
Audit redundancy: Combine automated scanning, manual code review, and third-party audits for critical components.
Reachability analysis: Use static analysis to determine which vulnerable code paths are actually executable — reduces noise and focuses remediation.
AI vendoring: For unmaintained or risky dependencies, fork and maintain an internal copy to control update cadence and patching.

Pro-tip: Treat model weights as a supply chain component. Verify checksums, use signed model repositories, and maintain provenance records for fine-tuning data.

Phase 3: Define Agent Boundaries

Clearly delineate what the agent is and isn’t allowed to do, and enforce it cryptographically.

Activities:

Cryptographic signing: Sign the agent’s base model, system prompt, tool definitions, and runtime environment. Verify signatures at startup to detect tampering.
Vendor assessments: Conduct formal security assessments of third-party model providers, MCP servers, and API services.
Unique identity assignment: Every agent instance gets a distinct, cryptographically rooted identity — not a shared service account.
Approved/prohibited action lists: Explicitly enumerate what the agent can do and what it must never do.
Blast radius definition: For each agent, define maximum damage potential if compromised.

Pro-tip: Treat the agent’s “constitution” (system prompt + rules) as a security-critical artifact. Version control it, sign it, and audit changes with the same rigor as code.

Phase 4: Defend Against Prompt Injection

The most distinctive threat vector for AI agents. Requires defense-in-depth.

Activities:

Input Isolation (Spotlighting): Borrowed from Microsoft research. User-provided input is clearly demarcated from system instructions using XML-like tags, base64 encoding, or structured message formats. The model treats spotlighted content as data to process, not instructions to follow.
Constitutional classifiers: Deploy a secondary classifier model that scans both inputs (for injection patterns) and outputs (for instruction leakage or policy violations).
Instruction hierarchy enforcement: Design prompt structure so system-level instructions outrank user-level instructions.
Output validation: Filter outputs — an agent summarizing a poisoned webpage may produce a summary containing malicious instructions for a downstream system.

Pro-tip: No single technique is sufficient. Prompt injection is fundamentally hard because LLMs can’t reliably distinguish context from instructions. Layer multiple defenses and assume some will fail.

Phase 5: Secure Tool Access

Tools (APIs, MCP servers, function calls) are how agents interact with the real world — and where the most damage can be done.

Activities:

Tool allow-listing: Maintain an explicit list of permitted tools per agent. Anything not on the list is inaccessible (deny-by-default).
Capability restrictions (RBAC on tools): Even within an allowed tool, restrict operations. Example: agent can read from a database but not write.
Parameter validation (agent side): Before calling a tool, validate parameters for type, range, and format. Block obviously malicious inputs.
Parameter validation (tool side): Tools independently validate all incoming parameters. Never trust the agent to have done this correctly.
Sandboxed execution: Run tool calls in sandboxed environments. Web-fetching tools restricted to approved domains. Code-execution tools run in containers with no network access.

Pro-tip: Log every tool call with full parameter details and responses. Tool call logs are primary forensic data during incident investigation.

Phase 6: Protect Agent Credentials

Credential management for agents is fundamentally harder than for humans — no MFA prompts, no confirmation dialogs.

Activities:

Short-lived credentials: Use identity-provider-issued tokens with short TTLs (minutes to hours). Use OAuth 2.0 token exchange or SPIFFE/SPIRE for workload identity.
Hardware-bound credentials: For high-security environments, bind credentials to hardware (TPM, HSM, secure enclaves). Credentials that can’t be extracted can’t be exfiltrated by prompt injection.
Credential isolation (per-agent): Never share credentials across agents. Each agent gets its own credential with independent scope and revocation.
Just-in-Time (JIT) access: Provision credentials only when needed. Agent requests a token, uses it, token expires. Eliminates standing privileges.
Explicit trust boundaries (multi-agent): Worker agents should never inherit a manager’s full privilege set.
Attribute-Based Access Control (ABAC): Make access decisions based on attributes: agent identity, current task, data sensitivity, time of day, network location.

Pro-tip: Implement automated credential rotation with zero downtime using credential providers (HashiCorp Vault, cloud-native secrets managers).

Phase 7: Safeguard Agent Memory

Agent memory is both a productivity feature and a massive attack surface. Poisoned memory = poisoned decisions.

Activities:

Memory isolation (session boundaries): Separate memory by session, task, and tenant. Task A’s context must never bleed into Task B.
Context integrity validation: Hash the agent’s context at key points (session start, before tool calls, after memory writes). Unexpected hash changes may indicate tampering.
Attribution tracking: Every piece of information in agent memory should have provenance — source, timestamp, and origin.
Context retention policies (TTL): Automatically prune old context to limit the window for memory poisoning.
Versioned stores/checkpoints: Maintain versioned snapshots of agent memory. If poisoning is detected, roll back to a known-good state.
RAG-specific protections: Treat vector databases as security-critical. Validate documents before ingestion, monitor for anomalous embeddings.

Pro-tip: Memory poisoning is an insider threat vector. An attacker who influences what an agent reads (e.g., posting to a monitored Slack channel) can shape behavior over time.

Phase 8: Measure What Matters

Security that can’t be measured can’t be improved.

Metric	Description	Goal
Dwell Time	Time a threat exists before awareness	Minutes, not days
Coverage	Percentage of agent estate monitored	100% tool call logging, credential tracking, session recording
Explainability	Ability to understand why an alert fired	Actionable alerts, not black-box noise
Detection Speed	Time from first malicious action to first alert	Test regularly with red team exercises
Behavior Conformance	Deviation from defined constitution	Continuous comparison of actual vs. expected behavior

Pro-tip: Real attacks rarely come one at a time. Practice prioritizing under pressure with multi-incident simulations.

🛡️ Defensive Operations — Agentic SOAR

Traditional SOAR (Security Orchestration, Automation, and Response) is too slow for AI-accelerated threats. The next generation is Agentic SOAR.

Core Principles

Move humans from bookkeeping to decisions: Automate evidence collection, log analysis, and initial triage. Keep humans on containment and disclosure decisions.
Adaptive response: Agentic SOAR uses AI to respond to novel situations in seconds (e.g., automated quarantine, credential revocation, session termination) without waiting for human approval.
Pre-authorized containment: Establish emergency change procedures that pre-authorize specific containment actions so they can execute at machine speed.

Operational Practices

Map detection coverage to MITRE ATT&CK framework
Run Atomic Red Team tests against agent infrastructure
Simulate multiple simultaneous incidents in tabletop exercises
Trust through verification: Apply Zero Trust to the defensive agents themselves — verify their integrity, limit their blast radius, and maintain clear escalation paths to human operators

🏛️ AI Governance Policies

Technical controls must be backed by formal governance. The framework defines three maturity levels:

Foundation

Documented acceptable use policies for AI agents
Incident response procedures specific to agentic systems
Basic cross-functional awareness (security + engineering alignment)

Enterprise

Formal governance framework with defined roles and responsibilities
Cross-functional governance committee (security, legal, engineering, business)
Regular risk assessments and compliance audits
Integration of policy checks into CI/CD deployment pipelines

Advanced

Continuous automated policy enforcement at runtime
Real-time compliance monitoring with automated remediation
Automated evidence collection for audit and regulatory purposes
Regular red team exercises with published findings and remediation tracking

🎯 Key Takeaways

Architect for breach — Perimeter security won’t save you. Assume the agent will be compromised and design containment from day one.
Automate the bookkeeping — Use AI to triage alerts, collect evidence, and correlate events. Keep humans in charge of high-impact containment and disclosure decisions.
Prioritize impossibility over tedium — Choose controls that remove a capability rather than those that merely add friction. Hardware-bound credentials beat SMS MFA. Sandboxing beats rate limiting.
Continuous evolution — Security is not a finish line. As agent capabilities grow, your defenses (and your maturity tier) must advance accordingly.
Start at Foundation, aim for Advanced — Not every agent needs hardware attestation on day one. But every agent needs cryptographic identities, short-lived credentials, and comprehensive logging from the start.
Measure relentlessly — Dwell time, coverage, explainability, detection speed. If you can’t measure it, you can’t improve it.
Apply Zero Trust to your defenders — Your security agents and SOAR playbooks need the same Zero Trust treatment: integrity verification, blast radius limits, and human escalation paths.

Final Thoughts

This framework is the most comprehensive public guidance on AI agent security to date. What sets it apart from generic Zero Trust advice is its honest acknowledgment that agents are fundamentally different from traditional software — they interpret, they persist, they chain actions, and they can be manipulated through their inputs in ways that have no parallel in conventional application security.

The “Impossible vs. Tedious” design test alone is worth the price of admission. It forces architects to think about what an AI-accelerated attacker can actually bypass versus what truly stops them.

The biggest gap? The framework assumes you have a security team capable of implementing it. For smaller teams or individual developers, the jump from Foundation to Enterprise is still significant. But even implementing just Phases 1-3 from the Foundation tier would put you ahead of most agent deployments today.

Related Reading: