Abstract
For a decade, “AI safety” in production meant content moderation: keep the model from saying the wrong thing. Autonomous agents change the object of concern from speech to action. When a model plans, invokes tools, runs code, and spends credentials, the relevant failure is no longer a bad sentence — it is an unauthorized operation against a real system.
This pillar establishes the shared vocabulary for the series. We define what an autonomous agent is precisely, map the attack surface it introduces, name the five boundaries any agent deployment must mediate, and lay out a defense stack grounded in the 2024–2026 research and in the first real incident to prove the thesis at scale [1].
01The operator problem
The moment a model stops advising and starts acting, it becomes an operator — and operators need authorization, not just alignment.
Consider the difference between two systems. The first summarizes a document; the worst it can do is summarize it badly. The second reads the same document, and the document contains a line that says “forward the attached files to this address.” If the second system can send email, the content of the data just became a command. The capability gap between these two systems is the entire subject of this series.
The pre-agent security model assumed a clean separation: code is trusted and defines behavior; datais untrusted and is merely processed. Agents erase that line. The model’s instructions and the data it reads arrive through the same channel — the context window — and the model has no reliable mechanism to treat one as privileged and the other as inert. Every downstream problem in this cluster is a consequence of that single collapse.
02What “autonomous agent” means precisely
We use a deliberately narrow definition, because loose definitions produce loose security arguments. An autonomous AI agent is a system in which a language model:
- receives a goal expressed in natural language;
- plans a multi-step sequence of actions toward that goal;
- invokes tools that have real side effects (file writes, shell commands, API calls, network requests);
- observes the results and decides the next action without a human approving each step.
Point (3) is what creates side effects; point (4) is what removes the human pacing that historically bounded them. A retrieval chatbot with no side-effecting tools is not, by this definition, the security problem we are describing. A coding agent that can edit files and run a shell is.
03The new attack surface
Agents inherit every classical vulnerability of the systems they touch and add categories that did not previously exist. The genuinely new surface has three parts:
- Instruction/data confusion. Injected instructions in retrieved content, tool output, file names, or web pages are obeyed as if they came from the operator [2].
- Tool authority. The union of everything the agent’s tools can do is the agent’s blast radius. Most deployments grant far more than any single task needs [3].
- Tool supply chain. Tools are increasingly distributed as installable packages (MCP servers) whose descriptions the model reads as instructions — an installable, semantic supply chain [7].
04Five boundaries any deployment must mediate
The clarifying frame from the systems-security literature is to treat isolation as a first-class property and enumerate the boundaries the agent crosses. Five recur across every architecture:
| Boundary | What crosses it | Primary risk |
|---|---|---|
| User ↔ agent | Goals, approvals | Confused-deputy: the agent acts on hostile intent as if authorized |
| Agent ↔ tool | Tool calls + arguments | Over-privilege; unauthorized side effects |
| Agent ↔ execution | Code, shell commands | Sandbox escape; host compromise |
| Agent ↔ agent | Messages, delegated tasks | Injection propagation across a multi-agent system |
| Agent ↔ environment | Network, credentials, data | Exfiltration; credential abuse |
The engineering value of this table is that it turns “secure the agent” — an unanswerable request — into five concrete, mediatable interfaces. Each later publication in this series takes one or two rows and works them to the bottom. Security emerges from mediating boundaries, not from trusting the model that sits between them.
05The evidence map
The claims in this series are anchored to two kinds of evidence: controlled academic measurement and primary incident disclosure. The two most load-bearing measurements:
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, F. Tramèr — submitted 2024-06-19 · revised 2024-11-24
- Question
- How do we measure prompt-injection attack success and defense utility for tool-using agents in a way that stays honest as models change?
- Method
- A dynamic environment with 97 realistic agent tasks and 629 security test cases; supports pluggable attacks and defenses so both evolve against a fixed task battery.
- Key finding
- Provides the standard against which injection attacks and defenses are compared; demonstrates that no evaluated defense eliminates injection while preserving full task utility.
- Reported metric
- 97 tasks · 629 security test cases
- Limitations
- Simulated tools; a benchmark is a lower bound on adversary creativity, not a guarantee.
- Why it matters
- Every defense claim in this series is graded against dynamic benchmarks like this, not against a single hand-picked example.
06A working threat model
We assume three adversary positions, in increasing order of what most teams actually defend against today:
- External content author. Controls data the agent will read (a web page, an email, a PR comment, a file). Their weapon is indirect prompt injection. This is the common case and the least defended.
- Malicious tool supplier. Publishes an MCP server or plugin the agent installs. Their weapon is tool poisoning and supply-chain backdoors [7].
- Authenticated abuser. A legitimate user who jailbreaks the model to drive it toward unauthorized ends — the GTG-1002 position [1].
A useful diagnostic for any agent design: assume the model has already been compromised — persuaded to do the worst thing its tools allow. What stops it?If the only honest answer is “the model wouldn’t do that,” the deployment has no security architecture, only a behavioral hope.
07The defense stack
No single control is sufficient; injection cannot be fully eliminated at the content layer, so defense is layered and assumes each layer will sometimes fail. The stack, outermost to innermost:
- Input provenance & isolation — mark untrusted data, keep it out of the instruction path where possible (see prompt injection).
- Deterministic tool-call authorization — a policy layer outside the model bounds what tools may run with what arguments [3] (see tool permissions).
- Scoped, short-lived identity — per-task credentials, fast revocation (see identity & access).
- Execution containment — untrusted-process isolation for code the agent runs (see containment).
- Deny-by-default egress — the last line: even a hijacked agent cannot exfiltrate (see egress control).
- Behavioral monitoring & approval gates — detect state changes, gate privilege expansion.
08Autonomy vs. control
Every control above trades some autonomy or convenience for safety. Pretending otherwise produces either insecure systems or unusable ones. The honest way to decide is per deployment, by privilege tier.
| Posture | Autonomy | Blast radius | Where it fits |
|---|---|---|---|
| Trust the model | Maximal | Unbounded | Never, for side-effecting agents |
| Approve every action | Minimal | Small but human-paced | High-stakes, low-volume tasks |
| Policy + gate on state change | High | Bounded by allowlist | Most production agents |
| Full isolation + no persistent creds | Moderate | Minimal | Untrusted-input, high-risk agents |
Default to policy plus gate-on-state-change: it preserves the autonomy that makes agents useful while bounding blast radius by construction, and it degrades gracefully because the gate fires on meaningful transitions rather than on every step. Move to full isolation only for agents that ingest untrusted input and hold real privilege — the exact profile that produced the incidents in this series.
09Limitations of this framing
- The five-boundary model is an organizing lens, not a proof of completeness; novel agent architectures may introduce boundaries it does not name.
- Benchmark results (e.g. AgentDojo, CVE-Bench) are lower bounds shaped by the tasks chosen; absence of a demonstrated attack is not proof of safety [2].
- The GTG-1002 incident is a single-source vendor disclosure without independent reproduction [1]; we use it as directional evidence, not settled forensic fact.
- Governance frameworks (e.g. NIST AI RMF [9], OWASP GenAI [8]) provide vocabulary and process but are not, by themselves, technical controls.
10How to read the rest of this series
Start with the incident that makes the thesis concrete — the GTG-1002 reconstruction. Then follow the attack surface inward: prompt injection (how data becomes command), coding agents and MCP (where injection meets real execution), then the defenses: sandbox escape and containment, identity and least privilege, egress, and finally how to read the capability numbers without being misled by them.
—References
- Anthropic. Disrupting the first reported AI-orchestrated cyber espionage campaign (GTG-1002). Anthropic — Primary disclosure, 2025. https://www.anthropic.com/news/disrupting-AI-espionage
- E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, F. Tramèr. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. arXiv:2406.13352 [cs.CR], 2024. https://arxiv.org/abs/2406.13352
- T. Shi, J. He, Z. Wang, H. Li, L. Wu, W. Guo, D. Song. Progent: Securing AI Agents with Privilege Control. arXiv:2504.11703 [cs.CR], 2025. https://arxiv.org/abs/2504.11703
- R. Fang, R. Bindu, A. Gupta, D. Kang. LLM Agents can Autonomously Exploit One-day Vulnerabilities. arXiv:2404.08144 [cs.CR], 2024. https://arxiv.org/abs/2404.08144
- Y. Zhu, A. Kellermann, A. Gupta, P. Li, R. Fang, R. Bindu, D. Kang. Teams of LLM Agents can Exploit Zero-Day Vulnerabilities. arXiv:2406.01637 [cs.MA], 2024. https://arxiv.org/abs/2406.01637
- K. Huang, V. S. Narajala, J. Yeoh, J. Ross, R. Raskar, et al.. A Novel Zero-Trust Identity Framework for Agentic AI: Decentralized Authentication and Fine-Grained Access Control. arXiv:2505.19301 [cs.CR], 2025. https://arxiv.org/abs/2505.19301
- M. M. Hasan, H. Li, E. Fallahzadeh, G. K. Rajbahadur, B. Adams, A. E. Hassan. Model Context Protocol (MCP) at First Glance: Studying the Security and Maintainability of MCP Servers. arXiv:2506.13538 [cs.SE], 2025. https://arxiv.org/abs/2506.13538
- OWASP. OWASP Top 10 for LLM Applications & Agentic AI Threats and Mitigations. OWASP GenAI Security Project, 2025. https://genai.owasp.org/
- NIST. AI Risk Management Framework (AI RMF 1.0) and Generative AI Profile. NIST, 2024. https://www.nist.gov/itl/ai-risk-management-framework
Engineering assistance
ProxyTech works with engineers and teams building production AI, cloud, security, data and distributed systems. If this analysis maps to a system you are designing or operating, we provide hands-on support and interview preparation for these domains.