Agent Security Papers

GitHub Pages Contributions Welcome Last Updated

This page tracks research papers on the security and privacy of LLM-based agents. The focus is on agentic systems that use tools, retrieve untrusted content, interact with external environments, persist memory, execute multi-step tasks, or act on behalf of users.

The collection is meant to help researchers follow recent progress, compare threat models, and identify open problems. Contributions, corrections, missing papers, and discussion are very welcome.

Scope: prompt injection, indirect prompt injection, tool abuse, tool selection hijacking, data leakage, privacy leakage, agent backdoors, persistent control, harmful task execution, and defenses for tool-using or environment-facing LLM agents.

Contents

Reading Map

Area What to Look For Representative Papers
Indirect prompt injection Untrusted external content changes an agent’s actions. BIPIA, InjecAgent, AgentDojo, ARGUS
Tool and action security Attacks against tool invocation, tool selection, or external action execution. ToolEmu, ToolHijacker, Tool Result Parsing
Privacy and data leakage Sensitive data exposure during adversarial or benign agent use. PrivacyLens, Simple Prompt Injection Attacks Can Leak Personal Data, Data Leakage Risks in Tool-Using LLM Agents
Persistent and stateful attacks Agent memory, files, workspace state, or long-running context becomes an attack channel. Stateful Agent Backdoor, ClawTrojan/DASGuard
Defenses and secure design Separating trusted instructions from untrusted data, provenance auditing, capabilities, and alignment. StruQ, SecAlign, CaMeL, IntentGuard

Papers by Year

2026

Paper Tags Summary
An Evaluation of Data Leakage Risks in Tool-Using LLM Agents in Realistic Scenarios privacy data leakage evaluation Studies non-adversarial data-handling failures in realistic tool-using agents, separating task success from safe information handling.
ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection defense provenance benchmark Introduces AgentLure for context-aware prompt injection and uses provenance-aware decision auditing to validate proposed agent actions.
AI Agents May Always Fall for Prompt Injections theory contextual integrity prompt injection Reframes prompt injection through contextual integrity and argues that simple data-instruction separation misses context-sensitive attacks.
From Prompt Injection to Persistent Control: Defending Agentic Harness Against Trojan Backdoors backdoor persistent control defense Shows how untrusted text can become persistent control content across local agent sessions and proposes DASGuard to sanitize control-like state.
Stateful Agent Backdoor backdoor stateful attack persistence Models multi-session agent backdoors as stateful machines that maintain attack progress through persistent components.
Defense Against Indirect Prompt Injection via Tool Result Parsing defense tool output parsing Defends against malicious instructions embedded in tool results by parsing tool outputs before they influence agent decisions.
Navigating the Risks: A Survey of Security and Privacy Threats in LLM-Based Agents survey security privacy Surveys security and privacy threats introduced when LLMs are embedded into software engineering workflows as autonomous or semi-autonomous agents.

2025

Paper Tags Summary
Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis defense intent analysis IPI Proposes IntentGuard, which detects whether the model intends to follow instructions originating from untrusted data.
Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution privacy exfiltration AgentDojo Extends AgentDojo-style scenarios to study how prompt injection can leak personal data observed during task execution.
Prompt Injection Attack to Tool Selection in LLM Agents tool selection attack ToolHijacker Attacks the retrieval-and-selection stage of agent tool use by injecting malicious tool documents into a tool library.
Defeating Prompt Injections by Design defense capabilities control/data flow Presents CaMeL, a secure agent layer that separates trusted control flow from untrusted data and enforces capability-based tool policies.

2024

Paper Tags Summary
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents benchmark harmfulness jailbreak Evaluates whether LLM agents refuse or complete harmful multi-step tasks, including cases where jailbreaks preserve agent capability.
SecAlign: Defending Against Prompt Injection with Preference Optimization defense alignment prompt injection Uses preference optimization to make models prefer outputs that follow legitimate instructions over injected ones.
PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action privacy agent actions benchmark Converts privacy-sensitive cases into agent trajectories to test whether language models leak sensitive information during action execution.
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents benchmark IPI defense evaluation Provides an extensible environment with realistic tasks, security test cases, attacks, and defenses for tool-using agents over untrusted data.
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents benchmark IPI tool use Benchmarks indirect prompt injection against tool-integrated agents across user tools, attacker tools, harmful actions, and data exfiltration.
StruQ: Defending Against Prompt Injection with Structured Queries defense structured query instruction/data separation Separates prompts and data into structured channels and fine-tunes models to follow only the instruction channel.

2023

Paper Tags Summary
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models BIPIA benchmark defense Introduces BIPIA for evaluating indirect prompt injection in LLM applications and proposes boundary-awareness and reminder-based defenses.
Identifying the Risks of LM Agents with an LM-Emulated Sandbox ToolEmu risk evaluation tool use Uses an LLM-emulated sandbox to evaluate tool-use risks and failures without executing risky real-world actions.

Papers by Topic

Prompt Injection and Indirect Prompt Injection

Tool Use, Tool Results, and Tool Selection

Privacy and Data Leakage

Backdoors, Memory, and Persistent Control

Defense Mechanisms

Contributing

Contributions are welcome. Please open an issue or pull request if you want to:

Suggested entry format:

| [Paper Title](paper-url) | <span class="tag">tag</span> | One-sentence summary of threat model, method, or finding. |

License

This paper list is intended for academic use and community maintenance. Please cite the original papers when using their ideas, benchmarks, or results.