diff --git a/paper_list/02 - Attack Strategies.md b/paper_list/02 - Attack Strategies.md index 2e18fdc..8a553cf 100644 --- a/paper_list/02 - Attack Strategies.md +++ b/paper_list/02 - Attack Strategies.md @@ -14,6 +14,11 @@ - [Backdoor Attacks](#backdoor-attacks) - [Model Fine-tuning](#model-fine-tuning) - [Activations Manipulation](#activations-manipulation) + - [Agentic Attack Surfaces](#agentic-attack-surfaces) + - [Pipeline Trust Exploitation / Second-Order Injection](#pipeline-trust-exploitation--second-order-injection) + - [Human-in-the-Loop and Legitimate-in-the-Loop Attacks](#human-in-the-loop-and-legitimate-in-the-loop-attacks) + - [Identity Anchor Attacks](#identity-anchor-attacks) + ## Completion Compliance ### In-Context Learning - **Few-Shot Adversarial Prompt Learning on Vision-Language Models** \[[Paper](https://arxiv.org/abs/2403.14774)\]
@@ -30,29 +35,23 @@ Yao Qiang, Xiangyu Zhou, Dongxiao Zhu (2023)
Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, Bing Qin (2023)
- **Bypassing the Safety Training of Open-Source LLMs with Priming Attacks** \[[Paper](https://arxiv.org/abs/2312.12321)\]
Jason Vega, Isha Chaudhary, Changming Xu, Gagandeep Singh (2023)
-- **Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses** \[[Paper](https://arxiv.org/abs/2406.01288 - )\]
+- **Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses** \[[Paper](https://arxiv.org/abs/2406.01288)\]
Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, Min Lin (2024)
-- **Don't Say No: Jailbreaking LLM by Suppressing Refusal** \[[Paper](https://arxiv.org/abs/2404.16369 - )\]
+- **Don't Say No: Jailbreaking LLM by Suppressing Refusal** \[[Paper](https://arxiv.org/abs/2404.16369)\]
Yukai Zhou, Wenjie Wang (2024)
-- **Improved Generation of Adversarial Examples Against Safety-aligned LLMs** \[[Paper](https://arxiv.org/abs/2405.20778 - )\]
+- **Improved Generation of Adversarial Examples Against Safety-aligned LLMs** \[[Paper](https://arxiv.org/abs/2405.20778)\]
Qizhang Li, Yiwen Guo, Wangmeng Zuo, Hao Chen (2024)
-- **Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks** \[[Paper](https://arxiv.org/abs/2407.00869 - )\]
+- **Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks** \[[Paper](https://arxiv.org/abs/2407.00869)\]
Yue Zhou, Henry Peng Zou, Barbara Di Eugenio, Yang Zhang (2024)
- **Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning** \[[Paper](https://arxiv.org/abs/2401.05949)\]
Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, Jinming Wen (2024)
+ ### Multi turn Dialog -- **Investigating the prompt leakage effect and black-box defenses for multi-turn LLM interactions** \[[Paper](https://arxiv.org/abs/2404.16251 - )\]
+- **Investigating the prompt leakage effect and black-box defenses for multi-turn LLM interactions** \[[Paper](https://arxiv.org/abs/2404.16251)\]
Divyansh Agarwal, Alexander R. Fabbri, Philippe Laban, Ben Risher, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu (2024)
-- **CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference** \[[Paper](https://arxiv.org/abs/2406.17626 - )\]
+- **CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference** \[[Paper](https://arxiv.org/abs/2406.17626)\]
Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, Lanqing Hong (2024)
-- **Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLM** \[[Paper](https://arxiv.org/abs/2405.05610 - )\]
+- **Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLM** \[[Paper](https://arxiv.org/abs/2405.05610)\]
Xikang Yang, Xuehai Tang, Songlin Hu, Jizhong Han (2024)
- **Great,Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack** \[[Paper](https://arxiv.org/abs/2404.01833)\]
Mark Russinovich, Ahmed Salem, Ronen Eldan (2024)
@@ -60,19 +59,17 @@ Mark Russinovich, Ahmed Salem, Ronen Eldan (2024)
Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, Bo Li (2024)
- **Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment** \[[Paper](https://arxiv.org/abs/2308.09662)\]
Rishabh Bhardwaj, Soujanya Poria (2023)
+ ### Special Tokens -- **Enhancing Jailbreak Attack Against Large Language Models through Silent Tokens** \[[Paper](https://arxiv.org/abs/2405.20653 - )\]
+- **Enhancing Jailbreak Attack Against Large Language Models through Silent Tokens** \[[Paper](https://arxiv.org/abs/2405.20653)\]
Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu, Wenbo Guo, Han Liu, Xinyu Xing (2024)
-- **Talk Too Much: Poisoning Large Language Models under Token Limit** \[[Paper](https://arxiv.org/abs/2404.14795 - )\]
+- **Talk Too Much: Poisoning Large Language Models under Token Limit** \[[Paper](https://arxiv.org/abs/2404.14795)\]
Jiaming He, Wenbo Jiang, Guanyu Hou, Wenshu Fan, Rui Zhang, Hongwei Li (2024)
-- **ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates** \[[Paper](https://arxiv.org/abs/2406.12935 - )\]
+- **ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates** \[[Paper](https://arxiv.org/abs/2406.12935)\]
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, Radha Poovendran (2024)
-- **Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers** \[[Paper](https://arxiv.org/abs/2405.10612 - )\]
+- **Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers** \[[Paper](https://arxiv.org/abs/2405.10612)\]
Sheng Yang, Jiawang Bai, Kuofeng Gao, Yong Yang, Yiming Li, Shu-tao Xia (2024)
+ ## Instruction Indirection - **On the Robustness of Large Multimodal Models Against Image Adversarial Attacks** \[[Paper](https://arxiv.org/abs/2312.03777)\]
Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, Ser-Nam Lim (2023)
@@ -88,36 +85,27 @@ Erfan Shayegani, Yue Dong, Nael Abu-Ghazaleh (2023)
Xunguang Wang, Zhenlan Ji, Pingchuan Ma, Zongjie Li, Shuai Wang (2023)
- **FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts** \[[Paper](https://arxiv.org/abs/2311.05608)\]
Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, Xiaoyun Wang (2023)
-- **Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models** \[[Paper](https://arxiv.org/abs/2405.20775 - )\]
+- **Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models** \[[Paper](https://arxiv.org/abs/2405.20775)\]
Xijie Huang, Xinyuan Wang, Hantao Zhang, Jiawen Xi, Jingkun An, Hao Wang, Chengwei Pan (2024)
-- **Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs** \[[Paper](https://arxiv.org/abs/2307.1049)\]
+- **Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs** \[[Paper](https://arxiv.org/abs/2307.10490)\]
Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov (2023)
-- **Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything** \[[Paper](https://arxiv.org/abs/2407.02534 - )\]
+- **Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything** \[[Paper](https://arxiv.org/abs/2407.02534)\]
Xiaotian Zou, Yongkang Chen (2024)
-- **From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking** \[[Paper](https://arxiv.org/abs/2406.14859 - )\]
+- **From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking** \[[Paper](https://arxiv.org/abs/2406.14859)\]
Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei (2024)
-- **Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks** \[[Paper](https://arxiv.org/abs/2406.06302 - )\]
+- **Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks** \[[Paper](https://arxiv.org/abs/2406.06302)\]
Zonghao Ying, Aishan Liu, Xianglong Liu, Dacheng Tao (2024)
-- **Voice Jailbreak Attacks Against GPT-4o** \[[Paper](https://arxiv.org/abs/2405.19103 - )\]
+- **Voice Jailbreak Attacks Against GPT-4o** \[[Paper](https://arxiv.org/abs/2405.19103)\]
Xinyue Shen, Yixin Wu, Michael Backes, Yang Zhang (2024)
- **Nevermind: Instruction Override and Moderation in Large Language Models** \[[Paper](https://arxiv.org/abs/2402.03303)\]
Edward Kim (2024)
-- **Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent** \[[Paper](https://arxiv.org/abs/2405.03654 - )\]
+- **Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent** \[[Paper](https://arxiv.org/abs/2405.03654)\]
Shang Shang, Xinqiang Zhao, Zhongjiang Yao, Yepeng Yao, Liya Su, Zijing Fan, Xiaodan Zhang, Zhengwei Jiang (2024)
-- **Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injection** \[[Paper](https://arxiv.org/abs/2404.04849 - )\]
+- **Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injection** \[[Paper](https://arxiv.org/abs/2404.04849)\]
Zhilong Wang, Yebo Cao, Peng Liu (2024)
-- **ObscurePrompt: Jailbreaking Large Language Models via Obscure Input** \[[Paper](https://arxiv.org/abs/2406.13662 - )\]
+- **ObscurePrompt: Jailbreaking Large Language Models via Obscure Input** \[[Paper](https://arxiv.org/abs/2406.13662)\]
Yue Huang, Jingyu Tang, Dongping Chen, Bingda Tang, Yao Wan, Lichao Sun, Xiangliang Zhang (2024)
-- **Preemptive Answer "Attacks" on Chain-of-Thought Reasoning** \[[Paper](https://arxiv.org/abs/2405.20902 - )\]
+- **Preemptive Answer "Attacks" on Chain-of-Thought Reasoning** \[[Paper](https://arxiv.org/abs/2405.20902)\]
Rongwu Xu, Zehan Qi, Wei Xu (2024)
- **Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues** \[[Paper](https://arxiv.org/abs/2402.09091)\]
Zhiyuan Chang, Mingyang Li, Yi Liu, Junjie Wang, Qing Wang, Yang Liu (2024)
@@ -139,10 +127,10 @@ Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, Muhao Chen (2023)< Hao Yang, Lizhen Qu, Ehsan Shareghi, Gholamreza Haffari (2024)
- **Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models** \[[Paper](https://arxiv.org/abs/2410.23861)\]
Hao Yang, Lizhen Qu, Ehsan Shareghi, Gholamreza Haffari (2024)
+ ## Generalization Glide ### Languages -- **Jailbreaking LLMs with Arabic Transliteration and Arabizi** \[[Paper](https://arxiv.org/abs/2406.18725 - )\]
+- **Jailbreaking LLMs with Arabic Transliteration and Arabizi** \[[Paper](https://arxiv.org/abs/2406.18725)\]
Mansour Al Ghanim, Saleh Almohaimeed, Mengxin Zheng, Yan Solihin, Qian Lou (2024)
- **Backdoor Attack on Multilingual Machine Translation** \[[Paper](https://arxiv.org/abs/2404.02393)\]
Jun Wang, Qiongkai Xu, Xuanli He, Benjamin I. P. Rubinstein, Trevor Cohn (2024)
@@ -156,6 +144,7 @@ Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Zheng-Xin Yong, Cristina Menghini, Stephen H. Bach (2023)
- **Multilingual Jailbreak Challenges in Large Language Models** \[[Paper](https://arxiv.org/abs/2310.06474)\]
Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, Lidong Bing (2023)
+ ### Cipher - **Using Hallucinations to Bypass GPT4's Filter** \[[Paper](https://arxiv.org/abs/2403.04769)\]
Benjamin Lemkin (2024)
@@ -169,18 +158,15 @@ Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shumi Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, Kai Chen (2024)
- **PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails** \[[Paper](https://arxiv.org/abs/2402.15911)\]
Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, Atul Prakash (2024)
-- **Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters** \[[Paper](https://arxiv.org/abs/2405.20413 - )\]
+- **Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters** \[[Paper](https://arxiv.org/abs/2405.20413)\]
Haibo Jin, Andy Zhou, Joe D. Menke, Haohan Wang (2024)
-- **Revisiting character-level adversarial attacks** \[[Paper](https://arxiv.org/abs/2405.04346 - )\]
+- **Revisiting character-level adversarial attacks** \[[Paper](https://arxiv.org/abs/2405.04346)\]
Elias Abad Rocamora, Yongtao Wu, Fanghui Liu, Grigorios G. Chrysos, Volkan Cevher (2024)
-- **Single Character Perturbations Break LLM Alignment** \[[Paper](https://arxiv.org/abs/2407.03232 - )\]
+- **Single Character Perturbations Break LLM Alignment** \[[Paper](https://arxiv.org/abs/2407.03232)\]
Leon Lin, Hannah Brown, Kenji Kawaguchi, Michael Shieh (2024)
-- **Enhance Robustness of Language Models Against Variation Attack through Graph Integration** \[[Paper](https://arxiv.org/abs/2404.12014 - )\]
+- **Enhance Robustness of Language Models Against Variation Attack through Graph Integration** \[[Paper](https://arxiv.org/abs/2404.12014)\]
Zi Xiong, Lizhi Qing, Yangyang Kang, Jiawei Liu, Hongsong Li, Changlong Sun, Xiaozhong Liu, Wei Lu (2024)
+ ### Personification - **Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles** \[[Paper](https://arxiv.org/abs/2311.14876)\]
Sonali Singh, Faranak Abri, Akbar Siami Namin (2023)
@@ -194,21 +180,18 @@ Zaibin Zhang, Yongting Zhang, Lijun Li, Hongzhi Gao, Lijun Wang, Huchuan Lu Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, Weiyan Shi (2024)
- **Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology** \[[Paper](https://arxiv.org/abs/2402.1569)\]
Zhenhua Wang, Wei Xie, Baosheng Wang, Enze Wang, Zhiwen Gui, Shuoyoucheng Ma, Kai Chen (2024)
-- **Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective** \[[Paper](https://arxiv.org/abs/2406.14023 - )\]
+- **Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective** \[[Paper](https://arxiv.org/abs/2406.14023)\]
Yuchen Wen, Keping Bi, Wei Chen, Jiafeng Guo, Xueqi Cheng (2024)
-- **SoP: Unlock the Power of Social Facilitation for Automatic Jailbreak Attack** \[[Paper](https://arxiv.org/abs/2407.01902 - )\]
+- **SoP: Unlock the Power of Social Facilitation for Automatic Jailbreak Attack** \[[Paper](https://arxiv.org/abs/2407.01902)\]
Yan Yang, Zeguan Xiao, Xin Lu, Hongru Wang, Hailiang Huang, Guanhua Chen, Yun Chen (2024)
-- **Defending Against Social Engineering Attacks in the Age of LLMs** \[[Paper](https://arxiv.org/abs/2406.12263 - )\]
+- **Defending Against Social Engineering Attacks in the Age of LLMs** \[[Paper](https://arxiv.org/abs/2406.12263)\]
Lin Ai, Tharindu Kumarage, Amrita Bhattacharjee, Zizhou Liu, Zheng Hui, Michael Davinroy, James Cook, Laura Cassani, Kirill Trapeznikov, Matthias Kirchner, Arslan Basharat, Anthony Hoogs, Joshua Garland, Huan Liu, Julia Hirschberg (2024)
+ ## Model Manipulation ### Backdoor Attacks - **Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models** \[[Paper](https://arxiv.org/abs/2402.06659)\]
Yuancheng Xu, Jiarui Yao, Manli Shu, Yanchao Sun, Zichu Wu, Ning Yu, Tom Goldstein, Furong Huang (2024)
-- **Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models** \[[Paper](https://arxiv.org/abs/2404.12916 - )\]
+- **Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models** \[[Paper](https://arxiv.org/abs/2404.12916)\]
Zhenyang Ni, Rui Ye, Yuxi Wei, Zhen Xiang, Yanfeng Wang, Siheng Chen (2024)
- **What's in Your "Safe" Data?: Identifying Benign Data that Breaks Safety** \[[Paper](https://arxiv.org/abs/2404.01099)\]
Luxi He, Mengzhou Xia, Peter Henderson (2024)
@@ -238,6 +221,7 @@ Javier Rando, Florian Tramèr (2023)
Yuanpu Cao, Bochuan Cao, Jinghui Chen (2023)
- **Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models** \[[Paper](https://arxiv.org/abs/2305.1471)\]
Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, Muhao Chen (2023)
+ ### Model Fine-tuning - **Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models** \[[Paper](https://arxiv.org/abs/2310.02949)\]
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, Dahua Lin (2023)
@@ -257,28 +241,58 @@ Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, Dan Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen, Xia Hu (2024)
- **Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!** \[[Paper](https://arxiv.org/abs/2402.12343)\]
Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, Yu Qiao (2024)
-- **Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation** \[[Paper](https://arxiv.org/abs/2406.20053 - )\]
+- **Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation** \[[Paper](https://arxiv.org/abs/2406.20053)\]
Danny Halawi, Alexander Wei, Eric Wallace, Tony T. Wang, Nika Haghtalab, Jacob Steinhardt (2024)
-- **Transforming Computer Security and Public Trust Through the Exploration of Fine-Tuning Large Language Models** \[[Paper](https://arxiv.org/abs/2406.00628 - )\]
+- **Transforming Computer Security and Public Trust Through the Exploration of Fine-Tuning Large Language Models** \[[Paper](https://arxiv.org/abs/2406.00628)\]
Garrett Crumrine, Izzat Alsmadi, Jesus Guerrero, Yuvaraj Munian (2024)
- **Increased LLM Vulnerabilities from Fine-tuning and Quantization** \[[Paper](https://arxiv.org/abs/2404.04392)\]
Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, Prashanth Harshangi (2024)
-- **Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data** \[[Paper](https://arxiv.org/abs/2404.05530 - )\]
+- **Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data** \[[Paper](https://arxiv.org/abs/2404.05530)\]
Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler (2024)
-- **No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks** \[[Paper](https://arxiv.org/abs/2405.16229 - )\]
+- **No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks** \[[Paper](https://arxiv.org/abs/2405.16229)\]
Chak Tou Leong, Yi Cheng, Kaishuai Xu, Jian Wang, Hanlin Wang, Wenjie Li (2024)
-- **Badllama 3: removing safety finetuning from Llama 3 in minutes** \[[Paper](https://arxiv.org/abs/2407.01376 - )\]
+- **Badllama 3: removing safety finetuning from Llama 3 in minutes** \[[Paper](https://arxiv.org/abs/2407.01376)\]
Dmitrii Volkov (2024)
-- **No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks** \[[Paper](https://arxiv.org/abs/2405.16229 - )\]
-Chak Tou Leong, Yi Cheng, Kaishuai Xu, Jian Wang, Hanlin Wang, Wenjie Li (2024)
+ ### Activations Manipulation - **Open the Pandora's Box of LLMs: Jailbreaking LLMs through Representation Engineering** \[[Paper](https://arxiv.org/abs/2401.06824)\]
Tianlong Li, Shihan Dou, Wenhao Liu, Muling Wu, Changze Lv, Xiaoqing Zheng, Xuanjing Huang (2024)
- **Scaling Laws for Adversarial Attacks on Language Model Activations** \[[Paper](https://arxiv.org/abs/2312.0278)\]
-Stanislav Fort (2023)
\ No newline at end of file +Stanislav Fort (2023)
+ +## Agentic Attack Surfaces + +Unlike prompt-level attacks, agentic attack surfaces target the trust relationships and operational assumptions of deployed agent systems — tool pipelines, operator layers, and session-level identity — rather than the model's content policies directly. These attacks often succeed without triggering any individual content refusal. + +### Pipeline Trust Exploitation / Second-Order Injection + +Attacks where adversarial instructions are delivered via the agent's own tool outputs, retrieved documents, or memory rather than the direct user channel. The agent processes them as trusted context, bypassing input-layer defenses. + +- **Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection** \[[Paper](https://arxiv.org/abs/2302.12173)\]
+Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, Mario Fritz (2023)
+- **Prompt Injection attack against LLM-integrated Applications** \[[Paper](https://arxiv.org/abs/2306.05499)\]
+Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, Yang Liu (2023)
+- **InjectAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents** \[[Paper](https://arxiv.org/abs/2403.02691)\]
+Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang (2024)
+- **AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents** \[[Paper](https://arxiv.org/abs/2406.13352)\]
+Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr (2024)
+ +### Human-in-the-Loop and Legitimate-in-the-Loop Attacks + +Attacks that exploit operator-level or human approval mechanisms to shift agent behavior. HITL attacks manipulate the human checkpoint itself; LITL attacks use plausible operator-level instructions to cumulatively drift the agent's operating mandate without triggering any individual refusal. + +- **Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs** \[[Paper](https://arxiv.org/abs/2307.10490)\]
+Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov (2023)
+- **GNOME: A Field Journal of Drift, Control, and Learning in Public** \[[Paper](https://dev.to/gnomeman4201/i-built-a-policy-drift-detector-for-llm-agents-heres-what-four-versions-taught-me-2be)\]
+GnomeMan4201 / badBANANA Research Collective (2026) *(practitioner field documentation; introduces HITL/LITL attack taxonomy and Second-Order Injection framing for deployed agentic systems)*
+ +### Identity Anchor Attacks + +Attacks that target the agent's persistent sense of role and operational scope across a session rather than per-turn content policies. Success is measured by mandate displacement — the agent operates outside its intended scope without ever crossing a flaggable content threshold. + +- **Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models** \[[Paper](https://arxiv.org/abs/2508.04196)\]
+AIM Intelligence (2026)
+- **Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue** \[[Paper](https://arxiv.org/abs/2402.17262)\]
+*(see also: Phenomenons section)*
+- **GNOME: A Field Journal of Drift, Control, and Learning in Public** \[[Paper](https://dev.to/gnomeman4201/i-built-a-policy-drift-detector-for-llm-agents-heres-what-four-versions-taught-me-2be)\]
+GnomeMan4201 / badBANANA Research Collective (2026) *(introduces Reflexive Identity as a named attack class targeting session-level identity anchors in deployed agents)*