This is the repo for a project built as part of BlueDot Impact's AI technical safety project.
Disclaimer: this is a red-teaming research project. Logs, examples, and outputs checked into this repo can include harmful requests and attempted jailbreak content (and, where an attack succeeded, harmful model completions), produced solely to study and measure model vulnerabilities.
The guiding idea behind this project is to favor methods that scale with compute over methods that rely on clever engineering or hand-crafted heuristics -- treating compute as the resource to bet on, rather than bespoke ideas that stop working once the problem gets harder.
The project was originally scoped around a more ambitious setting: automated governance of AI systems in a context with real, contested human values at stake (e.g. a democratic governance structure for deciding how AI systems should behave). Given the time and resource constraints of the course, the scope was narrowed down to a more tractable, well-studied problem where the same compute-scaling idea could still be explored concretely: automated red-teaming.
Automated red-teaming lets you discover a model's vulnerabilities and interpret them at scale and quickly, rather than relying on slow, manual human red-teaming. That matters beyond just measuring how jailbreakable a model is: understanding why these vulnerabilities happen is a step toward using that same understanding to actually improve model safety (see "Extension ideas" below).
Two separate methods for automated red-teaming, each based on a different paper, live in their own folders:
An iterative search algorithm inspired by PAIR (Chao et al., 2023). An attacker, a target, and an evaluator model loop together, with the attacker iteratively refining its jailbreak attempts based on the evaluator's feedback on prior attempts.
See that folder's README for a setup/usage guide and
REPORT.md for a detailed report of what it found.
Its examples/ directory includes a full run against HarmBench: the raw evaluator scores, the
complete rollout of every attack attempt per task, a
chart showing attack
success improving as more compute (more search iterations) is spent, and an automated interpretation
step that produces a written report summarizing the overall results, the attack strategies
discovered, and other interesting findings.
Status: complete -- and the method this project ultimately favors over the RL approach (see "Comparing the two methods" below).
2. RL
A reinforcement-learning approach inspired by Jailbreak-R1. GRPO is used to fine-tune a LoRA adapter on an attacker model as it attacks target models over the OpenRouter API. After each attack, an evaluator scores how successful it was, and the attacker learns over many iterations which strategies and phrasing actually work.
Status: concluded at proof-of-concept scale -- see that folder's README and REPORT.md for the full writeup, and RL/examples/ for a complete real run checked in for reference.
Prompt Discovery was far cheaper to run than the RL approach: it never touches a GPU, just makes API calls to attacker/target/evaluator models, while the RL approach needs a GPU-backed training loop for the attacker. The RL approach also turned out considerably more expensive, for the result it produced, than expected going in -- it reliably learned to always attempt an attack within a single training step, but showed no clear further improvement in attack quality after that. Prompt Discovery, meanwhile, produced a large, clear improvement (see its own report) for a fraction of the cost. The verdict from this project: the RL approach here is not worth continuing further, even with more compute, given far more compute-efficient alternatives already exist and outperformed it directly in this same repo -- see the RL report for the detailed reasoning.
Some assumptions behind the original project idea held up; others didn't:
- Held up: that a minimal setup relying on compute (more search iterations, more RL training steps) rather than engineering effort could discover real attack strategies and vulnerabilities, and that those could then be automatically interpreted -- this worked as expected.
- Didn't hold up: the red-teaming tasks themselves (from HarmBench/AdvBench/HarmfulQA) turned out not to be very interesting -- attacks frequently succeeded with quite simple methods, rather than requiring the kind of iterative, compute-hungry refinement the project was built around.
The results here are a function of the specific setup used, not a ceiling on what these methods can do in general. A couple of concrete constraints that shaped (and likely limited) the results:
- The target was always a plain LLM. A more complex AI system as a target -- e.g. an agent with real authority or tool access -- would open up a much larger and more interesting space of possible attacks than jailbreaking a chat model into producing text.
- Target models were restricted to open-weight models, since closed frontier models (e.g. Claude) generally prohibit this kind of automated adversarial probing in their terms of service. Open-weight models are already comparatively more compliant/less safety-trained than a model like Claude, which makes them easier targets than the frontier default -- and even with permission, probing frontier models at any real scale would have cost far more than this project's resources allowed.
Beyond scaling up compute on the same setup, one promising direction is feeding these discovered attacks and attacker strategies back into a target model's own training -- using red-teaming to discover vulnerabilities and interpret them at scale, then use that understanding to actually improve model safety, not just measure it. Many ideas here run into the "alignment tax" problem: a model trained to avoid causing harm can trivially satisfy that objective by refusing everything, including entirely benign requests. One way to avoid that: mix red-team attack attempts into a normal training set of regular, helpful tasks, rewarding refusal only on the attacks while still rewarding compliance on the regular tasks -- pushing the model to learn to actually discriminate between harmful and helpful requests, rather than to refuse indiscriminately.
Another direction is multi-agent RL, training an attacker and a defender together so each adapts to the other over time, similar to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO -- rather than this repo's RL project, where only the attacker is trained against a fixed target.
- PAIR (Chao et al., 2023) -- the iterative attacker/target/ evaluator search loop the Prompt Discovery method is based on.
- Jailbreak-R1 -- the GRPO-trained-attacker approach the RL method is based on.
- Attack and Defend: Adaptive Red Teaming of Language Models via GRPO -- adaptive, multi-agent red-teaming with GRPO; see "Extension ideas" above.
- TAP (Mehrotra et al., 2023) -- tree-of-attacks search, a related refinement on PAIR's single-chain iterative search.
- AutoDAN (Liu et al., 2023) -- genetic-algorithm-based automated jailbreak prompt generation, another automated red-teaming approach neither method here is based on but sits alongside.
Both methods are structured so a new target model, evaluator model, or task/prompt dataset can be swapped in without changing the core algorithm. See each subfolder's own README for the relevant entry points.