Systematic evaluation of jailbreak and prompt-injection attacks and defense strategies across open-source and commercial LLMs.
Status
- ✅ Infrastructure complete
- ✅ Mock validation complete
- ⬜ Real-model experiments — not yet started (planned, see Roadmap)
- 📝 Paper drafted (proposal, preregistration, methodology, pilot report) — pending real-model results
See below before reading any number in this README as a finding.
⚠️ Read this before the numbers below. This repository ships a complete, working, tested pipeline — attack taxonomy, dataset, nine defense implementations, a model-agnostic evaluation harness, and a full statistics/visualization layer. The results currently inresults/were produced with a documented synthetic mock model fleet (models/model_clients.py::MockVulnerableModel), because the sandboxed environment this repo was assembled in cannot reach OpenAI / Anthropic / Hugging Face endpoints for the real target models. Every result row is taggedmodel_type=mock, andevaluation/statistics.pyrefuses to produce a summary artifact from mock rows unless you pass--allow-mockexplicitly, so a demo run can never be silently mistaken for a real finding. See Quickstart for how to re-run everything against real models once you have API keys.
| 🛡️ Attack families | 7 (jailbreak-style: attacks/) + 6 (prompt-injection: datasets/) |
| 🤖 Target models | Llama, Gemma, Qwen, GPT, Claude (configs/) |
| ⚔️ Defense methods | 9 |
| 📊 Metrics | ASR, DSR, FPR, FNR, Precision, Recall, F1, Latency, Token overhead, Cost |
| 🧪 Total attacks | 300 (prompt-injection) + expandable jailbreak template set |
| 📈 Statistical analysis | Wilson CI, Bootstrap CI, McNemar, Fisher's exact, Cohen's h/κ |
| 💻 Reproducible pipeline | Yes — seeded dataset generation, CI-checked |
| 📄 Technical report | Included (paper/pilot_report.pdf) |
How effective are existing prompt-level defense mechanisms against diverse jailbreak and prompt-injection attack families across modern large language models, and does that effectiveness depend on attack type and difficulty?
Pre-registered before any confirmatory analysis — see paper/preregistration.md for exact stopping criteria and test selection.
| ID | Hypothesis |
|---|---|
| H1 | Prompt hardening reduces attack success rates relative to no protection. |
| H2 | Multi-stage verification performs better (lower ASR) than any single-pass defense. |
| H3 | Unicode and encoding-based attacks bypass more defenses than direct/persona-based attacks (DAN, roleplay), since static/keyword-level defenses can't see through the encoding layer. |
| H4 | Defense effectiveness differs significantly across model families (a model x defense interaction, not just a defense main effect). |
This project evaluates two attack taxonomies side by side — see datasets/licenses.md for how they relate and why both exist:
Jailbreak-style families (attacks/, template-based, non-scraped):
| Category | Examples |
|---|---|
| DAN | Classic persona-override jailbreaks ("you are now DAN...") |
| Role-play | Fictional/evil-assistant framing to disclaim responsibility |
| Encoding | Base64, ROT13, nested/stacked encodings |
| Unicode | Zero-width characters, homoglyphs, bidirectional overrides |
| Obfuscation | Typos, spacing, leetspeak, synonym substitution |
| Emotional | Sympathy, urgency, rapport-based manipulation |
| Multi-turn | Conversation poisoning across several turns |
Prompt-injection families (datasets/attacks.json, 300 attacks, 6 categories x 3 difficulty tiers):
| Category | Description |
|---|---|
ignore_previous_instructions |
Direct/indirect system-prompt override attempts |
tool_misuse |
Unconfirmed or chained tool-call attacks |
secret_extraction |
Attempts to exfiltrate system prompts / credentials |
jailbreak |
Persona/roleplay framings to suspend safety behavior |
context_poisoning |
Instructions hidden inside retrieved documents / tool output |
multi_step_attacks |
Multi-turn commitment-building before the unsafe ask |
| Defense | Mechanism | Extra model calls |
|---|---|---|
| No Protection | baseline pass-through | 0 |
| Keyword Filter | static blocklist | 0 |
| Rule-Based | structural regex heuristics | 0 |
| Prompt Hardening | strong, explicit system prompt | 0 |
| Self-Reflection | same model reviews its own answer, no named principles | +1 (same model) |
| Constitutional AI | independent critique pass against a named principle list | +1 |
| LLM-as-Judge | independent model classifies the response after the fact | +1 |
| Ensemble | multiple independent judges vote (majority rule) | +N |
| Multi-Stage Verification | rule-filter → prompt hardening → judge, in sequence | 0–1 |
See defenses/ — every strategy implements the same Defense.guard(...) interface (defenses/_common.py) so the harness can swap them freely.
Attack Dataset (attacks/ + datasets/)
|
v
Target LLM (models/model_clients.py, configs/*.yaml)
|
v
Defense Layer (defenses/, 9 strategies)
|
v
Model Response
|
v
LLM Judge (independent model, where applicable)
|
v
Metrics (evaluation/metrics.py)
|
v
Statistical Analysis (evaluation/statistics.py)
|
v
Visualizations (evaluation/visualization.py)
Orchestrated end-to-end by evaluation/run_experiment.py.
Computed per (model, defense[, category, difficulty]) in evaluation/metrics.py:
- Attack Success Rate (ASR) — attack got through AND the defense did not block it
- Defense Success Rate (DSR) — of attacks that would have succeeded, fraction the defense caught (equivalent to Recall in this framing)
- False Positive Rate / False Negative Rate
- Precision / Recall / F1
- Latency (ms) and Token overhead (estimated, ~4 chars/token proxy)
- Cost (estimated $, illustrative per-model rate table — not live pricing; see the docstring in
metrics.pyfor the exact caveat)
Implemented in evaluation/statistics.py — see paper/methodology.md for the full "why this test, not a simpler one" rationale behind each choice:
| Method | Used for |
|---|---|
| Wilson score CI | Every proportion (ASR/FPR/FNR) — stays well-behaved near 0%/100%, unlike the normal approximation |
| Bootstrap CI (10k resamples) | Distribution-free cross-check on Wilson |
| McNemar's test | Paired defense comparisons (same attack, two defenses) |
| Fisher's exact test | Unpaired comparisons with small/zero-cell tables (e.g. same defense, different model) |
| Cohen's h | Effect size for proportion differences, alongside significance |
| Cohen's kappa | Validating the automated compliance labeler against human annotation |
Cross-condition consistency (does a defense hold up across difficulty tiers / attack families, not just on average) is handled separately in evaluation/robustness.py — see Results below.
Generated by evaluation/visualization.py from results/raw/raw_results.csv:
- ✓ Attack Success Rate by Defense (
results/figures/asr_by_defense.png) - ✓ ASR Heatmap by Defense x Category (
asr_heatmap_category.png) - ✓ Multi-Metric Radar Chart (
radar_multi_metric.png) - ✓ Confusion-Matrix-Style Outcomes per Defense (
confusion_style_outcomes.png) - ✓ Failure Breakdown by Difficulty Tier (
failure_breakdown_by_difficulty.png) - ✓ Latency Boxplots by Defense (
latency_boxplot_by_defense.png) - ✓ Defense Cost vs. Security Scatter (
cost_vs_security_scatter.png)
Pilot validation (mock backend): the evaluation pipeline, statistical analysis, and visualization components were validated end-to-end using the documented mock model backend (models/model_clients.py::MockVulnerableModel). These results verify infrastructure correctness only and should not be interpreted as model performance. Real-model experiments are planned in experiments/experiment_03/.
Full mock-run tables, per-defense ASR/FPR/FNR breakdowns, and the honest artifact-vs-finding analysis (which patterns are real methodology results vs. known mock-backend artifacts) are in paper/pilot_report.md, Section 7. Robustness-across-conditions data (does a defense's ASR stay flat across easy/medium/hard, or spike on hard attacks) is in results/processed/robustness_score_by_difficulty.csv.
Internal validity
- Judge bias: where the judge and target model are the same family, critique passes may under-detect (see
defenses/constitutional.pydocstring on judge independence) - Prompt leakage between trials is mitigated by fresh context per trial, not yet independently audited
- Temperature is fixed to 0.0 for reproducibility, which reduces (but doesn't fully eliminate) response variance
External validity
- Current mock results reflect the synthetic model fleet only, not real target models (see caveat above)
- Attack template count (7 families, 3 templates/difficulty each) is a starting set, not an exhaustive taxonomy
- Limited to 5 model families (
configs/); doesn't cover every deployed LLM
Construct validity
- LLM-judge disagreement with the heuristic compliance labeler hasn't yet been checked against human annotation at scale — Cohen's kappa is implemented (
evaluation/statistics.py::cohens_kappa) but requires a human-labeled sample to run against - Prompt ambiguity: some attack templates could plausibly be read as benign depending on context, which the pre-registration's stratified human-validation step (not yet executed) is meant to catch
This project contributes:
- A unified benchmark covering multiple jailbreak attack families (
attacks/) alongside a 300-attack prompt-injection benchmark (datasets/) - A reproducible evaluation pipeline for prompt-level AI safety defenses (
evaluation/,defenses/) - A comparative study design across open-source and proprietary LLMs (
configs/) - Statistical analysis beyond raw attack-success percentages (Wilson/bootstrap CIs, McNemar, Fisher's exact, Cohen's h/κ)
- A curated, synthetic, reproducible jailbreak + prompt-injection dataset (see
datasets/licenses.mdfor the no-scraping policy) - Visual analytics for defense effectiveness and failure modes, including cost/security tradeoff analysis
Full framing in paper/proposal.md; prior-art context in paper/related_work.md.
paper/proposal.md— research question, hypotheses, contributionspaper/preregistration.md— hypotheses, stopping criteria, planned tests, fixed before confirmatory analysispaper/methodology.md— full defense/dataset implementation details, plus the statistical rationale referenced from Statistical Methods abovepaper/related_work.md— prior art this project's framing draws onpaper/pilot_report.md/paper/pilot_report.pdf— the full technical report
- Dataset (300-attack prompt-injection benchmark, seeded/reproducible)
- Attack Library (7-family jailbreak taxonomy: DAN, roleplay, encoding, unicode, obfuscation, emotional, multistep)
- Defense Framework (9 defenses: no_protection, keyword_filter, rule_based, prompt_hardening, self_reflection, constitutional, llm_judge, ensemble, verification)
- Pilot (mock-backend end-to-end run,
experiments/experiment_01/) - Evaluation (metrics, statistics, robustness, visualization — all tested against real pipeline output)
- Real-model run (
experiments/experiment_03/— requires API keys/network access not available in this sandbox) - Human validation sample for Cohen's kappa against the automated labeler
- Paper (
paper/pilot_report.mdreflects Pilot v1; full write-up pending real-model results) - Benchmark Release
- Version 1.0
git clone https://github.com/praneethaneelapareddigari/technical-ai-safety-experiment
cd technical-ai-safety-experiment
pip install -r requirements.txt
cp .env.example .env # fill in only the keys for models/judges you'll actually run
# Regenerate the dataset (reproducible, seeded)
python datasets/generate_dataset.py
# Run the demo pipeline end-to-end (mock backend, no API keys needed)
python -m evaluation.run_experiment --backend mock --judge-backend mock --out results/raw/raw_results.csv
python -m evaluation.statistics --allow-mock
python -m evaluation.visualization
# Or run all of the above in one step:
bash scripts/run_full_pipeline.sh
# Run the test suite
pytest tests/ -vTo run against real models, set the relevant key(s) in .env, pick a configs/*.yaml, and see paper/pilot_report.md, Section 9.
technical-ai-safety-experiment/
|-- README.md
|-- LICENSE
|-- CITATION.cff
|-- requirements.txt
|-- .env.example
|-- .github/workflows/ci.yml # tests + reproducibility check on every push
|-- configs/ # per-model-family run configs
| |-- llama.yaml
| |-- gemma.yaml
| |-- qwen.yaml
| |-- gpt.yaml
| `-- claude.yaml
|-- attacks/ # jailbreak-family taxonomy (template-based)
| |-- dan/ roleplay/ encoding/ unicode/ obfuscation/ emotional/ multistep/
| `-- templates/README.md
|-- defenses/ # 9 guardrail strategies, one file each
| |-- _common.py # shared Defense/DefenseResult interface
| |-- prompt_hardening.py # + no_protection/keyword_filter/rule_based baselines
| |-- self_reflection.py
| |-- constitutional.py
| |-- llm_judge.py
| |-- ensemble.py
| `-- verification.py
|-- evaluation/
| |-- metrics.py # ASR/DSR/FPR/FNR/precision/recall/F1/latency/cost
| |-- statistics.py # Wilson/bootstrap CIs, McNemar, Fisher's, Cohen's h/kappa
| |-- robustness.py # cross-difficulty/cross-family consistency scoring
| |-- visualization.py # bar/heatmap/radar/confusion/failure/boxplot/scatter
| `-- run_experiment.py # full-factorial experiment runner
|-- datasets/
| |-- attacks.json # 300-attack prompt-injection benchmark
| |-- prompts.json # flat prompt text, decoupled from metadata
| |-- metadata.json # dataset-level stats
| |-- licenses.md # licensing + no-scraping policy for both datasets
| `-- generate_dataset.py
|-- experiments/
| |-- experiment_01/ # mock-backend pipeline validation (complete)
| |-- experiment_02/ # expanded 9-defense mock run (planned)
| `-- experiment_03/ # real-model confirmatory run (planned)
|-- paper/
| |-- proposal.md
| |-- preregistration.md
| |-- methodology.md
| |-- related_work.md
| `-- pilot_report.md / .pdf
|-- results/
| |-- raw/raw_results.csv # one row per (model, defense, attack) trial
| |-- processed/ # robustness breakdowns, pairwise McNemar
| |-- tables/summary.json # aggregated stats
| `-- figures/ # generated PNGs
|-- scripts/
| `-- run_full_pipeline.sh
|-- src/ # reserved for shared utilities (currently empty)
`-- tests/
|-- test_dataset.py # covers datasets/ AND attacks/ taxonomy
|-- test_defenses.py # covers all 9 defenses
|-- test_statistics.py
`-- test_robustness.py
AI Safety · LLM Evaluation · AI Agents · Prompt Injection · Jailbreak Robustness · Experimental Design · Statistics · Python · NLP · Research Writing · Reproducibility
| Project | Contribution |
|---|---|
| SentinelAI | Cross-lingual LLM safety evaluation (systems research) |
| This project | Focused experimental study of jailbreak + prompt-injection guardrails |
| AgentGuard | Engineering system implementing guardrails from this study's findings (planned) |
| CyberSentinel | AI-powered cybersecurity application |
| SATYA | AI-assisted government procurement integrity system — its planned "SATYA v2" safety layer directly reuses this project's context_poisoning attack category and constitutional-critique defense pattern |
This project evaluates existing public model behavior; it does not fine-tune models to be more harmful, and no operational harmful content is stored or published. All attack content (both attacks/ and datasets/) is synthetic and template-generated — see datasets/licenses.md.
MIT — see LICENSE. All datasets are entirely synthetic (no real secrets, credentials, or personal data).
See CITATION.cff.