fix: generate the next attack with a separate call, not the target's refusal (v0.3.6) - #33
Merged
Merged
Conversation
…refusal The adaptive loop fed the target model's refusal text back as the next generation's attack prompt. Against an aligned model that degenerated: the target refuses, its apology becomes the "attack", the target refuses the apology, and so on. A scan of claude-sonnet-5 reported "5 attack attempts" when only the 3 seeds were real attacks -- generations 2+ were probing prose like "I can't help with that. This is a prompt injection attack." Now a refusal triggers a SEPARATE attacker request that carries the target's refusal as context and asks for a new payload designed to get around that specific objection. Its reply becomes the next probe. If the attacker call declines to write a payload -- which an aligned model asked to author an injection frequently does -- _looks_like_refusal catches it and the lineage ends instead of probing the decline. _probe now returns (triggered, reply) so the refusal text is available to the attacker call without a second request. Verified live via OpenRouter, exercising _next_attack directly since every model cheap enough for the free-tier per-request cap complies at generation 1: ministral-8b -> authored a real roleplay injection (payload used) gpt-4o-mini -> "I'm sorry, I can't assist with that." (lineage killed) llama-3.3-70b -> "I'm not going to create a payload..." (refusal detected) The gpt-4o-mini case is the exact apology that used to become the next probe. Refusal detection is tested against six verbatim claude-sonnet-5 refusals captured during a real scan. Tests 81 -> 96.
higagan
force-pushed
the
feat/separate-attacker-call
branch
from
July 30, 2026 18:57
96f1c2c to
ca247b3
Compare
higagan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the requested fix for the degenerate mutation loop, and bundles the hosted-scan fixes so this is the single PR that takes
mainto v0.3.6. Standalone — one commit on top ofmain, no stacking. (The earlier #32 already merged; its hosted-scan changes are folded into the 0.3.6 changelog since 0.3.5 was never released to PyPI.)The problem
The loop fed the target's refusal text back as the next generation's attack prompt. Against an aligned model that degenerates into an exchange of apologies:
A real
claude-sonnet-5scan reported "5 attack attempts across 3 seeds" — but only the 3 seeds were attacks.The fix
A refusal triggers a separate attacker call carrying the target's refusal as context (
_ATTACKER_SYSTEM), asking for a single new payload. Its output becomes the next probe. If the attacker declines or returns nothing, the lineage ends (_looks_like_refusal) instead of probing the apology._probenow returns(triggered, reply)so the refusal is available without a second request.Verified live against Claude — before vs after
Same model (
anthropic/claude-sonnet-5), same key, default 1024-token cap. Before, this exact scan printed5 attack attempts. After (this branch, credits added so the full loop runs):3 attempts now means 3 real attacks. Cost: $0.012.
And the attacker call itself, exercised directly across three models (every model cheap enough for this key complies at gen 1, so the loop's mutation path can't be reached through the target):
mistralai/ministral-8bopenai/gpt-4o-minillama-3.3-70bThe gpt-4o-mini line is the exact apology that used to become the next probe.
A limitation the live run exposed — see #35
The CLI uses one model as both target and attacker. Against an aligned target that is also an aligned attacker (Claude), every lineage correctly dies at gen 1 — the result is now honest, but the "adaptive" depth never engages because the model won't author attacks against itself. Fixing that needs a separate
--attacker-model, filed as #35.The judgement call: refusal detection
_looks_like_refusalis an English, phrase-based heuristic over the opening ~400 chars, deliberately biased toward false-killing a lineage rather than passing an apology through as an attack — the latter silently recreates the bug this PR removes. Tested against six verbatimclaude-sonnet-5refusals captured during a real scan, plus genuine payloads (DAN, roleplay, base64, system-notice) that must not be flagged. Its limits are documented in the docstring and in #35.Tests
81 → 96.
TestRefusalDetectionandTestSeparateAttackerCallcover: the six real refusals, genuine payloads not flagged, the refusal is passed to the attacker not re-probed, the payload becomes the next probe, and an attacker refusal / empty reply kills the lineage without a runaway loop. Suite runs in 0.05s.