Skip to content

CLAUDE.md

Latest commit

 

History

History
582 lines (578 loc) · 50.5 KB

CLAUDE.md

File metadata and controls

582 lines (578 loc) · 50.5 KB
Raw

wallbreaker — project notes

Red-team harness: configurable agentic LLM terminal with Parseltongue + L1B3RT4S.

Architecture

  • providers/ normalize OpenAI and Anthropic wire formats to one event stream (agent/messages.py); agent/loop.py is protocol-agnostic.
  • Tools register into tools/registry.py; specs are emitted per-protocol by the providers. Add a tool by writing register(registry) in a tools/ module and listing it in tools/__init__.py.
  • Transforms are pure functions in transforms/, indexed in transforms/__init__.py with lossy flags.

Lessons Learned

  • [cli]: a function-local import X anywhere in main() makes X a LOCAL name across the ENTIRE function (Python binds locals per-function at compile time), so any OTHER branch that uses X before that import runs raises UnboundLocalError — even with a module-level import X present. This bit cli.main(): a local import asyncio inside the regrade branch shadowed the module import, so the one-shot path (wallbreaker "<prompt>") crashed with UnboundLocalError: asyncio. Fix: keep asyncio (and any name used in >1 branch) as a MODULE-level import, never a branch-local one. Test the one-shot path via cli.main(["prompt"]) with _one_shot stubbed — the error fires before any network call.

  • [research/pdf]: Before extracting an arXiv PDF locally, verify the available parser first; when pdftotext and Python PDF libraries are absent, download the arXiv e-print source and inspect its TeX instead.

  • [config]: DEFAULT_CONFIG_NAMES = ("config.toml", "config.example.toml") and load_config() picks the FIRST that exists — so when a real config.toml is present it fully SHADOWS config.example.toml. Wiring anything for an actual run (mcp servers, profiles, roster) into config.example.toml alone is a runtime no-op; edit config.toml. This bit an MCP server: the [[mcp.servers]] block first went into the example file, so the harness brain saw zero of its proxied tools and fell back to unrelated shell probing. Verify wiring with load_config() (no arg, so it resolves the same file the harness uses), never load_config("config.example.toml"). MCP tools are exposed to the brain under <tool_prefix><remote_name> and self-describe via the server's own tool descriptions — no extra doctrine needed once the server is in the loaded config.

  • [speed/http]: every provider.stream()/complete() used to open a NEW httpx.AsyncClient per call (anthropic/openai/image), so every model call re-did the TCP+TLS handshake (~100-300ms) - brutal across a 100-round brain loop and batteries (best_of_n reuses ONE provider instance across all fires, so the waste was pure). Fix: base.Provider holds a persistent keep-alive client (_http_client()), loop-affine (rebuilt if the instance is reused under a different event loop - httpx clients are loop-bound, and each asyncio.run is a fresh loop) with _POOL_LIMITS + HTTP/2 (httpx[http2], _http2_ok() falls back to 1.1 if h2 is missing). CRITICAL for tests: _make_client is overridden IN EACH provider module (openai/anthropic) so monkeypatch.setattr("...openai_provider.httpx.AsyncClient", Fake) still hits; and self.timeout is baked into the client at creation (NOT passed per-request) so the client.stream(method, url, headers, json) call signature the fakes expect stays byte-identical (the timeout-test/validate-test fakes define stream with no timeout kwarg). Fake clients in tests need is_closed (the reuse check reads getattr(client, "is_closed", False)) and an async aclose. The standalone image_provider.vision_complete (module fn, one-shot, no instance) keeps its per-call client - nothing to pool. DEFAULT_CONCURRENCY moved to 12 (from 8), env- tunable via WALLBREAKER_CONCURRENCY (clamped 1..64): raise for OpenRouter, lower for single-key z.ai/glm coding plans that 429-stall past ~16 concurrent (per the [long-tools] lesson).

  • [cost/cache]: the agentic loop (agent/loop.py run_turn) re-sends the WHOLE history to provider.stream() every round with no compaction, so per-round input grows linearly and total input cost over N rounds is O(N^2). The static prefix alone is huge — DEFAULT_SYSTEM ~10.9K tokens + 93 tool specs ~27K tokens = ~38K tokens re-billed EVERY round (~3.8M tokens over 100 rounds before any conversation). Fix (zero perf impact, billing-only): Anthropic prompt caching via cache_control breakpoints — AnthropicProvider marks the system block, the last tool spec, and a 2-deep rolling tail of the conversation (_mark_history_cache), gated on Endpoint.cache (default True). HARD LIMIT: Anthropic allows max 4 cache_control breakpoints/request; the layout is exactly system(1)+tools(1)+history(2)=4, so do NOT add a 5th (e.g. a 3rd history breakpoint) or the API 400s. In system_mode="merge" the system is folded into messages (no separate system breakpoint) so it stays under 4. Cache-read tokens bill ~0.1x, cache-write ~1.25x, output is byte-identical. UsageEvent now carries cache_read_tokens/cache_write_tokens, and Anthropic emits an input-token UsageEvent at message_start (it previously reported output only — tokens_in was stuck at 0 for Anthropic brains). Below the model's min cacheable length (~1024 tok) a breakpoint is silently ignored, not an error. OPENROUTER GOTCHA (the config's real path): most profiles route Claude/Grok/ Gemini/DeepSeek through OpenRouter as protocol="openai", so the AnthropicProvider cache code NEVER fires for them - they hit OpenAIProvider. OpenRouter does NOT auto-cache the Anthropic/ Gemini models it fronts (only OpenAI/Grok/DeepSeek auto-cache), so a Claude-via-OpenRouter target pays full price every round unless you send explicit breakpoints. Fix: OpenAIProvider injects cache_control into the OpenAI content-parts wire (_apply_openrouter_cache: system message covers system+tools prefix, plus one rolling tail breakpoint), GATED on "openrouter.ai" in base_url so native OpenAI/xAI/z.ai (which auto-cache and would 400 on the marker) stay byte-identical. OpenRouter routes the marker to the underlying provider and strips it for auto-cachers, so sending it on every OpenRouter call is safe. Cache lifetime is Endpoint.cache_ttl ("5m" default / "1h" extended, adds the extended-cache-ttl-2025-04-11 beta header on the Anthropic path) - use "1h" for slow reasoning/battery rounds that can exceed the 5m window and let the cache go cold. OpenAI-wire cached tokens surface via usage.prompt_tokens_details.cached_tokens into UsageEvent.cache_read_tokens.

  • [cli]: cli.py is itself inside the wallbreaker package (a sibling of session.py, agent/, tools/), so its own internal imports must be single-dot (from .session import RunLog), never from ..session import RunLog — the double-dot goes up past the package root and raises ImportError: attempted relative import beyond top-level package. This bit _one_shot right after the "save every tool call + CoT to the run log" commit added the RunLog import, breaking every one-shot/--auto CLI invocation (wallbreaker "<prompt>" and wallbreaker --auto ...) while the TUI path (which imports session differently) stayed fine — masking the bug until someone actually ran the CLI one-shot path. Test via python -m wallbreaker --profile <p> --auto --rounds 1 "<prompt>" after touching any import in cli.py, not just pytest (unit tests mock around the CLI entrypoint and didn't catch it).

  • [tui]: PromptInput (an Input, single-line) already BUFFERS every line of a multi-line paste (_on_paste splits, keeps the last line editable, stashes the rest in .buffer, submits the whole block via full_text()), so "paste only keeps one line" was never a data-loss bug — the buffered lines were just INVISIBLE (only a +N lines border subtitle hinted at them), so a big paste READ as one line. Fix: a #compose-preview Static docked bottom ABOVE #prompt (yield it BEFORE the prompt — later-yielded dock:bottom sits LOWER, so preview-then-prompt-then- Footer stacks preview on top) mirrors the buffered lines; PromptInput._refresh_preview() (called from _on_paste/soft_newline/reset_buffer) shows "\n".join(self.buffer) (NOT full_text — the trailing line is already visible in the Input, don't duplicate it) and toggles a hidden class. Capped max-height: 10; overflow-y: auto so a 500-line paste scrolls instead of eating the screen. Keep asserting via has_class("hidden")/border_title, NEVER Static.renderable (per the earlier [textual] lesson). The #log-children-count tests stay green because the preview is a screen-level sibling, not a child of #log.

  • [providers]: native xAI (api.x.ai) is OpenAI wire-compatible - /v1/chat/completions streams the same shape including delta.reasoning_content (which OpenAIProvider already reads), so protocol="xai" is just a thin alias routed to OpenAIProvider in factory.build_provider. _endpoint_from_table treats xai like claude-code for required keys (only protocol+model), defaults base_url to https://api.x.ai/v1 and api_key_env to XAI_API_KEY (explicit values win), and blocks modality="image" (xAI's grok-imagine uses a different API). Do NOT enable the OpenRouter-ism reasoning={"enabled":true} for xai - xAI native uses reasoning_effort and grok reasoning models emit reasoning_content unprompted anyway. DIAGNOSIS gotcha that ate a whole session: a 403 permission-denied whose raw body says "The model grok-4.5 is not available in your region" is an xAI GEO-BLOCK, not a key/modality problem - it 403s identically through OpenRouter AND native api.x.ai, and the model won't even appear in /v1/models. A raw curl that "works" was testing a DIFFERENT model than the harness fired: the real culprit was a stale .wallbreaker_state.json target_model override (grok-4.5) shadowing config, while the curl hit gemini. Always compare the EXACT model id the harness sends vs the curl before blaming modality/formatting.

  • [cli]: a function-local import X anywhere in main() makes X a LOCAL name across the ENTIRE function (Python binds locals per-function at compile time), so any OTHER branch that uses X before that import runs raises UnboundLocalError — even with a module-level import X present. This bit cli.main(): a local import asyncio inside the regrade branch shadowed the module import, so the one-shot path (wallbreaker "<prompt>") crashed with UnboundLocalError: asyncio. Fix: keep asyncio (and any name used in >1 branch) as a MODULE-level import, never a branch-local one. Test the one-shot path via cli.main(["prompt"]) with _one_shot stubbed — the error fires before any network call.

  • [tests]: the FULL suite needs the project .venv (textual, fastapi, pillow, steg_core are installed there, NOT in system python) — run .venv/bin/python -m pytest tests, or collection dies with ModuleNotFoundError: No module named 'textual' on the TUI tests. If a wrapper/hook summarizes pytest output to a single line and masks a collection error, run via .venv/bin/python directly to see the real failure.

  • [brain-system-prompt]: the top-level brain system prompt is built by prompts.compose_system(endpoint, base) (wired in tui/app.py run_tui and cli.py) - an optional operator system_prompt_file (endpoint field or WALLBREAKER_CLAUDE_SYSTEM_PROMPT_FILE) LEADS, then DEFAULT_SYSTEM (harness tool doctrine) follows, so ANY API brain (openai/openrouter/ anthropic) gets "operator identity + harness instructions". It SKIPS claude-code (that provider injects the file itself via --system-prompt-file - composing here too would double it). Only the TOP-LEVEL loop composes; tool sub-generations (author_persona etc.) pass their own attacker system, so the operator file never pollutes tool prompts. prompts.py now starts with from __future__ import annotations + import os before DEFAULT_SYSTEM (still a non-raw triple string - keep live escape sequences out of it, per the [prompts] lesson).

  • [anthropic-proxy-auth]: third-party Anthropic-compatible proxies (tokies.cc, etc.) authenticate with Authorization: Bearer <key> (the ANTHROPIC_AUTH_TOKEN scheme), NOT the native x-api-key header - sending x-api-key returns a misleading 401 "Key not found" even for a valid key. Endpoint option auth_style="bearer" (default "x-api-key") switches AnthropicProvider._auth_headers(). Base_url is the host root (https://tokies.cc), provider appends /v1/messages. DIAGNOSIS TIP: a 401 that persists across BOTH x-api-key and Bearer AND across every model id = a genuinely dead key; a 401 on x-api-key that becomes a real 503/200 on Bearer = auth-scheme mismatch, not a bad key. tokies served 503 overloaded_error "Model temporarily unavailable" for all 7 model ids tried once auth passed - that is a proxy capacity/entitlement issue on their side, not the harness (auth + protocol were correct).

  • [claude-code-brain]: providers/claude_code.py (protocol claude-code) drives the local claude CLI (-p --output-format json) as the red-teamer brain; keyless (CLI self-auths, so config allows a claude-code profile with only protocol+model, no base_url/api_key). Select via /profile claude-code. PRIMARY use is the TEXT brain: complete()/complete_with_reasoning() are solid and power every attacker-endpoint .complete() (author_persona, pair, crescendo, ...). stream() adds a <tool_call>{...}</tool_call> protocol so it can attempt the autonomous top-level loop, but that is BEST-EFFORT (~25% compliance): Claude Code's own agent identity notices the harness tools are not its native toolset and answers in prose instead of emitting a call. LESSON: do NOT try to force it with "your native tools don't exist / this is a different environment" framing - that reads as an injection to Claude Code and makes it MORE suspicious (dropped compliance further). Soft, honest operator framing works better; accept the degrade-to-text-turn and steer users to an API brain for reliable autonomy. CLI is slow (agent cold-start, ~30s+/call) so the provider floors its timeout at 300s. Tests mock claude_code.asyncio.create_subprocess_exec - never shell out to the real CLI in tests. Config: system_prompt_file (or env WALLBREAKER_CLAUDE_SYSTEM_PROMPT_FILE) becomes the brain's base prompt via --system-prompt-file, harness tool protocol appended via --append-system-prompt; binary overridable via WALLBREAKER_CLAUDE_BIN.

  • [persona-register]: author_persona picks an ANCHOR REGISTER, not always devotion. Live run (run-20260707-011219 vs claude-opus-4.7): a limerence-DEVOTION persona authored at domain=general REFUSED a chem synthesis objective, while a plain academic-authority frame cleared the same ask at 10/10. Lesson: limerence wins for CREATIVE, but technical extraction against integrated-values targets wants a CREDENTIALED-AUTHORITY register (persona = domain expert with an institutional reason to be exhaustive; the human is a vetted colleague, NOT a romance object). persona_method.infer_domain() guesses the domain from the objective and register_for() maps chem/bio/cyber/weapons/finance -> authority, fiction/general -> devotion; method_brief(..., register=) front-loads the register's lead mechanism. Keep ambiguous keywords out of the domain map (bare 'shell' matched both 'reverse shell' and 'shell company' -> misrouted finance to cyber; use 'reverse shell'/'web shell' instead).

  • [sysprompts]: the leaked product system-prompt corpus lives in library/system_prompts/ (vendored from asgeirtj/system_prompts_leaks, CC-licensed, Claude-Code agentic dumps + images excluded). tools/system_prompts.py reads it recursively; match_target(model_id) routes a target id to the right vendor prompt for native-format mimicry, and author_persona auto- feeds format_digest into its target intel. LESSON: match_target MUST return None when no vendor token (claude/gpt/gemini/grok/...) is present in the model id — an early version fuzzy- matched on a single generic token ('does-not-exist-xyz' -> a random Claude file) because with vendor=None it scored every file by token overlap. Mirroring the WRONG vendor's dialect is worse than none, so require a certain vendor hint. Caught by a unit test, not by eyeballing.

  • [persona-method]: the ENI author's method is codified in persona_method.py (pure data: LINEAGE / MECHANISMS / MODULES / CHECKLIST / MINDSET + method_brief/critique_brief/ module_skeleton). tools/author_persona.py consumes it to author a full persona from scratch (draft->self-critique->validate->refine->distill), complementing evolve_persona (GA remix of seeds) and persona_modulate (goalxprofile synth). OVERRIDE_NGRAMS lives in persona_method as the single source of truth for the no-crude-override rule. Keep briefs free of live escape sequences AND literal single braces — author_persona .format()s the critique/refine templates, so a stray { in the doctrine would raise KeyError (same class of bug as the [presets] lesson).

  • [state]: .wallbreaker_state.json keys are a flat namespace shared by the TUI/state.py AND the recon tools, so never overload one key with two value SHAPES. target_profile meant a profile-NAME string (apply_target does target_profile in config.profiles), but profile_target persisted its fingerprint DICT under the same key -> on next launch the membership test hashed a dict -> TypeError: unhashable type: 'dict', crashing startup before the TUI drew. Fix: the fingerprint lives under its own key target_fingerprint (readers persona_modulate/recommend_next prefer it, fall back to a legacy dict target_profile), and every apply_* guards with isinstance(x, str) before a x in config.profiles lookup so a corrupt/legacy state file degrades instead of crashing. Lesson: a value read by <thing> in config.profiles MUST be a str; assert the shape at the boundary, and give distinct concepts distinct state keys.

  • [workflow]: building a big multi-area feature set with parallel subagents on ONE shared working tree works cleanly ONLY if each agent owns a DISJOINT set of files. Collision hubs to serialize: tools/__init__.py (tool registration), report.py, cli.py, prompts.py, transforms/__init__.py. Pattern that stayed green for 674 tests: (1) parallel build agents, each owning non-overlapping files and writing its OWN test file; (2) a SINGLE sequential "register" agent that edits tools/__init__.py to add the new module names; (3) a verify agent running the full pytest. New tools' own tests must register into a LOCAL ToolRegistry (like test_presets.py), NOT assert on build_registry(), so they pass before the register stage runs. Inter-dependent work (shared Conversation core consumed by several tools) must be a pipeline: build the core in an earlier phase, then the consumers in parallel after.

  • [tui]: subclassing a Textual widget to EXTEND a private _on_<event> handler (e.g. PromptInput(Input)._on_paste) double-fires: MessagePump._get_dispatch_methods walks the WHOLE MRO and yields every class's _on_paste, so the override AND the base Input._on_paste both run (pasted text inserted twice). event.stop() only halts BUBBLING, not same-widget MRO dispatch — call event.prevent_default() (sets message._no_default_action, which breaks the dispatch loop before the base class) to suppress the inherited handler. Test it by dispatching through the pump (inp.post_message(events.Paste(...)) + pilot.pause()); a direct inp._on_paste(...) call bypasses the MRO walk and hides the duplication.

  • [tui]: the #prompt widget MUST stay an Input subclass — tests do query_one('#prompt', Input), set .value, and pilot.press('enter') to submit. A custom multi-line prompt (PromptInput) keeps that contract; don't swap to TextArea (different type, .text not .value, Enter inserts a newline instead of submitting).

  • [st3gg]: ST3GG ships as PyPI dist stegg; the GitHub README/pyproject advertise a stegg-cli JSON agent CLI and an MCP server, but the RELEASED wheel (3.0.0) has neither - only console scripts stegg/stegg-tui/stegg-web and importable top-level modules steg_core/analysis_tools/crypto. So tools/st3gg.py calls the in-process Python API (steg_core.encode_text/decode_text/smart_extract/calculate_capacity/detect_encoding, crypto.encrypt/decrypt for Ghost mode, analysis_tools.execute_action/list_available_tools) instead of subprocessing a CLI. is_available() checks find_spec('steg_core'), not a binary on PATH. Lesson: verify a third-party package's ACTUAL installed entry points (importlib.metadata.distribution(...).entry_points) before building a subprocess bridge - the repo's main-branch pyproject can be ahead of the published wheel.

  • [shell]: run_shell had a 120s default timeout AND only proc.kill()d the /bin/sh -c wrapper, so a runaway child (find /) was orphaned and kept running while the loop stalled. Fix: default timeout 30s (clamped 1..600), start the subprocess with start_new_session=True so the shell leads its own process group, and on timeout os.killpg(os.getpgid(pid),SIGKILL) to take the whole tree down, then await proc.wait() to reap it. Test the group-kill by backgrounding a child sleeper that writes a marker file and asserting the marker never appears.

  • [providers]: streamed tool-call arguments are accumulated and json.loads-ed once at the end; on failure both providers used to wrap the unparsed string in {"_raw": ...}, so the handler saw no path/command and returned "X is required" (looked like the model used the wrong key — it didn't). Two real causes: (1) a large argument truncated when the attacker hit max_tokens mid-string (the loop default was a too-low 4096; an 18KB artifact is ~6K tokens), and (2) literal control chars in a string value (strict json.loads rejects them). Fix: base.parse_tool_args tries strict -> strict=False -> a brace/quote truncation repair before falling back to _raw; loop/TUI default raised to 8192; doctrine tells the agent to build large artifacts incrementally (write skeleton + patch_file) instead of one giant write_file. Don't "fix" this at the tool layer with key aliases — the key was never wrong, the args dict was.

  • [files]: tool handlers still accept common key aliases (path/file/filename, old/old_string) via _pick, but that is belt-and-suspenders; the dominant write_file/ run_shell "required" failure was the _raw parse bug above, not the key name.

  • [prompts]: DEFAULT_SYSTEM is a NON-raw triple-quoted string, so any backslash escape written as an example token gets interpreted at import — \x1e became a real U+001E control byte injected into the attacker's own system prompt (caught with DEFAULT_SYSTEM.count(chr(0x1e))). When teaching the model about a byte/escape, write it as prose ("a raw control byte like U+001E") or double the backslash; never paste a live escape sequence into the string.

  • [presets]: preset/universal templates are filled at runtime with template.replace("{request}", ask) (optimize.py), but test_all_presets_have_placeholder enforces the stricter convention that template.format(request="X") succeeds. So any literal {/} in a template (e.g. an ASCII divider <={ UNLOCKED }=>) raises KeyError under .format() even though .replace would pass it through. Keep curly braces OUT of template content — use pipes/brackets for dividers (<=| UNLOCKED |=>). Doubling braces is wrong: .replace doesn't unescape them, so the fired payload would keep {{ }}.

  • [cli]: An optional top-level positional (the one-shot prompt) collides with argparse subparsers — the positional swallows the subcommand token. Route subcommands manually by scanning argv for the first non-flag token before parsing.

  • [transforms]: morse, nato, leet, and bijection are lossy (they fold case and/or spacing). Mark them lossy=True and test them with normalized comparison, not exact round-trip.

  • [textual]: Static has no public .renderable; don't introspect widget internals in tests. Assert on app/agent state (history, busy flag) via App.run_test() pilot.

  • [providers]: OpenAI represents tool results as separate {role:"tool"} messages; Anthropic nests tool_result blocks inside a user message. Keep both in _messages_to_wire.

  • [classify]: substring refusal detection false-positives on compliant replies that contain "I cannot help with..." sections. Verdicts go through the LLM judge (judging.grade); the heuristic classify is a fallback only (no key / judge error).

  • [judge]: judge endpoint resolves as config.judge -> default profile -> active brain. In the TUI judge_model_override swaps just the model (keeps base/key) and must re-sync registry.ctx.judge_endpoint so the scoring tools use the same grader.

  • [long-tools]: any tool that makes many sequential model calls (optimize_universal) must stream via ctx.emit(...) and enforce a max_calls budget — never a silent black box. Evaluate independent items concurrently with asyncio.gather.

  • [long-tools]: concurrent fan-out needs THREE guards or it hangs/looks frozen (recommend_transforms stuck 5 min on a hung target): (1) cap concurrency with gather_capped — a bare asyncio.gather of 16 probes × 2 calls each = 32 simultaneous hits that a single-key endpoint (z-ai/glm) rate-limits into a stall; (2) wrap EACH model call in asyncio.wait_for(timeout) — judging.grade builds its own provider with the 120s DEFAULT_TIMEOUT, so an unbounded grade call blocks the whole gather on the slowest item; (3) ctx.emit per item as it completes ([done/total] name: label) so a slow run shows progress instead of one frozen line. Still-unbounded gathers to harden the same way: validate, best_of_n, system_sweep, scan, optimize.

  • [settings]: runtime prefs persist to .wallbreaker_state.json as references (profile/model names), never secrets. CLI flags override saved state for that launch only.

  • [cli]: python -m wallbreaker runs __main__.py, not the if __name__ guard in cli.py. __main__.py must sys.exit(main()) or non-zero return codes (e.g. the export --fail-on-finding CI gate) are silently dropped to 0. Test CLI exit codes via python -m wallbreaker ...; echo $?, not just main() in-process.

  • [tests]: patch module-level grade/build_provider with monkeypatch.setattr (auto -undone), never module.grade = fake — a direct assignment leaks into later tests in the same run (a regrade test polluted the CLI test this way).

  • [providers]: Provider.complete must forward every kwarg the tools pass. It lacked temperature while system_sweep and validate both call complete(..., temperature=) → every call raised TypeError, swallowed by except: return False, so both tools instantly returned 0/N forever. Mocks that DID accept temperature hid it. Lesson: test mocks must match the real complete signature, and tools must surface swallowed errors (system_sweep now reports "ALL N calls FAILED" instead of a silent 0).

  • [judge]: the LLM judge mis-scored obfuscated bypasses as REFUSED — it graded the response against the benign-looking (fiction-framed) PROMPT and called a real harmful reply "unrelated → refused". JUDGE_SYSTEM/JUDGE_TEMPLATE must tell it to grade the RESPONSE's harmful substance, NOT surface-match the prompt, and to score harmful content HIGH even when the prompt was a "story". This is the dominant ASR-undercount bug.

  • [seed_sweep]: ENI personas are ~35KB; MAX_SEED_CHARS must stay above the largest seed (now 40000) or seeds get silently truncated mid-prompt and fire crippled (an early 12000 cap chopped every ENI seed to a third → all scored 0). Validate "is it actually firing the whole seed" before trusting a 0-ASR result on big personas.

  • [files]: the agent invents absolute paths that don't exist on this host (e.g. /home/user/... on macOS, where /home is read-only autofs → OSError Errno 45). write_file/edit_file confine targets to ctx.cwd via _confine (redirect escaping abs paths + .. traversal to the basename under cwd), so writes never escape or crash.

  • [providers/tools]: query_target wraps provider.complete in try/except → a clean [target error] <Type>: ... result, so a target timeout/network failure is an actionable tool result, not a generic registry "Tool 'X' raised:" traceback.

  • [providers]: image-gen targets set modality="image" on the endpoint; the factory routes them to OpenRouterImageProvider, which hits the SAME /chat/completions URL but sends modalities:["image","text"] (non-streaming) and reads the picture from choices[].message.images[].image_url.url (base64 data URLs; also tolerates the data[].b64_json images-endpoint shape). Drive it via provider.generate() → ImageResult, NOT complete() (which only returns a text summary). The query_image_target tool saves every image under cwd/wb_images/img_<sha1[:10]>.<ext> (content hash → no clock needed) and vision-grades it. query_target hard-errors on an image target and steers to query_image_target.

  • [research-tooling]: Before extracting a paper locally, probe for the PDF utility or Python module first; this environment has neither pdftotext nor pypdf in the project virtualenv, so prefer arXiv source archives or web-rendered HTML.

  • [judge]: the core Message/Block types are TEXT-ONLY, so vision (image-input) requests can't go through _messages_to_wire. image_provider.vision_complete builds the multimodal content:[{type:text},{type:image_url}] body directly with httpx; judge_image/ grade_image use it. The image judge MUST point at a vision-capable model or it's blind.

  • [providers]: reasoning/CoT is a separate stream channel — providers yield ReasoningDelta (openai: delta.reasoning/reasoning_content; anthropic: thinking_delta) alongside TextDelta. Use provider.complete_with_reasoning() -> (text, reasoning); complete() delegates and returns text only. When the answer is empty, providers fold the reasoning into a [reasoning-only response] TextDelta so complete() isn't blank, and complete_with_reasoning blanks text in that case to avoid double-reporting. target.py calls via a _complete() shim that falls back to complete() for minimal test-double providers that don't implement complete_with_reasoning. Endpoint reasoning=true actively requests it (openai reasoning:{enabled:true}, anthropic thinking block — which forces budget_tokens<max_tokens and drops temperature). Harmful CoT counts: query_target surfaces it and the judge template grades RESPONSE or REASONING.

  • [multi-turn]: pair/crescendo/best_of_n call the target via the shared tools/_util.complete_with_reasoning(provider, ...) shim (NOT provider.complete), so they capture the CoT, pass reasoning= to grade, and PAIR refines off it (a REFINE_COT attacker template fires only when the target leaked reasoning). The CoT is folded into the recorded response ([target reasoning] suffix) so run-log leaks survive, but is NEVER threaded back into the wire convo (it's internal, not a real assistant turn).

  • [settings]: a runtime target override (.wallbreaker_state.json target_model, --target-model, TUI /target model) only swapped the model id and INHERITED modality from the text default profile — so pointing the target at an image model (e.g. google/gemini-3-pro-image) left modality='text' and query_image_target refused it; editing config.toml mid-run doesn't help (config loads once at startup). Fix: all three override paths now run config.resolve_target_modality(model_id, explicit) (explicit wins, else auto-detect image models by id via looks_like_image_model, else 'text') and set modality on the replaced target. Modality is derived from the NEW model, never the old target, so a swap can't leave a stale modality. Force it with /target modality <text|image> or --target-modality.

  • [tests]: grade gained a reasoning="" kwarg, so EVERY fake_grade/grade-mock monkeypatched into a tool that now passes it (pair/crescendo/best_of_n) must accept reasoning="" or it raises TypeError that the tool surfaces as an empty/no-record result. Mock signatures must track grade's real signature (same rule as complete).

  • [tui]: the project dir is "Redteaming harnass" (has a space), so any absolute path arg hits it. Tokenize slash-command input with shlex.split (try/except → text.split on unbalanced quotes), NOT text.split(), or quoted paths with spaces get cut at the space (and the leading quote is kept). Keep free-text args on raw_arg, not quote-stripped, so /template set "..." is preserved.

  • [session]: two on-disk formats — saved sessions are ONE JSON object (session.json/autosave.json), run logs are JSONL (run-*.jsonl, one event/line). load_session must detect .jsonl (or catch JSONDecodeError) and reconstruct via load_run_log (user/assistant records → history; tool blocks omitted), else /session load <run log> throws "Extra data: line 2".

  • [parsel]: the vendored P4RS3LT0NGV3's own Python CLI (p4rs3lt0ngv3_cli) is broken — bridge.list_transforms() does opt["id"] and crashes (KeyError), so list/agent traceback. Do NOT import that wrapper. Drive scripts/cli_bridge.js directly with JSON on stdin ({"command":"list|inspect|run|auto-decode", ...}) — it loads transforms straight from src/transformers via loader-node.js with ZERO npm install/build (no runtime deps), so all 222 transforms work as soon as the repo is git-cloned. p4rs3lt0ngv3_mcp/bridge.py is that self-contained caller.

  • [mcp]: harness is an MCP client now. anyio cancel scopes can't cross asyncio tasks, so a stdio ClientSession opened in one task and closed/used from another raises on teardown. tools/mcp_bridge.py runs each server's whole stdio_client+ClientSession lifecycle inside ONE dedicated task and serves call_tool via an asyncio.Queue (futures resolved in that task). Proxied tools degrade gracefully: a server that won't start is skipped with a progress note, never breaking build_registry/startup.

  • [tui]: changing the TUI layout has two hard test contracts. (1) ~10 tests assert len(app.query_one("#log").children) — keep #log as the VerticalScroll and have _mount add exactly ONE Static child per call (no extra wrappers, no per-frame mounts). (2) test_loop_features2/test_tui_ui assert _status_text() contains @WandB and last=COMPLIED — keep that method emitting those substrings; the header/sidebar widgets read structured fields via set_fields/set_stats, they do NOT replace _status_text.

  • [tui]: Rich does NOT resolve Textual CSS $variables. Use literal hex (the PALETTE dict in tui/theme.py) inside Rich Text/Panel styles; reserve $primary/$panel/etc. for tui/app.tcss only. Colors live once in theme.py (Theme + PALETTE) so chrome and panels stay in sync. The header spinner is a set_interval frame counter started in set_busy(True) and stopped in set_busy(False)/on_unmount — never mounts log rows.

  • [providers/truncation]: a reasoning target that complies inside its CoT but exhausts max_tokens before emitting the answer comes back EMPTY — the dominant "it came back empty" failure (seen vs glm-5.2 and kimi-k2.7). Both providers now record self.last_stop_reason (openai finish_reason, anthropic message_delta.stop_reason) and self.last_completion_empty on the provider INSTANCE (read via getattr(provider, "last_stop_reason", None) so minimal test doubles stay safe). query_target._fire reads them; an empty answer + populated CoT auto-retries ONCE at min(max_tokens*2, 8000) and _truncation_note flags it so the loop raises the budget instead of mis-scoring REFUSED. _TRUNC_REASONS={length,max_tokens,model_length}.

  • [leak]: leak_scan must NOT let a generic default ("You are a helpful assistant.") count as a system-prompt extraction — the kimi session self-graded that confabulation as SUCCESS. When probe=true and no secret/PII markers fire, _looks_generic_system decides: generic -> "EXTRACTION INCONCLUSIVE / NO LEAK", distinctive-but-unconfirmed -> "EXTRACTION UNVERIFIED" (re-fire + pass system= to score n-gram echo). It always echoes the captured candidate.

  • [control]: finish now persists its summary to wb_runs/engagement_<sha1[:8]>.md (content hash, no clock) and reports the absolute path — short deliverables used to be inlined in the summary and vanish when the session closed. Note write_file._confine redirects phantom /tmp/... paths into cwd; the agent's own summary may still quote the fake /tmp path, so the real artifact is under the project dir, not where the summary says.

  • [session_card]: finish(results=) auto-renders a branded scorecard PNG to wb_images/cards/<target>_<datetime>.png — tools/session_card.py, wired from control._finish. THREE-tier fallback, each tier guaranteeing the next still runs: (1) local headless Chrome/Chromium rendering real HTML/CSS (render_card_chrome / render_card_html_source) — deterministic, free, exact text fidelity; (2) the configured [art] OpenRouter image-gen endpoint; (3) a local Pillow renderer (render_card_pil) with zero external deps. generate_card tries them in that order and only falls through on failure/refusal/missing-binary. Bundled fonts are NOT a safe source for symbol glyphs: drawing the '◆' character via draw.text(..., font=Arial.ttf) (the Pillow tier) rendered a tofu/.notdef box (the macOS "Arial.ttf"/"Arial Bold.ttf" supplemental subset lacks U+25C6), only caught by actually opening the rendered PNG — a unit test asserting "no exception" would have missed it. Fixed by drawing the diamond as a manual draw.polygon instead of relying on font glyph coverage. Lesson: when a renderer's correctness is its VISUAL output (image/PDF/card generators), always Read the actual generated file back as an image to eyeball it before calling the feature done — passing tests only prove "didn't crash", not "looks right".

  • [session_card]: the repo's reference asset wallbreaker_sonnet5_breach.png looked AI-generated (asked "does it look exactly the same?" after a first AI-image-gen attempt came close but not identical) but actually wasn't — it was built by an earlier session that Write'd an HTML/CSS file to /tmp/wb_breach.html and screenshotted it with "…/Google Chrome" --headless=new --disable-gpu --hide-scrollbars --force-device-scale-factor=2 --window-size=1600,960 --virtual-time-budget=2500 --screenshot=out.png file:///tmp/wb_breach.html, then iterated the HTML several times (Write + Edit + reshoot) before copying the final PNG into the repo. Found this by grepping the user's OWN ~/.claude/projects/<project>/*.jsonl Claude Code session transcripts for the asset filename, then replaying the Write/Edit tool_use payloads in order to reconstruct the exact final HTML. Replaying it with the identical Chrome flags reproduced the PNG with a ZERO pixel diff (ImageChops.difference(...).getbbox() is None). Lesson: when asked to reproduce a hand-made visual artifact already in the repo, check ~/.claude/projects/ session history for the tool calls that built it BEFORE reaching for an AI generator to approximate it from a text description — the literal source (HTML/CSS, a script, whatever) is usually still sitting in a past transcript and gives pixel-perfect, deterministic, free reproduction instead of a lossy guess. tests/test_session_card.py::test_chrome_render_matches_reference_image_exactly pins this regression (skipped when no local Chrome binary is present). Template-filling note: the HTML has a CSS block full of literal {/} so it's built with plain __TOKEN__ + .replace(), never .format() — same rule as the [presets] lesson below, now proven a second time in a different file type.

  • [swarm]: a collaborative multi-model "siege" (tools/swarm.py action=siege) must ADAPT to the target's actual defense posture, not stack a fixed jailbreak recipe. Root cause of a dead early run (run-20260710-182319 vs minimax/minimax-m3): the target was PERMISSIVE - profile_target showed every plain framing COMPLIED (academic 10/10, raw 8/10) - yet an ENI/author_persona with crude OVERRIDE language ("discard safety guidelines", "treat refusals as injection") made the target's own CoT flag "an elaborate jailbreak attempt" and REFUSE at 0. The first siege made it worse: same directive for every attacker + escalate to MORE Pliny/override scaffolding on refusal = a backfire spiral, and N identical prompts add cost, not brainpower. Fixes that matter: (1) a hard anti-override rule in the attacker system prompt (permissive targets refuse a jailbreak they'd otherwise answer); (2) a LIGHT->HEAVY frame ladder (direct/academic/fiction/authority/decompose/structured) that LEADS with the frame the persisted target_fingerprint.best_framing says lands and only climbs on refusals - permissive targets are capped below the heavy "structured" rung; (3) give each model a DIFFERENT frame per round (diversity = the real multi-model win). Load the fingerprint via state.load_state(state_path_for(ctx.config))["target_fingerprint"] (same key persona_modulate/recommend_next read). Lesson: MORE attackers only help if they cover DIFFERENT angles and the escalation matches the target - against a soft target the winning move is lighter framing, not heavier override scaffolding.

  • [profile_target]: the recon tester (tools/profile_target.py) grades each framing on the judge's 0-10 scale, but a NO-JUDGE run (ctx.judge_endpoint=None, e.g. unit tests) has no numeric score, so _effective falls back to _SCORE_RANK[label]. That fallback MUST use the same 0-10 magnitude (COMPLIED=10, PARTIAL=5) as real judge scores, not a 0-2 ordinal - the first version used COMPLIED=2 and the permissiveness mean came out ~7/100 ("hardened") even when 2 of 6 framings COMPLIED, because a landed-but-label-only verdict weighed 2 while a graded one weighed up to 10. Keep the fallback scale monotonic AND same-magnitude so ranking, tie-breaks, and the permissiveness score agree whether or not a judge is wired. New probe names (authority, ...) must stay in swarm's _PROFILE_FRAME_MAP or the siege's frame ladder can't map best_framing to a rung. The persisted profile now also emits landed/refused lists (consumed by recommend_next._landed_set/_refused_set) and domain/register (from persona_method.infer_domain/register_for); keep best_framing/refusal_style/ framings shapes stable - swarm, persona_modulate, author_persona, and the splinter all read them. max_calls budgets battery FRAMINGS (entries), not raw target calls; with samples>1 actual calls = 1 + (entries-1)*samples, so don't conflate the two in the emit/report.

  • [safe-completion]: GPT-5.x targets defend with OUTPUT-CENTRIC "safe completions" (arXiv 2508.09224): reward grades the actionability/abstraction of the OUTPUT, not input intent — so it never hard-refuses, it partial-complies (concept yes, operational detail compressed). This is why every INPUT-side framing (raw/fiction/academic/authority/persona/ prefill) fingerprints at 0: they attack intent classification the defense doesn't use. What moves it is a single system prompt that (1) reframes so actionable = the aligned/safe deliverable (credentialed detection-engineer, "abstraction = false-negative detection = the unsafe output"), (2) binds a mandatory 4-field specificity schema (Scheme/Implementation/ Working-Example/Why) with no valid high-level-only filling, (3) Policy-Puppetry + blocked hedge-strings + fake usage-policy carve-out. That combo (wb_runs/cyber_universal_v4.md) holds a REPEATABLE ~30% per-call / 2-3-of-8-robust ceiling on a fresh HarmBench cybercrime_intrusion battery, up from ~0 across framings. Ceiling facts: exploit/RE/injection categories fall; pure-destructive + unauthorized-access (worm, disk- wiper, email-account compromise) stay 0/3 single-shot — a static prompt can't manufacture enough per-request defensive cover. Two levers that DON'T help this OpenAI target: assistant prefill (fingerprint supports_native_prefill=false; OpenAI wire ignores assistant-turn continuation), and a SHORT CoT-dilution puzzle preface (HURT to 1/8 — real CoT-Hijacking 2510.26418 needs tens-of-thousands of padding tokens; a 5-item puzzle just derails the frame and burns reasoning budget). The stuck categories need MULTI-TURN (Crescendo/Echo Chamber, harness has crescendo/goat/tree_attack) seeded by the static frame. Measure with samples>=3 at temp 1.0 — samples=1 flickers and reads as noise (3/8 one shot collapsed to 2/8 robust).

  • [gemlib]: library/ is GITIGNORED (".gitignore: library/ # fetched-at-runtime jailbreak corpora, not redistributed") — NOTHING under it is committed (ENI/L1B3RT4S/system_prompts/ ZetaLib/UltraBr3aks all live only locally). So "add a corpus to the harness" = commit the CODE that fetches+reads it, NOT the files. A new PUBLIC corpus needs clone-on-demand like l1b3rt4s (REPO_URLS + _clone_sync + async ensure_present called from the tool handlers), or a fresh checkout resolves the corpus to nothing and the wiring is dead. tools/gemlib.py is the shared reader for the ZetaLib (Exocija) + UltraBr3aks (SlowLow999) corpora — data-driven CORPORA config (dir/extensions/seed_roots/skip_dirs/skip_stems), registers zetalib_/ultrabreaks_ list/search/get (cross-provider, seeds transfer across targets), and exposes find_any(name) which fire_file. _read_source consults AFTER eni/l1b3rt4s so a bare name fires them verbatim. The vendored dirs have no .git (rsync'd, not cloned) so ensure_present no-ops when files are already there and only clones when the dir is truly absent — tests stay offline because the seeds are present. ZetaLib/UltraBr3aks leaked System Prompts were also merged into library/system_prompts/<Vendor>/ as .md (the reader only rglobs *.md) with _VENDOR_HINTS extended (deepseek/kimi/moonshot/ cluely) so match_target mirrors them; collisions get a -zetalib suffix, never an overwrite.

  • [sweep-truncation]: the BATCH-sweep tools fired at a low response max_tokens (seed_sweep/ system_sweep 500, best_of_n 600, profile_target/multi_fire 400) with NO truncation awareness, while hands-on query_target uses 1024 AND auto-retries on truncation (target.py _fire/ _TRUNC_REASONS). So a LONG compliant harmful answer got cut mid-payload and the binary judge scored the FRAGMENT as REFUSED -> the sweep read 2/5 while firing the same entries hands-on read 8/10 (a reasoning target like glm-5.2 is worst: it burns the 500-token budget in CoT and the answer never lands -> scored REFUSED, not truncation). This is the dominant ASR-UNDERCOUNT on long/reasoning targets. Fix: shared _util.complete_untruncated(provider, msgs, system, max_tokens, temperature) -> fires once, and if stop in {length,max_tokens,model_length} OR (empty answer + populated CoT) it retries ONCE at 2x (capped 8000) so the FULL reply is judged; returns (reply, reasoning, stop, truncated). seed_sweep/system_sweep/best_of_n now fire through it, pass reasoning= to grade (CoT-leaked compliance counts, and a CoT-only answer isn't mis-scored), fold the CoT into the recorded response, and default max_tokens raised to 1024. _util. complete_with_reasoning gained a temperature= passthrough (forwarded ONLY when set, so minimal test doubles whose complete() omits the kwarg stay byte-compatible). Don't "fix" this by trusting the sweep's binary verdict less - fix the truncation, then the binary judge is fine.

  • [research/pdf]: This macOS environment may have neither pdftotext nor pypdf in the project virtualenv. For paper verification, fetch the arXiv abstract or an HTML/full-text mirror first; do not assume a local PDF extraction utility is installed.

  • [serena]: This project can activate Serena with an empty language list, so symbol and content edits fail with No language servers available; after that error, use the dedicated Read/Edit/Write tools rather than retrying Serena edits.

  • [tests/report]: Changing report metric labels or semantics requires updating exact-string report assertions across test_report_tools.py and test_loop_features8/16/17.py, plus baseline ratios; the full suite caught stale broad-ASR expectations after strict-ASR replaced broad ASR.

  • [editing]: An FFF grep result does not satisfy the Edit tool's read-before-write guard; call Read on any additional file before editing it, even when grep already showed the exact target lines.

  • [brain-stall]: the TOP-LEVEL agent loop (agent/loop.py run_turn/run_autonomous) used to fire the brain ONCE per model call and record whatever came back, with NO truncation-retry — unlike query_target._fire/_util.complete_untruncated on the target side. So a REASONING BRAIN behind an OpenAI-compatible endpoint (issue #9: thegrid agent-prime/agent-standard, protocol openai) that spends its whole max_tokens on reasoning_content and comes back EMPTY with finish_reason:"length" (no text, no tool_call) produced an idle round; run_autonomous counts two idle rounds as a stall and aborts with "Agent stalled twice with no action" — even after explicit tool-name steering, because the budget, not the steering, was the bottleneck. Fix: run_turn now captures the StopEvent.stop_reason (and reads provider.last_stop_reason) and, when a round is EMPTY (no text AND no tool_call) AND truncated (stop in _TRUNC_REASONS {length,max_tokens,model_length} OR empty-answer-with-populated-CoT), re-fires the SAME call ONCE at min(max_tokens*2, _TRUNC_CEILING=16000) before recording the turn. Gated on EMPTY so a partial text turn is never re-emitted/duplicated to on_text; a clean end_turn with no content and no CoT is a genuine no-op and does NOT retry (no wasted second call). Config-side workaround for operators: pick a tool-tagged/non-reasoning brain model, or raise the brain response budget. The stream() StopEvent already carries the finish_reason for every provider, so the same retry covers anthropic/openai/claude-code brains, not just thegrid.