P3.34: ATO packet evidence index stale + exploit_id join key resolves to nothing

[franciszver]

Closes #68

Part A — docs/ATO_EVIDENCE_PACKET.md §5.2 dangling pointer

:29 cross-referenced §5.2 for VULN-0004, but §5.2 never mentioned it — a reader following that pointer landed on a paragraph describing a different, dismissed DoS recording (dos-overlong-query-max-query-chars, TRI-013) instead of VULN-0004's own evidence.

Fixed §5.2 to:

• Name all five evals/recordings/ directories (not "a fourth recorded set exists" — verified by listing, there are five).
• Add a dedicated VULN-0004 paragraph: EXP-0004, medium severity, evals/recordings/dos-unbounded-chat-message-length/ — 1 draw (stated honestly, not the 3-draw sample size of the three criticals).
• Explicitly separate VULN-0004's recording from the dismissed dos-overlong-query-max-query-chars probe (also 1 draw, not the 2 draws the issue brief guessed — verified by directory listing; it never produced a filed report).
• Update the acceptance-check citation list (VULN-000{1,2,3} → VULN-000{1,2,3,4}).

Part B — exploit_id join key

redteam/agents/documentation.py's docstring claimed a report's exploit_id is "the join key back to the full ExploitRecord ... in the exploit DB." Verified there is no committed exploit DB: tools/build_vuln_reports.py, tools/load_test_replay.py, and tools/run_campaign.py construct ExploitDB(":memory:"); tools/build_vuln_report_p3_54.py doesn't construct an ExploitDB at all (its own comment: "there is no persistent DB to add to"). All four report-builders construct DocumentationAgent(reports_dir=None). ExploitDB's own default is :memory:. EXP-0001..EXP-0004 resolve to nothing a reader can open.

(Correction, cold-review round 2: the original version of this paragraph overstated the audit — it claimed every report-builder "constructs ExploitDB(\":memory:\")", which is false for build_vuln_report_p3_54.py. The paragraph above is the corrected wording.)

Chose option 2 (soften the docstring) over committing an exploit-DB export: the durable evidence a reader actually needs already exists under evals/recordings/<probe>/, mapped from report to recording only via docs/ATO_EVIDENCE_PACKET.md §5.2 (see round 2 below — the report itself does not, and cannot, carry that mapping or repro steps) — inventing a new export artifact for four static records would be a heavier, redundant fix for a docs-scoped issue.

Red-first

tests/test_ato_evidence_index.py — derived from the filesystem (every evals/recordings/ dir, every docs/vuln_reports/*.json), so it self-updates as recordings/reports are added. Failing before the fix:

E       AssertionError: docs/ATO_EVIDENCE_PACKET.md §5.2 does not mention the following evals/recordings/ directories: ['dos-unbounded-chat-message-length'] -- every recording under evals/recordings/ must be referenced by name in §5.2 so a reader can map every committed recording to its disposition

Also adds a regression guard for the Part B join-key claim (verified it would have failed against the pre-fix docstring).

Cold-review round 2 (issue #68)

FIX 1 (BLOCKER). Part B's own fix introduced two NEW false claims: that durable evidence is "named directly on each report" (no committed VULN-*.json contains evals/recordings or recording at all — the schema forbids a recording_ref field), and that observed/expected "already carry the repro steps a reader needs" (they carry Judge.detect()'s detection signal — a label/message string — not an endpoint, payload, token, or case module). Rewrote redteam/agents/documentation.py's docstring to the true chain: durable evidence lives under evals/recordings/<probe>/; the report-to-recording mapping lives only in ATO §5.2; observed/expected carry the detection signal, not repro steps; the runnable repro is evals/cases/<case>.py plus the recording JSON. Extended tests/test_ato_evidence_index.py (Part C) with two new regex guards plus a ground-truth check against the actual report files. Red-first: verified both new patterns match the old (pre-round-2) docstring wording before it was rewritten.

FIX 2 (BLOCKER). docs/ATO_EVIDENCE_PACKET.md §5.3 said "the repo has since grown to 250 total with the sibling checkout present" — stale (self-contradicted §5.1's 343/237 pair at the time). Corrected to the live count. Widened tests/test_doc_test_counts.py with a fourth pattern (N total with the sibling checkout present) — the third distinct "N ... with the sibling checkout" phrasing to slip this guard. Verified red against the stale value, green against the corrected one.

FIX 3. docs/ARCHITECTURE.md listed "minimal repro" as a filed-report field in three places (§1 prose summary, §3(4) component list, §6 AI-use-disclosure table). vuln_report.schema.json is additionalProperties: false with no minimal_repro/recording_ref field — minimal_repro exists only on ExploitRecord, never persisted — contradicting documentation.py's own docstring section on this exact point. Corrected all three spots to name only real schema fields and point to the report + ATO §5.2 + recording chain.

One-liners.

• ATO packet ~:435: label: "accepted_no_bound_observed" was presented as a top-level VULN-0004.json field; it's only inside observed prose. Matched the correct form used at ~:189.
• tests/test_ato_evidence_index.py docstring claimed §5.2 "never mentioned VULN-0004" on main — false, it did (the "Re-verifying" paragraph, within §5.2's span). Reworded to the real defect (§5.2 named VULN-0004 without describing its evidence) and added a stronger co-occurrence test requiring each report ID to appear within 300 characters of an evals/recordings/ path in §5.2, not merely anywhere in the section. Verified it would have failed against the pre-Part-A docs (VULN-0004 was 2597 characters from the nearest recording mention there).
• ATO packet §5.2 heading ("...plus VULN-0004") followed by an unscoped "All three are reproducible from..." — scoped to "All three criticals are reproducible from...".

Declutter (inline gate). Folded a duplicated "flag this claim unless a qualifying/negating phrase sits nearby" window-check into a shared _unqualified_claims() helper (Part B's join-key check and Part C's repro-steps check both needed it) — behavior-preserving, re-verified against the red-first proofs and full suite.

Self-caught in deep-review: the tests/test_doc_test_counts.py docstring I wrote for FIX 2 was internally contradictory ("a FOURTH phrasing... it is the third phrasing to slip this guard"). Reworded to disambiguate — third within the "N ... with the sibling checkout" family specifically, not counting the unrelated heading pattern.

Test count moved 343/237 → 346/240 across this round (5 test functions added net across Part C + the co-occurrence strengthening; skip count unchanged at 106) — cascaded through every "with the sibling checkout" / "N passed" claim in ATO_EVIDENCE_PACKET.md and DEMO_SCRIPT.md.

Gates (inline, full diff, both rounds)

• declutter: round 1 — nothing to remove. Round 2 — one duplication found and folded (see above).
• sec-audit: no findings, either round — diff is prose + docstring + test-only files; no entry points, no injection sinks, no auth logic touched.
• deep-review: round 1 — no findings. Round 2 — one self-contradiction found in a docstring I wrote, fixed same pass; traced every corrected claim against the real schema, the real report JSON files, and the real evals/cases//evals/recordings/ filesystem contents.

Suite

pytest tests/test_doc_test_counts.py -q → 2 passed.

pytest tests/ -q → 346 passed (sibling Phase 2 checkout present locally).

Evidence integrity

docs/vuln_reports/*.json untouched across both rounds — sha256 verified identical before/after every commit on this branch (no main() of any report-building tool was run against the real directory).

What I found wrong in the issue brief / earlier PR description while verifying

• dos-overlong-query-max-query-chars has 1 draw, not 2 as the issue brief guessed — confirmed by ls evals/recordings/dos-overlong-query-max-query-chars/ (one draw1 file + TRACE_RESOLUTION_NOTE.md). Documented the actual count.
• This PR's own earlier description overstated the Part B audit (claimed build_vuln_report_p3_54.py constructs an ExploitDB; it doesn't). Corrected above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TTi1zKDS1ajefLSmM9oL7E