P3.37: harden recording_ref derivation and widen the doc guard - #80
Merged
Conversation
…-shaped and .. refs (#79)
…ding_ref phrasing (#79) Widened window from 10 to 40 chars (matching the sibling carries-no branch) so the original ARCHITECTURE.md:77 phrasing (17 intervening chars) is caught. Also required 'field' within 10 chars after recording_ref on both branches -- an unqualified 0-40 widen produced a false positive against documentation.py's own section heading ('has no minimal_repro.steps, and how ## recording_ref works'), which isn't a stale-claim at all. Mutation-verified against all three original ARCHITECTURE.md phrasings (see PR description).
…A-262 lookahead (#79) '\Z' is correct under Python re (which jsonschema's Draft202012Validator uses) but is not portable JSON Schema: Draft 2020-12 mandates ECMA-262 regex semantics, under which '\Z' matches a literal 'Z' character, and Go's RE2 rejects it outright. Switched to '(?![\s\S])', a portable negative-lookahead anchor with identical semantics in both dialects -- verified byte-for-byte equivalent to '\Z' under this repo's jsonschema for bare end, trailing newline, trailing space, and no-trailing-slash cases. Only Python consumers exist today, but this keeps the contract itself dialect-neutral.
…hardening round (#79) 389 -> 392 (with sibling checkout), 283 -> 286 passed (CI shape), per tests/test_doc_test_counts.py. Net +3 from FIX 1's three red-first tests (directory-with-slash, directory-without-slash, .. segment); FIX 2 and FIX 3 added no new tests.
4 tasks
franciszver
added a commit
that referenced
this pull request
Jul 26, 2026
Red-first: extends tests/test_release_notes.py with 13 new failing assertions covering the ten corrections from the final overclaim review, then rewrites docs/RELEASE_NOTES_v3.0.0.md to satisfy them: BLOCKERS: 1. Quotes planning/KICKOFF_PROMPT.md's HARD CONSTRAINT in full, including "A single-agent or linear pipeline FAILS the assignment" -- previously elided without an ellipsis. Stops calling "(separate process/context)" a parenthesised aside; it is the brief's own operational definition of architectural independence. 2. Issue #73 is CLOSED (a documentation-correction issue, not an implementation tracker) -- reworded to state the gap was raised and resolved at the documentation level, and OS-process isolation is not currently scheduled, rather than implying #73 tracks future work. 3. Scopes the upstream #130 argument to VULN-0002 (document_citations: [] matches #130's Ask exactly); VULN-0003 is described as the alongside-a-DocumentCitation shape #130 pre-emptively called "harmless today", with the stronger argument for why that premise doesn't hold for this shape either (mirrors the correction already posted to upstream #170). 4. VULN-0001 is scoped to the shipped default (copilot_per_user_token_enabled=False) -- a real introspection validator replaces the permissive default when the flag is True. VULN-0004 is unconditional (no flag gates it). 5. Corrects "the only path any test exercises" -- test_judge_agent.py:236 constructs JudgeAgent(scorer=corrupting_scorer) for a drift test. NON-BLOCKING: 6. Qualifies "after these findings were filed" -- true for three of four; VULN-0004 was filed later the same day v2.1.0 was tagged. 7. Re-cites judge.py:44-53 (the scorer-seam docstring paragraph) instead of the mis-ranged :44-47. 8. Corrects "scans red_team.py the same way" -- the Judge-side AST scan resolves relative imports; the Red-Team-side scan checks absolute module paths only, so `from .judge import X` would pass undetected. 9. Adds "(temperature 0)" to the 3/3 draw-count claim, matching what the upstream filings state. 10. Fixes a quoted word substitution: config.py:251 says "mechanism", not "alternative". Also, consequences of #77/#79 merging into main during the rebase: - vuln reports are now 14-field artifacts (recording_ref added); updated the enumeration and field count everywhere they're cited. - Each report now names its own evals/recordings/ directory via recording_ref; ATO §5.2 remains the human-readable index, not the sole resolution path. - Restated the exploit_id limitation precisely: exploit_id alone still only resolves in-process, but the evidence trail no longer depends on that join now that recording_ref is on the artifact itself. - Re-derived test counts post-rebase across RELEASE_NOTES/ ATO_EVIDENCE_PACKET/DEMO_SCRIPT: 435 passed locally (sibling present), 329 passed / 106 skipped in CI. Rebase note: docs/ATO_EVIDENCE_PACKET.md and docs/DEMO_SCRIPT.md conflicts were resolved by taking main's post-#78/#79/#80 versions (both merges' own count bumps), then re-deriving counts fresh rather than carrying any number from either side forward.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #79
Small, surgical hardening round from the focused re-review of PR #78. Three one-line-scoped fixes.
FIX 1 — directory-shaped
recording_refnow fails loudly_recording_ref_for(redteam/agents/documentation.py) previously tookparts[-2]unconditionally, so a directory-shaped ref (with or without a trailing slash) silently produced a schema-valid but WRONG pointer:Now rejects (raises
DocumentationAgentError):/(directory-shaped).jsonsuffix..path segmentDoes NOT re-anchor on a literal
recordingssegment or relativise against the repo root — that was PR #78's own fix, deliberately preserved (see the docstring, which explains why:tools/load_test_replay.py's scratch tempdir breaks that invariant). Regression-checked:No regression — 0/3-filed-reports failure mode from re-anchoring does NOT reappear.
FIX 2 — doc guard now catches the 17-char phrasing it was written for
tests/test_ato_evidence_index.py's_NO_RECORDING_REF_CLAIM_REallowed only 10 chars between "has no" and "recording_ref", missing the originaldocs/ARCHITECTURE.md:77phrasing (17 chars:deliberately has no `minimal_repro`/`recording_ref` field). Widened to.{0,40}(matching thecarries nobranch).A bare widen produced a false positive against
redteam/agents/documentation.py's own section heading ("...has no ``minimal_repro.steps``, and how ## ``recording_ref`` works..."), so both branches now also requirefieldwithin 10 chars afterrecording_ref— this matches every real "stale claim" phrasing (which all end in "...recording_ref` field") without matching the heading (which says "works", not "field").Mutation-tested against all three original (pre-PR-#78)
ARCHITECTURE.mdphrasings::74-77(prose summary): now caught (was missed before):182-187(component list): legitimately out of scope — this location's original text only says "the schema has nominimal_reprofield" and never claimsrecording_refis absent, so it was never a stale claim:318(AI-use table): still caught (was already caught before, unaffected)docs/ARCHITECTURE.mdrestored byte-identical after mutation testing (git status --porcelainclean throughout).FIX 3 — portable regex anchor in the schema
contracts/v1/vuln_report.schema.json'srecording_refpattern used\Z, correct under Python'sre(whichjsonschema'sDraft202012Validatoruses) but not portable ECMA-262 (JSON Schema's mandated dialect) —\Zthere matches a literalZ, and Go's RE2 rejects it outright. Switched to(?![\s\S]), a portable negative-lookahead with identical semantics in both dialects. Verified byte-for-byte equivalent to\Zunder this repo'sjsonschema(4.26.0) for every probed case (bare end, trailing\n, trailing space, no trailing slash) — a trailing newline is still rejected. Extended the in-schema comment to record the rationale for future editors.Docs
docs/ATO_EVIDENCE_PACKET.md/docs/DEMO_SCRIPT.mdtest-count claims updated 389→392 (with sibling checkout) / 283→286 passed (CI shape), for the net +3 tests from FIX 1's red-first suite (tests/test_doc_test_counts.pyenforces this).docs/RELEASE_NOTES_v3.0.0.mduntouched (open PR #72).Verification
392 passed(was 389 baseline; +3 from FIX 1's tests)286 passed, 106 skipped(was 283/106 baseline)docs/vuln_reports/*.json(4 files): byte-identical before/after —git diffempty, sha256 unchangedTest plan
..segment (quoted failing output in commitb8f8e3b)python -m pytest -q→ 392 passedpython tools/load_test_replay.py --iterations 5→ exploit_ids=5, signals=0 (no regression)(?![\s\S])byte-equivalent to\Zunder jsonschema 4.26.0