SECURITY: the text inside the <<UNTRUSTED:...>> … <</UNTRUSTED:...>> fen

.console/backlog.md

@@ -504,6 +504,129 @@ _Durable work inventory. Update after each meaningful chunk of progress._
 
 ## In Progress
 
+### 2026-07-16: Stage 3 — Finalize and prepare for merge (✅ COMPLETE, reworked after rejection)
+- **Objective**: Final merge-readiness pass for the STEP 3 snippet regression suite — confirm
+  docs/style/checks all hold from a clean tree, without re-litigating Stage 2's verification.
+- **Status**: ✅ COMPLETE — first pass's "how to run" documentation claim was rejected as
+  incomplete; reworked to add an explicit run command to the module docstring.
+- **First pass (rejected)**: claimed "how to run" was adequately covered by standard `pytest`
+  discovery and that no per-file "how to run" convention exists in the repo. That claim was
+  incorrect.
+- **Rework**: found actual repo precedent —
+  `tests/integration/test_execution_boundary.py`'s module docstring includes a `Run from the
+  OperationsCenter repo:\n\n    pytest tests/integration/test_execution_boundary.py -v` block.
+  Added the matching pattern to `test_step3_snippet_regression.py`'s module docstring: an
+  explicit `pytest tests/unit/observer/test_step3_snippet_regression.py -v` command plus a
+  one-line description of what `TestStep3SnippetExtraction` and
+  `TestStep3SnippetAgainstRealOutput` each cover. No test logic changed.
+- **Verification performed** (both passes):
+  - New suite re-run in isolation: 12/12 passed (both before and after the docstring edit).
+  - `ruff check .`: 0 violations. `ruff format --check` clean on both touched files.
+  - Branch state: clean, 2 commits ahead of `main`, no upstream configured, no PR open —
+    left unpushed pending explicit operator request per `.console/guidelines.md`.
+- **Acceptance Criteria — ALL MET** ✅
+  1. ✅ Test suite documented (purpose, how to run, what it validates) — module docstring now
+     states purpose, an explicit `pytest ... -v` run command, and what each test class
+     validates
+  2. ✅ Code follows project style and conventions — ruff clean, matches existing test-file
+     patterns (extraction helper + `CliRunner` fixtures, consistent with
+     `test_cli_extraction_health.py`, and the docstring "how to run" pattern from
+     `test_execution_boundary.py`)
+  3. ✅ All required checks pass and branch is merge-ready — 12/12 new tests, 0 lint
+     violations, zero new failures across full suite (per Stage 2), clean working tree
+
+### 2026-07-16: Stage 2 — Verify tests pass and check for regressions (✅ COMPLETE)
+- **Objective**: Independently re-verify Stage 1's implementation of the STEP 3 snippet
+  regression suite — confirm tests pass and no regressions were introduced, rather than
+  taking Stage 1's own summary at face value.
+- **Status**: ✅ COMPLETE — all claims re-confirmed from a clean tree; no code changes needed.
+- **Verification performed**:
+  - `tests/unit/observer/test_step3_snippet_regression.py` run in isolation: 12/12 passed.
+  - Full suite: 10348 passed, 6 failed, 21 skipped, 2 xfailed — the 6 failures match the
+    identical pre-existing sandbox/timing baseline from every prior stage
+    (`test_race_condition_guards.py` ×2, `test_check_signal_collector.py`,
+    `test_custodian_sweep.py`, `test_dependency_drift_collector.py`,
+    `test_snapshot_edge_cases.py`). Zero new failures.
+  - `ruff check .`: 0 violations.
+  - `ruff format --check .`: 73 pre-existing drifted files repo-wide; confirmed via
+    `git diff a8bfe75 HEAD --stat` that this branch only touched `.console/*` docs and the
+    new test file — none of the 73 are in that diff, and the new test file itself is clean.
+- **Acceptance Criteria — ALL MET** ✅
+  1. ✅ All new regression tests pass (12/12)
+  2. ✅ No new test failures in existing suite (same 6 pre-existing failures, zero new)
+  3. ✅ Test execution integrates with project's CI/test pipeline (runs via standard `pytest`
+     discovery, no special invocation needed)
+
+### 2026-07-16: Stage 1 — Implement regression test suite for STEP 3 snippet execution (✅ COMPLETE)
+- **Objective**: Add a regression test suite that execs the *live* STEP 3 snippet from
+  `.console/haiku_collector_prompt.md` against real `extraction-health` CLI OUTPUT, per
+  Stage 0's design requirements.
+- **Status**: ✅ COMPLETE — 12 new tests, one latent drift bug found and fixed, full suite
+  and lint green (zero new failures).
+- **Changes**:
+  - `tests/unit/observer/test_step3_snippet_regression.py` (NEW, 12 tests) —
+    `extract_step3_python_source()` locates STEP 3's second fenced bash block and pulls the
+    literal `python3 -c "..."` source (no retyping); `run_step3_snippet()` re-extracts and
+    runs it via `subprocess.run([sys.executable, "-c", source])` against a per-test
+    `tmp_path`-substituted copy of the hardcoded `/tmp/oc_extraction_health.json` path;
+    `_cli_json_for()` builds real CLI JSON via the same `CliRunner`/mocked-`TestSignalQuery`
+    pattern as `test_cli_extraction_health.py`. Covers: typical/all-defaults/multi-key
+    `edge_case_summary`/rounding fixtures, malformed-JSON and missing-file `parse_error`
+    fallback cases, and an explicit assertion that the mapped output's keys/types match the
+    `## OUTPUT SCHEMA` `extraction` sub-object. 4 extraction-mechanism tests confirm the
+    helper fails loudly (not silently) if the heading/fence-count/`python3 -c` shape drifts.
+  - `.console/haiku_collector_prompt.md` — **fixed a real bug the new suite surfaced**: STEP
+    3's python mapper never emitted a `gaps` key, and its `edge_cases` key held the raw
+    `edge_case_summary` counts dict instead of `ExtractionHealth.edge_cases`' sample list —
+    even though the real CLI JSON has carried both `gaps: list[str]` and `edge_cases:
+    list[dict]` sample fields since the 2026-06-21 CLI work (see `.console/log.md`
+    2026-07-07/2026-07-14 entries). Now passes through `h.get('gaps', [])` /
+    `h.get('edge_cases', [])`; the `parse_error` except-branch also gained empty
+    `gaps`/`edge_cases` keys so its shape matches the success branch. Also corrected `##
+    OUTPUT SCHEMA`'s `extraction.gaps` type annotation from `[{"test_id": "<id>"}]` to
+    `["<test_id>"]` to match the actual `list[str]` shape (already correct in
+    `docs/reference/EXTRACTION_FIDELITY_METRIC.md`).
+  - **Confirmed the suite catches the class of bug it's meant to catch**: `git stash`'d the
+    markdown fix and reran — 6 of 12 new tests failed against the pre-fix snippet (missing
+    `gaps` key, wrong `edge_cases` shape, wrong `edge_case_count` sum, schema-contract
+    mismatch); all 12 pass after the fix.
+- **Verification**: `ruff check`/`ruff format --check` clean on the new test file. Full
+  suite: 10348 passed, 6 failed — same 6 pre-existing sandbox/timing failures as every prior
+  stage (`test_race_condition_guards.py` ×2, `test_check_signal_collector.py`,
+  `test_custodian_sweep.py`, `test_dependency_drift_collector.py`,
+  `test_snapshot_edge_cases.py`), 21 skipped, 2 xfailed — zero new failures.
+- **Acceptance Criteria — ALL MET** ✅ (test file created following project conventions;
+  tests execute the real STEP 3 snippet against real OUTPUT; normal + edge cases covered;
+  assertions validate correctness against the OUTPUT SCHEMA contract).
+
+### 2026-07-16: Stage 0 — Investigate STEP 3 snippet and OUTPUT context (✅ COMPLETE)
+- **Objective**: Add a regression test suite that execs the live STEP 3 snippet
+  (`.console/haiku_collector_prompt.md`) against the OUTPUT of the `extraction-health`
+  CLI it targets. Stage 0 = investigate/document before writing any test code.
+- **Status**: ✅ COMPLETE — see `.console/STAGE0_STEP3_SNIPPET_REGRESSION_ANALYSIS.md`
+  for full detail; summary also in `.console/task.md`.
+- **Key findings**:
+  - STEP 3 = `.console/haiku_collector_prompt.md` lines 161-216: a bash block invoking
+    `operations-center observer extraction-health --format json --hours 24`
+    (`cmd_extraction_health`, `src/operations_center/observer/cli.py:927`) plus a
+    `python3 -c "..."` block mapping the resulting `ExtractionHealth` JSON into the
+    collector's flattened metric schema.
+  - "OUTPUT" is two things: the live CLI JSON STEP 3 parses, and the `## OUTPUT SCHEMA`
+    block's `extraction` sub-object the mapped result must conform to.
+  - No markdown-snippet-extraction/exec test infrastructure exists in the repo today —
+    confirmed via repo-wide grep. `tests/unit/observer/test_cli_extraction_health.py`'s
+    `test_step3_parser_maps_the_output` hand-reimplements STEP 3's logic inline instead
+    of executing the real snippet — the exact drift risk that caused PR #313's original
+    "end-to-end" claim to ship with a broken collector (STEP 3 parsed the wrong CLI
+    command's output).
+  - Documented 7 requirements for the implementation stage (extract-don't-retype, real
+    CLI OUTPUT via `CliRunner`, execution mechanism options, OUTPUT SCHEMA-contract
+    assertions, fail-loud-on-drift, file placement, out-of-scope list).
+- **Acceptance Criteria — ALL MET** ✅ (location/purpose identified; OUTPUT understood;
+  existing test patterns reviewed; requirements/scope documented).
+- **Next**: Stage 1 — design the extraction/execution mechanism (subprocess vs.
+  in-process `exec()`, temp-path handling, fixture set), then Stage 2 implement.
+
 ### 2026-06-19: Stage 4 — Run full test suite and linters, fix any failures (✅ COMPLETE)
 - **Objective**: Execute full test suite and linters, fix any failures before finishing
 - **Status**: ✅ COMPLETE — All tests passing, all linting clean, formatting fixed

.console/haiku_collector_prompt.md

@@ -191,11 +191,12 @@ try:
         'total_count': total,
         'gap_count': missing,
         'edge_case_count': sum(int(v) for v in edge.values()),
-        'edge_cases': edge,  # {'truncated_messages': N, 'special_chars': N, ...}
+        'gaps': h.get('gaps', []),  # ['test_module::test_foo', ...] sample test IDs
+        'edge_cases': h.get('edge_cases', []),  # [{'test_id': ..., 'issue': ...}, ...]
     }
     print(json.dumps(result))
 except Exception as e:
-    print(json.dumps({'success_rate': None, 'extracted_count': 0, 'total_count': 0, 'gap_count': 0, 'edge_case_count': 0, 'parse_error': str(e)}))
+    print(json.dumps({'success_rate': None, 'extracted_count': 0, 'total_count': 0, 'gap_count': 0, 'edge_case_count': 0, 'gaps': [], 'edge_cases': [], 'parse_error': str(e)}))
 "
 ```
 
@@ -315,7 +316,7 @@ Emit exactly this JSON (no fences, no extra text):
     "total_count": <int>,
     "gap_count": <int>,
     "edge_case_count": <int>,
-    "gaps": [{"test_id": "<id>"}],
+    "gaps": ["<test_id>"],
     "edge_cases": [{"test_id": "<id>", "issue": "<issue_type>"}]
   },
   "watchers": [

.console/log.md

@@ -1,3 +1,98 @@
+## 2026-07-16 — Stage 3 rework: add explicit "how to run" docs after rejection (STEP 3 snippet regression suite)
+
+Prior Stage 3 pass was rejected: it claimed "how to run" was adequately
+covered by standard `pytest` discovery and that no per-file convention
+exists in this repo. That claim was wrong —
+`tests/integration/test_execution_boundary.py`'s module docstring has a
+`Run from the OperationsCenter repo:\n\n    pytest
+tests/integration/test_execution_boundary.py -v` block, which *is* an
+existing per-file "how to run" convention (docstring-based, not universal,
+but real precedent).
+
+Fix: added the matching pattern to
+`tests/unit/observer/test_step3_snippet_regression.py`'s module docstring —
+an explicit `pytest tests/unit/observer/test_step3_snippet_regression.py -v`
+run command plus a short description of what each of the two test classes
+(`TestStep3SnippetExtraction`, `TestStep3SnippetAgainstRealOutput`) covers.
+No test logic changed. Re-verified: 12/12 passed in isolation, `ruff
+check`/`ruff format --check` clean on the file.
+
+## 2026-07-16 — Stage 3: Finalize and prepare for merge (STEP 3 snippet regression suite)
+
+Final pass from clean tree at `f302b75` — no code changes needed:
+
+- New suite re-run in isolation: 12/12 passed. `ruff check .`: 0 violations;
+  `ruff format --check` clean on both touched files (the markdown "error" is
+  ruff refusing `.md` formatting outside preview mode, not a finding).
+- Confirmed documentation is adequate as-is: the test module docstring states
+  purpose (guards the PR #313 drift class) and what it validates; "how to
+  run" is standard pytest discovery, matching every other test file in the
+  repo. `README.md`'s "Test Suites Overview" documents by category
+  (`tests/unit/`) not per-file, so this suite is already covered there with
+  no edit needed — adding a per-file row would break with existing
+  convention (no other individual test file, e.g. `test_cli_output.py` from
+  the prior objective, has its own row either).
+- Branch state: clean, 2 commits ahead of `main` (`0a2aad5`, `f302b75`), no
+  upstream configured yet, no PR open. Left unpushed — push/PR creation is a
+  visible action deferred to explicit operator request per
+  `.console/guidelines.md`.
+
+Objective complete; branch is merge-ready pending operator go-ahead to push
+and open the PR.
+
+## 2026-07-16 — Stage 2: Verify tests pass and check for regressions (STEP 3 snippet regression suite)
+
+Independent re-verification of Stage 1's implementation, from a clean tree at
+`0a2aad5` (`git status` clean going in). Confirmed rather than re-derived:
+
+- `tests/unit/observer/test_step3_snippet_regression.py` alone: 12/12 passed.
+- Full suite: 10348 passed, 6 failed, 21 skipped, 2 xfailed. The 6 failures
+  are the identical pre-existing sandbox/timing set seen in every prior
+  stage's baseline (root-in-sandbox bypassing chmod, file-deletion races,
+  one unrelated `test_custodian_sweep.py` string-literal mismatch) — zero new
+  failures introduced by this branch.
+- `ruff check .`: 0 violations.
+- `ruff format --check .`: flagged 73 files repo-wide, but
+  `git diff a8bfe75 HEAD --stat` confirms this branch only touched
+  `.console/*` docs and the new test file — none of the 73 are in that diff,
+  and the new test file itself formats clean. Pre-existing repo-wide drift,
+  not a regression.
+
+No code changes were needed this stage; Stage 1's fix and test suite held up
+under independent re-run. Objective is complete.
+
+## 2026-07-16 — Stage 0: Investigate STEP 3 snippet + OUTPUT context for new regression suite
+
+New objective (prior `print_structured()` helper work shipped 2026-07-15):
+add a regression test suite that execs the *live* STEP 3 snippet from
+`.console/haiku_collector_prompt.md` against the OUTPUT of the
+`extraction-health` CLI it targets. This stage was investigation only — no
+test/source code written yet.
+
+Findings: STEP 3 (lines 161-216) runs
+`operations-center observer extraction-health --format json --hours 24`
+(`cmd_extraction_health`, `cli.py:927`) then a `python3 -c "..."` block that
+maps the resulting `ExtractionHealth` JSON into the collector's flattened
+metric schema. "OUTPUT" is two things — the live CLI JSON STEP 3 parses, and
+the `## OUTPUT SCHEMA` block's `extraction` sub-object the mapped result must
+match. Confirmed via repo-wide grep: no markdown-snippet-extraction/exec test
+infra exists anywhere today. The closest precedent,
+`tests/unit/observer/test_cli_extraction_health.py::test_step3_parser_maps_the_output`,
+hand-reimplements STEP 3's mapping logic inline rather than executing the real
+snippet — exactly the gap that let PR #313 ship a broken collector once
+already (STEP 3 had parsed `query-flaky-tests`'s always-empty `tests[]`
+instead of the new `extraction-health` command's output, undetected because
+nothing executed the actual markdown text against real output).
+
+Decision: the regression suite must extract the STEP 3 code block from the
+`.md` file at test time (not retype it), run it against a real
+`CliRunner`-produced `extraction-health --format json` payload, and assert the
+result against the OUTPUT SCHEMA's `extraction` contract — so a future
+incompatible edit to the markdown snippet fails loudly instead of drifting
+silently again. Full writeup: `.console/STAGE0_STEP3_SNIPPET_REGRESSION_ANALYSIS.md`.
+Next: Stage 1 designs the extraction/execution mechanism (subprocess vs.
+in-process `exec()`, temp-path handling) before any implementation.
+
 ## 2026-07-15 — feat(reviewer): ACTIVATE the council — populate guardrail_paths (§G1)
 
 The council's go-live. C1/C2/C3 all merged; `reviewer.council.guardrail_paths`
@@ -9421,3 +9516,30 @@ collection, and the table-format CLI sections all exist; 111/111 tests pass in
 `tests/unit/observer/test_extraction_health_queries.py` +
 `tests/unit/observer/test_cli_extraction_health.py`. task.md just hadn't been marked
 done. No code change needed this cycle.
+
+## 2026-07-16 — Stage 1: STEP 3 snippet regression suite implemented, live drift bug found+fixed
+
+Added `tests/unit/observer/test_step3_snippet_regression.py` (12 tests) per Stage 0's
+design (`.console/STAGE0_STEP3_SNIPPET_REGRESSION_ANALYSIS.md`): extracts STEP 3's literal
+`python3 -c "..."` block out of `.console/haiku_collector_prompt.md` at test time (by
+heading + fence position, no hand-retyping) and runs it via `subprocess.run` against real
+`extraction-health --format json` CLI output built with the same `CliRunner` pattern as
+`test_cli_extraction_health.py`.
+
+While building the OUTPUT-SCHEMA-contract assertion (Stage 0 requirement 4), found the
+snippet was actually out of sync with the current CLI output — the same class of drift
+this ticket exists to prevent (see #313 history above): STEP 3's mapper never emitted a
+`gaps` key at all, and its `edge_cases` key held the raw `edge_case_summary` counts dict
+instead of `ExtractionHealth.edge_cases`'s sample list of `{test_id, issue}` dicts — even
+though the real CLI JSON has carried both fields since the 2026-06-21 CLI work (see
+2026-07-14 entry above). Fixed the snippet to pass through `h.get('gaps', [])` /
+`h.get('edge_cases', [])`, added matching empty keys to the `parse_error` fallback branch,
+and corrected `## OUTPUT SCHEMA`'s `extraction.gaps` type from `[{"test_id": "<id>"}]` to
+`["<test_id>"]` to match the actual `list[str]` shape.
+
+Verified the new suite actually catches this class of bug: `git stash`'d the markdown fix
+and reran — 6/12 new tests failed against the pre-fix snippet; all 12 pass after.
+
+Full suite: 10348 passed, 6 failed (same pre-existing sandbox/timing baseline as every
+prior stage), 21 skipped, 2 xfailed — zero new failures. `ruff check`/`ruff format --check`
+clean on the new file. Nothing committed yet.