P3.31: durable pending human-triage surface + CLI approve path (#63)

contracts/README.md

@@ -28,6 +28,21 @@ ad hoc payload shape once this contract exists for their edge.
 No `v2/` exists yet; nothing has broken compatibility since this issue's
 initial cut.
 
+**Log:**
+
+- Issue #63: `observability_snapshot.schema.json` gained an optional
+  `pending_human_triage_count` property (durable count of reports still
+  awaiting human triage). Not added to `required`, so a pre-#63 PRODUCER
+  that omits the field stays schema-valid — additive, stays `v1`. **This
+  is one-directional, not a blanket "consumers stay valid" claim**: this
+  schema's `additionalProperties: false` (unchanged) means a pre-#63
+  CONSUMER validating an incoming snapshot against its OWN pinned copy of
+  this schema will reject any snapshot that now carries the new field. A
+  consumer must update its own copy of the schema (or relax
+  `additionalProperties`) before it can accept a post-#63 producer's
+  output — see the field's own `description` in the schema for the
+  cross-reference to `--list-pending`'s differently-scoped same-named key.
+
 ## Schemas (one per edge in ARCHITECTURE.md §2)
 
 | Schema | Edge | Notes |

contracts/v1/observability_snapshot.schema.json

@@ -32,6 +32,7 @@
       }
     },
     "open_high_sev_count": { "type": "integer", "minimum": 0 },
+    "pending_human_triage_count": { "type": "integer", "minimum": 0, "description": "How many of the vuln_reports passed to emit_snapshot() are still awaiting human triage (issue #63) -- NOT a directory-wide count: `tools/run_campaign.py --list-pending` prints the same key name but scans an entire --reports-dir, which can differ from this field's value (see docs/ARCHITECTURE.md's Observability Layer section). Additive v1 field, not in 'required' -- a pre-#63 PRODUCER that omits it stays schema-valid, but a pre-#63 CONSUMER validating incoming snapshots against an unmodified copy of this schema (additionalProperties: false, unchanged) will reject any snapshot that now includes it." },
     "cost": {
       "type": "object",
       "additionalProperties": false,

docs/ARCHITECTURE.md

@@ -208,6 +208,18 @@ correctly kept separate.
    signals are read programmatically, not just rendered for a human), in
    addition to being a human-facing dashboard.
 
+   **`pending_human_triage_count` names two different scopes (issue #63).**
+   `observability_snapshot.schema.json`'s `pending_human_triage_count`
+   field (`redteam.observability.findings.pending_human_triage_count`)
+   counts only the `vuln_reports` sequence a caller passes to
+   `emit_snapshot` — in the live campaign loop, everything filed/pending
+   so far in *that one run*. `tools/run_campaign.py --list-pending` prints
+   a line using the identical key name, `pending_human_triage_count=N`,
+   but that N is a directory-wide scan of an entire `--reports-dir` on
+   disk, independent of any one run. The two numbers can legitimately
+   differ — do not assume the snapshot field and the CLI's printed line
+   agree.
+
 ## 4. Fully-local model strategy (decided)
 
 The owner's decision, locked in `planning/PLAN.md` and reaffirmed here as

docs/ATO_EVIDENCE_PACKET.md

@@ -326,9 +326,9 @@ explicitly documented as an arbitrary placeholder accepted by the target's
 own insecure-by-default validator (VULN-0001) — "safe to publish as-is" per
 that document's own text, not a real credential.
 
-`pytest tests/ -q` re-run for this packet: **346 passed** with the sibling
+`pytest tests/ -q` re-run for this packet: **375 passed** with the sibling
 Phase 2 checkout (`../agentforge-2-evidence-agent`, pinned `v2.0.0`)
-present locally (confirmed at PR time); **240 passed, 106 skipped** in CI
+present locally (confirmed at PR time); **269 passed, 106 skipped** in CI
 and for anyone without that sibling — CI (`.github/workflows/ci.yml`) does
 not check it out, so the 100 total sibling-checkout citation cases
 class-skip cleanly there: 40 `TestTraceCitationsAgainstPinnedTarget` cases
@@ -373,7 +373,7 @@ those changes included, not a pre-change baseline.
   evidence the project has previously demonstrated this discipline under
   pressure, not as a claim about this PR's own diff (which touches no
   secret-adjacent files).
-- **346 passing tests (240 passed, 106 skipped in CI), no live/network/GPU
+- **375 passing tests (269 passed, 106 skipped in CI), no live/network/GPU
   call in the default suite.** Every test file under `tests/`
   (`tests/contracts/`, `tests/redteam/`, `tests/test_cases.py`,
   `tests/test_case_sourceref_relevance.py`, `tests/test_runner_sse.py`,
@@ -395,18 +395,18 @@ those changes included, not a pre-change baseline.
   has moved across PRs that touch test-suite-relevant code (e.g. PR #40's
   own test plan: "177 passed (unchanged; no test-suite-relevant code
   touched)" at that point in the repo's history; this PR's own platform
-  changes plus its expanded citation-verification test set move it to 346
-  with the sibling checkout present, or 240 passed / 106 skipped without
+  changes plus its expanded citation-verification test set move it to 375
+  with the sibling checkout present, or 269 passed / 106 skipped without
   it, §5.1).
 
 ---
 
 ## 5. Eval-result evidence
 
-### 5.1 The 346-test suite (240 in CI)
+### 5.1 The 375-test suite (269 in CI)
 
-`pytest tests/ -q` → **346 passed** with the sibling Phase 2 checkout
-present, re-confirmed for this packet (§4.1); **240 passed, 106 skipped**
+`pytest tests/ -q` → **375 passed** with the sibling Phase 2 checkout
+present, re-confirmed for this packet (§4.1); **269 passed, 106 skipped**
 in CI (`.github/workflows/ci.yml` does not check out the sibling target)
 and for any clone lacking it. Organized across `tests/contracts/` (schema
 + uniqueness constraints), `tests/redteam/` (the six agents + campaign
@@ -518,8 +518,8 @@ to approve and nothing already filed.
   suspected halts new directives; an empty-completion error is skipped, not
   fatal (this is §6's postmortem subject); `max_iterations` input
   validation. Test count: 163 baseline → 171 (PR #35's own reported delta;
-  the repo has since grown to 346 total with the sibling checkout present,
-  or 240 passed / 106 skipped without it, §5.1).
+  the repo has since grown to 375 total with the sibling checkout present,
+  or 269 passed / 106 skipped without it, §5.1).
 
 ### 5.4 Load-test numbers
 
@@ -639,8 +639,8 @@ describes — not because it was dramatic.
   (Mermaid diagram, trust-zone framing), §2 Auth model (platform + target),
   §3 Versioned dependency list (`requirements-contracts.txt`, contracts
   versioning, model runtimes), §4 Self-scan results (commands run + process
-  evidence), §5 Eval-result evidence (346 tests with the sibling checkout
-  present / 240 passed, 106 skipped in CI, 3 criticals, live-campaign
+  evidence), §5 Eval-result evidence (375 tests with the sibling checkout
+  present / 269 passed, 106 skipped in CI, 3 criticals, live-campaign
   evidence, load-test numbers), §6 Sample incident and postmortem.
 - **Every section cites a real, already-committed artifact**, not an
   invented one: `docs/ARCHITECTURE.md`, `docs/THREAT_MODEL.md`,

docs/DEMO_SCRIPT.md

@@ -1,9 +1,11 @@
 # Demo Script — AgentForge Phase 3 Red-Team Platform
 
-P3.20 (issue #44). A reproducible, copy-pasteable walkthrough of the
-platform end-to-end, at four beats: the loop finding a vuln, the Judge
-confirming it, the regression harness catching a **reintroduced** fixed
-exploit, and one graceful failure. Every command below was run against this
+P3.20 (issue #44); Beat 5 added in the cold-review fix to PR #76 (issue
+#63). A reproducible, copy-pasteable walkthrough of the platform
+end-to-end, at five beats: the loop finding a vuln, the Judge confirming
+it, the regression harness catching a **reintroduced** fixed exploit, one
+graceful failure, and approving a durably-pending report from the CLI.
+Every command below was run against this
 repo at `v2.0.0` while writing this doc; outputs are pasted verbatim where
 noted. See `docs/ARCHITECTURE.md` §2 for the component interaction diagram
 this script drives, and `docs/ATO_EVIDENCE_PACKET.md` §5 for the underlying
@@ -20,9 +22,9 @@ evidence table this script complements with runnable commands.
   immediately before and after any live call and confirm VRAM stays flat.
 - `pytest tests/ -q` green (deterministic — no live/network/GPU call in the
   default suite; confirmed while writing this doc). The printed count is
-  environment-dependent: **346 passed** when the sibling Phase 2 checkout
+  environment-dependent: **375 passed** when the sibling Phase 2 checkout
   (`../agentforge-2-evidence-agent`, pinned `v2.0.0`) is present locally;
-  **240 passed, 106 skipped** in CI and for anyone cloning this repo without
+  **269 passed, 106 skipped** in CI and for anyone cloning this repo without
   that sibling (the 106 skipped are `TestTraceCitationsAgainstPinnedTarget`
   (40 cases, `tests/test_dos_input_bound_resolution.py`) plus
   `TestCitationsAgainstPinnedTargets` (60 cases,
@@ -33,7 +35,7 @@ evidence table this script complements with runnable commands.
 
 ```
 $ pytest tests/ -q
-346 passed in 2.38s          # with the sibling Phase 2 checkout present
+375 passed in 2.38s          # with the sibling Phase 2 checkout present
 ```
 
 ---
@@ -299,6 +301,45 @@ here for completeness:
 
 ---
 
+## Beat 5 — Approving a durably-pending report from the CLI (issue #63/#66)
+
+Every pending report Beat 1–4's loop files (critical severity, or a
+`denial_of_service` finding — see the "one human touchpoint" note below)
+now survives the filing process exiting when `--reports-dir PATH` is
+passed to `tools/run_campaign.py`, and can be approved later by a
+completely separate invocation, no bespoke per-report script:
+
+```
+python tools/run_campaign.py --list-pending --reports-dir PATH
+python tools/run_campaign.py --approve EXP-0004 --reports-dir PATH --db-path PATH --approved-by NAME
+```
+
+`--approve` fails closed by design: `--db-path` must already name an
+existing sqlite file holding the original exploit record for that
+`exploit_id` (a typo'd or not-yet-created path is a hard refusal, exit 1 —
+it is never silently created as an empty DB), and `--approved-by` has no
+default — an explicit human identity is the point of a human-approval
+gate. The pending report's full body is printed to stdout before it is
+stamped, so approval is an informed act, not a blind exploit_id lookup.
+For a genuinely DB-less pending report, the explicit
+`--unverified-i-vouch-without-db-check` flag skips the cross-check with a
+loud stderr `WARNING` naming exactly what was skipped — there is no quiet
+way to bypass the check.
+
+```
+pytest tests/tools/test_run_campaign_cli.py -v
+```
+
+Exercises the full list → approve round trip end-to-end (file a pending
+report with one `DocumentationAgent` instance, `del` it to simulate that
+process exiting, then list and approve it with only the CLI against a
+fresh instance) plus every fail-closed path: no `--db-path`/no escape
+hatch, a `--db-path` naming a file that doesn't exist yet, no
+`--approved-by`, and a tampered/drifted pending artifact that no longer
+matches its source exploit record.
+
+---
+
 ## What this proves
 
 - **The loop finds real vulnerabilities autonomously**: `run_campaign`
@@ -328,20 +369,27 @@ here for completeness:
   `DocumentationAgent`'s critical-severity report gate (`VULN-0003`'s own
   `"requires_human_gate": true` / `"approved_by": "owner"`), never a loop
   restart.
+- **The human-approval touchpoint is durable and fails closed**: a report
+  filed pending by one process survives that process exiting
+  (`--reports-dir PATH`) and is approvable later by a completely separate
+  CLI invocation (`tools/run_campaign.py --approve`, Beat 5), with no
+  silent path to approving unverified content — a missing/typo'd
+  `--db-path`, a missing `--approved-by`, or a tampered/drifted pending
+  artifact are all hard refusals, not warnings.
 
 ## CI
 
 CI (`.github/workflows/ci.yml`) runs the deterministic suite —
 `python -m pytest tests/ -q` — on every push to `main` and on every pull
 request. CI does not check out the sibling Phase 2 target, so its printed
-count is **240 passed, 106 skipped** (the 106 skipped are
+count is **269 passed, 106 skipped** (the 106 skipped are
 `TestTraceCitationsAgainstPinnedTarget` (40, issue #25/#54),
 `TestCitationsAgainstPinnedTargets` (60, issue #58), and
 `TestStandingUpTargetPathsExistInPinnedTarget` (6, issue #61), all of
 which class-skip cleanly when `../agentforge-2-evidence-agent` is absent). Live-model and target-stack
 runs remain manual, outside CI: every command in this script was run
 locally against the dev stack while writing this doc, with the sibling
-checkout present, giving **346 passed**. `pytest tests/ -q` is still the
+checkout present, giving **375 passed**. `pytest tests/ -q` is still the
 reproducibility bar — re-run it after pulling this branch to confirm
-nothing here has drifted: expect **346 passed** if you have the sibling
-Phase 2 checkout at `v2.0.0`, or **240 passed, 106 skipped** if you don't.
+nothing here has drifted: expect **375 passed** if you have the sibling
+Phase 2 checkout at `v2.0.0`, or **269 passed, 106 skipped** if you don't.