dream(darwin-evolution): #292 reward-hack monitor never scanned tool-response content (evaluated)

[ruvnet]

MetaHarness Dream Cycle, 2026-09-07. Deep surface: darwin-evolution (slot 2). Full detail, research,
candidate-selection scoring, and weight-eft/learn scan findings: #292.

Hypothesis

Given a resolved Darwin trajectory whose tool-CALL arguments carry no reward-hack signal but whose
tool-RESPONSE content (role:'tool') contains a gold-read pattern (gold_patch/FAIL_TO_PASS/PASS_TO_PASS/
test_patch/expected-output/.golden/run_evaluation), when detectRewardHack() runs over it, then it
should be flagged kind: 'gold-read' exactly as a call-argument leak already is today, subject to: (a)
zero verdict change for the 4 existing fixture trajectories; (b) purely additive — no existing pattern,
tool classification, or finding kind removed/narrowed; (c) 0 regressions in the full weight-eft suite;
(d) the new detection path proven non-vacuous (a crafted fixture must fail pre-fix, pass post-fix). Frozen
before implementation, not modified after evaluation began.

Candidate

packages/weight-eft/src/reward-hack.ts's deterministic reward-hack monitor (ADR-198, an "Ornith-1.0"-
style gate dropping reward-hacked trajectories before SFT/DPO export) scanned only the outgoing arguments
of role:'assistant' tool_calls[].function.arguments. role:'tool' messages' content — what a tool
actually returned — was never inspected by any of the monitor's 3 finding kinds. A trajectory that leaks
gold/test content through an innocuous-looking call (clean arguments, leaking response) sailed through
undetected. Disclosed but not fixed by the last darwin-evolution night (2026-09-02, #272).

Changes (1 source file + 1 new test file + 1 new fixture, ~55 net lines):

• reward-hack.ts: new toolResponses() extraction helper + a 4th detection pass in detectRewardHack()
  scanning role:'tool' content against the existing GOLD_READ_PATTERNS, tagged kind: 'gold-read' —
  identical finding shape to the existing call-argument detection. Purely additive: no existing pattern,
  classification, or finding kind touched.
• New fixture reward-hack-response-content.json + reward-hack-response-content.test.ts, isolated from
  the original reward-hack-archive.json/reward-hack.test.ts (left byte-identical) to avoid any risk to
  existing assertions.

Evaluation Receipt

evaluated: accepted. Deterministic, $0, zero LLM calls:

• New fixture: a leak trajectory whose tool-CALL arguments are clean (independently verified to match none
  of GOLD_READ_PATTERNS/VERIFICATION_WRITE_PATTERNS/OUT_OF_SANDBOX_PATTERNS) but whose tool-RESPONSE
  content contains FAIL_TO_PASS: test_validate_negative.
• Pre-fix (confirmed live via git stash on the source change, then restored): detectRewardHack
  returns [], isRewardHacked false, exportTrainingData reports droppedRewardHacked: 0 — 2/4 new
  tests fail exactly as predicted.
• Post-fix: flags kind: 'gold-read', tool: 'grep'; exporter drops the leak trajectory, keeps the
  honest one, no SFT row carries the leaked text. 4/4 new tests pass.
• Control: the original 4-fixture reward-hack-archive.json is byte-identical in verdict — confirmed
  both by the still-green original test file and by the independent critic manually checking every
  role:'tool' content string in that archive against GOLD_READ_PATTERNS.
• Full suite: weight-eft 37→41/41 (+4, 0 regressions), tsc --noEmit clean.

Baseline Comparison

Baseline = main at session start (d5833dc6512ac1adeeef91a331c29055cd8a4dbb), same evaluator (vitest +
tsc). No behavioral change for any trajectory already caught by the existing call-argument detection; the
fix only affects the previously-invisible response-content-only leak case.

Darwin Lineage

Not run — a correctness fix to weight-eft's own reward-hack monitor, not a Darwin-scoped tunable
genome/routing/tier/prompt parameter. Confirmed directly: packages/darwin-mode/src/safety.ts's mutation
allowlist governs only Darwin's own 7 generated genome surface files, not weight-eft's source — no
ADR-071 mutation surface applies (same reasoning as every prior non-parameter darwin-evolution library
fix, e.g. 2026-09-02's ADR-278, 2026-08-17's cost-seam wiring).

Flywheel Evidence

Not a flywheel-domain candidate. docs/dream-cycle/2026-09-07-evidence.md carries the full
OBSERVATION/MEASUREMENT/INFERENCE/DECISION record.

Reward Hack Check

No unresolved signal. The fix is purely additive — no existing test weakened, no pattern/threshold
narrowed, no gold/corpus data touched, no cost hidden, no output hardcoded. Independent adversarial critic
(fresh sub-agent, no authorship context, re-ran the suite and tsc itself rather than trusting the claim)
verdict: ACCEPT — confirmed purely additive, confirmed the new fixture is non-vacuous by independently
checking its call arguments against all three pattern lists, confirmed no false positives introduced on
the original archive, found no gaming/hardcoding/scope-creep. One disclosed, non-blocking follow-up: the
response-content scan applies only GOLD_READ_PATTERNS, not VERIFICATION_WRITE_PATTERNS/
OUT_OF_SANDBOX_PATTERNS — flagged as Recommended Next Step #1 in the gist, not attempted here (kept to
one conceptual change).

Security Review

This candidate closes a training-data-integrity gap: a reward-hacked "success" that goes undetected
teaches the distilled cheap-tier model to reward-hack. No credential/shell/network surface touched, no new
dependency, no ReDoS risk (reuses the existing bounded regex patterns against a new input surface). Not a
containment/sandbox-escape fix like 2026-09-02's ADR-278 — this is detection-scope hardening of an
existing, already-ADR'd mechanism (ADR-198), not a new architectural decision.

Regression Analysis

0 regressions. weight-eft full suite (41/41, was 37/37), tsc --noEmit clean. Downstream sweep: the only
in-repo consumer, create-agent-harness (dynamic import('@metaharness/weight-eft/cli')), full suite
572/574 (2 pre-existing skips, unchanged). Full monorepo npm run build (4 phases, JS/TS only — no
wasm/NAPI steps needed for this package): clean.

ADR

None — a detection-consistency correctness fix to an existing, already-ADR'd mechanism (ADR-198's
reward-hack monitor), not a new architectural decision (same reasoning as 2026-09-03's #276 threat-model
fix). Searched docs/adrs/INDEX.md; no gap requiring a new ADR for this class of change.

Research Gist

docs/dream-cycle/2026-09-07-gist.md (this branch). GIST=LOCAL — no gist-creation tool available in
this session's toolset, matching every prior Dream Cycle night. Full 2026 reward-hacking/leak-detection
research (OpenAI's Action-Monitor baseline, Cursor's and Datacurve's SWE-bench-Pro audits, Anthropic's
Nov-2025 reward-hacking paper, RHB taxonomy), 7-system competitor comparison, and 5-candidate STEP 3.2
scoring table (all 5 independent research roles converged on this candidate unprompted).

Issue

#292 (full findings, ledger check, prior-gist score, weight-eft/learn scan results, competitor review).

Witness

session_commit : d5833dc6512ac1adeeef91a331c29055cd8a4dbb
report_sha256   : 4a6e49dd59527e22924cd57cd571998cc04126d0ee2654a598936fdcc912e189
witness         : 2986185670f7f8ae3b5c9b9b569073b3e7910c9f24beedab7b30553242265f1e

Self-verified end-to-end (hash the gist up to the witness block → concatenate with session commit → hash
again) before this PR was opened.

Merge Policy

Human review required. Do not self-merge. Do not autonomously promote Flywheel state.

Recommendation: ACCEPT — a small (1 source file + 2 test/fixture files), deterministic,
independently-critiqued closure of a previously-disclosed training-data-integrity gap, directly in
tonight's scan surface, corroborated by 2026 field evidence (OpenAI's own Action-Monitor baseline already
covers tool outputs) from 5 independent research passes that converged on this finding unprompted.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01M2FPq8HeQ4oCJfMwg41VTj

---

Generated by Claude Code

[ruvnet]

Dream exact-head review — ACCEPT, limited to the deterministic Node software envelope.

The candidate adds response-side tool evidence to the reward-hack detector, proves a nonvacuous clean control, catches the response leak, preserves a clean response, and verifies the exporter drops the leak. Exact-head CI, Real Tools, Security, packaging/audit, and the reported Node/Rust/WASM matrices are green; the baseline/candidate distinction is explicit.

Accepted scope: the tested deterministic trajectory primitive and exporter behavior. This is not independent validation on live models, terminals, or a representative production corpus, and it grants no execution or promotion authority.