Finding
PatchBench, arXiv:2609.04075, submitted 2026-09-03, shows that PoC-crash-only validation can materially overstate autonomous vulnerability patch quality. The originating team reports an average 1.83x solve-rate inflation across 11 state-of-the-art agents when evaluation relies on the original PoC alone. They also report substantial similarity to historical developer patches in 25% of agent patches and a recurring failure mode where agents suppress the observed crash without repairing the root cause.
Treat all results as originating-team measured until RuV reproduces them.
RuV implication
Dream Machine and MetaHarness must not promote security patches merely because the triggering crash disappears or the existing regression suite passes. Security repair evaluation needs an independent root-cause contract.
Proposed evaluation contract
For security-patch candidates, freeze before candidate generation:
- vulnerability class and affected invariant
- original PoC
- at least one transformed or transplanted PoC that preserves the vulnerability but changes superficial crash structure
- regression suite
- negative controls that should remain valid
- root-cause oracle or independently reviewed security property
- patch-diff similarity check against any historical or reference fix available to the evaluator
- performance and compatibility budget
A candidate is SECURITY_ACCEPT only when the root-cause property is repaired, all transformed attacks fail, negative controls remain valid, and protected behavior does not regress.
Benchmark
Compare:
- crash-only validation
- crash plus regression tests
- root-cause validation with transformed or transplanted attack cases
Report apparent solve rate, root-cause solve rate, false promotion rate, historical-patch similarity, regressions, runtime, model cost, evaluator cost, and exact reproduction artifacts.
Falsification
If transformed attacks do not materially change promotion decisions on RuV-native security tasks, retain the simpler existing evaluator and record the negative result.
Acceptance
The stronger evaluator must reduce false security promotion by at least 50% on the frozen adversarial corpus, preserve legitimate patch acceptance within 3 absolute percentage points, and add no authority to the candidate agent. Evaluator artifacts are immutable after outcomes become visible. No autonomous merge or deployment.
Finding
PatchBench, arXiv:2609.04075, submitted 2026-09-03, shows that PoC-crash-only validation can materially overstate autonomous vulnerability patch quality. The originating team reports an average 1.83x solve-rate inflation across 11 state-of-the-art agents when evaluation relies on the original PoC alone. They also report substantial similarity to historical developer patches in 25% of agent patches and a recurring failure mode where agents suppress the observed crash without repairing the root cause.
Treat all results as originating-team measured until RuV reproduces them.
RuV implication
Dream Machine and MetaHarness must not promote security patches merely because the triggering crash disappears or the existing regression suite passes. Security repair evaluation needs an independent root-cause contract.
Proposed evaluation contract
For security-patch candidates, freeze before candidate generation:
A candidate is
SECURITY_ACCEPTonly when the root-cause property is repaired, all transformed attacks fail, negative controls remain valid, and protected behavior does not regress.Benchmark
Compare:
Report apparent solve rate, root-cause solve rate, false promotion rate, historical-patch similarity, regressions, runtime, model cost, evaluator cost, and exact reproduction artifacts.
Falsification
If transformed attacks do not materially change promotion decisions on RuV-native security tasks, retain the simpler existing evaluator and record the negative result.
Acceptance
The stronger evaluator must reduce false security promotion by at least 50% on the frozen adversarial corpus, preserve legitimate patch acceptance within 3 absolute percentage points, and add no authority to the candidate agent. Evaluator artifacts are immutable after outcomes become visible. No autonomous merge or deployment.