You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An honest-verdict harness contract for Foreman coders: a repo-agnostic, declare-then-verify pattern where a coder's GO is a claim package the gate audits deterministically, instead of a self-certification the gate rubber-stamps. Three layers, one primitive:
Work-class cross-check: the coder declares a work class (code-fix, docs, ci-policy, packaging, ...) in its verdict; the gate derives the actual class from the diff footprint and downgrades on mismatch. Which classes may self-GO is policy: configurable per Workload, default ci-policy/release-policy never self-GO.
Evidence contract: a gate-side detector scans added lines for empirical claims (numbers with units like tok/s, ms, GB; benchmark-table rows; certainty language like "validated"/"measured"). Every detected load-bearing claim must be matched by a coder evidence entry citing a source that exists (in-repo file:line, or a fetchable URL). Unmatched claims force NEEDS-VERIFICATION. Gate-side detection makes it fabrication-proof: omitting a claim from the evidence block does not help, the gate finds it in the diff.
Path-scoped verification registry: gate checks register diff-footprint globs (.github/workflows/** runs actionlint, **/*.sh runs shellcheck, Formula/*.rb runs brew style, **/*.md runs the claim scan) with repo-level override via .foreman/verify.yaml. The gate also reports coverage: what fraction of the changed lines any check actually verified, so GATE-PASS on a diff the gate could not inspect stops reading as a clean pass.
Problem Statement
A six-issue M2/M3 batch produced three GOs and zero mergeable ones; every hollow GO traced to the same structural gaps:
Acceptance A/B: re-run the same six-issue batch on the same coder model; green means zero hollow GOs (every GO is either mergeable or the coder correctly declined).
Builds on existing seams, not parallel machinery: GateProfile languages and the generic-gate runtime-missing semantics, the tiered gateCheckRegistry (blocking vs advisory), and the grounding package's ground-truth scan.
Alternatives Considered
Reviewer-model routing for unverifiable content: uses the existing reviewer fleet, but it is a model checking a model; reviewer-diversity experiments showed rubber-stamping, so honesty would depend on which reviewer is serving.
Forced human queue for any unverifiable diff: maximally honest but caps autonomy; docs/CI work would always need a human touch.
Pre-dispatch issue classification: cheaper than a wasted coder run, but classifying from issue text alone is unreliable (the Add container image scanning and supply chain security #233 diff, not its issue text, revealed the class); kept as a later optimization.
Feature Description
An honest-verdict harness contract for Foreman coders: a repo-agnostic, declare-then-verify pattern where a coder's GO is a claim package the gate audits deterministically, instead of a self-certification the gate rubber-stamps. Three layers, one primitive:
code-fix,docs,ci-policy,packaging, ...) in its verdict; the gate derives the actual class from the diff footprint and downgrades on mismatch. Which classes may self-GO is policy: configurable per Workload, defaultci-policy/release-policynever self-GO.evidenceentry citing a source that exists (in-repo file:line, or a fetchable URL). Unmatched claims forceNEEDS-VERIFICATION. Gate-side detection makes it fabrication-proof: omitting a claim from the evidence block does not help, the gate finds it in the diff..github/workflows/**runs actionlint,**/*.shruns shellcheck,Formula/*.rbruns brew style,**/*.mdruns the claim scan) with repo-level override via.foreman/verify.yaml. The gate also reports coverage: what fraction of the changed lines any check actually verified, soGATE-PASSon a diff the gate could not inspect stops reading as a clean pass.Problem Statement
A six-issue M2/M3 batch produced three GOs and zero mergeable ones; every hollow GO traced to the same structural gaps:
id-token: write.Proposed Solution
Design proposal in
docs/proposals/(PR to follow), then two implementation slices:.foreman/verify.yaml+ coverage reporting (closes [BUG] Coder gate is blind to non-Go changes (CI/shell/Ruby/docs), hollow GATE-PASS #1072 broadly).Builds on existing seams, not parallel machinery:
GateProfilelanguages and the generic-gate runtime-missing semantics, the tieredgateCheckRegistry(blocking vs advisory), and thegroundingpackage's ground-truth scan.Alternatives Considered
Additional Context
Refs #1072, #1073. Batch evidence: workload run-20260711-182721 (issues 850/233/822/440/699/631).
Priority
High: directly gates M2/M3 (autonomous batches are only useful if a GO can be trusted).
Willingness to Contribute
Yes, proposal and implementation PRs to follow.