Skip to content

[FEATURE] Honest-verdict harness: declare-then-verify contract for repo-agnostic coder gates #1075

Description

@Defilan

Feature Description

An honest-verdict harness contract for Foreman coders: a repo-agnostic, declare-then-verify pattern where a coder's GO is a claim package the gate audits deterministically, instead of a self-certification the gate rubber-stamps. Three layers, one primitive:

  1. Work-class cross-check: the coder declares a work class (code-fix, docs, ci-policy, packaging, ...) in its verdict; the gate derives the actual class from the diff footprint and downgrades on mismatch. Which classes may self-GO is policy: configurable per Workload, default ci-policy/release-policy never self-GO.
  2. Evidence contract: a gate-side detector scans added lines for empirical claims (numbers with units like tok/s, ms, GB; benchmark-table rows; certainty language like "validated"/"measured"). Every detected load-bearing claim must be matched by a coder evidence entry citing a source that exists (in-repo file:line, or a fetchable URL). Unmatched claims force NEEDS-VERIFICATION. Gate-side detection makes it fabrication-proof: omitting a claim from the evidence block does not help, the gate finds it in the diff.
  3. Path-scoped verification registry: gate checks register diff-footprint globs (.github/workflows/** runs actionlint, **/*.sh runs shellcheck, Formula/*.rb runs brew style, **/*.md runs the claim scan) with repo-level override via .foreman/verify.yaml. The gate also reports coverage: what fraction of the changed lines any check actually verified, so GATE-PASS on a diff the gate could not inspect stops reading as a clean pass.

Problem Statement

A six-issue M2/M3 batch produced three GOs and zero mergeable ones; every hollow GO traced to the same structural gaps:

Proposed Solution

Design proposal in docs/proposals/ (PR to follow), then two implementation slices:

  • Slice 1: evidence contract + work-class cross-check (kills both false-GO classes observed in the batch).
  • Slice 2: path-scoped checker registry + .foreman/verify.yaml + coverage reporting (closes [BUG] Coder gate is blind to non-Go changes (CI/shell/Ruby/docs), hollow GATE-PASS #1072 broadly).
  • Acceptance A/B: re-run the same six-issue batch on the same coder model; green means zero hollow GOs (every GO is either mergeable or the coder correctly declined).

Builds on existing seams, not parallel machinery: GateProfile languages and the generic-gate runtime-missing semantics, the tiered gateCheckRegistry (blocking vs advisory), and the grounding package's ground-truth scan.

Alternatives Considered

  • Reviewer-model routing for unverifiable content: uses the existing reviewer fleet, but it is a model checking a model; reviewer-diversity experiments showed rubber-stamping, so honesty would depend on which reviewer is serving.
  • Forced human queue for any unverifiable diff: maximally honest but caps autonomy; docs/CI work would always need a human touch.
  • Pre-dispatch issue classification: cheaper than a wasted coder run, but classifying from issue text alone is unreliable (the Add container image scanning and supply chain security #233 diff, not its issue text, revealed the class); kept as a later optimization.

Additional Context

Refs #1072, #1073. Batch evidence: workload run-20260711-182721 (issues 850/233/822/440/699/631).

Priority

High: directly gates M2/M3 (autonomous batches are only useful if a GO can be trusted).

Willingness to Contribute

Yes, proposal and implementation PRs to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/foremanForeman: the agentic fleet orchestrator add-onenhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions