SOTA: qualify evaluator and tool interface instruments before benchmark promotion

[ruvnet]

Finding

Two Sep 3 frontier results expose one benchmark failure class.

1. arXiv:2609.04198 reports shared endpoint observer repeatability far below preregistered gates despite clean engineering. Same window ranking agreement was reported at Spearman 0.400 versus a required 0.90, with byte identical next day replay at 0.78 versus 0.99.
2. arXiv:2609.03966 reports interface induced trajectory censoring where valid emitted tool calls can be lost by the serving template and parser interaction, producing zero observed tool activity despite capable weights.

RuV impact

MetaHarness, Dream Machine, Ruflo, Cognitum routing, RVM security evaluation, Core Memory experiments, and RuVector benchmarks can all optimize against an instrument that is not stable enough to certify the candidate.

Proposed primitive

Add a dependency free InstrumentQualificationReceipt in @metaharness/bench with two independent checks:

1. repeat ranking reliability using deterministic Spearman correlation across matched repetitions
2. interface preservation from well formed emitted tool call to parsed call to executed call

The receipt must fail closed on insufficient samples, malformed rankings, non finite values, duplicate identities, or silent interface loss. Output is evidence only with authority: none.

Promotion gates

No benchmark may use the instrument as promotion evidence unless explicit policy thresholds pass. Defaults are not silently invented by the library.

Minimum reproduction program:

1. stable synthetic control
2. unstable ranking control
3. parser censorship control where emitted valid calls are not parsed
4. partial execution control
5. malformed input and resource bound tests
6. one real MetaHarness host plus one shared model endpoint

Report model and endpoint identifiers, adapter and template versions, seeds, sample size, correlations, call counts, latency, cost, failures, and reproduction steps.

Governance

Research, implementation, adversarial review, security, testing, reproducibility, and release roles are independent. No autonomous merge, deployment, credential changes, or evaluator threshold mutation after outcomes are observed.

Acceptance

A candidate implementation must detect every synthetic silent censorship case, reject insufficient or malformed reliability evidence, preserve stable controls, add zero runtime dependencies, keep existing bench behavior unchanged, and pass repository CI and security gates.

[ruvnet]

2026-09-08 update: Harbor Adapters and Harbor-Index (arXiv:2609.04298, submitted 2026-09-03) should be used as an interoperability and parity arm rather than creating another evaluator subsystem. The originating team ports more than 80 agent benchmarks, evaluates 8 models across 54 benchmarks with Terminus-2 plus native harnesses, and curates 82 audited hard tasks from 29 benchmarks. No evaluated model and harness exceeds 30% on Harbor-Index; the strongest reported configuration reaches 28.0%.

Add one MetaHarness qualification track that requires adapter parity against the benchmark origin, oracle validation, and matched native versus common-harness result ranges before an adapter can supply promotion evidence. This complements the current instrument receipt: instrument stability checks whether the measurement path is repeatable; Harbor-style parity checks whether adaptation preserved benchmark semantics.

Do not import another orchestration layer unless the parity experiment shows MetaHarness adapters materially diverge. Promotion target for any common adapter is oracle correctness on deterministic controls, native versus adapted task success within the preregistered equivalence margin, identical task corpus and verifier digests, and explicit adapter version in every receipt.