feat(eval): report quality-constrained cost per localization

[fishmingyu]

Problem

CodeNib records trajectory tokens and, when LiteLLM can price a response, a runtime USD estimate. That is not yet a reproducible product or paper metric:

• runtime prices can change without leaving a versioned pricing source;
• the query-sweep record omits prompt, completion, and cache-write token fields;
• local models correctly have no API-dollar estimate;
• mean cost alone hides failed localizations and denominator differences;
• "cost per resolved query" would overstate a localization-only benchmark.

Part of #415 M5.

Metric Contract

The report uses cost per successful localization, not issue resolution.

• Default quality: answer_blocks recall@5.
• A query is successfully localized when its score reaches a declared threshold.
• Cost per success is total spend across all attempted paired queries divided by successful localizations, so misses are charged.
• An arm is quality-qualified only when the lower bound of its paired quality-delta confidence interval is above a declared non-inferiority margin relative to grep_only.
• Token counts remain primary. USD is either a preserved runtime estimate or an offline projection from an immutable pricing snapshot; the report labels the source.
• GitHub/Copilot premium requests are not inferred from model tokens.

Scope

• Load direct and orchestrator-sharded cell directories with strict duplicate, model, query, arm, and denominator audits.
• Preserve prompt, completion, cache-read, and cache-write token counts in every query-sweep cell going forward.
• Add a versioned pricing schema with model aliases, per-token classes, effective/retrieved dates, currency, and source URLs.
• Reprice cells offline only when the required raw token fields exist; never infer missing historical input/output splits.
• Report attempted queries, infrastructure success, localization success, quality, raw token classes, cost per attempt, cost per success, paired deltas, confidence intervals, and qualification status.
• Support optional shared build-cost amortization at declared query horizons without mixing it into model-call cost.
• Emit machine-readable JSON plus concise Markdown suitable for artifact evaluation and figure generation.
• Validate the report against the existing five-model, 500-query synthesis runtime results without rerunning model calls.

Out of Scope

• Patch generation or SWE-bench resolution.
• Claims about actual Copilot AI Credits or premium requests.
• Inventing API prices for self-hosted Qwen/Gemma deployments.
• Estimating GPU, queueing, or operator cost without an explicit supplied rate.
• Reconstructing missing historical token classes from total tokens and USD.

Acceptance Criteria

• Synthetic tests cover quality failures, infrastructure failures, unmatched pairs, duplicate cells, cache pricing, missing token classes, and zero-success arms.
• Existing cloud runs produce recorded-cost reports with explicit provenance; local runs produce token-per-success reports with USD marked unavailable.
• A future split-token fixture can be deterministically repriced from a pinned catalog.
• Changing a pricing snapshot changes projected USD but never raw token or localization metrics.
• The full unit tier and pre-commit pass.

[fishmingyu]

Draft implementation is available in #427.

The retained five-model audit completed without new model calls:

• 8,500 input cells; 7,500 analyzed after the explicit arm allowlist
• 500 strictly paired queries and 25 repository clusters per model
• quality guard: paired localization Recall@5 with repository-clustered bootstrap confidence intervals
• selected token-minimizing admitted policy: Haiku eager; Gemini, Gemma, Qwen3.5-9B, and Qwen3.5-27B compact
• token cost per successful localization is 49.2%, 68.1%, 87.5%, 53.9%, and 54.4% lower than paired grep/read, respectively
• USD is reported only when the exact provider/model/token-class pricing contract is available; local and incomplete records remain unavailable

One Qwen3.5-27B eager retry lacks metered usage, so that arm's aggregate cost is deliberately unavailable rather than silently underestimated. The report claims localization cost only, not issue-resolution rate or Copilot-credit savings.