methodology: preregistration for corrective Qwen3.6-vs-3.8 study (official-sampler fix, 6-seed crossover, clustered inference) - #47
Draft
Lightheartdevs wants to merge 16 commits into
Conversation
The p1_testwrite family produced discovery-only loops that shipped no gradeable artifact. Add a strict system prompt, require CHANGELOG.md / decisions.md / research.md via --require-files, and tighten the per-cell stuck threshold to 250 iterations for that family only. Every other family keeps the historical 500-iteration default and an empty require-files set, so their behaviour is unchanged. Ran live in the qwen3.8-27b-q4-t1-t3 campaign: p1_testwrite think-xhigh went 0/3 to 3/3.
…6T14:23:09Z) New entry benchmarks/qwen36-vs-qwen38-27b-2026-08: 802 frozen cells across 9 sampler/mode arms plus a 19-cell Q8_0 control, every number derived from the frozen dataset mmbt-frozen-dataset-v2.csv (sha256 d2ed0bec..., stamped 2026-08-16T14:23:09Z; supersedes the 746-cell freeze #1). Finding: the one large, robust effect is a no-think delivery regression, not a capability gap. 3.8's identical-call-loop rate is +28.0 pp at the matched sampler (Fisher p=4.9e-09) and +29.2 pp at the vendor points (p=1.1e-11), disappears in think mode, and reproduces at Q8_0 (6/19, Wilson CI excludes zero). Conditional on delivery and after grader corrections the models are within noise at the matched sampler (86.0% vs 86.2% no-think); the vendor-point conditional gap (~15 pp) survives correction and is left unresolved. Corrections: three verified MMBT grader defects, corrected in a non-destructive overlay (165 per-cell records, digest e332cd2c...) - D1 word-gate tokenizer mismatch (62 counter-dependent verdicts invalidated), D2 p3_pm keyword literalism (36 verdict flips; the repo's own tested fix had been scoped to gemma4 cell names and never ran on these campaigns), D3 p2_triage brief/ground-truth contradiction (24 flips). grade.json, briefs and ground truth untouched. Retractions: five earlier conclusions from this investigation are retracted in claims.yaml (operator-label abort rate, p3_pm and p2_triage family results, the 9-3 family win split, the "default vs default" thinking comparison). Seven new claims added, all provisional or held. KNOWN-LIMITATIONS gains freeze-extract, replicate-depth, seed-correlation, missing-think-arm, empty-hand-rating, p3_market, and Q8_0 quantization caveats. Also includes the campaign harness fix (bench_autopilot.py: scope sandbox cleanup to the running arm instead of a global docker wipe) and the campaign configs this investigation ran on (cfg-offspec.json, qwen3.8-27b-q4-t03-matched-mmbt.json, fleet ensure-scripts). Validation: verify_overlay.py green (idempotence, leniency invariant 0 violations over 686 cells, write guard, 853 protected files unmutated, empty post-freeze-divergence ledger); build_results.sh deterministic rebuild byte-identical across runs; validate_fixes.py 174 checks, 0 failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rvives The repo-wide "*.csv text eol=lf" attribute was normalizing the freeze-2 extract in the object store (python csv.writer emits CRLF terminators), so the committed blob hashed f28a37b7... instead of the stamped d2ed0beca5b68e9ca63788e452235f23a299af06639893f558bef19f784cf018 that every document in the entry cites, and sha256sum on a fresh checkout would fail. Pin the file binary per the existing policy for SHA-anchored corpus files and recommit the verbatim bytes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… B1-B11 bias list) House-convention audit doc for the entry: what is locked across both models, what varies, what the comparison can and cannot support, and the full bias list - including the three that cut against the headline (B2 residual quant thinness, B5 matched-regime asymmetry, B11 unresolved recoverable workspaces). All numbers at freeze #2 (802 cells, 2026-08-16T14:23:09Z). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ective study This commit is the preregistration boundary — methodology before results. No corrective-campaign cell has run; everything here is frozen before any new result ingestion, per PREREGISTRATION.md (sha256 f5be6189bc3b43409dab3c070a29b43a42bcd2948446e1b1a81c20b87f489a0c). - benchmarks/qwen36-vs-qwen38-corrective-2026-08/PREREGISTRATION.md: binding protocol (arms incl. PRIMARY official-nothink, six fixed seeds 101-601, Towers 1/3 host crossover at 500 W, automated loop terminator at exact consecutive run >= 30, fixed N, statistical analysis plan, claim-language contract). - benchmarks/qwen36-vs-qwen38-corrective-2026-08/protocol/CARD-EVIDENCE.md: firsthand pinned-revision card fetches — both README sha256s (Qwen3.6-27B @ 6a9e13bd, Qwen3.8-27B @ 1d4bf0f2), the byte-identical generation_config.json bodies, and the exact quoted sampler lines with line numbers (3.6 L635-637; 3.8 L252-253, worked example L451-461). - benchmarks/qwen36-vs-qwen38-27b-2026-08/findings.md, AUDIT.md: prepended EXPLORATORY notices per protocol section 9 — the V36 vendor no-think arm ran the thinking sampler in non-thinking mode (off-spec), and cell-level Fisher there is descriptive only. Notices only; no number altered; the 802-cell corpus and its receipts remain untouched as exploratory evidence. - tooling/graders/v2/: D1 wc -w counting contract (+-3% band), D2 p3_pm semantic R3, D3 p2_triage brief-aligned ground truth + brief-parsed vocabulary, CHANGES.md provenance, tests. v1 graders untouched. - tooling/tasks/v2/: briefs stating the counting contract and the offline p3_market mirror; v1 briefs untouched. - tooling/fixtures/: frozen offline p3_market corpus (36 snapshot pages + hashed manifest + unfetchable ledger), deterministic stdlib mirror, and the protocol section 3 determinism gate (passed). - tooling/corrective/: loop terminator (metric identical to protocol section 4), delivery validator, evidence manifest + fixed-N balance checker, cell supervisor with quarantine/rerun ledger, crossover runner, arm configs, and 32 tests (64 green repo-wide with graders/v2). configs/family_artifacts.json corrected at freeze: artifacts the briefs explicitly mark optional are no longer delivery-gated (they would have misclassified brief-compliant deliveries as non-delivered). - tooling/harness.py: --sandbox-network (default bridge = zero behavior change) + sandbox_runtime.sandbox_network receipt field, for the offline p3_market family. - tooling/scripts/run_microbench.sh: BENCH_SEED -> --seed per-cell seed plumbing, BENCH_TASK_ONLY family allowlist, and resume-skip acceptance of automated terminal labels (loop-run30/timeout). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lightheartdevs
marked this pull request as draft
August 16, 2026 18:49
… manifests for the corrective crossover ensure-q36-tower1-only.sh / ensure-q38-tower3-only.sh mirror the phase_a lane scripts line for line: identical llama-server flags, only the model path + alias (and hosting tower) swapped, plus one added guard refusing to start while the opposite-phase bench container is running (same GPU, same remote loopback port 11434). Both verified read-only (--check) against the live fleet: model file sha256s, image digest + revision, llama-server binary sha256, GPU UUID, 500 W cap, tunnel unit sha256s all match the pins. phase_a.json / phase_b.json are BENCH_SERVING_MANIFEST payloads (harness.py record_environment embeds them and resolves the lane by topology.lanes[].coordinator_port — resolution verified for both ports in both manifests with the harness function itself). Both towers hold both GGUFs locally; every model file sha256 was measured on both towers 2026-08-16 (q36 ff6941de…, q38 bee238bb… on tower1 AND tower3), so the crossover swaps hosts, never artifacts. No model copies were needed. run_crossover.sh --dry-run preflight against the current phase_a fleet: power caps + phase_a endpoint aliases PASS; ODS drain FAIL (ods-llama-server id 834bdb6c901d still serving on tower1, recorded for restoration); phase_b endpoint checks FAIL until the swap is performed. Campaign not started. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…l s.7)
BUILD ONLY: nothing here has been executed against the fleet. DSV4 keeps
serving; the window opens only after the orchestrator drains it.
- run_quant_pilot.sh: bounded-window runner. 2 models x {UD-Q4_K_XL, Q8_0}
x seeds 101/211 x 12 families = 96 cells on Tower2's two RTX PRO 6000s,
GPU<->model assignment crossed between the seeds; two co-residency waves
per seed (Q4 then Q8), containers torn down between waves. Fail-closed
preflight (refuses while deepseek-v4-flash-0731 runs; pinned hostname /
GPU UUIDs / 500 W caps / drained memory / image Id+revision / free ports
/ model size+sha256), llama-server argv byte-identical to the historical
ensure-*-only.sh lanes except --model/--alias, per-container pinned
binary sha proof, per-seed serving manifest via BENCH_SERVING_MANIFEST,
cells through cell_supervisor.py (loop terminator, 3 h ceiling, rerun
ledger), window ledger + evidence manifest build/check at close, and the
orchestrator-owned DSV4 restore reminder printed at window close.
- configs/quant-pilot.json: exploratory arm at the official-nothink
sampler, thinking off; per-model quant + artifact pins verified today:
q38 Q4 bee238bb / q38 Q8_0 a680f44a (local file hashed; byte-identical
to unsloth/Qwen3.8-27B-GGUF Q8_0 at pinned rev fe1e2a23) / q36 Q4
ff6941de (all three match the HF LFS oids at their pinned revisions).
- download-qwen36-q8_0.sh: written, NOT run. Qwen3.6-27B Q8_0 is absent
from /mnt/bulk/models; pinned to unsloth/Qwen3.6-27B-GGUF rev 82d411ac,
Qwen3.6-27B-Q8_0.gguf, 28595763424 bytes, sha256 f93f517f...; resume +
size + full-sha256 verification before install into
/mnt/bulk/models/qwen3.6-27b-q8_0/.
- evidence_manifest.py: expected_cells takes model-level quant with
fallback to arm-level (default-preserving; existing manifest rows are
byte-identical). Covered by a new test_evidence_manifest.py case; all
33 corrective tests pass.
- README.md: quant-pilot machinery documented; the "deliberately NOT
configured" limitation replaced with the runner pointer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ap for the corrective study Firsthand nvidia-smi collection on tower1/tower3 (RTX 5090, driver 595.84, cap 500 W of 600 W max) and tower2 (2x RTX PRO 6000, driver 595.58.03, cap 500 W), plus the live-verified tunnel endpoint map (18101 -> Qwen3.8, 18103 -> Qwen3.6, /v1/models + 1-token completion probes). Part of the PREREGISTRATION section 5 manifest/ pinning pass; model/image/binary pins remain in tooling/deployments/qwen-corrective/phase_a.json + phase_b.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…configs -> phase_a serving manifest Closes the campaign-blocking wiring gap recorded at fleet-prep: run_microbench.sh hard-coded the v1 briefs and never passed --sandbox-network, so p3_market would have run its live-web brief on the default bridge (PREREGISTRATION section 3 violation) and the six v2 families would have run briefs the v2 graders do not parse (section 8). Default-preserving env wiring (BENCH_TASK_BRIEFS=v2, BENCH_SANDBOX_NETWORK), set per cell by cell_supervisor.py (network for p3_market only, per tooling/fixtures/README.md step 3). Verified by dryrun_v2_wiring_capture.sh (5 scenarios incl. env-unset byte-identical historical behavior) + existing dryrun_argv_capture.sh regression. Arm configs gain the launch-time serving_manifest key (deployments README step 5, phase_a). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the remaining section 5 pins for the corrective study: harness sha256 (HEAD and on-disk, matching), the full v2 grader set (incl. tests + ground truth), v1 exploratory-reference graders, task briefs (v1 + v2), p3_market fixtures + serve_fixtures.py, pinned card README/generation_config shas (copied from protocol/CARD-EVIDENCE.md), Q8_0 pilot model pins (q38 Q8_0 verified by direct hash on tower2; q36 Q8_0 remote pin from quant-pilot.json, not yet downloaded), llama.cpp image digest + llama-server sha (copied from phase_a.json), and the legacy-harness exclusion sha fdbc1584. Serving manifests, host inventory, and card evidence remain pinned at their already-committed paths, referenced not duplicated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preregistration boundary - methodology before results
Corrects the defects that put PR #46 on hold. Committed BEFORE any new result ingestion; the 802-cell freeze in #46 is relabelled exploratory (insertion-only notices).
Results will follow in a separate PR that must reproduce via the committed pipeline.
🤖 Generated with Claude Code