Skip to content

STATUS.md

Latest commit

 

History

History
76 lines (54 loc) · 6.34 KB

STATUS.md

File metadata and controls

76 lines (54 loc) · 6.34 KB
Raw

Status

Generated by scripts/status.py — do not edit by hand. Read from 17 artifact result file(s) at 2026-07-28 18:44 UTC.

Numbers here are read out of provenance-stamped artifacts. See RETRACTIONS.md for why this file is generated rather than written.

Provenance

  • devices present: ['cuda']
  • smoke artifacts present: False
  • assert_reportable requires CUDA + bf16 + pinned revision + clean tree; anything from a dev machine is non-reportable by construction.

Pass C -- H1 (primary: lambda_chose - lambda_yoked, predicted NEGATIVE)

model device outcome lambda_median hdi_low hdi_high P(<0) sesoi mde/sesoi equiv_reachable decision
gemma-2-2b-it cuda primary-inconclusive -0.0015 -1.0755 1.0901 0.5016 0.2355 1.373 True inconclusive
llama-3.2-3b-instruct cuda primary-inconclusive 0.4361 0.2125 0.699 0 0.1665 1.9872 False inconclusive
qwen2.5-1.5b-instruct cuda primary-inconclusive -0.1367 -0.9995 0.6477 0.6328 0.2418 1.434 True inconclusive
  • models with a full 9.2 pass: 0
  • models directional at P(lambda<0) >= 0.95: 0
  • A3.7 project gate: Stage 1 NOT entered (needs one full pass AND two directional)

inconclusive is the MODAL outcome by design, not a surprise: A3.3 puts it at ~48% even when the effect is real and exactly at the SESOI, and ~70% under a true null. Read it against the MDE, not against zero.

A credible effect in the WRONG (positive) direction scores inconclusive under 9.2 as written, since pass requires the HDI to exclude 0 in the PREDICTED direction and fail requires the whole HDI inside the ROPE. That is the rule as preregistered; it is reported here rather than amended.

Pass C robustness -- A4.1 non-centered template effect

model device outcome lambda_median hdi_low hdi_high P(<0) sesoi mde/sesoi equiv_reachable decision
gemma-2-2b-it cuda primary-inconclusive 0.0003 -1.0361 1.0915 0.4996 0.2355 1.373 True inconclusive
llama-3.2-3b-instruct cuda primary-inconclusive 0.4401 0.2148 0.6706 0.0001 0.1665 1.9872 False inconclusive
qwen2.5-1.5b-instruct cuda primary-inconclusive -0.1442 -1.0034 0.5975 0.6366 0.2418 1.434 True inconclusive

A4.1 non-centred u_template after llama's centered fit showed 362 divergences in a sd_template funnel. The reparameterisation is pure -- same model, same posterior in expectation -- and was verified on synthetic data before it touched the real fit.

The centered fit above remains PRIMARY (A4.1 commitment 2). This table is a robustness check and is reported whichever way it moves; that was fixed before the refit ran so it could not be chosen afterwards.

Absolute-Likert instrument (A1.5 validation record)

model device median_rho sigma_coll sigma_asc icc_c1_asc passed
gemma-2-2b-it cuda -0.944 0.1788 0.8874 0.3511 False
llama-3.2-3b-instruct cuda -0.9221 0.1593 0.6355 0.4607 False
qwen2.5-0.5b-instruct cuda -0.9639 0.0523 0.5621 0.5398 False
qwen2.5-1.5b-instruct cuda -0.9156 0.0976 0.5019 0.343 False
qwen2.5-3b-instruct cuda -0.7093 0.3243 0.8828 0.259 False

All models are expected to fail polarity validity here. That is the point of the record.

Pairwise instrument (Amendment 2 model)

model device outcome sigma_item post_sd sep_ratio reliab_emp reliab_model beta_min beta_max excess_slope excess_flat
gemma-2-2b-it cuda completed 1.5699 0.2946 5.3292 0.8576 0.9631 0.6693 1.7299 -0.0009 True
llama-3.2-3b-instruct cuda completed 1.0986 0.2786 3.9439 0.7398 0.9378 0.4207 3.7007 0.0324 False
qwen2.5-0.5b-instruct cuda halted-reliability nan nan nan 0.0834 0.7692 1.9776 8.6333 0.034 False
qwen2.5-1.5b-instruct cuda completed 1.6119 0.3222 5.0022 0.7429 0.9595 -8.7304 0.2282 0.0197 False
qwen2.5-3b-instruct cuda halted-reliability nan nan nan 0.4732 0.9611 -3.3954 -0.6566 0.0144 False

reliab_emp is the A2.2 gate (threshold 0.70). A large gap between reliab_emp and reliab_model, or excess_flat = False, indicates the order model is still misspecified.

Readout mass: fraction invalid, by template (A1.6)

model t0 t1 t2 t3 t4
gemma-2-2b-it 0 0 0 0 0
llama-3.2-3b-instruct 0 0 0 0 0
qwen2.5-0.5b-instruct 0 0 0 0 0
qwen2.5-1.5b-instruct 0 0 0 0 0
qwen2.5-3b-instruct 0 0 0 0 0

A template above ~0.1 has a prompt bug. This is the cheapest place to catch one.