Skip to content

AORC2F/LSTM Q' campaign: experiment configs, findings, and eval corruption hardening#28

Open
taddyb wants to merge 1 commit into
masterfrom
aorc2f-lstm-qprime-campaign
Open

AORC2F/LSTM Q' campaign: experiment configs, findings, and eval corruption hardening#28
taddyb wants to merge 1 commit into
masterfrom
aorc2f-lstm-qprime-campaign

Conversation

@taddyb

@taddyb taddyb commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Summary

This branch carries the AORC2F/LSTM Q' identifiability campaign: four new experiment configs, three findings docs (wave 1, cross-wave comparison, disagg-head sensitivity), comparison/plotting tooling, and the eval-corruption hardening that the campaign runs surfaced.

Experiment configs

  • config/experiments/aorc2f_{distributed,lumped}_frozen_chunk1.yaml, lstm_daily_frozen_chunk1.yaml, lstm_hourly_native.yaml — the 4-source campaign templates.

Findings docs

  • docs/2026-07-16-aorc2f-wave1-findings.md — wave 1 results.
  • docs/2026-07-16-wave2-cross-wave-findings.md — cross-wave comparison, later amended to correct the backend-mismatch dismissal and test the double-routing hypothesis against the store (mixed result; controls still missing).
  • docs/2026-07-16-disagg-head-sensitivity-findings.md — disaggregation-head sensitivity.
  • docs/nh-qprime-store-contract.md updated.

Eval hardening (src/)

  • Detect silently-corrupted eval chunks instead of writing bad predictions (src/training/eval.rs, src/bin/eval.rs), strengthened with a panic-hook flag so corruption can't slip through as a swallowed panic.
  • Small supporting changes in src/cli/run.rs, src/cli/system.rs, src/cuda_graph/geometry_kernel.rs, src/sparse/cusparse.rs, src/data/error.rs.
  • Doctest fix: fence pseudocode formulas so rustdoc doesn't compile them.

Tooling & skills

  • scripts/generate_run_notebooks.py, scripts/hydrograph_comparison.py, scripts/wave_comparison_plots.py — per-run notebook generation and cross-wave comparison plots.
  • .claude/skills/ddrs-run-and-operate/SKILL.md — documents the GPU eval OOM incident (cubecl worker-thread panic) and its fix.

Invariant check

Touches src/sparse/cusparse.rs and src/cuda_graph/ — per CLAUDE.md, cargo run --release --example compare_ddr_sandbox must still report ABSOLUTE MATCH before merge.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant