feat: NH Q' store contract, hourly-native reading, and ddrs import#24
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds scripts/make_streamflow_fixtures.py and the three generated stores it produces under tests/fixtures/. Deterministic values let later Rust tests assert exact elements against the resolution-sniff reader. xarray normalises "hours since 1981-01-01 00:00:00" to drop the time component on write; the generator patches the zarr attr back after to_zarr() so the on-disk string is exactly "hours since 1981-01-01 00:00:00". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…E entry) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…its test, doc fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolution sniffed from CF time units at open. Daily path unchanged (read_slab is the old read_window_daily body, renames only). Hourly stores slice the native axis in read_window/read_test_window and 24h-mean in read_window_daily. Also adds Debug impl to StreamflowSource (manual, to avoid propagating the bound to ZarrArray internals) — required by the fixture test.
Review fixes on b1efd1e: bound the baseline-path transient to 32-day blocks, unit-test the hour-0/contiguity checks, pin qr_units, make hourly_to_daily_mean row-major friendly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… source Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… group Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ck, report labels Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the Task 9 disagg-guard negative test: run() wrote status=failed + exit_reason to the manifest but returned Ok, so every workflow failure exited 0. Manifest is still written before the error returns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Covers: change-control, debugging-playbook, failure-archaeology, architecture-contract, hydrology-reference, config-and-flags, build-and-env, run-and-operate, diagnostics-and-tooling, validation-and-qa, docs-and-writing, external-positioning, identifiability-campaign, proof-and-analysis-toolkit, research-frontier, research-methodology. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ities - ddrs-identifiability-campaign: description repointed to selective-equifinality paper (leakance gate CLOSED); Phase C NO-GO entry added; Phase B section marked CLOSED with observation-operator mechanism. - ddrs-failure-archaeology: 2026-07-06 NO-GO entry added (root cause: Σ-zeta observation operator is not invertible; every rival explanation REFUTED). - ddrs-research-frontier: Frontier 2 (leakance under fixed objective) marked CLOSED with strikethrough milestone. - ddrs-config-and-flags: leakance_losing_only, leakance_impervious_threshold, experiment.state_cache added; data_sources.attributes list form documented. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…kills index README.md: - data-source groups table now lists all 5 shipped groups (conus, conus-hourly, global, daily-lstm, hourly-lstm) with notes on each - new "Importing a Q' store" section documents ddrs import + hourly-native routing + link to nh-qprime-store-contract.md - leakance section retitled "NOT promotable"; 2026-07-06 NO-GO status added CLAUDE.md: - leakance status block replaced: GO-marginal → CLOSED/NO-GO with mechanism (Σ-zeta observation operator not invertible) and campaign doc links - workspace layout: checkpoint path corrected from flat .mpk to directory - architecture tree: rskan v0.1.0 → v0.1.3 .claude/skills/ddrs-overview/SKILL.md: new global index skill — routes any task to the correct sub-skill; surfaces the 5 critical facts every agent needs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…U arms 1-3) Pre-registers H1-H4 (geometry convergence, n bias-absorption, selective gradient alignment, gauge-local reachability) before any run. 3 arms: daily-lstm flat / daily-lstm disagg / hourly-lstm native, CPU NdArray, seed 42, standard windows. Infra: --backend flag on ddrs run + parameter- gradient probe adapted from probe_zeta_gradient. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…), dHBV2-UH common reference as sensitivity Per user decision: each arm's geometry is evaluated at its own eval-window median summed upstream Q' (the flow regime it actually routes); the common reference guards against conflating Q' magnitude disagreement with geometry divergence. Level-2 restricted to fully-covered upstream networks in all arms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
8 tasks: --backend flag on ddrs run, --params probe generalization, 3 tracked arm configs, smoke-timing gate, detached sequential runs, per-arm dumps+probes, 4-level convergence analysis, findings doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CPU arm uses NdArray (deterministic), forces sparse_solver=cpu and use_cuda_graphs=false, and skips the GPU pre-flight. Workflow internals were already backend-generic; dispatch/plot-post-step now bind the backend at runtime. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ality review) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ent probing
ProbeLeaves generalized to named leaves; leakance assert now conditional
on probing leakance params; generic grad_<name>_{abs,net} NetCDF writer.
Legacy leakance grad mode (no --params) byte-identical.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, hourly-lstm native)
…root-workspace pinned) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… (4 levels, H1-H4 verdicts) 6-stage pipeline (each cached to .npz) measuring parameter convergence across R1/R2/R3 LSTM-source arms: network + coverage closure (A), summed upstream Q′ per store (B), raw-param Spearman + realized geometry (C), routing skill vs baseline (D), gradient cosine alignment + gauge- distance reachability (E), verdict printing + figures (F). Verdicts.json + spread CDFs / gradient cosine bar chart / |g|-vs-distance figures written to --out. Dev cross-check verifies topological accumulation (headwater identity + brute-force outlet sum, both PASS). Geometry spread at common reference uses the sensitivity-set-aligned params (fix for mismatched set sizes between primary and sensitivity analysis sets). Run from ~/projects/ddr via `uv run python`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ean-Q disagreement - bfs_upstream_distance: raise default max_sweeps 5→200; add for/else warning when fixpoint not reached at exhaustion - stage_b: compute and save mean_q alongside median/p10/p90; cache-load falls through to recompute when mean_q missing from old cache - stage_c: use b_r12/b_r3["mean_q"] (not median_q) for H2 Q'-disagreement; cache-load branch reconstructs nested geo_spread_own / geo_spread_common dicts instead of returning flat np.load dict (KeyError on Stage F access) - stage_e: add bin_labels to np.savez_compressed; cache-load branch reconstructs nested h3 / h4 / bin_labels instead of flat dict - stage_a: assert np.all(local_down >= local_up) after eval-network remap - load_qprime_for_eval_network: replace .sel(time=boolean_mask) with .sel(divide_id=...).isel(time=np.flatnonzero(t_mask)) - H4 cell verdict: tighten SUPPORTED to violations==0 (was <=1) per spec Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on + minor hardening
- Stage B: np.median/percentile/mean → nan-safe variants (Q' stores carry NaN)
- Hourly aggregation: replace Python hour loop with vectorized np.add.at on
transposed arrays; NaN hours excluded from both sum and finite-count denominator
- Stage C geo_common cache: positional list()[i] → named dict access by key
- spearmanr guards: skip call + return nan when either input has <2 unique values
- H2 INCONCLUSIVE: now reachable when rho is NaN (was dead branch)
- Stage C params guard: sys.exit with actionable message on missing q_spatial/p_spatial
- Stage E npz: add sign_agree_{n,q,p} arrays alongside cosines
- _dev_crosscheck_topo: replace 28-line self-contradicting comment with 2-line
correct explanation (headwater = index never in local_down)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ch eval network) Load the hourly eval window in 30-day blocks (720 h × n_reaches per block) instead of all at once. Peak extra memory per block ≈ 0.38 GB vs the previous 70+ GB for the full 132,336 × 131,496-hour load. Adds --hourly-block-days CLI flag (default 30) for equivalence verification. Equivalence confirmed: block-30 vs single-block produces max_abs_diff = 0.000e+00 on all four stage_b arrays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…TED, H4 INCONCLUSIVE (n, not geometry, is the cross-source-stable parameter) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…asured wall times Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mmon-mode-removed H3 + noise ceiling Registered verdicts unchanged; audit metrics reported alongside under verdicts.json 'audit' key. Per /tmp/experiment-handoff-lstm-equifinality-audit.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ngraded to INCONCLUSIVE Metric asymmetry disclosed (like-for-like n rel-spread 0.4512 > all geometry, reversing H1's direction); timing-axis H2 also negative (rho -0.233/-0.211); CM-removed H3 shows n's alignment was entirely common mode (R1-R3 residual 0.050) against the -1/(k-1) null; noise ceilings 0.14-0.63 mark the gradient instrument as underpowered. Registered verdicts unchanged. Script's wrong auto-assessment replaced with null-aware reporting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StreamflowStoresniffs daily vs hourly from the CF time units at open (days since/hours since, anything else hard-errors). Hourly stores are sliced natively inread_window/read_test_window(no repeat-24, no disagg) and 24h-mean-aggregated inread_window_daily(chunked in 32-day blocks to bound baseline-path memory). The daily path is byte-identical —read_slabis the old body with renames only, verified mechanically in review.ddrs import <store> [--name <group> | --dry-run]: validates any store against the new DDR Q' store contract (schema, units, time axis, sample read, COMID coverage vs the resolved adjacency) and registers it as aconfig/sources/group. The NH-side producer (forward_merit.py) stays in the neuralhydrology repo; the contract doc is the inter-repo interface.kan_head.disaggregation+ hourly-native source is a config error at dataset open; dataset open logsstreamflow resolution: Daily|Hourlyas a smoke-run self-check. Bonus fix surfaced by the negative test:ddrs runnow exits non-zero on workflow failure (previously always exited 0 with only the manifest recording the failure).daily-lstm(CudaLSTM, 288,421 divides, 1981→2020) andhourly-lstm(MTS-LSTM, 197,088 divides, 350,640 hourly steps) ship as source groups; all four unit-catchment stores in/mnt/ssd1/data/icechunk/passddrs import --dry-run.Test Plan
parse_cf_units/axis-validation units, 10 fixture-backedhourly_streamflowintegration tests (exact-value pins incl. daily repeat-24 regression), 4import_cmdintegration testscompare_ddr_sandbox→ ABSOLUTE MATCH (invariant 1)streamflow resolution: Daily/Hourlyconfirmed in run logs (~22 s/batch daily, ~46 s/batch hourly)Spec:
docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md· Plan:docs/superpowers/plans/2026-07-01-nh-qprime-import.md🤖 Generated with Claude Code