feat(ingest): constraint equations + unit-generation inference (honest negative on v1 inversion)#52
Merged
Merged
Conversation
Adds lhs REAL to predispatch_constraint (ALTER-if-missing pattern) and threads it through both PD7Day and PredispatchIS parsers, plus the upsert (already DO UPDATE, now includes lhs). Verified LHS column presence/name against live PD7Day and PredispatchIS downloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds constraint_equation_terms (latest-version-only, no effective-date history) plus scripts/ingest_constraint_equations.py, which downloads DUDETAILSUMMARY + the three SPD*CONSTRAINT tables from AEMO's MMSDM archive and builds duid/interconnector/region terms. Verified against a real MMSDM_2026_05 download: DUDETAILSUMMARY's END_DATE "current" sentinel (2999/12/31) exceeds pandas' Timestamp range, so latest-row selection compares the fixed-width date strings directly rather than parsing them. ENERGY-bidtype only (FCAS out of scope); ~6/423 connection points have no current DUID mapping and are dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds pure app/unit_inference.py: merge-based interconnector-term substitution, region-term constraints excluded outright, and vectorized groupby counting to isolate constraints with exactly one unknown DUID term. TDD fixtures cover solvable/2-unknown/region/missing-mwflow cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds scripts/validate_unit_inference.py: runs infer_unit_generation on short-lead stored predispatch data for a settled trading day, compares against NEMWEB Next_Day_Dispatch's DISPATCH,UNIT_SOLUTION (verified on a live file; TOTALCLEARED aggregated 5-min -> 30-min via ceil to match predispatch's period-ending convention), and reports per-DUID n/corr/MAE. Real-data validation across three samples (2026-07-09 short-lead day, 2026-05-15 short-lead day, and the full 2026-05-10..16 week, all at short lead) found zero constraints with exactly one unknown DUID term among those actually binding -- the minimum DUID-term count on any constraint that was both binding and registered in the terms table was 7. No inferred MW was produced in any sample, so no correlation/MAE could be computed; see PR/task notes for the full analysis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
parse_dudetailsummary_csv now filters to END_DATE >= today (string compare handles the 2999/12/31 sentinel) before picking the latest row, so fully retired connection points are dropped instead of mapping to their last historical DUID. insert_constraint_equation_terms now does a transactional delete + insert so terms AEMO removes from an equation don't linger as phantom rows (the table is a latest-snapshot by design). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tder311
marked this pull request as ready for review
July 10, 2026 01:49
This was referenced Jul 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
First slice of backsolving unit behaviour from AEMO constraint equations — and an honest empirical result about how far the classic trick gets you on today's NEM.
Built
predispatch_constraint.lhs(both PD7Day and PredispatchIS parsers; upsert refills on conflict). LHS flows live from now on; historical bulk refill deliberately deferred (needs a ~15GB archive re-pass — cheap to run later if we pursue the joint solver below).constraint_equation_terms— latest-snapshot equation definitions from MMSDM (SPD tables + current-only DUDETAILSUMMARY connection-point→DUID mapping, truncate-and-load semantics): 129,142 terms / 5,070 constraints (123k DUID, 5.9k interconnector).app/unit_inference.py— pure single-unknown inversion: substitute known interconnector flows, solvemw = (lhs − Σ known)/factorwhere exactly one DUID term is unknown. Fully tested (10 cases incl. exclusions and multi-constraint agreement).scripts/validate_unit_inference.py— short-lead inference vs realised Next_Day_Dispatch unit MW.The honest result
Zero solvable instances across three real windows (2026-07-09, 2026-05-15, full week 2026-05-10→16 at ~2h lead). Binding constraints are overwhelmingly multi-unit group constraints — minimum 7 DUID terms among binding-and-registered constraints. Two structural causes:
C_N_NESBESS_150_G, bound 11× in our data, absent from the May archive).Decision this PR poses
The mechanism and data layer are correct and durable (LHS + equation terms are useful regardless). Whether to build the joint least-squares solver — the version of this that plausibly fires — is a deliberate call, not momentum. Parked pending that call.
Tests
510 passed / 0 failed. Review round fixed: current-only DUID mappings (retired connection points dropped, discriminating two-generation test) and replace semantics for the terms snapshot (phantom-term purge).
🤖 Generated with Claude Code