feat(inference): joint least-squares unit-generation inference + Network page validation view#53
Merged
Merged
Conversation
Add app/joint_inference.solve_unit_generation: per (run, interval) it stacks all constraint equations into A*g = b (interconnector/region terms substituted to the RHS, unresolvable-known constraints dropped) and solves for the unit vector with 0 <= g <= MAXAVAIL via scipy.optimize.lsq_linear. Identifiability is a first-class output: each unit is flagged good / weak / unidentifiable from the null space of A (unit vectors with a null-space projection are structurally non-estimable) and the pseudoinverse row-norm. Adds scipy to requirements. Pure module, fully TDD-covered (11 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add scripts/validate_joint_inference.py: downloads PD7Day runs from NEMWEB Current for a settled-day window, parses the full CONSTRAINTSOLUTION and INTERCONNECTORSOLUTION tables in-memory, keeps each run's first 4 forward hours, jointly solves per (run, interval), and scores 'good'-quality units against realised Next_Day_Dispatch TOTALCLEARED at 30-min resolution. Includes a calibration gate (zero-DUID constraints must reconstruct from interconnector flows alone) and MAXAVAIL bounds from bid_per_offer per (30-min interval, duid); the solver now accepts interval-keyed bounds. Region terms are excluded: the live PD7Day file carries no region/demand table and the ENERGY-terms snapshot contains zero region terms, so the exclusion costs no equations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Next_Day_Dispatch directory listings reference files by absolute path (/Reports/CURRENT/...); joining them onto the directory URL produced a 404 for every file. Add resolve_report_url handling absolute-path, full-URL, and bare-filename HREFs, and let the joint validation runner skip (with a warning) any single day whose realised-dispatch file cannot be fetched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds inferred_unit_generation (good/weak quality only; unidentifiable rows carry no usable MW and are dropped) with an idempotent upsert helper, plus pure short-lead-selection/tracking-stats functions in joint_inference.py. Adds --persist to validate_joint_inference.py to seed history offline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ycle After the existing price/interconnector/constraint insert, backsolves unit MW from this run's unfiltered CONSTRAINTSOLUTION frame (predispatch_constraint only stores binding rows) and upserts good/weak results. Wrapped so a solve failure logs and never breaks price/network ingestion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GET /api/network/unit-inference/units ranks stored DUIDs by observed correlation vs realised dispatch (not the solver's quality flag alone, since some 'good' units track poorly). GET .../series returns the paired short-lead inferred vs realised chart data for one DUID, 404 if unknown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DUID picker (labelled duid . corr . n, non-tracking units greyed but selectable) plus a dual-line inferred-vs-realised chart, stats chip, and n_equations provenance line, following NetworkPage's existing conventions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fetch_terms/fetch_bounds (and aggregate_bounds_to_30min) move from scripts/validate_joint_inference.py into app/joint_inference.py so both the script and data_ingester import them from the app layer -- scripts depend on app, never the reverse. Live-hook log now also reports the short-lead split. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Covers the zero-variance NaN-correlation path (pure function + /units endpoint: null observed_corr, tracking false, sorts last). Network page now renders both endpoints' message fields as provenance lines. Drops the dead residual fallback in insert_inferred_unit_generation and restores db: NEMDatabase annotations on the moved fetchers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The working version of constraint-equation unit backsolving (supersedes #52's validated-dead single-unknown approach): per predispatch run/interval, stack ALL ~900 published constraint LHS values into a bounded least-squares system (scipy lsq_linear, 0 ≤ g ≤ bid MAXAVAIL) and solve for unit MW jointly, with SVD null-space identifiability as first-class output (good/weak/unidentifiable — rank-deficient units are flagged, never guessed).
Evidence
Shipped
app/joint_inference.py— pure solver + identifiability + tracking stats; DB-aware fetchers in a marked section.inferred_unit_generationtable (good/weak rows), live-populated by the PD7Day ingest cycle (solve failure logs, never breaks ingestion) +--persistseeding runner (seeded 2026-06-19→07-10 locally).GET /api/network/unit-inference/units+/series— observed corr/MAE vs 30-min SCADA, NaN-safe JSON, 404 unknown DUID.duid · corr · n, non-tracking greyed), realised-vs-inferred dual-line chart, stats chip, provenance lines.Review + verification
Two review gates (solver maths incl. hand-derived SVD fixture check; persistence/UI round with NaN-corr and isolation checks), fix rounds applied. Backend 604 / frontend 137 tests green. Endpoints live-verified against the seeded local DB.
Next (separate PR, needs a go)
Forward unit profiles + constraint decomposition views (the forward rows are already being stored); fresher equation source to lift the ~200-constraint coverage cap.
🤖 Generated with Claude Code