AC power flow solver with neural network#2
Open
faezs wants to merge 27 commits into
Open
Conversation
Implements full AC power flow for grid-tied solar at 220V nominal (160-270V Pakistan range) with WAPDA sync and phase coupling. Components: - Chopaan.AC.Solver: Exact nonlinear solver using SBV with dReal backend - Full AC power flow equations (P_ij, Q_ij with conductance/susceptance) - Node power balance constraints - Dataset generation for neural network training (~10s/sample) - Chopaan.AC.NeuralDispatch: Fast neural network inference (~1ms) - PyTorch-based inference via subprocess - Architecture: 2N → 256 → 256 → 128 → 2(N-1) - Validation against exact solver - Chopaan.AC.Dispatch: Production streaming dispatcher - Integrates with Streamly for real-time telemetry - Async validation every N dispatches - Retrain detection when error exceeds threshold - train_dispatch.py: Training script for neural network - Loads SBV-generated samples from CSV - Trains with Adam optimizer and LR scheduling - Saves model weights and normalization params
…raining Major architectural changes based on review feedback: 1. **Solver (IPOPT instead of dReal)** - Replace SBV/dReal with IPOPT via CasADi for proper NLP optimization - Add objective function: minimize losses + curtailment penalty - Realistic sampling from solar/load profiles with time-of-day bias - Output optimal P,Q dispatch setpoints (not V,δ) 2. **Neural Network (P,Q outputs + uncertainty)** - Output P,Q setpoints that inverters actually control - MC Dropout for uncertainty quantification at inference - Persistent Unix socket server for ~1ms latency (not subprocess) - Fall back to IPOPT when uncertainty exceeds threshold 3. **Training (physics-informed loss)** - Physics constraints: P bounds, Q bounds, apparent power limits - Combined loss: MSE + λ × physics_violation - Proper normalization and denormalization 4. **Grid Sync (WAPDA compliance)** - Frequency monitoring: 50 Hz ± 0.5 Hz - Voltage monitoring: 220V ± extreme (160-270V) - RoCoF protection: < 1 Hz/s - Phase sequence verification - Islanding detection (2-of-3 indicators) - Reconnection sequence with delay and power ramp 5. **Inverter Commands** - Per-inverter P,Q setpoints with enable/disable - Power ramping during reconnection - Grid status in every dispatch command
Based on Manin-Marcolli's categorical framework for neural information
networks. The key insight is that Kirchhoff's current law (Proposition
2.10) is exactly the condition for a summing functor to be in the
equalizer.
New files:
- SummingFunctor.hs: Categorical types (DirectedGraph, SummingFunctor,
KirchhoffEqualizer) for power flow networks
- summing_functor_net.py: PyTorch GNN with KirchhoffProjection layer
that guarantees power balance BY CONSTRUCTION
Architecture:
Node Features → GAT Layers → Edge Flow Predictor → Kirchhoff Projection
The Kirchhoff projection is differentiable:
Φ' = Φ - A^T (A A^T)^{-1} A Φ
where A is the incidence matrix. This projects edge flows to the
equalizer subspace where conservation holds at all non-slack nodes.
Updated:
- NeuralDispatch.hs: Added NetworkType (Dense/Functor), dispatchGraph
- train_dispatch.py: Added --use-functor flag for graph-based training
- package.yaml: Exposed Chopaan.AC.SummingFunctor module
Based on Manin-Marcolli §2.3.2 and Corollary 2.20: a summing functor Φ ∈ Σ^prop_C(G) is completely determined by its values on corollas. Key insight: Instead of global projection (O(n³)), we compose locally: 1. Each node v has a CorollaModule that outputs flows on incident edges 2. LOCAL conservation is enforced within each corolla (not post-hoc) 3. Nodes processed in topological order 4. Grafting: upstream outputs become downstream inputs This IS Kirchhoff's law, but encoded in composition rather than projection. New files: - Properad.hs: Haskell types for properad composition - Properad, CorollaValue, ProperadFunctor - enforceLocalConservation: per-node conservation - processInOrder: topological composition - properad_net.py: PyTorch implementation - CorollaModule: local neural module per vertex - ProperadPowerFlowNet: topological composition - O(n) instead of O(n³) Comparison: Projection: Global GNN → All edges → Project (O(n³)) Properad: Local corollas → Compose in order → Done (O(n)) The properad composition ∘_E ensures output of upstream = input to downstream on shared edges. Conservation is local, global consistency follows from compositional structure.
Two key extensions based on Manin-Marcolli: 1. MESHED GRIDS (§2.3.2 relaxation) - Properad requires DAG, but real grids have loops - Solution: decompose into spanning tree + cycle edges - Tree edges: process via properad composition (O(n)) - Cycle edges: iterate until convergence (Gauss-Seidel style) - CorollaModule has cycle_correction for loop flows 2. RESOURCE OPTIMALITY (§3.3) - Optimization IS an adjoint functor - ρ : C → R assigns resources to configurations - β : R → C finds optimal config (LEFT ADJOINT) - Adjunction: MorC(β(A), C) ≃ MorR(A, ρ(C)) - The neural network learns β! The universal property means: - Any config achievable with resources A factors through β(A) - β(A) is provably optimal for those resources New files: - ResourceOptimality.hs: Haskell types for adjunction - PowerResource, PowerFlowConfig (objects of R and C) - ResourceFunctor (ρ), OptimalityFunctor (β) - isOptimal, factorsThroughOptimal (verify adjunction) - CycleDecomposition, iterateToConvergence - meshed_properad_net.py: PyTorch implementation - CycleDecomposition: find spanning tree + cycles - OptimalityFunctor: neural network IS β - Damped iteration for cycle convergence - ResourceOptimalDispatcher: complete β ⊣ ρ
Implements Manin-Marcolli §6: Hopfield dynamics on networks where power flow is viewed as a dynamical system converging to attractors representing optimal dispatch. The neural network learns the transition matrix T such that fixed points minimize cost while satisfying physical constraints. Key features: - Categorical threshold functor (·)₊ enforces feasibility by construction - Learnable transition matrix T captures network coupling - Damped dynamics ensure convergence to stable operating points - Energy-based training objective (attractors are local minima) - Soft threshold for differentiable training, hard for inference
Haskell fixes: - SummingFunctor.hs: Implement negateFlow for PowerPair (was placeholder id) - Properad.hs: Fix field accessors (edgeSource→edgeSrc, dgEdges→graphEdges) - HopfieldDynamics.hs: Fix field accessors and add Set import - ResourceOptimality.hs: Implement economic dispatch with merit order, add defaultOptimalityFunctor, compute line ratings from node injections - Dispatch.hs: Add missing drKirchhoffSatisfied field Python fixes: - properad_net.py: Fix topological sort edge lookup bug (was O(n²) nested loop) - meshed_properad_net.py: Proper reserve margin calculation, physics-based loss formula with R_line constant instead of magic number - hopfield_dynamics.py: Vectorize edge adjacency via incidence matrix, topology-aware ExternalInput with incidence masking - summing_functor_net.py: Vectorize EdgeFlowToNodeSetpoints using scatter_add All implementations are now complete with proper physics and no placeholders.
faezs
force-pushed
the
claude/ac-power-flow-neural-NnNtM
branch
from
d8a3ec5 to
de75b95
Compare
24h dispatch env on the 4-node test grid with single PV inverter. Action: P,Q setpoints normalized to [-1,1]. Reward: cost + voltage violation + Kirchhoff residual. Uses PKR tariff with peak hours 18-22.
Header-only PufferLib-style env mirroring chopaan_env.py. Static-inline init/allocate/reset/step over caller-owned buffers, xorshift32 RNG, no allocations in the hot path. Smoke-tested with gcc -O2.
Adds chopaan-gen-grids executable that samples K distribution feeders through mgenv (Grid.Sample.sampleGridSpec + generateGrid) and writes a JSON manifest with edge_src/tgt, per-unit edge resistance, node type (slack/PV/PQ), PV ratings, and load profiles. mgenv pulled in via source-repository-package on its master branch. Also includes the SoA-layout C vec env (chopaan_vec.h) staged for the Hopfield settling loop that will consume these grids.
hopfield-c/ compiles chopaan's HopfieldDynamics.hopfieldStep (Manin-Marcolli Eq 6.2) to a branch-free C function using con-kitty/categorifier-c. F.hs instantiates the step at fixed size (4-node radial feeder, 3 edges, P/Q per edge), unrolls K=8 settling iterations, and Categorify.expression lowers it to Cat; Main.hs writeCFiles emits hopfield_step.c/.h as the env's per-step kernel.
mgenv-gen/ holds GenJson.hs — uses mgenv's Grid.Sample (sampleThis -> generateGrid) to emit grids.json for the env, built with cabal under GHC 8.6.5 (same as chopaan lib). Documents the GHC split: mgenv/chopaan are 8.6.5, hopfield-c/categorifier is 9.0.1, so they are sibling cabal projects in the repo rather than one build. mgenv supplies topology; categorifier supplies the Hopfield physics.
cabal run hopfield (GHC 9.0.1) lowered F.hs through the Categorifier plugin to
Cat and emitted generated/hopfield_step.{c,h}: a branch-free SSA kernel,
input_double[13] -> output_double[7], implementing the 8x-unrolled categorical
Hopfield settling. Verified: gcc compiles it and driver_demo.c runs (flows
settle, gridImport=16.26).
Records the two cabal.project fixes that unblocked the build (drop orphaned
cmk/connections pin; pin index-state 2022-03-01) as a patch + README notes.
F.hs now takes the graph as data: a 4x3 directed incidence matrix B, per-edge conductances g=1/R, and node P/Q injections. It builds adjacency A=B^T B, conductance-weighted coupling T_ee'=coup*A*g, and external input Th=B^T inj from the incidence, then runs the 8x-unrolled Hopfield settling. Regenerated hopfield_step.c is input_double[30] -> output_double[7]. Verified topology-sensitive: zeroing edge e2's incidence column + conductance changes settled flows and gridImport (-77.8 -> -46.7). So mgenv's generated graph drives the categorified C kernel as data.
pufferlib-env/chopaan/ wires the categorified C kernel (hopfield_step.c) as the
per-step physics of a grid-dispatch env over an mgenv incidence-matrix topology:
- chopaan.h : c_step packs B + conductances + injections into the kernel's
input_double[30], settles, prices grid import (reward).
- binding.c : env_binding.h glue; pulls hopfield_step.c into the extension TU.
- chopaan.py : pufferlib.PufferEnv wrapper (Box(6) obs, Box(2) action).
- build.sh : builds the native demo and the CPython extension (binding.so).
Verified: native demo ~2.8 M steps/sec single-threaded; the puffer C-extension
(built as setup.py's Extension does) runs 4 vec envs x 24-step episodes through
vec_init/reset/step/log/close with returns ~ -72. The graph is mgenv's; the
dynamics are the lowered categorical Hopfield morphism.
Non-destructive path to build mgenv-json via cabal/GHC 8.6.5: inlined
opt-expect (RL.MDP, Env.MonadEnv) and lcirc (LCirc.{LCirc,Cospan,Spider} with
a real Spider Frobenius def), removed unused concat-hardware subdir, and a
build script supplying libnuma + zlib (foreign libs nix shell omits). Cleared
6 resolve/link blockers; remaining wall is singletons-2.5.1 third-party bit-rot
(Int/Integer NameU) under hgeometry. Recorded so it survives container restart.
This GHC 8.6.5's template-haskell has Uniq=Integer, so singletons' qNewUnique (:: q Int) doing 'NameU n -> return n' fails Int/Integer. Vendor singletons-2.5.1 with 'return (fromIntegral n)' and register it as a local package in cabal.project. Build now compiles past it.
cabal run mgenv-json now produces actual mgenv radial feeders (Euclidean-MST trees, real wire resistances + PV/load samples) -> mgenv-gen/sample-output/ grids.json. Full reproduction in mgenv-gen/build-recipe/: mgenv-src.patch (diff vs pristine tarball), patched cabal.project/freeze/mgenv.cabal/package.yaml, GenJson.hs, vendored singletons patch, inlined opt-expect + lcirc modules, and build/run scripts. Key fixes: inlined opt-expect & lcirc (private dep) incl. a real Spider def; patched singletons NameU Int/Integer; bumped streamly 0.6.1->0.8.0; fixed mgenv source bugs (absToUTC import, tedges typo); stubbed broken dynamics scaffolding; excluded orthogonal diagrams/servant modules (files kept). GenJson builds the GridSpec' directly since mgenv's Randomizable GeoC/LifeTime are unimplemented.
mgenv_grids.h is codegen'd from real mgenv output (32 distinct 4-node Euclidean-MST feeders: directed incidence B, normalized conductances, PV ratings, loads). binding.c's my_init reads PufferLib's per-env seed and sets grid_id, so each vec env trains on a distinct mgenv feeder; c_reset loads it via ch_load_grid. Verified end-to-end: 8 vec envs -> 8 distinct 24h returns through the CPython extension; ~2.5 M steps/sec rotating across grids with the categorified Hopfield kernel as physics. Closes the loop: mgenv (cabal) samples topology -> categorifier lowers the Hopfield step to C -> PufferLib env settles that kernel over each mgenv graph.
…ernel) A Haskell dear-imgui app to play with the feeders like a game. The renderee is an algebraic-graphs Graph Int (a real mgenv 4-node feeder from grids_4node.json, loaded via aeson in Feeder.hs). Each frame Main.hs packs the graph's incidence + injections + conductances + alpha and calls the categorified Hopfield kernel over FFI (Kernel.hs -> cbits/hopfield_step.c, the same C Categorifier lowered F.hs to and the RL env runs) — no physics reimplementation. Settled edge flows are drawn with the ImGui draw list; nodes are draggable (SDL mouse), edges recolour by flow, sliders tune PV setpoint / damping / per-edge conductance / hour, and Next cycles the mgenv feeders. Cost score = grid import x tariff. Headless container can't build/run a GUI; ships as source + cabal + README with system deps (SDL2/OpenGL) and dear-imgui version caveats.
…sampler - chopaan-viz/cabal.project: isolate the viz build from the parent chopaan project (otherwise cabal walks up and tries to build mgenv/Shpadoinkle). - hopfield-c/gen_fhs.py: codegen a size-N graph-parameterized F.hs for Categorifier, unrolling the Hopfield settle into named bindings (not a tuple, which caps ~7) so it scales past the 4-node toy. F_N8.hs is the N=8 output (94-double input -> 15-double output). - mgenv-gen GenJson.hs: sample feeder size from a plausible LV distribution (Gaussian ~6, clamped [nMin..nMax]) instead of a fixed 4; edges follow as the EMST tree. - gitignore dist-newstyle.
Start migrating mgenv's generation core off GHC 8.6.5 so the dear-imgui renderer can import it directly. Key unlock: hgeometry (unportable, maxes at 0.9.0.0) is only used for Delaunay->MST = the Euclidean MST, reimplemented with Prim's (Geometry/EMST.hs, no geometry deps). PLAN.md documents the full analysis: ConCat/streamly/astro are all in dynamics (strippable), sampleGridSpec is unused, leaving a monad-bayes + algebraic-graphs closure. Also: Feeder now invokes the real mgenv-json binary live (subprocess) instead of a static JSON, pending the direct-import migration.
The full generation closure now builds on GHC 9.0.1 with only monad-bayes +
algebraic-graphs: Physics.Units (pure-Double reimpl, drops astro/dimensional),
Geometry.EMST (Prim's, drops hgeometry/singletons), Physics.{Transmission,PV,
Consumption,Storage} + Grid.HH stripped to spec+sampler (drop streamly/ConCat/
dynamics), Grid.Sample stripped to the generateGrid path, Prob.Randomizable
ported to monad-bayes 1.1 (MonadSample -> MonadDistribution). All 9 modules
compile under ghc-9.0.1. This removes the GHC wall so the dear-imgui renderer
can import Grid.Sample directly. Also snapshots the data-driven kernel
orchestration (build_kernel/gen_fhs/gen_grids_h) and the N-generic env.
…ess) Now that mgenv's generation core builds on GHC 9.0.1, the renderer links it (cabal.project adds ../mgenv-gen/migrate-9.0.1). Feeder.sampleFeeders runs mgenv's real generateGrid in-process via sampleIO, flattening SampledGrid (TransmissionSpec/HHSpec labelled graph) to the renderer record. Also fixes Main.hs draw calls against the real dear-imgui 2.1 API discovered by building: getForegroundDrawList/imCol32 from DearImGui.Raw, Ptr ImVec2 marshaled with 'with', addText_ with CString.
…erified) The whole renderer compiles and links as a 41MB ELF on GHC 9.0.1: migrated mgenv-gen9 (9 modules), Feeder (direct import of Grid.Sample, in-process generateGrid), Kernel (FFI to the categorified hopfield_step.c), Main (dear-imgui 2.1.3). BUILD.md records the confirmed solver constraints (index-state 2023-06, dear-imgui <2.2, sdl2 <2.5.4) and system libs (SDL2/GLEW/GLU/X11). Can't run a GUI headless, but the dear-imgui API usage and the direct mgenv import are now compiler-verified. Migration plan marked done.
…n + system libs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPinEEC8gDDuAkCdGbF7AZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements full AC power flow for grid-tied solar at 220V nominal (160-270V Pakistan range) with WAPDA sync and phase coupling.
Components:
Chopaan.AC.Solver: Exact nonlinear solver using SBV with dReal backend
Chopaan.AC.NeuralDispatch: Fast neural network inference (~1ms)
Chopaan.AC.Dispatch: Production streaming dispatcher
train_dispatch.py: Training script for neural network