Stabilize constrained aircraft recovery and interruption-safe CFD training - #41
Open
iamdarshg wants to merge 156 commits into
Open
Stabilize constrained aircraft recovery and interruption-safe CFD training#41iamdarshg wants to merge 156 commits into
iamdarshg wants to merge 156 commits into
Conversation
Replace per-tensor .item() reductions in multiobjective_gradients.py (gradient_l2_norm, gradient_cosine_similarity, _gradient_dot_product, _validate_gradient_tensor) with a single torch.cat over present grads, one isfinite().all() finiteness gate, and one vector_norm / torch.dot, each with a single .item() sync. The guard_dot correctness gate in aircraft_diffusion_cfd.py is evaluated as one torch.dot over concatenated aligned flats (same fail-closed < -1e-8 semantics). Cuts ~18-20k GPU->CPU syncs/update to O(1) per telemetry metric (per-call: 195->2 syncs for a norm, 327->6 for a cosine on the real tensor set; ~3.6x/2.2x per-call wall time). Telemetry feeds only last_* dicts/callbacks; norms match the prior per-tensor computation to ~3e-8 relative (float32 ulp), so the optimizer step is unchanged. Any nonfinite gradient still raises NonFiniteGradientError. Co-Authored-By: Claude <noreply@anthropic.com>
Groundwork for threading the 33 per-update direct-solver SDF evaluations. scipy's distance_transform_edt releases the GIL during the C computation (measured ~3.6x on 8 threads for 96^3), so per-thread workspaces let the SDF+q computations run concurrently. Replaces the single shared-buffer _EDT_WORKSPACES guarded by _EDT_WORKSPACE_LOCK with a thread-local workspace map; each workspace is 2x float64[96^3] + int32[3,96^3] ~= 25 MiB. Co-Authored-By: Claude <noreply@anthropic.com>
P1: compute_all_link_distances now crops the SDF+q computation to the solid bounding box (margin 2) and stacks the 26 shifted neighbor slices into one vectorized tensor, bit-exact vs the per-direction loop over all 27 channels. P2: _heuristic_metrics labels connected components on the solid-bbox crop (largest_component_fraction invariant). P3: aircraft validity runs on a thread pool overlapped with the GPU LBM solve. Direct-objective values are identical (verified torch.equal). Co-Authored-By: Claude <noreply@anthropic.com>
3a2e826 renamed _EDT_WORKSPACES -> _THREAD_EDT_WORKSPACES (thread-local); test_sdf_utils.py still imported the old global dict, so the module failed collection. Rework both tests against the thread-local API via _edt_workspace(shape): workspace-reuse identity is preserved (same shape on the same thread returns the same buffers), and the reservation test now asserts the reserved workspace's buffer shapes. Co-Authored-By: Claude <noreply@anthropic.com>
Cache the full-grid Fourier encoding in LatentTo3DConverter as a persistent=False buffer, invalidated on numel/device/dtype/bands change, and use it in forward_flat_indices (via index_select) and forward. Bit-identical parity (torch.equal) with the previous per-call re-encode; the identity path (bands <= 0) is never cached. Co-Authored-By: Claude <noreply@anthropic.com>
Merge the detached metric-only no_grad passes into the grad-enabled loops of _backward_full_grounded_threshold_margin and _backward_full_grounded_coordinate_loss. - threshold margin: accumulate the raw margin sums detached inside the existing per-chunk backward loop; per-chunk backward(retain_graph=...) stays byte-identical (bit-exact loss and gradients). - coordinate loss: one grad-enabled loop accumulates detached metric sums, grad-carrying per-batch dice masses, and per-chunk bce/margin losses; one final backward on total_chunk_loss + w_dice * dice_obj (detached analytic dice coefficients). Safe because coordinate gradient checkpointing is on in training; returned loss bit-identical, gradients last-ulp (~1e-7 relative). Co-Authored-By: Claude <noreply@anthropic.com>
Raise the coordinate-decoder chunk size to the largest bit-exact value and
hoist the per-chunk latent expansion.
(a) coordinate_chunk_size 16384 -> 65536 (config.yaml + ModelConfig fallbacks).
The brief's 131072 target was measured and rejected: cuBLAS picks a
different GEMM kernel at N>=98304, giving a deterministic 1-ULP fp32
difference (breaks the mandatory bit-exact gate). 16384/32768/65536 are
mutually bit-exact (torch.equal on forward outputs); 65536 is the largest
such size and captures the same forward-path speedup (13% faster
forward_flat_indices). Full-lattice decode: 54 -> 14 chunks. The two
Task-4 backward methods are not edited; they read model_config's chunk
size, so their loop granularity changes with identical per-voxel
arithmetic (loss last-ULP ~1e-8 rel, same category Task 1/4 accepted).
(b) hoist latent[:, None, :] out of the per-chunk loop. _decode_latent_
coordinate_chunk and _checkpointed_coordinate_chunk accept an optional
pre-expanded view (None = exact old behavior); forward/forward_flat_
indices build it once per call. Bit-exact: outputs torch.equal, all
gradients zero-diff, including the view-input activation_checkpoint path.
(c) gradient checkpointing stays ON: disabling it peaks 9.4 GiB (> 8 GiB box).
(d) split-input-Linear measured 18% slower + LOW parity; not implemented.
Tests: 105 passed (converter-cache, multiobjective, recovery-review, cli,
consistency-model). Full report in .superpowers/sdd/optimization-plan/task-5-report.md.
Co-Authored-By: Claude <noreply@anthropic.com>
…decoder hoist) Restore coordinate_chunk_size to pre-Task-5 values in all three sites: - CLI/config.yaml: 65536 -> 16384 - ModelConfig fallbacks (x2): 65536 -> 32768 The 65536 bump was measured net-neutral-to-slightly-negative on the training update (checkpointed backward +7% slower, offset by forward -13%) and added +0.56 GiB peak VRAM on an 8 GiB box. Label: NOT WORTH IT. The hoist (b) is the clean bit-exact win and is kept unchanged. Parity (vs a6af990, dual-import probe, all torch.equal): - config_value + ModelConfig().coordinate_chunk_size == 16384 - converter forward (eval + training/checkpoint path, full lattice + flat indices) byte-identical; latent.grad and param.grad byte-identical - both Task-4 backward methods (_backward_full_grounded_coordinate_loss, _backward_full_grounded_threshold_margin) losses torch.equal, grads zero-diff, at 4-chunk granularity - 96^3 config-path forward on CUDA byte-identical at chunk 16384 Co-Authored-By: Claude <noreply@anthropic.com>
…offload (Task 7) - read each per-update loss tensor .item() exactly once and reuse the float in the totals, the progress postfix (every 5 updates), and the metrics callback; loss tensors untouched - _append_jsonl: append+flush only (drop per-append fsync/full-file sha256/ recount); in-memory running record count; move the durable-prefix sha256 to the run-state save path (build_run_state stamps it over the recorded offset) so resume integrity is preserved and _reconcile_updates_log still validates - offload_optimizer_state_between_steps: false (full-update profile at 96^3 with checkpointing shows no OOM; peak ~6.7 GiB of 8.0 GiB) - drop the per-10-batches torch.cuda.empty_cache() allocator sync - update 3 reconcile/resume tests to stamp the digest via the _updates_log_reconciliation_metadata code path build_run_state uses
Two coupled changes, both required for a green PR 41 push. NUMERICS: run the neural-network GEMMs (coordinate decoder + diffusion encoder/attention) on TF32 tensor-core math when experiment.tf32_gemm_math is set. The D3Q27 LBM/SPSA solver path is elementwise-only (no matmul/ einsum), so this CANNOT change solver arithmetic -- only NN GEMM precision. Measured 2026-08-17: decode fwd+bwd 1.511x, full-update mean 36.2 vs 49.3 s/u (-27%), steady-state ~31-34 vs ~44. Gradients within GRAD_ATOL (0.2-0.4x); decoder output drifts ~2e-4 rel (49x the 4e-6 refactor-parity gate, benign at geometry level). fp32 storage unchanged. Wired at trainer construction so solver-only/converter-only tests keep IEEE fp32. ON by explicit user decision 2026-08-17; the claim-bearing run's paper documents this precision mode. FIX: gate the deferred SPSA-read path on CUDA via the new _direct_solver_supports_deferred_reads (mirrors _direct_solver_supports_batch). The committed deferred_solver_reads: true lever routed every SPSA probe through simulate_aerodynamics_deferred, bypassing the mocked _direct_measured_objective_for_single and breaking the CPU unit tests that observe the sequential per-solve structure. On a CPU simulator there are no GPU->CPU syncs to defer, so the sequential loop is canonical; stub simulators (plain object()) fall back to it instead of raising AttributeError. test_constrained_recovery_review.py: 7 failures -> 0 (35 passed); affected surface 173 passed. Co-Authored-By: Claude <noreply@anthropic.com>
Two coupled changes, both required for a green PR 41 push. NUMERICS: run the neural-network GEMMs (coordinate decoder + diffusion encoder/attention) on TF32 tensor-core math when experiment.tf32_gemm_math is set. The D3Q27 LBM/SPSA solver path is elementwise-only (no matmul/ einsum), so this CANNOT change solver arithmetic -- only NN GEMM precision. Measured 2026-08-17: decode fwd+bwd 1.511x, full-update mean 36.2 vs 49.3 s/u (-27%), steady-state ~31-34 vs ~44. Gradients within GRAD_ATOL (0.2-0.4x); decoder output drifts ~2e-4 rel (49x the 4e-6 refactor-parity gate, benign at geometry level). fp32 storage unchanged. Wired at trainer construction so solver-only/converter-only tests keep IEEE fp32. ON by explicit user decision 2026-08-17; the claim-bearing run's paper documents this precision mode. FIX: gate the deferred SPSA-read path on CUDA via the new _direct_solver_supports_deferred_reads (mirrors _direct_solver_supports_batch). The committed deferred_solver_reads: true lever routed every SPSA probe through simulate_aerodynamics_deferred, bypassing the mocked _direct_measured_objective_for_single and breaking the CPU unit tests that observe the sequential per-solve structure. On a CPU simulator there are no GPU->CPU syncs to defer, so the sequential loop is canonical; stub simulators (plain object()) fall back to it instead of raising AttributeError. test_constrained_recovery_review.py: 7 failures -> 0 (35 passed); affected surface 173 passed. Co-Authored-By: Claude <noreply@anthropic.com>
CLI/run_with_resource_monitor.py and CLI/watch_training_progress.py import psutil, and tests/test_resource_monitor.py + tests/test_watch_training_progress.py import those modules transitively. psutil was missing from CLI/requirements.txt, so the CI test job errored at collection with ModuleNotFoundError for every commit. Pre-existing infra break; TF32 merge is unaffected. Verified locally: 6 passed. Co-Authored-By: Claude <noreply@anthropic.com>
CLI/run_with_resource_monitor.py and CLI/watch_training_progress.py import psutil; tests import those modules transitively. Missing from CLI/requirements.txt broke CI test collection (ModuleNotFoundError) on every commit. Also pushed to PR 41 (455af83). Co-Authored-By: Claude <noreply@anthropic.com>
test_watch_training_progress imports training_tui, which does `from rich import box`. The ubuntu CI runner installs only requirements-dev.txt, so `rich` was undeclared there and collection failed. Local runs passed because rich was present in the fat dev env. Co-Authored-By: Claude <noreply@anthropic.com>
Mirror of PR-41 fix 1067c42: test_watch_training_progress imports training_tui, which imports `rich`. CI installs requirements-dev.txt only, so rich must be declared. Co-Authored-By: Claude <noreply@anthropic.com>
…uped cat _gradient_dot_product and _flatten_present_gradients issued one fp32->fp64 conversion kernel per parameter (~10k launches/update across the ~10 telemetry calls). Replace with a single concatenation + one conversion per dtype group via _cat_grouped_to_dtype. fp32->fp64 is exact, reshape preserves logical row-major order, and within a single-dtype group the element order is identical, so outputs are byte-for-byte the per-parameter path (parity probe 30/30 identical incl. fp64/fp16 edge cases, 1e308 overflow behavior; 17/17 unit tests pass). Mixed dtypes are still converted exactly per group. Co-Authored-By: Claude <noreply@anthropic.com>
…ss (lever 2) sparse_voxel_reconstruction_loss gathered per-row via flat_probabilities[row][mask]. The backward of a boolean-mask gather (IndexBackward0) calls torch.nonzero, a device->host sync, ~83x/update -> the ~3.5s/update GPU-idle stall between backward and the optimizer direct_copy burst (largest remaining addressable cost). Replace with index_select on precomputed long indices: same elements, same order, same means and gradients (proven bit-identical: 28/28 pattern-level checks plus 8/8 full-function loss+grad checks across batch sizes and the empty-row guard path), but the backward (IndexSelectBackward0) scatters on-device with no sync. Verified: build/perf/baseline/probe_mask_gather_parity.py and probe_svrl_full_parity.py; 6/6 existing sparse/reconstruction unit tests pass. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…lect The committed sparse-loss fix (f654b28) removed one IndexBackward0 source, but the trace still showed 228 IndexBackward0 events (216 in backward) whose backward does torch.nonzero (device->host sync) + _index_put_impl_. Three remaining sites gathered a differentiable tensor via x[mask]: - balanced_voxel_bce_with_logits (losses[mask]) — the BCE term called by sparse_voxel_reconstruction_loss. - grounded_threshold_margin_loss (penalty[mask]) — also had a bool(...any().item()) host-side guard. - _backward_full_grounded_coordinate_loss chunk loop (bce[mask]) — 2 gathers per chunk, 54 chunks = ~108 backward nonzero+sync sequences/update. All three now precompute torch.nonzero indices from the non-grad target mask in the forward and gather via index_select, whose backward scatters on device (no host sync). The coordinate loop computes the indices once per chunk and reuses them for the metric-only and grad-carrying sums. Gathered elements are identical and in the same order -> values and gradients bit-identical (micro-probe across 1-D..5-D shapes incl. all-False masks, plus 8 unit tests). Trace before/after: #1 idle gap changed from radix_sort -> CatArrayBatchedCopy (3.79s) to CPU-autograd elementwise/cat gaps; IndexBackward0 count 228 -> 0; backward GPU busy 12.7s -> 8.3s. Net wall neutral within run noise because the coordinate-loss backward is CPU-autograd-bound (gradient-checkpoint recompute), which this commit documents as the next bottleneck. Co-Authored-By: Claude <noreply@anthropic.com>
The no_grad fast_inference + full 54-chunk decode that produced the SPSA base is bit-identical to the later replay decode (same noise, model, steps, no optimizer step between). Build the grad-mode replay decode once, BEFORE the SPSA solves; the SPSA base (direct_logit_snapshot) is its detached copy, and the optimizer backward reuses the same tensor. Removes one fast_inference + one full converter decode per update. A/B (measure_steady_su, --warmup 1 --iterations 3, back-to-back runs): HEAD steady: 32.09 / 35.18,31.25 -> ~31.7 median edited steady: 30.32 / 28.74 -> ~29.5 median By-construction parity: identical RNG stream (noise created at same point, fast_inference deterministic given fixed noise), bit-identical decode values (established by prior microbench), grad_direct telemetry unchanged (0.25), no crash across all runs. Co-Authored-By: Claude <noreply@anthropic.com>
The guard walk walked all 54 coordinate-decoder chunks to compute the param-space guard direction (~4.5 s/u). It now decodes only the two chunks with the largest guard-gradient L2 via a new converter.forward_voxel_mask; the main decode graph is untouched and the optimizer backward still walks every chunk. Measured param-space cosine vs the exact direction: C=1 0.979, C=3 0.993, so C=2 interpolates ~0.986 -- the replay-side projection keeps 98%+ of the exact rejection. That projection has never fired in measured updates (cosine +0.40 -> +0.17, toward orthogonal), so the practical risk is negligible; the walk itself drops from ~4.5 s to <0.3 s. Steady s/u: 28.74 -> 26.14 (measure_steady_su, last steady update). Co-Authored-By: Claude <noreply@anthropic.com>
Merge the experiment branch's optimization stack into the recovery PR so the PR is the single up-to-date source of truth for every landed change. Optimizations merged in (all parity-gated, steady state ~26.1 s/u): - C=2 guard walk: top-2 coordinate-decoder chunk backward (0.99 cosine vs exact) with tiling guard + exact full-walk fallback for tiny grids (unit-test compatibility, fixes PR finding-#5 tiny-grid interaction) - tf32_gemm_math: NN GEMMs on TF32 (solver path elementwise-only, so D3Q27 arithmetic unchanged); documented for the claim-bearing run - grouped-cat fp64 gradient telemetry (multiobjective_gradients.py) - batched guard-dot fp64 hoist: one deferred GPU->CPU sync per update - deferred SPSA solver reads: one batched scalar read for all 33 solves - drop redundant no_grad full-lattice decodes (~2.2-2.5 s/u) - boolean-mask gather -> index_select in sparse voxel loss (bit-identical) - CUDA-graph decoder MLP capture/replay (kernel_fusion_graph.py, OFF) Test gate: full suite 498 passed, 2 skipped (test_gui deselected). Co-Authored-By: Claude <noreply@anthropic.com>
Enable torch._dynamo.compiled_autograd behind model.compiled_autograd
(opt-in, default off) so Inductor fuses the checkpointed coordinate-decoder
recompute-forwards and elementwise backward ops into fewer kernels. Targets
the backward's ~26k small launches.
Measured on the real 96^3 trainer from step-1305 (build/perf/
compiled_autograd_*.py):
- parity: loss bit-identical; all 276 trained-param grads within 2.1e-7
max abs (GRAD_ATOL 5e-4, ~2400x headroom); 42/276 bit-identical
- steady-state: 9.092 -> 8.185 s/u (1.111x) on warm-cache probe; warmup
overlapped with Triton JIT (46.1s vs 46.3s)
Co-Authored-By: Claude <noreply@anthropic.com>
The D3Q27 MRT moment projection runs torch.tensordot/matmul against the 27x27 moment_basis in both collide paths (608/659/666 single, 1254/1288/1292 batch), so the trainer's global torch.backends.cuda.matmul.allow_tf32=True would have silently changed solver arithmetic despite the old "elementwise-only" comment. - Add _ieee_fp32_math decorator (save/restore allow_tf32=False) and apply it to collide_and_stream + collide_and_stream_batch: solver GEMMs are now force-IEEE, independent of the trainer's TF32 setting. - Trainer TF32 block: fix the false comment and add an explicit else that pins IEEE when experiment.tf32_gemm_math is off, so a TF32 flag leaked from an earlier trainer/process cannot leak into a non-TF32 run. Verified (build/perf/tf32_solver_parity.py): every solver quantity -- final field, per-solve + accumulated momentum-exchange force, Cd/Cl (single and C=2 batch paths), and the end-to-end DirectSolverSPSAFunction loss + backward gradient -- is BIT-IDENTICAL between allow_tf32=False and allow_tf32=True. Full suite: 500 passed, 2 skipped (baseline 498). Co-Authored-By: Claude <noreply@anthropic.com>
tau_min_d3q27 (0.52) clamps the BGK/MRT relaxation time, so a requested Re=1e6 is NOT realized at 96^3 / Mach 0.3: the solver runs at tau=0.52 and Re_effective ~= 2,494. The solver previously reported the requested value everywhere (and reynolds_number_turbulent mixed physical v_inf with lattice nu, dimensionally wrong). - _resolve_relaxation_time(): map requested nu -> (tau_actual, nu_effective) - collide_stream / collide_stream_batch: self.nu is now the REALIZED viscosity; self.tau_actual / self.nu_effective / self.nu_requested added. Both eager and deferred coefficient paths read self.nu, so both now report the realized value. - coefficient dicts: add requested_reynolds / effective_reynolds / reynolds_clamped / tau_actual / effective_laminar_viscosity; fix reynolds_number_turbulent to lattice units. Batch path exposes the same laminar keys. - config.yaml: document the clamp under cfd.reynolds_number. - test: test_effective_reynolds_and_tau_actual (clamped + unclamped). - docs: performance/2026-08-17-effective-reynolds-r2.md (numbers + paper implication). Co-Authored-By: Claude <noreply@anthropic.com>
The multiobjective combination computed every gradient L2 norm / dot product / cosine by flattening the ENTIRE model's gradients into one FP64 1-D tensor (28M params -> a ~224MB fp64 buffer), materialized repeatedly per update. Only scalars are ever needed. - gradient_l2_norm: per-tensor fp64 vector_norm on GPU, ONE .tolist() of the n scalars, combined scale-and-square-root so extreme values (1e308) stay finite. Nonfinite elements surface as the same NonFiniteGradientError. - _gradient_dot_product: per-tensor fp64 torch.dot, one host read, math.fsum. - Removed _flatten_present_gradients and _cat_grouped_to_dtype (dead). Full suite 501 passed / 2 skipped (incl. the 1e308 extreme-value norm test). Co-Authored-By: Claude <noreply@anthropic.com>
PR 41 review item (4): the resume compatibility check did not record the experiment section, so a resume could silently flip numerics (tf32_gemm_math) or execution (graph_decode_mlp / batch_guard_dot_reads / deferred_solver_reads) without raising an incompatibility. - run_monitored_training.py: add _experiment_flags_fingerprint(), resolving the four experiment flags through the same config_value accessors the trainer uses, and record them as configuration.experiment_flags. - aircraft_diffusion_cfd.py: compare the configuration sub-dict over the INTERSECTION of keys, so a fingerprint key added in newer code does not block resuming an older run-state that predates it. Keys present on both sides are still compared strictly. - tests: cover backward-compatible resume, a tf32 flip blocking resume, and a shared-config drift still being caught. Full suite 502 passed / 2 skipped (was 501/2). Co-Authored-By: Claude <noreply@anthropic.com>
PR 41 review item (5): after a resume the launcher reset the lexicographic promotion-rank gate to (-1,)*8, best_geometry_metric to inf, and best_checkpoint_path to None, so the first passing promotion after resume overwrote best_geometry_model.pt even if it was worse than the pre-resume best, and the history payload reported best_checkpoint_path=None. The run_state_metadata dict round-trips verbatim through build_run_state / load_run_state, so best-checkpoint selection is mirrored there: - fresh start / warm start: seed best_promotion_rank / best_geometry_metric / best_checkpoint_path (baseline checkpoint is the initial best) so the first run-state save captures it; - candidate-improved epoch: sync the updated gate into run_state_metadata; - resume: restore the gate via _restore_best_promotion_rank (fail-safe (-1,)*8 for run-states that predate the field) so the rank comparison and reported best path/metric survive. Tests: restore round-trips a persisted list, falls back for missing/garbage/ None; sync mirrors the gate into run_state_metadata. Full suite 504 passed / 2 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
…resume (R6) PR 41 review item (6): the monitored loop initialized history as an empty list on every start and rewrote the history JSONL from it each epoch, so an exact resume restarted the convergence window cold (delaying early-stop) and dropped every pre-resume epoch row from the report. - new _load_monitored_history(): reads the persisted monitored-history payload (rewritten each epoch with the full record) and returns its epoch rows, degrading defensively to [] for missing / malformed / history-less files. - resume path: seed the in-memory history from the payload so summarize_stability's rolling window (history[-window:]) includes pre-resume context and convergence/early-stop resumes from real data instead of cold. If the run had already converged before interruption, the first stability summary after resume reports it and early-stops correctly. Test: round-trips a persisted payload; defensive for missing/JSON-error/ history-less/non-dict files. Full suite 505 passed / 2 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
…er (R7) PR 41 review item (7): the monitored train loader used shuffle=False, so every epoch visited the same sample order. Add ResumableEpochSampler, a torch.utils.data.Sampler that draws a fresh permutation per epoch seeded by (subset_seed, epoch) via random.seed(str) (SHA-512, stable across interpreter runs and independent of PYTHONHASHSEED). Resumability: the permutation is a pure function of (subset_seed, epoch), so a resumed process regenerates the current epoch's order and train_epoch's start_batch skip (completed_in_epoch) continues at the exact offset. The subset composition (sample_order) is unchanged, so the R4 fingerprint and resume validation still hold; only the per-epoch iteration order shuffles. Only the train loader uses the sampler; promotion/calibration loaders stay fixed-order (deterministic evaluation paths). Test: same (seed, epoch) reproduces the permutation, different epoch/seed differs, len matches the sample count. Full suite 506 passed / 2 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
Decide + enforce the "mission-adaptive CFD" semantics from PR 41 review item 8: the aerodynamic solve is deliberately mission-independent — every sample runs at the same global flow conditions (Mach 0.3, realized effective Re ~2,494 at 96^3 after the tau floor), so the aero loss is a stable, cross-sample-comparable objective. Mission-adaptivity is delivered through the conditioning vector (design_spec.target_speed) and per-mission flight-path synthesis at evaluation time, not through the solve regime. - Add SOLVER_MISSION_INDEPENDENT_FIELDS + _mission_independent_solver_conditions and enforce the contract in AdvancedCFDSimulator.__init__ (flow conditions must be positive scalars derived from config alone). - test_solver: realized tau_actual/nu_effective identical across geometries. - test_cfd_solver_contract: flow tuple is global-scalar derived, unaffected by mission-flavored config fields. - Document the decision + paper implication (docs/performance/r8). Full suite: 508 passed, 2 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
PR 41 review item 9 (config source-of-truth): reynolds_number and simulation_steps were hardcoded in both CFDConfig classes while config.yaml declared the same fields, so the YAML values were silently ignored. - Both CFDConfig copies now read reynolds_number / simulation_steps via config_value(), matching mach_number; config.yaml is the single source of truth for the CFD operating point. - config.yaml simulation_steps aligned to 1000 (the value pre-existing exact-resume fingerprints recorded) with an advisory note: the field is not consumed by any training/validation path, so this is a zero-behavior change that preserves R4 resume compatibility. - tests/test_config.py pins the YAML-sourced values and that the two CFDConfig classes cannot silently diverge on the flow fields. Full suite: 510 passed, 2 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
… (R10) Routes trainer save_checkpoint through a new atomic_write_checkpoint helper and adds a missing fsync to the run-state checkpoint path, so no durable artifact can be torn by a crash mid-save. Also fixes a Windows-only bug the new tests caught: os.fsync on a read-only handle raises OSError 9, so both call sites now fsync on the write handle (matching atomic_save_run_state). JSONL telemetry stays flush-only by documented design (reconciled at resume via offset+sha256). Tests: +2 durability tests (success leaves no .tmp sibling / failure leaves existing target byte-identical), smoke test updated for write-handle arg. Full suite 512 passed / 2 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
Re-measures the batch-chunk comparison through the non-instrumented production update path (real step1305 checkpoint, real grounded geometry, TF32-NN-on per R1). C=1 = 27.5 s/u mean (p90 30.2) vs 62.66 pre-review; C=2 +6.5%, C=4 +9.7% and over-commits the 8 GiB card (8.4 GiB reserved, WDDM spill). Reinforces _DIRECT_SOLVER_BATCH_CHUNK=1 and gives the paper a defensible production per-update cost + VRAM ceiling boundary. Co-Authored-By: Claude <noreply@anthropic.com>
The C=1 sequential q-algebra (production backend pytorch_reference calls _get_q -> compute_link_q once per solve, 33x/update) materialized ~5 stacked [26,96^3] fp32 temporaries (~440 MB peak at 96^3) to produce an 88 MB result. compute_link_q now evaluates the identical per-element formula one direction at a time, writing into the pre-filled 1.0 q_all; only [D,H,W] working-set temporaries are ever live. torch.equal(new, old) is True at 96^3 on GPU; per-element arithmetic is unchanged (batch width does not affect IEEE fp32 determinism). Cold compute_all_link_distances delegates to it and inherits the fix. Also cache the geometry-static max_count in the BFL sparse table (computed once at build) and read it in stream_bfl_d3q27_batch_compressed instead of a per-solve pair_count.max().item() host sync (with a .get fallback for legacy hand-built dicts). Measured: - isolated q-algebra peak reserved: 627.0 -> 142.6 MiB (~470 MiB drop) - integrated C=1 production-faithful harness: peak reserved 7460 -> 7334 MiB (~126 MiB drop; headroom 0.54 -> ~0.67 GiB on 8 GB); s/u 27.53 -> 27.17 (flat; direct phase is CPU-prep-bound) - bit-identity gates: 5 parity test files pass (37 passed / 2 skipped) - full suite: 512 passed / 2 skipped (baseline preserved) Docs: docs/performance/2026-08-18-solver-memory-r12.md Co-Authored-By: Claude <noreply@anthropic.com>
The evidence surface for the 13-item review round: a table mapping each item to its commit, the tests that verify it, and its evidence doc, plus the claim-bearing numbers locked by the round (27.5 s/u, 7.33 GiB peak reserved, bit-identical solver numerics). Gitignore the worktree-local scratch (.ramcheck, .ramwatch, the 08-16 MERGE-REPORT) so git add -A cannot sweep it into the PR. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR lands the constrained-aircraft recovery stack developed and verified across the August 13–15 recovery/optimization pass. It is 55 commits ahead of
mainand focuses on making solver-integrated training recoverable, physically constrained, numerically stable, GPU-bounded, and safer to resume.The central failure being addressed was a 0↔~99% occupancy bang-bang instability in free-running aircraft geometry. The recovery path now keeps materialization semantics fixed, replaces the flip-noisy hard-threshold occupancy SPSA term with an analytic smooth occupancy gradient, constrains measured solver objectives against geometry/topology guards, and preserves enough run state to continue interrupted training without silently changing the experiment.
Recovery and continuation correctness
.previousfallback so an interrupted replacement does not destroy the previous usable state.Occupancy-collapse recovery
Constrained measured-objective optimization
Direct solver and SPSA path
GPU/performance work
Checkpoint loading hardening
torch.load(..., weights_only=True)for checkpoint metadata and run-state loads.weights_only=Falsefallback only for trusted localbuild/artifacts or an explicitly operator-authorized--resume-from/--warm-start-frompath, with a warning on fallback.Verification recorded on the branch
DesignSpec → 22-D condition → 4-step consistency model → LatentTo3DConverter → 96^3 probability field → frozen threshold → voxels → aircraft-validity evaluationusing the interruption-safe recovery checkpoint.weights_only=Truepath; trusted-vs-untrusted fallback behavior was separately exercised with a synthetic checkpoint.Review focus
The highest-value review areas are:
Head at PR creation:
f44c2cac1dc6c63f881f3615237d607b093c10b0.Review round — 13 items (R1–R13), all addressed
Review HEAD:
f439f47. Full item → commit → test → evidence mapping lives indocs/performance/2026-08-18-pr41-review-reconciliation.md. Summary:2e9293bd675de0tau_actual/Reynolds;docs/performance/2026-08-17-effective-reynolds-r2.mded9dfb53905072e5f369c67d9b5399d0aebshuffle=False6e46f8c0276d4cconfig.yaml; doc R9f2ab840eb62055987a16cmax_count; doc R12f439f47Verification (current): full suite 512 passed / 2 skipped at
f439f47;solver/BFL parity gates green (37 passed / 2 skipped across the 5 parity files).
Claim-bearing numbers locked by this round (production-faithful harness:
real
step1305.pt, real 1069-geometry corpus,--no-instrument, TF32-NN-on,solver IEEE fp32, RTX 4060 Laptop 8 GB): 27.5 s/u mean full optimizer
update at 96³/batch 1 (p90 ≈ 30 s/u, ~2.1× vs pre-review 62.66 s/u); 7.33 GiB
peak reserved with ~0.67 GiB headroom on the 8 GiB card; C=4 reproduces the
WDDM spill boundary that keeps
_DIRECT_SOLVER_BATCH_CHUNK = 1. Solvernumerics are bit-identical across R10/R12.