From 304f45c866185407448da5e2944cb8ac107fbe57 Mon Sep 17 00:00:00 2001 From: Clawrence Date: Tue, 14 Jul 2026 07:49:23 -0400 Subject: [PATCH] bench: independent memory-ladder re-measurement + sweep_mode at-scale caveat Fresh CPX51 run: LU/LDLT ladder 626k-2.8M DOFs (ratio 0.58-0.60, cert numbers reproduced within 2.5%), rank sweep 1-16 (memory vs ranks quantified on both metrics), coax-vs-sphere per-DOF constant (2.8x), Nf=22 workload shape, and a measured negative: sweep_mode loses to per-frequency LDLT refactorization at 2.26M DOFs. Co-Authored-By: Claude Fable 5 --- bench/MEMLADDER-2026-07-14.md | 78 +++++++++++++++++++++++++++++++++++ bench/memladder.sh | 46 +++++++++++++++++++++ docs/SOLVER_IMPROVEMENTS.md | 5 +++ 3 files changed, 129 insertions(+) create mode 100644 bench/MEMLADDER-2026-07-14.md create mode 100644 bench/memladder.sh diff --git a/bench/MEMLADDER-2026-07-14.md b/bench/MEMLADDER-2026-07-14.md new file mode 100644 index 0000000..be4b560 --- /dev/null +++ b/bench/MEMLADDER-2026-07-14.md @@ -0,0 +1,78 @@ +# Memory-vs-DOF ladder — independent re-measurement (2026-07-14) + +One fresh Hetzner CPX51 (16 vCPU / 32 GB), this repo's `bench/Dockerfile.bench` image, +`bench/memladder.sh`. Motivated by three review questions: does factor memory scale +"in a straight line" with DOFs; does more MPI ranks mean more memory; and is the +coax-vs-sphere per-DOF difference real. All memory below is measured, with the metric +named: **factor** = MUMPS INFOG(22) "memory effectively used" summed over ranks; +**RSS** = peak resident set summed over ranks (the whole job, what +`memTimeEstimation.runTimesMems` records as `memCost`). + +## 1. Same geometry, fixed ranks: LU vs LDLT ladder + +`testRun` sphere geometry, degree 3, Nf=1, 8 ranks: + +| DOFs | LU factor | LDLT factor | ratio | LU→LDLT solve time | max dS vs LU | +|---|---|---|---|---|---| +| 626,088 | 5,147 MB | 3,205 MB | 0.62 | 37.5 → 16.5 s * | 4.4e-16 | +| 898,902 | 7,283 MB | 4,336 MB | 0.60 | 25.7 → 22.4 s | 1.1e-15 | +| 1,231,185 | 9,863 MB | 5,939 MB | 0.60 | 35.8 → 29.3 s | 5.6e-16 | +| 1,803,117 | 15,619 MB | 9,148 MB | 0.59 | 61.1 → 46.0 s | 7.9e-16 | +| 2,258,487 | 19,495 MB | 11,580 MB | 0.59 | 74.1 → 60.6 s | 4.5e-16 | +| 2,801,391 | 24,021 MB | 14,035 MB | 0.58 | 111.0 → 74.1 s | 8.9e-16 | + +\* first case in the session; LU time includes one-time JIT compilation — treat its +timing (not memory) as an outlier. + +Per-DOF factor memory rises 8.2 → 8.6 MB/kDOF across the range: mildly superlinear, +as sparse-direct theory predicts. The 2.8M LU row independently reproduces the +certification table (24,021 vs 24,646 MB, −2.5%; solve 111 vs 120 s). + +## 2. Rank count vs memory (same problem, stock LU, 898,902 DOFs) + +| ranks | factor | RSS (whole job) | solve | +|---|---|---|---| +| 1 | 6,348 MB | 8.3 GB | 126.2 s | +| 4 | 6,605 MB | 10.6 GB | 41.5 s | +| 8 | 7,283 MB | 12.3 GB | 25.7 s | +| 16 | 8,997 MB | 15.4 GB | 16.4 s | + +More ranks means more total memory on both metrics (+42% factor, +86% RSS from +1→16 ranks) buying a 7.7x speedup. On many-rank cluster jobs the per-rank overhead +dominates whole-job memory, which is why a factor-level change (LDLT) moves job RSS +much less than it moves the factor itself. + +## 3. Geometry sets the per-DOF constant + +Single rank, stock LU, same box, same day: + +| case | DOFs | factor | per kDOF | +|---|---|---|---| +| coax cable-port (idx 3, h=1/3.5, deg 3) | 544,731 | 10,912 MB | 20.0 MB | +| testRun sphere (h=0.48, deg 3) | 966,117 | 6,823 MB | 7.1 MB | + +The coax matrix costs **2.8x more memory per DOF** than the sphere matrix — memory +vs DOFs cannot be read across different geometries. (The 10,912 MB coax number also +reproduces the SOLVER-UPDATE table exactly, from a freshly regenerated matrix.) + +## 4. Nf=22 workload shape (2,258,487 DOFs, 8 ranks, LDLT) + +| config | factor | RSS (whole job) | total solve | +|---|---|---|---| +| `{'symmetric': True}` | 11,636 MB | 20.1 GB | 1,104 s | +| `{'symmetric': True, 'sweep_mode': True}` | 11,556 MB | 25.2 GB | 1,915 s | + +**Negative result — do not chase:** at this scale `sweep_mode` is a net LOSS with a +plain LDLT factor (1.7x slower, slightly more memory, despite zero re-anchors): the +LDLT refactorization is cheap enough (~60 s) that FGMRES-iterating on a stale anchor +costs more than just refactorizing per frequency. sweep_mode's measured win remains +the fp32-anchor configuration at laptop scale (see SOLVER_IMPROVEMENTS.md); with fast +fp64 factors at multi-million DOFs, prefer plain per-frequency factorization. + +## Reproduce + +```bash +docker build -f bench/Dockerfile.bench -t scatt3d-bench bench/ +docker run --rm -v $(pwd):/work -w /work/bench scatt3d-bench bash memladder.sh +# results stream to bench/memladder.log; per-case logs in bench/memladder_work/ +``` diff --git a/bench/memladder.sh b/bench/memladder.sh new file mode 100644 index 0000000..1768862 --- /dev/null +++ b/bench/memladder.sh @@ -0,0 +1,46 @@ +#!/bin/bash +# Memory-vs-DOF dispute ladder (2026-07-14). +# Settles: (a) does LU factor memory vs DOFs follow a line on ONE geometry at fixed ranks, +# (b) do more ranks cost more memory (per metric), (c) coax-vs-sphere per-DOF constant. +# Metrics per case: MUMPS INFOG(16/17) estimates + INFOG(21/22) effective (the doc's metric) +# AND whole-process RSS sum over ranks (the friend's memTimeEstimation.py metric). +exec > /work/bench/memladder.log 2>&1 +source /usr/local/bin/dolfinx-complex-mode 2>/dev/null +cd /work/bench +mkdir -p memladder_work && cd memladder_work +cp ../bench_driver.py . + +run() { # tag np h solver_json + local tag=$1 np=$2 h=$3 js=$4 + echo "=== $tag np=$np h=$h ($(date -u +%H:%M:%SZ)) ===" + if [ "$np" = 1 ]; then + SCATT3D_SRC=/work/Scatt3D timeout 14400 python3 bench_driver.py "$h" 3 1 "$js" "$tag" > "case_$tag.log" 2>&1 + else + SCATT3D_SRC=/work/Scatt3D timeout 14400 mpirun -np "$np" python3 bench_driver.py "$h" 3 1 "$js" "$tag" > "case_$tag.log" 2>&1 + fi + echo "EXIT $? tag=$tag" + grep -E 'BENCH_RESULT|MUMPS factor memory|Killed|out of memory|INFOG\(1\)|Traceback' "case_$tag.log" | head -6 + rm -rf data3D/bench${tag}* +} + +# 1) sphere LU + sym lines, fixed np=8 (same geometry family the 2.8M cert used) +for H in 0.6 0.5 0.42 0.35 0.31 0.28; do + run "LUh${H}n8" 8 "$H" '{}' + run "SYMh${H}n8" 8 "$H" '{"symmetric": true}' +done + +# 2) rank A/B at h=0.5 (np=8 already covered by the ladder) +for NP in 1 4 16; do + run "LUh0.5n${NP}" "$NP" 0.5 '{}' +done + +# 3) sphere at ~545k dofs, single rank — direct comparison to the doc's coax 545k point +run "LUh0.48n1" 1 0.48 '{}' + +# 4) coax idx3 h=1/3.5 deg3 (the doc's 545k coax), single rank, LU +echo "=== COAX3 lu np=1 h=1/3.5 ($(date -u +%H:%M:%SZ)) ===" +SCATT3D_SRC=/work/Scatt3D timeout 14400 python3 ../cableport_validate.py 3 '{}' memlu 0.2857142857142857 3 1 > case_coax3lu.log 2>&1 +echo "EXIT $? tag=coax3lu" +grep -E 'CABLEPORT_RESULT|MUMPS factor memory|ndofs|Traceback' case_coax3lu.log | head -6 + +echo MEMLADDER_DONE diff --git a/docs/SOLVER_IMPROVEMENTS.md b/docs/SOLVER_IMPROVEMENTS.md index c6db40b..4598127 100644 --- a/docs/SOLVER_IMPROVEMENTS.md +++ b/docs/SOLVER_IMPROVEMENTS.md @@ -34,6 +34,11 @@ nearby ("anchor") frequency is an excellent preconditioner. In sweep mode the so the anchor factorization and runs FGMRES at subsequent frequencies; it re-factorizes (re-anchors) automatically only when iterations exceed `sweep_max_it` (default 25) or the solve fails — so worst case it degenerates to the direct solver, never below it. +**Measured caveat (2026-07-14, bench/MEMLADDER-2026-07-14.md):** with a plain fp64 LDLT +factor at 2.26M DOFs / Nf=22 / 8 ranks, sweep_mode is a net LOSS (1.7x slower than +refactorizing per frequency, zero re-anchors) — the cheap factor makes per-frequency +refactorization the better deal at scale. sweep_mode's measured win is the fp32-anchor +configuration at laptop scale below. This IS a converging iterative solver for this problem — it sidesteps the reason all ~20 previous iterative attempts failed (see below) by using spectral information the sweep has already paid for. The denser the frequency grid, the bigger the win (the measured sweeps