Support Single Precision mode across the model#940
Draft
suvarchal wants to merge 8 commits into
Draft
Conversation
Make working precision configurable via -DUSE_SINGLE_PRECISION=ON. Default build stays double precision and unchanged. - WP is now preprocessor-selected (4/8) in o_PARAM; central MPI_WP parameter in MOD_PARTIT mirrors it (MPI_REAL vs MPI_DOUBLE_PRECISION). - CMake gains USE_SINGLE_PRECISION option with per-compiler default-real flag flip (Intel -r4, GNU drops -fdefault-real-8, Cray -s real32, NVHPC -Mr4); CVMix forced OFF in SP (double-only, out of scope). - Replace MPI_DOUBLE_PRECISION/MPI_DOUBLE with MPI_WP at WP-data sites; convert WP comm/gather buffers from real(real64) to real(kind=WP). - NetCDF stays double on disk; WP reads routed through generic nf_*_vara_x so existing meshes/restarts/forcing keep working (convert-on-I/O). - mod_transit and reference-density locals converted to WP; io_meandata vector rotation round-trips through WP temps; iceberg subsystem made WP-consistent. - Startup banner prints WP byte size and SINGLE/DOUBLE mode. Co-authored-by: Cursor <cursoragent@cursor.com>
The KPP divide-by-zero guard epsln=1.0e-40 is a denormal in real32 and is flushed to zero under Intel -O3 (FTZ/DAZ) in single-precision builds, defeating the /(wm+epsln), /(hbl+epsln), /(dVsq+Vtsq+epsln) guards and producing 0/0 NaN at the cold start (Wvel NaN at step 1 on core2). Make epsln precision-dependent: 1.0e-20 (normal, FTZ-safe) in single precision, 1.0e-40 (unchanged) in double precision. The double-precision build is bit-identical.
Intel-compiled fesom.x stack-allocates large per-column automatic arrays and SIGSEGVs on global meshes (e.g. core2) under the default 8 MB stack limit. Wrap the test launch in /bin/sh -c "ulimit -s unlimited || ulimit -s 1048576; exec ..." so mpiexec and the ranks it spawns inherit a larger stack. Degrades gracefully where unlimited is disallowed; harmless for GNU/CI.
CVMix is a fixed double-precision library (kind cvmix_r8 = real64); it was forced OFF under USE_SINGLE_PRECISION. Link it in both precisions and convert WP<->cvmix_r8 at the CVMix call boundary in the wrappers (scalar args cast, array args via cvmix_r8 temp buffers; FESOM-side arrays stay WP). CVMix always runs in double precision, so the mixing-scheme menu is identical in single and double precision (enabling like-for-like comparison) and the default double-precision build is numerically unchanged. Verified on core2: KPP, PP, cvmix_PP, cvmix_KPP and cvmix_TKE all run in GNU and Intel single precision.
In a single-precision build (WP=4) a namelist.io field may request 8-byte output (e.g. the stock 'salt',1,'m',8). The mean-I/O layer already honours this -- the field is written as NF_DOUBLE with its mean accumulated in real64 -- but the samples are single-precision-sourced. Make the user aware: tally such fields as streams are defined and print one summary line at the end of ini_mean_io (rank 0 only), listing the affected variables. No per-stream or per-step output; silent in double-precision builds where accuracy > WP can never trigger. Add an SP-only ctest (integration_pi_sp_mpi2, label single_precision) that runs the pi mesh and asserts the SINGLE PRECISION MODE startup banner. To support it, add_fesom_test[_with_options] gains optional EXTRA_SUCCESS_MARKERS and LABEL arguments; default behaviour is unchanged.
No integration test exercised the CVMix vertical-mixing path. The stock namelist.oce ships mix_scheme='KPP' -- FESOM's native (non-CVMix) KPP -- so a CVMix-enabled build (CVMIX=ON, the default) never runs CVMix code by default; CI only ever covered native KPP. CVMix schemes such as cvmix_TKE (used at high resolution) went untested. Add integration_pi_cvmix_tke_mpi2 and integration_pi_cvmix_kpp_mpi2 (label cvmix): pi-mesh runs with mix_scheme set to 'cvmix_TKE' / 'cvmix_KPP'. CVMix is a fixed double-precision library (kind cvmix_r8 = real64) built into FESOM's tree via FetchContent; only the FESOM-side WP<->cvmix_r8 conversion differs by build. So each test covers the previously-untested FESOM-DP+CVMix path in a DP build and the FESOM-SP+CVMix path (the conversion shims from the CVMix-in-DP change) in an SP build -- run `ctest -L cvmix` in each build dir. Gated on CVMIX (default ON); no precision guard. Pass/fail is the clean run + artifacts, so a wrong scheme string or a CVMix-less build fails at setup. To support it, add_fesom_test[_with_options] gains an optional MIX_SCHEME argument that rewrites mix_scheme in the test's namelist.oce (whitespace- tolerant, comment-preserving, matching the harness's other rewrites). Verified: both schemes pass in GNU DP and GNU SP builds on pi, with CVMIX_TKE / CVMIX_KPP genuinely initialised (real64 vs real32 params in the logs).
…kflow
Precision is a property of the build, not the test: in a USE_SINGLE_PRECISION
build every integration test runs single precision; in a DP build every test
runs double. The previous per-test SP modelling (integration_pi_sp_mpi2 + a
single_precision label) was redundant and left the base pi tests unlabelled.
Harness (cmake/FesomTesting.cmake):
- Every fesom.x integration test now auto-asserts the build's startup banner
('SINGLE'/'DOUBLE PRECISION MODE', chosen by USE_SINGLE_PRECISION) as a
required success marker, so each test self-certifies it ran at the intended
precision -- a mis-configured build fails loudly on every test.
- Every integration test gets a base 'integration' label; the optional LABEL
arg (e.g. cvmix) is now appended, so cvmix tests carry 'integration;cvmix'.
`ctest -L integration` runs the suite, `ctest -L cvmix` isolates CVMix.
tests/integration/CMakeLists.txt:
- Remove the now-redundant integration_pi_sp_mpi2 (its guarantee holds for
every test in an SP build) and the single_precision label.
CI (.github/workflows/fesom2_sp_ctest.yml, new):
- SP-only, non-blocking (continue-on-error) workflow: builds full-config in
single precision (mirrors DP: MESHPARTITIONER + MESHDIAG ON) so the whole
local suite registers in SP, and runs it via `ctest -E '^(remote_|ecbundle_)'`.
DP is already covered by fesom2_ctest_runner.yml and the default preset (which
already runs the cvmix tests in DP), so no dp/sp matrix -- that would be a
third DP run. Promote SP to blocking later by removing continue-on-error;
widening to SP meshpart/meshdiag/remote needs no build change, only run
selection.
Verified: build-dp/build-sp reconfigure; integration_pi_sp_mpi2 gone;
'integration'/'cvmix' labels present, 'single_precision' gone; generated test
scripts carry the correct DOUBLE/SINGLE marker; a DP integration test passes
with the new required marker.
Line 51 began with a literal '-' (a diff fragment committed by mistake in #938), which terminated the `run: |` block scalar and made the whole workflow invalid YAML -- GitHub rejects it ("Invalid workflow file") and act cannot parse it. Restore the intended 8-space indentation so the line stays a comment inside the run block; the commented-out ctest invocation is preserved, not deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single precision capability,
Adds opt-in single-precision (-DUSE_SINGLE_PRECISION=ON) for standalone fesom.x. Default build stays double precision and unchanged. Mesh/restart/forcing files remain double on disk; WP arrays convert on I/O so existing features. CVMIX in SP is wip.
CORE2
Benchmark: core2 mesh, core2 forcing, 1 year, on levante 64-tasks-1-compute node, used KPP as CVMIX needs bit more work to be used in SP mode. Compiled with intel compiler.
SP: SYPD : 22.0574 years/day
DP: SYPD : 13.3427 years/day
speedup : ~ 1.69x
spatial plots are just last month (december)
PI January 1948, 1 month)
Benchmark: bundled pi mesh (~3.1k nodes), CORE2 forcing, 2 976 steps (Δt = 900 s), lean monthly I/O (sst, sss, ssh, a_ice). Two builds: build-sp (SP) and build-dp (DP). Local ubuntu, GNU + OpenMPI.
Global metric — SYPD (Simulated Years Per Day, from fesom.stats): higher = more simulated time per wall-clock day.
SP vs DP: ~1.21× faster wall time at both rank counts (+17% at mpi2, +19% at mpi8). SYPD: +21% (mpi2) and +23% (mpi8).
MPI scaling (same precision): mpi2→mpi8 ≈ 2.7× wall-time speedup and 2.7× SYPD (sub-linear vs 4× cores on this small mesh).
Largest mpi2 component gains (SP/DP): oce_mix_pres ~1.6×, oce_ssh_solve ~1.55×, oce_timestep_ale ~1.27×. ice_timestep ~neutral (~1.01×).
Field sanity (month 1, mpi2): SST max|Δ| ≈ 0.24 °C, SSS ≈ 0.05 psu, SSH ≈ 2.3 mm — SP and DP stay close.
