Skip to content

Update components to match GEOSgcm as of 2026-Jul-13 - #864

Merged
gmao-rreichle merged 3 commits into
developfrom
feature/update-components-2026Jul13
Jul 16, 2026
Merged

Update components to match GEOSgcm as of 2026-Jul-13#864
gmao-rreichle merged 3 commits into
developfrom
feature/update-components-2026Jul13

Conversation

@mathomp4

@mathomp4 mathomp4 commented Jul 13, 2026

Copy link
Copy Markdown
Member

In #863, GEOSldas was moved to the latest ESMA_env (for ISSM purposes).

Now that GEOSldas is using essentially the v12-ish GEOSgcm_GridComp, we should bring along all the other updates:

  • ESMA_cmake: v3.75.0 -> v4.41.0. (v3 is essentially static now, development is on v4 and v5)
  • ecbuild: geos/v1.4.0 -> geos/v3.13.1
  • NCEP_Shared: v1.4.0 -> v1.5.0
  • GMAO_Shared: v2.1.6 -> v3.0.0 (fix for Open MPI 5 amongst others)
  • GEOS_Util: v2.1.15 -> v3.0.0
  • MAPL: v2.69.1 -> v2.70.0

The ESMA_cmake update from v3 to v4 causes the non-zero-diff result (per testing by @biljanaorescanin).

@mathomp4's nightly regression testing recently moved from gcc14 to gcc15. This necessitates the GMAO_Shared update to v3.0.0 and GEOS-ESM/GEOSgcm_GridComp#1479.


Tested by @biljanaorescanin (see below). Non-zero differences are within "roundoff" (also verified by @gmao-rreichle).

Intriguingly, for standard Intel GLOBAL/assim tests, the binary ObsFcstAna output is zero-diff. Only the nc4 HISTORY output is non-zero-diff (with diffs ~1e-34).


This PR replaces #851 and #852.

Related PRs:
GEOS-ESM/GEOSgcm_GridComp#1479 (merged 16-July-2026)

@mathomp4 mathomp4 self-assigned this Jul 13, 2026
@mathomp4
mathomp4 marked this pull request as ready for review July 13, 2026 16:49
@mathomp4
mathomp4 requested a review from a team as a code owner July 13, 2026 16:49
@gmao-rreichle

Copy link
Copy Markdown
Contributor

Thanks, @mathomp4. I closed earlier PRs that are now superseded by the present PR.
What's confusing me is that #852 was labeled non-0-diff. #852 included the cmake change from v3 to v4.
Curious to know what @biljanaorescanin's testing will show.

@mathomp4

Copy link
Copy Markdown
Member Author

Thanks, @mathomp4. I closed earlier PRs that are now superseded by the present PR. What's confusing me is that #852 was labeled non-0-diff. #852 included the cmake change from v3 to v4. Curious to know what @biljanaorescanin's testing will show.

Ohhhhhh nuts. Yes it probably is non-zero-diff for Intel. I've updated the label.

I forgot that v3 → v4 had changes to Intel flags from @wmputman and a bit from myself. These were non-zero-diff in and of themselves. I've been in v4 land for so long with GEOSgcm that I forgot that!

Hmm. I mean, if need be, we could stage this? Everything but ESMA_cmake should probably be zero-diff if I'm remembering correctly. The bits I can think of that wouldn't be, I'm not even sure the LDAS cares about (or even compiles)

@mathomp4

Copy link
Copy Markdown
Member Author

Also, maybe it's NZD for GNU? I honestly don't do as much hand-wringing to keep GNU zero-diff. I mainly want GNU to run, zero-diff-ness doesn't matter as much there.

@gmao-rreichle

Copy link
Copy Markdown
Contributor

How about @biljanaorescanin first runs the LDAS tests with the PR "as is." If it's zero-diff, we're done.
If it's NZD, we inspect the output to verify "roundoff" changes and @biljanaorescanin should test again, but this time without the cmake change. If this second test (w/o cmake change) is zero-diff, it's as expected and we're done (assuming it is also within "roundoff"). If the second test is NZD or the diffs are not within "roundoff", we need to dig deeper.

@biljanaorescanin

Copy link
Copy Markdown
Contributor

How about @biljanaorescanin first runs the LDAS tests with the PR "as is." If it's zero-diff, we're done. If it's NZD, we inspect the output to verify "roundoff" changes and @biljanaorescanin should test again, but this time without the cmake change. If this second test (w/o cmake change) is zero-diff, it's as expected and we're done (assuming it is also within "roundoff"). If the second test is NZD or the diffs are not within "roundoff", we need to dig deeper.

Regression testing summary

The standard Intel model tests contain no significant numerical mismatches:

  • LDAS_CONUS and LDAS_GLOBAL pass within numerical tolerance.
  • LDAS_DEBUGCONUS is byte-identical.
  • LDAS_GLOBALCS contains one numerically minor file, but no significant mismatch.

=> The standard Intel model differences are consistent with negligible numerical/roundoff behavior under the configured tolerances.

The optimized GNU model tests show greater numerical divergence:

  • 8 files containsignificant numerical mismatches.
  • 5 additional files contain numerically minor differences.
  • LDAS_GNUDEBUGCONUS is data identical

I am working on without cmake update tests

@biljanaorescanin

biljanaorescanin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Without cmake repo all intel tests are zero diff.

But GNU fail, one test in run and others in comparisons.

Runtype	Clone	Build	Build Time	Model Run/Compare	Assim Run/Compare
-------	-----	-----	----------	-----------------	-----------------
gnuconus	pass	pass	26 min	pass/FAIL	-- / --
gnuglobal	--	--	--	FAIL/FAIL	pass/FAIL
gnuglobalcs	--	--	--	pass/FAIL	pass/FAIL
gnuglobalcnclm4	--	--	--	pass/FAIL	-- / --
gnudebugconus	--	pass	15 min	pass/pass	-- / --

GNU run time error is weird and I will have to see with @mathomp4. In short it points to routing code RIVER_ROUTING_HYD .
Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

@mathomp4

Copy link
Copy Markdown
Member Author

Without cmake repo all intel tests are zero diff.

But GNU fail, one test in run and others in comparisons.

I forgot, GNU will be non-zero-diff because I've moved to use GCC 15 in my testing rather than 14 (which is the old GNU compiler we were using). And GCC versions usually aren't zero-diff to each other.

But the crash shouldn't happen for sure.

I probably have (or can make) Baselibs for GCC 14 that could also run. The file it's dying on is the one that @weiyuan-jiang degraded to -O2 because of other issues. Maybe GCC 15 is compiling it in a different way?

@biljanaorescanin

Copy link
Copy Markdown
Contributor

I can just try in my clone to remove that o2 constrain https://github.com/GEOS-ESM/GEOSgcm_GridComp/blob/develop/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSroute_GridComp/CMakeLists.txt and see will test pass, if it doesn't we can do some of the other options.

@biljanaorescanin

biljanaorescanin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

I think I resolved the routing test issue, and the test now passes.
It was not related to the forced -O2 compilation flag or negative storage values.

The routing code was evaluating fractional powers for the entire storage array, including elements with zero storage. With GCC 15 and the new baselibs, that calculation path triggered a floating-point exception. The fix uses a safe positive temporary base for those elements and then explicitly restores the physically correct zero flow result.

Here is branch with fix and we can talk how to proceed.I will rerun all tests with this branch and no cmake update so we can take a look tomorrow.

Here is update of test runs

Tests use fix and no cmake.

Screenshot 2026-07-14 at 7 27 32 PM

Differences are across history files, rs and binary files. Not all roundoff.

@weiyuan-jiang

Copy link
Copy Markdown
Contributor

I think I resolved the routing test issue, and the test now passes. It was not related to the forced -O2 compilation flag or negative storage values.

The routing code was evaluating fractional powers for the entire storage array, including elements with zero storage. With GCC 15 and the new baselibs, that calculation path triggered a floating-point exception. The fix uses a safe positive temporary base for those elements and then explicitly restores the physically correct zero flow result.

Here is branch with fix and we can talk how to proceed.I will rerun all tests with this branch and no cmake update so we can take a look tomorrow.

Here is update of test runs

Tests use fix and no cmake.

Screenshot 2026-07-14 at 7 27 32 PM Differences are across history files, rs and binary files. Not all roundoff.

nice fix !

@gmao-rreichle
gmao-rreichle merged commit 6ea03cb into develop Jul 16, 2026
7 of 8 checks passed
@gmao-rreichle
gmao-rreichle deleted the feature/update-components-2026Jul13 branch July 16, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants