[bartgol]

Add a GitHub workflow that runs the standalone homme tests (gnu, intel, and cuda) on the self-hosted ghci runners whenever files in components/homme are touched.

[BFB]

---

Summary

Add .github/workflows/homme-testing.yml, which runs components/homme/utils/test_homme.py on the same self-hosted runners used by the eamxx workflows, for PRs touching components/homme/** (plus the workflow itself, get-cuda-arch, config_machines.xml, externals/ekat, and externals/scorpio). A manual workflow_dispatch with bless generates baselines.

The PR has 5 commits, one per logical change: (1) --machine-file option for test_homme.py, (2) build cprnc along with the test executables, (3) the ghci machine files, (4) the workflow, (5) a fix for a unit test that passes device pointers to MPI (cuda).

This is the first of a few stacked PRs. bfb testing (cxx vs f90, without baselines) is left to a follow-up PR stacked on this one.

Workflow

• Matrix: runner in {gnu, intel, cuda}. The runner label is also the name of the machine file; the CIME compiler for the cuda runner is gnugpu.
• Only the submodules homme needs (cime, ekat, scorpio) are checked out. The ssh urls in .gitmodules are rewritten to https for the clone.
• A Set test resources step splits the node, since homme tests do not reserve cpus/gpus: max ranks per test (USE_NUM_PROCS, passed with --cmake-opt: 4 on cpu, 2 on gpu) x threads per rank (2 on cpu, 1 on gpu) x number of concurrent tests (--run-parallel-level) must fit in the node size taken from CIME (MAX_MPITASKS_PER_NODE: 32 cores, or 2 gpus for gnugpu). OMP_NUM_THREADS and OMP_STACKSIZE are set explicitly, and the tests run with an unlimited stack (with few ranks per test, each rank holds a lot of data).
• On failure, the failed tests and the error messages of the first failing runs are printed, and the .err files are uploaded.

Machine files (cmake/machineFiles)

• ghci.cmake (common settings), plus ghci-gnu, ghci-intel, and ghci-cuda.
• Kokkos_ENABLE_DEPRECATED_CODE_5 must stay ON: with it OFF, Kokkos 5.1 aborts if the vector length of a TeamPolicy is not a power of 2, and Homme's CPU code uses the number of level packs (e.g., 3 for nlev20) as vector length in BoundaryExchange.cpp.
• HOMMEXX_MPI_ON_DEVICE is OFF for cuda, since the runners' MPI is not cuda-aware yet.

Changes to existing homme files (all opt-in; behavior is unchanged when the new options are not used)

• test_homme.py: new --machine-file (pick the machine file by name). It now passes the name of the machine file (including the -bfb suffix) to ctest_script.cmake via MACHINE_FILE. The EXTRA_CMAKE_OPTIONS argument is now shell-quoted (identical for a single option; with 2+ options the unquoted ; used to split the shell command).
• ctest_script.cmake: support for the option above (MACHINE_FILE).
• test_execs/CMakeLists.txt: the test-execs target now depends on cprnc, as check already does. This matters when cprnc is compiled from the CIME sources: ctest_script.cmake only builds test-execs, so the tests would otherwise miss cprnc. If cprnc is a prebuilt binary (found via CPRNC_DIR), it is an imported target and nothing extra gets built.
• src/share/compose/cedr_caas.cpp: in GPU builds with HOMMEXX_MPI_ON_DEVICE=OFF, the CAAS unit test skips the variant without a user reducer, which hands device buffers directly to MPI_Allreduce (production code always uses a reducer that stages through host memory). This was the likely cause of the compose_ut_test segfault on cuda (to be confirmed in CI).

Status / notes

• Seen in CI so far (before the split): intel (all test runs succeed, cprnc is built through the new dependency; only the tests that compare against baselines fail, since baselines do not exist yet), and cuda (builds and runs on H100; only the missing baselines fail, plus compose_ut_test, addressed by commit 5).
• Not yet seen: gnu has not run yet (the runner is busy with other workflows), and the latest head has not completed in CI.
• Baselines must be generated once per compiler (run the workflow manually with bless, normally on master) into /projects/e3sm/data/baselines/ghci/<compiler>/master/homme.
• No nightly schedule or CDash submission was added.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SaJsK9j78W8TBXGQqGLfDs

[Copilot]

🟡 Changes recommended

CUDA option handling, compiler environment propagation, MPI configuration, and test concurrency need correction before these jobs can run reliably.

4 open issues

• Quote CUDA CMake options before shell execution · New
• Export consistent MPI compiler overrides for builds · New
• Disable CUDA-aware MPI for ghci GPU runners · New
• Limit concurrent tests to prevent runner oversubscription · New

What changed in this PR

Add standalone HOMME testing to the existing self-hosted ghci infrastructure
for pull requests affecting HOMME and selected dependencies.

Changes:

• Add GNU/OpenMP, Intel/OpenMP, and GNU/CUDA workflow jobs, with manual
  baseline generation and CTest log uploads.
• Add shared and compiler-specific ghci machine configurations.
• Forward the requested compiler to CTest and prefer compiler-specific
  machine files, retaining the generic fallback.

File	Description
components/​homme/​utils/​test_homme.py	Forward the explicit compiler selection to CTest.
components/​homme/​cmake/​machineFiles/​ghci.cmake	Define shared runner and testing settings.
components/​homme/​cmake/​machineFiles/​ghci-intel.cmake	Configure Intel, OpenMP, and MKL.
components/​homme/​cmake/​machineFiles/​ghci-gnugpu.cmake	Configure CUDA and MPI compiler backends.
components/​homme/​cmake/​machineFiles/​ghci-gnu.cmake	Configure GNU, OpenMP, and OpenBLAS.
components/​homme/​cmake/​ctest_script.cmake	Select compiler-specific machine files when available.
.github/​workflows/​homme-testing.yml	Run the test matrix and support baseline generation.

_{Review effort: Balanced}

[bartgol]

CI status for head 998ffe7 (all jobs only fail on the missing baselines, as expected before bless):

• cuda: compose_ut_test now passes. The segfault came from the CAAS unit test handing device buffers to a non-cuda-aware MPI (fixed in the last commit).
• intel: besides the baseline tests, dirk_ut_test failed once (1 assertion out of 15866). That test seeds its random inputs from std::random_device (Catch's seed is 0) and uses tight tolerances, and nothing in this PR touches that code. The same non-bfb code passed dirk_ut_test in the intel job of the stacked PR Homme: add bfb jobs to the homme testing workflow, raise the ctest timeout, drop the log dumps #8958 (only baseline failures), so I'm treating it as a pre-existing seed-dependent flake, not something this PR introduced. I could not re-run the job here, since the run is still open: its gnu job sat in the queue for hours.
• gnu: it ran for the first time (in the stacked PR Homme: add bfb jobs to the homme testing workflow, raise the ctest timeout, drop the log dumps #8958's run), and only the baseline tests fail, like the other two.

---

Generated by Claude Code