CI: submit configure and build timings to CDash - #7246
Open
ax3l wants to merge 4 commits into
Open
Conversation
Our CDash dashboard showed test results only. Every build reported `hasconfigure=false` and `hascompilation=false`, so configure and build timings, and all compiler warnings and errors, were missing. The cause is that CDash only receives those results when the steps run *through* CTest, which records them in `Configure.xml` and `Build.xml`. All three CI systems called `cmake` and `cmake --build` directly and then ran `ctest -D ExperimentalTest -D ExperimentalSubmit`, which produces a `Test.xml` and nothing else. Add `Tools/CI/ctest_dashboard.sh`, a wrapper around the CTest script `Tools/CI/ctest_dashboard.cmake`, that drives the configure and build via `ctest_configure()` and `ctest_build()`. Using a CTest script rather than `ctest -D ExperimentalConfigure` is deliberate: the latter needs a `DartConfiguration.tcl`, which only exists after a first `cmake` run, so it would time a warm re-configure and under-report by minutes. The test step stays with the caller, so CI keeps its existing `ctest` command line including options such as `--no-tests=error`, which `ctest_test()` does not offer. All steps share one dashboard tag, so a single `-D ExperimentalSubmit` uploads them as one CDash build. User-facing: - CDash now shows configure and build times per CI build, and every compiler warning and error with its source file and line. - The GitHub Actions compile matrix reports to CDash for the first time, under site `GitHub-Actions`. Internal: - The wrapper appends `-DBUILDNAME`/`-DSITE` to the configure options. Without them `include(CTest)` labels `Test.xml` with the host name and a generic build name, and CDash files it as a separate build from `Configure.xml` and `Build.xml`. - Build parallelism moves from `cmake --build -j N` to `CMAKE_BUILD_PARALLEL_LEVEL`, since the CTest build step has no `-j`. - Submission still only happens on merges to `development`. Build-only jobs submit from the script via `CDASH_SUBMIT=ON`; jobs that also test submit from a step that runs even when the build or tests failed, so a broken build reaches the dashboard. - Not converted: Windows (PowerShell/cmd), the GNU Make CUDA job (no CMake), and the clang-tidy, CodeQL and sanitizer workflows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CxAc564RY4pPkGbhio7fBm
Follow-up to the previous commit, from review of the CDash wiring. Each item was reproduced against CTest 3.31 before and after the fix. - Submit a configure that fails before `include(CTest)`. Such a failure leaves no `DartConfiguration.tcl`, so the CI's own `ctest -D ExperimentalSubmit` aborts with exit 64 and discards the `Configure.xml` holding the CMake error. Everything above `CMakeLists.txt:199` is in that window: compiler detection, the `WarpX_DIMS` validation, all `option()` handling -- exactly the failures a dashboard is most useful for. The driver now submits these itself, driven by `CTestConfig.cmake` rather than the build directory, and the CI submit steps skip when `DartConfiguration.tcl` is absent. - Stop a CDash outage from failing a green build. `ctest_submit()` made `ctest -S` exit 255 on an upload error, which on pushes to `development` would fail all 14 GitHub build jobs whose builds had already succeeded. Note that `RETURN_VALUE` alone does not prevent this; `CAPTURE_CMAKE_ERROR` is what does. - Pass the build parallelism to `ctest_build(PARALLEL_LEVEL ...)` explicitly. It was reaching the compiler only by leaking through the environment, which also silently parallelized the later `--target pip_install` calls in the macOS and Intel jobs. An unset `CMAKE_BUILD_PARALLEL_LEVEL` now passes nothing rather than forcing a level, so Ninja keeps its own one-job-per-core default. - Restore `build_pyfull`'s build scope via the new `CDASH_BUILD_TARGET`. That job only ever built the `pip_install` target, which does not depend on the `warpx` app, so building the default target added a compile and link the job never did. - Correct the claim that the CTest build step has no `-j`, repeated in four places. `ctest_build()` does take `PARALLEL_LEVEL`. - Note that `ctest_start()` reuses a tag only within the same UTC day, so an Azure job straddling midnight splits into two dashboard builds. - `file(STRINGS ENCODING UTF-8)` so a non-ASCII option cannot be silently truncated, and require CMake 3.25 to match the project. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CxAc564RY4pPkGbhio7fBm
The `Clang pywarpx` job failed on this branch. The build itself was fine
-- `pip` reported "Successfully installed pywarpx-26.9" and the build
command exited 0 -- but CTest reported "15 Compiler errors" and the job
exited 255.
CTest finds build errors by matching regexes against the build log, and
one of its default patterns is `([^ :]+):([0-9]+): ([^ \t])`. Every one of
the 15 "errors" was a Python warning from `pip`:
setuptools/_distutils/dist.py:318: UserWarning: Unknown distribution
option: 'tests_require'
setuptools/dist.py:332: InformationOnly: Normalizing '26.09' to '26.9'
CTest's exception list covers `: warning` but not setuptools' warning
class names, so these are counted as errors. Applying the default
matchers to the job log by hand reproduces the count exactly, 15 of 15.
The cause was `CDASH_BUILD_TARGET=pip_install`, added in the previous
commit to stop the dashboard building targets this job never built. That
pulled pip's output inside the build step CTest scrapes. `build_pyfull`
now builds the default target and runs `--target pip_install` afterwards,
outside the dashboard step -- the same shape as the macOS and Intel jobs,
which pass. The cost is the extra app compile and link that the previous
commit set out to avoid; roughly a minute, and worth it here.
`CDASH_BUILD_TARGET` is removed rather than left unused: its only purpose
was this job, and a knob whose obvious use is a packaging target is a trap.
The restriction is now documented where the script builds.
Note that the concurrently failing `clang-tidy-1D` job is unrelated to
this branch: `apt.llvm.org` was unreachable while installing LLVM
("connect (101: Network is unreachable)").
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CxAc564RY4pPkGbhio7fBm
Review of the plumbing, not the design. `ctest -S` stays: the CTest command line drives an already-generated build tree, so it can only time a warm re-configure, and a dashboard client script is the documented way to drive the configure itself. It is also what VTK, ParaView, CMake, ITK, Trilinos and deal.II do. The wrapper around it was doing work CMake already does: - `separate_arguments(UNIX_COMMAND)` parses shell-quoted arguments back into one list element each and escapes embedded `;` on the way. That replaces the whole options-file mechanism: the temporary file, the `file(STRINGS ENCODING UTF-8)` read and the `string(REPLACE ";" "\;")` loop. The wrapper now hands the options over with `printf '%q '`. - `CTEST_SCRIPT_DIRECTORY` gives the source directory, so the shell no longer resolves it. - `list(APPEND)` adds `-DBUILDNAME`/`-DSITE` in one line. - The 15-line `-G` parser, which existed because `ctest_configure()` appends its own generator after our options and would override one passed there, becomes a guard that says so. The two jobs that passed `-GNinja` inline now set `CMAKE_GENERATOR`, like every other job. 81 -> 62 lines of actual code, and three mechanisms fewer to understand. No behaviour change and no CI YAML churn beyond those two `-GNinja`. Re-verified after the rewrite: option fidelity for values with spaces and semicolons, all four generator paths, parallelism set/unset/malformed, identical BuildName and Site across Configure.xml, Build.xml and Test.xml, and the exit codes -- success 0, build failure 255, early configure failure 255 with Configure.xml still uploaded, CDash unreachable 0. Considered and rejected: `CMakePresets.json`. Workflow presets produce no dashboard XML at all, test presets have no dashboard schema, and `ctest --preset X -D ExperimentalTest` ignores the preset's `binaryDir`, finds no tests and exits 0 -- green CI that tested nothing. Presets are worth having for local developer ergonomics, separately. `CTEST_USE_LAUNCHERS` would give exact per-file diagnostics instead of regex log scraping, but a signal-killed custom command passes silently under launchers until CMake 4.5 (cmake#27921), so it wants its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CxAc564RY4pPkGbhio7fBm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Our CDash dashboard shows test results only. Every build reports
hasconfigure=falseandhascompilation=false, so configure and build timings — and every compiler warning and error — are missing.CDash only receives those results when the steps run through CTest, which records them in
Configure.xmlandBuild.xml. All three CI systems calledcmakeandcmake --builddirectly and then ranctest -D ExperimentalTest -D ExperimentalSubmit, which produces aTest.xmland nothing else.This PR configures and builds through CTest in all three CI systems, and reports the GitHub Actions compile matrix to CDash for the first time.
Details
New
Tools/CI/ctest_dashboard.sh, a wrapper around the CTest scriptTools/CI/ctest_dashboard.cmake, drives the configure and build viactest_configure()andctest_build():CDASH_BUILD_NAME=CPU-3D CDASH_SITE=Azure CMAKE_BUILD_PARALLEL_LEVEL=2 \ Tools/CI/ctest_dashboard.sh build -DWarpX_DIMS=3 -DWarpX_FFT=ONUsing a CTest script rather than
ctest -D ExperimentalConfigureis deliberate: the latter needs aDartConfiguration.tcl, which only exists after a firstcmakerun, so it would time a warm re-configure and under-report by minutes.The test step stays with the caller, so CI keeps its existing
ctestcommand line — including options such as--no-tests=error, whichctest_test()does not offer — adding only-D ExperimentalTest. All steps share one dashboard tag, so a single-D ExperimentalSubmituploads them as one CDash build.User-facing:
GitHub-Actions.Internal:
-DBUILDNAME/-DSITEto the configure options. Without theminclude(CTest)labelsTest.xmlwith the host name and a generic build name, and CDash files it as a build separate fromConfigure.xmlandBuild.xml.cmake --build -j NtoCMAKE_BUILD_PARALLEL_LEVEL, which the driver passes on toctest_build(PARALLEL_LEVEL ...).development. Build-only jobs submit from the script viaCDASH_SUBMIT=ON; jobs that also run tests submit from a step that runs even when the build or the tests failed, so a broken build reaches the dashboard.include(CTest)is submitted by the driver itself. There is noDartConfiguration.tclat that point, so actest -D ExperimentalSubmitstep would abort and discard theConfigure.xmlholding the CMake error — and that window covers compiler detection, theWarpX_DIMSvalidation and alloption()handling.ctest_submit()runs withCAPTURE_CMAKE_ERROR(RETURN_VALUEalone does not suppress the non-zero exit).cmd), the GNU Make CUDA job (no CMake), and the clang-tidy, CodeQL and sanitizer workflows. These can follow separately.Testing
Verified locally against a real WarpX build and a minimal reproducer project:
Configure.xmlandBuild.xmlare produced and carry real cold-configure and build timings (27 s configure / 465 s build on a 1D+2D build).Build.xml), and the non-zero exit propagates.Configure.xml,Build.xmlandTest.xmlall carry the sameBuildNameandSite, so CDash groups them as one build.-DCMAKE_CXX_FLAGS="-Werror -Wall") and semicolons (-DWarpX_DIMS="1;2") survive the wrapper unchanged.-GNinja,-G Ninja, theCMAKE_GENERATORenvironment variable, and the default.CMAKE_BUILD_PARALLEL_LEVELreaches the build tool (cmake --build . --parallel "7"); unset or malformed values fall back to the generator's own default rather than to serial.pre-commit runpasses, and the Sphinx docs build cleanly.Exit-code semantics were checked explicitly, since they decide whether CI goes red:
0255, with errors inBuild.xmlinclude(CTest)255,Configure.xmluploaded0, warning onlyThe second commit fixes issues found in review; each was reproduced against CTest 3.31 before and after the fix.
The dashboard itself can only be confirmed after this merges to
development, since PR pipelines deliberately do not submit.🤖 Generated with Claude Code
https://claude.ai/code/session_01CxAc564RY4pPkGbhio7fBm