GitHub CI pipeline: In clang-17 x TSAN config symbolize reports with a sibling llvm-symbolizer version (17's crashes on our binaries, stalling tests); accordingly re-enable transport_test SHM-jemalloc sub-mode in that config. Also: temporary test-subset input, letting 2 parallel runs split one cell's test load.

Author: ygoldfeld

.github/workflows/main.yml

@@ -107,6 +107,15 @@ on:
         required: false
         default: ''
         type: string
+      # XXX Temporary validation aid; remove (with its 3 consumers below) when the sanitizer-validation
+      # campaign concludes.  Lets 2 parallel runs split the test load of one cell.
+      test-subset:
+        description: >-
+          If 'u-t-only': the transport_test runs no-op.  If 't-t-only': the unit-tests run no-ops.
+          Blank: everything runs.
+        required: false
+        default: ''
+        type: string
 
 jobs:
   # Impetus behind this is to set up at least one magic string used in 2+ places.
@@ -606,6 +615,19 @@ jobs:
           # detector CHECK-fails -- and its failure path hangs the process (observed: a 3-hour unit_test wedge).
           # Data-race detection (TSAN's main job) is unaffected; lock-inversion detection is the sacrifice.
           export TSAN_OPTIONS="disable_coredump=0 detect_deadlocks=0 suppressions=$SAN_SUPP_CFG"
+          if [ '${{ matrix.compiler.id }}' = 'clang-17' ]; then
+            # clang-17's llvm-symbolizer-17 crashes ("LLVM ERROR: Sections with relocations should have an
+            # address of 0") when symbolizing our binaries' TSAN reports, stalling the reporting thread 10+
+            # seconds each time (leading to, e.g., test-internal timeouts).  A sibling version's symbolizer
+            # handles the same binaries fine -- and nothing requires it to version-match the compiler.
+            for SYMB_VER in 16 18 15; do
+              if command -v llvm-symbolizer-$SYMB_VER > /dev/null; then
+                export TSAN_OPTIONS="$TSAN_OPTIONS external_symbolizer_path=$(command -v llvm-symbolizer-$SYMB_VER)"
+                echo "To avoid clang-17 symbolizer bugginess overrode symbolizer to [llvm-symbolizer-$SYMB_VER]."
+                break
+              fi
+            done
+          fi
           echo "TSAN_OPTIONS = [$TSAN_OPTIONS]."
         fi
         if [ "$SAN_SUPP" != '' ]; then
@@ -799,6 +821,10 @@ jobs:
         # key template groups (<=> ipc::session::Session compile-time config, possibly others) in separate
         # `.cpp`s, would benefit other environments, including user environments, and be less reliant on fairly
         # dynamic hardware conditions from GitHub/whomever.  Revisit that; ideally get rid of this as a result.
+        # Update: The below trick is likely insufficient (from numbers we've seen), so we split the handful of
+        # heaviest unit_test `.cpp`s per this to-do.  Combined with the below trick (but not without it, notably),
+        # 8/2026 GitHub build was OK.  The to-do remains: Ideally split more stuff, so that this can be removed
+        # (among other benefits).  transport_test is likely the guy to attack.
         # XXXrevisit soon, if there's time; or delete this line and leave TODO; whatever seems best at the end.
         if [ '${{ matrix.compiler.high-mem-build }}' != '' ] \
            && [ '${{ matrix.compiler.high-mem-build }}' != 'false' ] \
@@ -946,18 +972,8 @@ jobs:
       run: |
         # Build libraries and demos/tests with Conan.
         ${{ env.setup-run-env }}
-        #XXXtemp-debug!
-        # Sample RAM use in the background, so its output interleaves with the log as we go: post-mortem evidence
-        # regarding sporadic whole-VM kills (runner-shutdown message from GitHub) observed when several jumbo test
-        # TUs compile simultaneously in `-g` configs -- apparent OOM of the 16 GiB runner.
-        ( while true; do echo "[mem-sample] $(free -m | grep Mem:)"; sleep 15; done ) &
-        MEM_SAMPLER_PID=$!
-        #XXXtemp-debug-end
         # At least capnp compiler binary is built with our build-settings, so $BUILT_CMD_PREFIX is required.
         $BUILT_CMD_PREFIX conan build .
-        #XXXtemp-debug!
-        kill $MEM_SAMPLER_PID
-        #XXXtemp-debug-end
 
     - name: Install built targets with Makefile
       if: env.cell-selected == 'true'
@@ -1121,6 +1137,8 @@ jobs:
           && (!cancelled())
       run: |
         # Run unit tests.
+        # XXX Temporary; see test-subset input.
+        if [ '${{ github.event.inputs.test-subset }}' = 't-t-only' ]; then echo 'Skipped per test-subset.'; exit 0; fi
         cd ${{ env.install-dir }}/bin
         # Some newline issues with the possible additional args; so need to make a wrapper script
         # and then redirect, as desired, its output.
@@ -1252,6 +1270,8 @@ jobs:
       run: |
         # Prepare run script for [transport_test - Scripted mode] variations below.
         cat <<'EOF' > ${{ env.install-dir }}/bin/run_transport_test_sc.sh
+        # XXX Temporary; see test-subset input.
+        if [ '${{ github.event.inputs.test-subset }}' = 'u-t-only' ]; then echo 'Skipped per test-subset.'; exit 0; fi
         echo "Log level: [$1]."
         cd ${{ env.install-dir }}/bin/transport_test
         OUT_DIR_NAME=log_level_$1
@@ -1317,6 +1337,8 @@ jobs:
         # Prepare run script for [transport_test - Exercise mode] variations below.
         cat <<'EOF' > ${{ env.install-dir }}/bin/run_transport_test_ex.sh
         # Script created by pipeline during job.
+        # XXX Temporary; see test-subset input.
+        if [ '${{ github.event.inputs.test-subset }}' = 'u-t-only' ]; then echo 'Skipped per test-subset.'; exit 0; fi
         echo "Log level: [$1]."
         echo "Exercise sub-mode: [$2]."
         echo "Sub-mode snippet (none or '-shm-?'): [$3]."
@@ -1381,37 +1403,20 @@ jobs:
           && (!cancelled()) && (steps.transport_test_ex_shm_c.outcome == 'failure')
       run: /usr/bin/bash -e ${{ env.install-dir }}/bin/run_transport_test_ex.sh data shm_classic_log_level_data -shm-c
 
-    # Disabling this particular test run for the specific case of clang-17 in TSAN (thread sanitizer) config
-    # (in particular at least 2 other clangs+TSAN are exercised, so the TSAN coverage is still good).
-    # First the reason in detail: This run semi-reliably (50%+) fails at this point in the server binary:
-    #   2023-12-20 11:36:11.322479842 +0000 [info]: Tguy: ex_srv.hpp:send_req_b(1428): App_session [0x7b3800008180]:
-    #     Chan B[0]: Filling/send()ing payload (description = [reuse out-message + SHM-handle to modified (unless
-    #     SHM-jemalloc) existing STL data]; alt-payload? = [0]; reusing msg? = [1]; reusing SHM payload? = [1]).
-    #   LLVM ERROR: Sections with relocations should have an address of 0
-    #   PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
-    #   Stack dump:
-    #   0.  Program arguments: /usr/bin/llvm-symbolizer-17 --demangle --inlines --default-arch=x86_64
-    #   Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
-    #   ...
-    #   ==77990==WARNING: Can't read from symbolizer at fd 599
-    #   2023-12-20 11:36:31.592293322 +0000 [info]: Tguy: ex_srv.hpp:send_req_b(1547): App_session [0x7b3800008180]: Chan B[0]: Filling done.  Now to send.
-    # Sometimes the exact point is different, depending on timing; but in any case it is always the above
-    # TSAN/LLVM error, at which point the thread gets stuck for a long time (10+ seconds); but eventually gets
-    # unstuck; however transport_test happens to be testing a feature in a certain way so that a giant blocking
-    # operation in this thread delays certain processing, causes an internal timeout, and the test exits/fails.
-    # Sure, we could make some changes to the test for that to not happen, but that's beside the point: TSAN
-    # at run-time is trying to do something and fails terribly; I have no wish to try to work around that situation;
-    # literally it says "PLEASE submit a bug report [to clang devs]."
-    #
-    # TODO: Revisit; figure out how to not trigger this; re-enable.  For the record, I (ygoldfel) cannot reproduce
-    # in a local clang-17, albeit with libc++ (LLVM STL) instead of libstdc++ (GNU STL).  I've also tried to
-    # reduce optimization to -O1, as well as with and without LTO, and with and without -fno-omit-frame-pointer;
-    # same result.
+    # clang-17 + TSAN only:
+    #   In this config this run would semi-reliably (50%+) fail: TSAN reports during the run
+    #   invoke llvm-symbolizer-17 which crashes on our binaries ("LLVM ERROR: Sections with relocations should
+    #   have an address of 0" -- an LLVM bug; ticket filed), stalling the reporting thread 10+ seconds each
+    #   time; and a certain giant blocking operation in a particular thread then delays certain processing,
+    #   causing an internal timeout, and the test exits/fails.  Worked around by symbolizing with a sibling
+    #   llvm-symbolizer version in that config; see `setup-tests-env`.  If this becomes insufficient, or some other
+    #   such problem arises, it is defensible to disable this for the offending compiler, as there are as of this
+    #   writing a few TSAN configs that would remain (different compiler version(s)).
     - name: Run integration test [transport_test - Exercise mode - SHM-jemalloc sub-mode]
       id: transport_test_ex_shm_j
       if: |
         (env.cell-selected == 'true')
-          && (!cancelled()) && ((matrix.compiler.id != 'clang-17') || (matrix.build-test-cfg.sanitizer-name != 'tsan'))
+          && (!cancelled())
       run: /usr/bin/bash -e ${{ env.install-dir }}/bin/run_transport_test_ex.sh info shm_jemalloc -shm-j
 
     - name: Re-run with increased logging, on failure only