Skip to content

Commit f2a89e2

Browse files
bomly-guyclaude
bomly-guy
and
claude
authored
ci(smoke): parallel per-ecosystem goldens regeneration + build-tool subprocess timeouts (#286)
* ci(smoke): parallelize goldens regeneration into per-ecosystem slices Update Smoke Goldens previously ran the entire smoke suite in one job, so a single stalled scan (mvn against a cold ~/.m2 twice this week) killed the whole run and no goldens PR was opened. Actions minutes are no longer a concern now that the repo is public, so split regeneration into 21 matrix slices — one per ecosystem, with the slow network-heavy reachability cases isolated — each uploading its changed goldens as an artifact. A final open-pr job merges all artifacts and opens the PR even when some slices failed, listing the failed slices in the PR body so partial regenerations are visible instead of all-or-nothing. Slice patterns were verified against a stub test tree so every golden file is produced by exactly one slice. Both workflows now invoke go test directly instead of make smoke ARGS=... because the patterns use $ anchors that make would expand. Also close nightly Smoke coverage gaps: scan-bun, scan-recursive-monorepo, scan-java-maven-reachability, and TestPluginWorkflows/TestLiteVersion were not matched by any slice and never ran. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(detectors): bound build-tool invocations with a 10-minute timeout mvn/gradle/sbt resolve dependency graphs over the network and the pipeline context carries no deadline, so a stalled remote hung scans indefinitely — observed three times on 2026-07-17 with mvn against Maven Central in CI. Derive a bounded context (detectors.BuildToolContext, 10m) for every build-tool invocation, following the existing javaReadyTimeout pattern, and surface DeadlineExceeded in the wrapped error. Failures flow through the existing degrade-to-fallback paths. Also document the smoke CI slice-matrix contract in CLAUDE.md/AGENTS.md: new smoke tests must be registered in both workflow matrices, goldens owned by exactly one slice, $-anchored patterns, go test invoked directly because make expands $ in ARGS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci(smoke): cache ~/.m2 and Gradle home across smoke runs The three mvn stalls on 2026-07-17 all happened during cold-cache dependency downloads from Maven Central. Java derives user.home from passwd rather than $HOME, so every bomly invocation in a job already shares the runner-level ~/.m2 despite the harness's per-test HOME override — a workflow-level cache therefore warms all mvn invocations in both Smoke and Update Smoke Goldens. Rolling keys (unique per run, prefix restore) are used because smoke scans clone pinned repos at runtime, leaving no lockfiles in the workspace to hash. Gradle's distribution and module caches get the same treatment in the java slices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci(smoke): route all Maven repositories through Central in CI Root cause of the four scan-maven stalls on 2026-07-17: transitive POMs of the pinned example-java-maven demo (Spring 3.1.1, Keycloak 1.8.1, old slf4j/plexus/neo4j parents) declare long-dead repositories — snapshots.repository.codehaus.org, repository.jboss.org/maven2, static.springframework.org, oss.sonatype.org snapshots, and friends. Maven consults them during resolution; on the runner network path one accepts connections and never responds, and Maven's default read timeout is effectively unbounded. Write a settings.xml with mirrorOf '*' pointing at Central in the maven slices of both workflows. Every artifact the demos need lives on Central (verified: a cold resolve under the mirror succeeds with a byte-identical dependency tree), so dead-repo lookups become fast 404s instead of hangs. Maven resolves user.home from passwd, so the settings apply to bomly-spawned mvn despite the harness's per-test HOME override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci(smoke): full slice isolation in both matrices (CodeRabbit review) Adopt the review feedback and extend it: smoke.yml now mirrors the goldens matrix exactly — 21 slices with dedicated reachability slices and $-anchored patterns, including a TestScan$ top-level anchor so TestScanRecursive* no longer piggybacks on every slice that mentions TestScan (they run once, in the plugin slice). The goldens go slice names its TestAuditDiffAndExplain/TestLiteDiff/TestLiteExplain subtests explicitly so future non-go variants cannot silently land there, and golden staging uses --diff-filter=ACMR so a deleted file cannot break the cp loop. Verified by simulation against a stub test tree with go's real -run matcher: every golden owned by exactly one goldens slice, every smoke test runs in exactly one smoke slice, zero overlaps in either matrix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(detectors): widen java readiness probe; smoke: reject degraded goldens Goldens PR #287 exposed a silent-degradation path: the 5s java readiness probe flaked on a loaded CI runner, Ready reported "not ready: java readiness check timed out after 5s", the scan fell back from maven-detector to syft-detector, and -update happily wrote the degraded output as the new golden. Two fixes: - javaReadyTimeout 5s → 30s. The probe only runs when a java executable exists on PATH (LookPath gates it), so the wider bound costs nothing when java is absent and stops slow first JVM starts from flipping readiness. This is also the known local flake in the maven/gradle/sbt Ready tests under parallel make test. - The smoke harness now refuses to compare or write output containing a "timed out" reason anywhere in the JSON (normalizeJSON guard). Deterministic fallbacks ("executable not found", "exit status 1") remain legitimate golden content; timeout-induced degradation fails the case loudly instead of corrupting goldens — this also covers the new 10m build-tool timeout if it ever fires during regeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(smoke): pin every scan case to its expected detector via --detectors The engine skips fallback detectors outside the --detectors selector set (pipeline_resolve.go fallbackSelected), so pinning turns silent degradation into a hard failure: with --detectors maven and a broken java runtime the scan exits 3 with 'detector maven-detector: not ready', where the unpinned scan silently emits syft-detector output. - benchmark.Target gains a 'detectors' field, appended to SmokeArgs as --detectors; all 22 targets in scan_targets.json pinned to the detector their golden encodes. - The 13 explicit TestScan cases (reachability, workspaces, multimodule, sbom, monorepo) pinned inline. Cases whose golden legitimately encodes a fallback chain pin both links (sbt-native,sbt / swiftpm-native,swiftpm). - scan-maven-multimodule's golden regenerated: PR #285 had baked in a degraded syft-shaped golden (mvn exit 1 on the dead-repo network path pre-mirror); it is maven-native again, matching its pre-drift shape, and nightly smoke will now verify that instead of quietly agreeing with degraded output. - Pins verified output-neutral against 10 existing goldens locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci(smoke): pin Maven 3.9.16 — apt's 3.8.7 cannot resolve reactor siblings The pin-validation run surfaced the real cause of the multimodule degradation: ubuntu-24.04's apt maven is 3.8.7, and its dependency:tree fails on in-reactor sibling dependencies ('Could not find artifact dev.bomly.example:core:jar:1.0.0'), silently degrading the scan to the syft fallback — reproduced exactly in an ubuntu:24.04 container, and confirmed fixed with Maven 3.9.16 (resolves reactor siblings cold, mirror active, exit 0). Replace apt maven in both workflows with a checksum-verified 3.9.16 tarball from archive.apache.org. With the --detectors pin this failure mode was a loud slice failure instead of a corrupted golden — exactly the layering working as intended. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci(smoke): build CLI binaries once and share across the slice matrix Every matrix job compiled the full bomly binary in TestMain (and the lite binary on demand) — 21 redundant compiles per run. Both workflows now have a build job that runs make build once and uploads bin/ as an artifact; slices download it and point the harness at the prebuilt binaries via BOMLY_SMOKE_BINARY / BOMLY_SMOKE_LITE_BINARY. The harness falls back to building as before when the variables are unset, so local make smoke is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 74322bf commit f2a89e2

16 files changed

Lines changed: 788 additions & 110 deletions

File tree

‎.github/workflows/smoke.yml‎

Lines changed: 147 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -16,54 +16,105 @@ concurrency:
1616
cancel-in-progress: true
1717

1818
jobs:
19+
# Build the CLI binaries once and share them with every slice via an
20+
# artifact — 21 matrix jobs otherwise each compile the full binary (and the
21+
# lite binary) inside TestMain.
22+
build:
23+
name: Build CLI binaries
24+
runs-on: ubuntu-latest
25+
timeout-minutes: 15
26+
steps:
27+
- name: Check out repository
28+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
29+
30+
- name: Set up Go
31+
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
32+
with:
33+
go-version-file: go.mod
34+
cache: true
35+
cache-dependency-path: |
36+
go.sum
37+
38+
- name: Build bomly and bomly-lite
39+
run: make build
40+
41+
- name: Upload binaries
42+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
43+
with:
44+
name: bomly-binaries
45+
path: bin/
46+
if-no-files-found: error
47+
retention-days: 1
48+
1949
smoke:
2050
name: Smoke (${{ matrix.slice.name }})
51+
needs: build
2152
runs-on: ubuntu-latest
2253
timeout-minutes: 45
2354
strategy:
2455
fail-fast: false
2556
matrix:
57+
# Mirrors the slice layout of update-smoke-goldens.yml: one slice per
58+
# ecosystem, slow network-heavy reachability cases isolated in their
59+
# own slices, and `$`-anchored patterns (including the `TestScan$`
60+
# top level) so every test runs in exactly one slice. Keep the two
61+
# matrices in sync when adding smoke tests.
2662
slice:
2763
- name: go
28-
run: 'TestScan/scan-go|TestDiff/diff-go|TestExplain/explain-go|TestAuditScan/(scan-go-enrich|scan-go-audit|scan-go-audit-high)|TestAuditDiffAndExplain/(diff-go-audit|explain-go-enrich)|TestLiteScan/lite-scan-go|TestLiteDiff/lite-diff-go|TestLiteExplain/lite-explain-go'
64+
run: 'TestScan$/scan-go$|TestDiff/diff-go|TestExplain/explain-go|TestAuditScan/(scan-go-enrich|scan-go-audit)|TestAuditDiffAndExplain/(diff-go-audit|explain-go-enrich)|TestLiteScan/lite-scan-go|TestLiteDiff/lite-diff-go|TestLiteExplain/lite-explain-go'
65+
- name: go-reachability
66+
run: 'TestScan$/scan-go-reachability'
2967
- name: node
30-
run: 'TestScan/(scan-npm|scan-yarn|scan-pnpm|scan-github-actions)|TestDiff/diff-npm|TestAuditScan/scan-npm-audit'
68+
run: 'TestScan$/(scan-npm$|scan-npm-scope-runtime|scan-npm-workspaces|scan-yarn|scan-pnpm|scan-bun$|scan-github-actions)|TestDiff/diff-npm|TestAuditScan/scan-npm-audit'
69+
node: true
70+
- name: node-reachability
71+
run: 'TestScan$/scan-npm-reachability'
3172
node: true
3273
- name: java
33-
run: 'TestScan/(scan-maven|scan-gradle)'
74+
run: 'TestScan$/(scan-maven|scan-gradle|scan-recursive-monorepo)'
75+
java: true
76+
maven: true
77+
gradle: true
78+
- name: java-reachability
79+
run: 'TestScan$/scan-java-maven-reachability'
3480
java: true
3581
maven: true
3682
- name: python
37-
run: 'TestScan/(scan-python-pip|scan-python-pipenv|scan-python-poetry)'
83+
run: 'TestScan$/(scan-python-pip$|scan-python-pipenv|scan-python-poetry)'
84+
python: true
85+
- name: python-reachability
86+
run: 'TestScan$/scan-python-pip-reachability'
3887
python: true
3988
- name: uv
40-
run: 'TestScan/scan-python-uv'
89+
run: 'TestScan$/scan-python-uv'
4190
python: true
4291
uv: true
4392
- name: php
44-
run: 'TestScan/scan-composer'
93+
run: 'TestScan$/scan-composer'
4594
- name: ruby
46-
run: 'TestScan/scan-bundler'
95+
run: 'TestScan$/scan-bundler'
4796
- name: sbom
48-
run: 'TestScan/(scan-sbom-spdx|scan-sbom-cyclonedx)|TestDiff/diff-sbom|TestLiteScan/(lite-scan-sbom-spdx|lite-scan-sbom-cyclonedx)'
97+
run: 'TestScan$/scan-sbom|TestDiff/diff-sbom|TestLiteScan/lite-scan-sbom'
4998
- name: dotnet
50-
run: 'TestScan/scan-nuget'
99+
run: 'TestScan$/scan-nuget'
51100
dotnet: true
52101
- name: rust
53-
run: 'TestScan/scan-cargo'
102+
run: 'TestScan$/scan-cargo'
54103
- name: dart
55-
run: 'TestScan/scan-pub'
104+
run: 'TestScan$/scan-pub'
56105
dart: true
57106
- name: swift
58-
run: 'TestScan/(scan-cocoapods|scan-swiftpm)'
107+
run: 'TestScan$/(scan-cocoapods|scan-swiftpm)'
59108
- name: elixir
60-
run: 'TestScan/scan-mix'
109+
run: 'TestScan$/scan-mix'
61110
- name: scala
62-
run: 'TestScan/scan-sbt'
111+
run: 'TestScan$/scan-sbt'
63112
- name: cpp
64-
run: 'TestScan/scan-cpp-conan'
113+
run: 'TestScan$/scan-cpp-conan'
65114
- name: container
66115
run: 'TestContainer(Scan|Diff|Explain)|TestContainerAuditScan/container-scan-alpine-audit'
116+
- name: plugin
117+
run: 'TestPluginWorkflows|TestLiteVersion|TestScanRecursive'
67118
steps:
68119
- name: Check out repository
69120
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -113,16 +164,93 @@ jobs:
113164
if: matrix.slice.python
114165
run: python -m pip install --require-hashes -r .github/requirements-ci-tools.txt
115166

116-
- name: Install smoke-test dependencies
167+
# apt's Maven on ubuntu-24.04 is 3.8.7, which cannot resolve in-reactor
168+
# sibling modules during dependency:tree ("Could not find artifact
169+
# dev.bomly.example:core") and silently degraded the multimodule scan
170+
# to the syft fallback. Maven 3.9+ resolves reactor siblings without a
171+
# prior install, so pin a modern release (checksum-verified).
172+
- name: Install Maven
117173
if: matrix.slice.maven
174+
env:
175+
MAVEN_VERSION: 3.9.16
176+
MAVEN_SHA512: 831a8591fe20c8243b1dbe7d71e3244f31d1665b0804b2e825e38cbbe5ce0cafb8338851f90780735568773e0a6cd07bbec107cda0b896b008b861075358b6f6
118177
run: |
119-
sudo apt-get update
120-
sudo apt-get install -y maven
178+
curl -fsSL "https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz" -o /tmp/maven.tar.gz
179+
echo "${MAVEN_SHA512} /tmp/maven.tar.gz" | sha512sum -c -
180+
sudo tar -xzf /tmp/maven.tar.gz -C /opt
181+
echo "/opt/apache-maven-${MAVEN_VERSION}/bin" >> "$GITHUB_PATH"
182+
183+
# Transitive POMs of the pinned demo repos declare long-dead
184+
# repositories (codehaus, repository.jboss.org, oss.sonatype.org
185+
# snapshots, ...). Some of them accept connections and never respond,
186+
# and Maven's default read timeout is effectively unbounded, which has
187+
# stalled runs. Every artifact the demos need lives on Central, so
188+
# route all repository access there: dead-repo lookups become fast
189+
# 404s instead of hangs. Maven resolves user.home from passwd, so this
190+
# settings.xml applies to every bomly-spawned mvn despite the smoke
191+
# harness's per-test HOME override.
192+
- name: Route all Maven repositories through Central
193+
if: matrix.slice.maven
194+
run: |
195+
mkdir -p ~/.m2
196+
cat > ~/.m2/settings.xml <<'XML'
197+
<settings>
198+
<mirrors>
199+
<mirror>
200+
<id>central-mirror-all</id>
201+
<name>Force all repositories through Maven Central</name>
202+
<url>https://repo.maven.apache.org/maven2</url>
203+
<mirrorOf>*</mirrorOf>
204+
</mirror>
205+
</mirrors>
206+
</settings>
207+
XML
208+
209+
# Smoke scans clone pinned repos at runtime, so there are no lockfiles
210+
# in the workspace to hash — use a rolling key (unique per run, prefix
211+
# restore) to keep the newest cache warm. Java derives user.home from
212+
# passwd, not $HOME, so every bomly invocation shares the runner-level
213+
# ~/.m2 despite the harness's per-test HOME override; a warm cache
214+
# removes the cold-start Maven Central downloads that have stalled runs.
215+
- name: Cache Maven repository
216+
if: matrix.slice.maven
217+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
218+
with:
219+
path: ~/.m2/repository
220+
key: maven-repo-${{ runner.os }}-${{ matrix.slice.name }}-${{ github.run_id }}
221+
restore-keys: |
222+
maven-repo-${{ runner.os }}-
223+
224+
- name: Cache Gradle caches
225+
if: matrix.slice.gradle
226+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
227+
with:
228+
path: |
229+
~/.gradle/caches
230+
~/.gradle/wrapper
231+
key: gradle-home-${{ runner.os }}-${{ matrix.slice.name }}-${{ github.run_id }}
232+
restore-keys: |
233+
gradle-home-${{ runner.os }}-
234+
235+
- name: Download prebuilt binaries
236+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
237+
with:
238+
name: bomly-binaries
239+
path: ${{ runner.temp }}/bomly-bin
240+
241+
- name: Make prebuilt binaries executable
242+
run: chmod +x "${RUNNER_TEMP}/bomly-bin/bomly" "${RUNNER_TEMP}/bomly-bin/bomly-lite"
121243

244+
# Mirrors `make smoke` but invokes go test directly: slice patterns may
245+
# use $ anchors, which make would expand (e.g. `$|` becomes empty) if
246+
# passed through ARGS.
122247
- name: Run smoke tests
123248
id: smoke_tests
124249
continue-on-error: true
125-
run: make smoke ARGS="-run '${{ matrix.slice.run }}'"
250+
env:
251+
BOMLY_SMOKE_BINARY: ${{ runner.temp }}/bomly-bin/bomly
252+
BOMLY_SMOKE_LITE_BINARY: ${{ runner.temp }}/bomly-bin/bomly-lite
253+
run: go test -tags smoke ./test/smoke/ -v -count=1 -timeout 15m -run '${{ matrix.slice.run }}'
126254

127255
- name: Fail job if smoke tests failed
128256
if: steps.smoke_tests.outcome == 'failure'

0 commit comments

Comments
 (0)