Skip to content

ci: decouple Trivy CVE scan from the release path #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chrisleekr merged 3 commits into from
Sep 4, 2026
Merged

ci: decouple Trivy CVE scan from the release path #294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fe6702f
ci: decouple Trivy CVE scan from the release path
chrisleekr Sep 4, 2026
e86d39e
fix: address review - restore attestation-verify block and correct it…
chrisleekr Sep 4, 2026
7815f31
style(ci): drop diff-narration comments from the workflow files
chrisleekr Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 3 additions & 78 deletions .github/workflows/docker-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,6 @@
# - GHA cache is scoped per (arch, variant) so the four legs don't trample
# each other.
#
# Trivy scans each (arch, variant) manifest tag separately with the existing
# time-boxed .trivyignore.yaml allowlist.
#
# Defense-in-depth: every workflow input that flows into a `run:` block is
# passed through `env:` first to prevent script-injection at YAML-render time.
name: Docker Build & Push
Expand Down Expand Up @@ -65,11 +62,9 @@ on:

permissions:
contents: read
security-events: write
# Read-only attestations grant lets the `scan` job verify SLSA / SBOM
# attestations via `gh attestation verify` before scanning. The `merge`
# job overrides locally to add the `id-token: write` + `attestations: write`
# scopes required to publish Sigstore-signed attestations.
# The `merge` job overrides locally to add the `id-token: write` +
# `attestations: write` scopes required to publish Sigstore-signed
# attestations.
# Note: As SBOM file size is over 16MB, temporary disable SBOM attestations.
# attestations: read

Expand Down Expand Up @@ -333,73 +328,3 @@ jobs:
# subject-digest: ${{ steps.inspect.outputs.digest }}
# sbom-path: "sbom-${{ matrix.variant }}.cyclonedx.json"
# push-to-registry: true

scan:
name: Trivy scan (${{ matrix.variant }} / ${{ matrix.arch }})
needs: merge
runs-on: ubuntu-24.04
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
arch: [amd64, arm64]
variant: [orchestrator, daemon]
steps:
- name: Checkout (for .trivyignore.yaml)
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

- name: Login to Docker Hub
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

- name: Compute image tag
id: imgtag
env:
TAG_NAME: ${{ inputs.tag-name }}
VARIANT: ${{ matrix.variant }}
run: |
PACKAGE_VERSION="${TAG_NAME#v}"
echo "tag=${PACKAGE_VERSION}-${VARIANT}" >> "$GITHUB_OUTPUT"

# Regression gate: fail the workflow if the published image is missing
# its Sigstore-signed SLSA provenance + SBOM attestations. Catches
# silent drops from future refactors of the build / merge jobs.
# Note: As SBOM file size is over 16MB, temporary disable SBOM attestations.
# - name: Verify image attestations
# env:
# GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# IMAGE_REF: oci://${{ env.IMAGE_NAME }}:${{ steps.imgtag.outputs.tag }}
# REPO: ${{ github.repository }}
# run: |
# # Two separate calls: each fails if the requested predicate type is
# # absent, so this catches a regression that drops EITHER attestation.
# gh attestation verify "$IMAGE_REF" --repo "$REPO" \
# --predicate-type https://slsa.dev/provenance/v1
# gh attestation verify "$IMAGE_REF" --repo "$REPO" \
# --predicate-type https://cyclonedx.org/bom

- name: Scan image with Trivy
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
env:
TRIVY_PLATFORM: "linux/${{ matrix.arch }}"
with:
image-ref: "${{ env.IMAGE_NAME }}:${{ steps.imgtag.outputs.tag }}"
format: sarif
output: "trivy-results-${{ matrix.variant }}-${{ matrix.arch }}.sarif"
severity: CRITICAL,HIGH
limit-severities-for-sarif: true
ignore-unfixed: true
# Daemon image ships a full toolchain (nvm, Docker CLI, apt, etc.) so
# CVEs are a permanent fact of life: scan it for visibility in the
# GitHub Security tab but don't block CI. Orchestrator still gates.
exit-code: ${{ matrix.variant == 'daemon' && '0' || '1' }}
trivyignores: .trivyignore.yaml

- name: Upload Trivy SARIF to GitHub Security
uses: github/codeql-action/upload-sarif@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
if: always()
with:
sarif_file: "trivy-results-${{ matrix.variant }}-${{ matrix.arch }}.sarif"
category: "trivy-container-scan-${{ matrix.variant }}-${{ matrix.arch }}"
9 changes: 4 additions & 5 deletions .github/workflows/release-please.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,12 @@ jobs:
name: Docker
needs: release-please
if: needs.release-please.outputs.release_created == 'true'
# Ceiling for the reusable docker-build.yml: it needs security-events:write
# (Trivy SARIF upload) and id-token:write (Sigstore/attestations). Matches
# the grant the retired release.yml gave this path; the release-please job's
# PR/issues write scopes are deliberately NOT granted here.
# Ceiling for the reusable docker-build.yml: it needs contents:read to
# check out the tag and id-token:write for Sigstore/attestations. The
# release-please job's PR/issues write scopes are deliberately NOT granted
# here.
permissions:
contents: read
security-events: write
id-token: write
uses: ./.github/workflows/docker-build.yml
with:
Expand Down
120 changes: 120 additions & 0 deletions .github/workflows/trivy-scan.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# Container CVE scanning for the published Docker Hub images.
#
# Separate from docker-build.yml so a CVE verdict never gates publishing or the
# release automation downstream of it: these images are already on Docker Hub
# by the time anything here runs.
#
# Scheduled rather than release-triggered because Trivy verdicts move when
# advisories are published, not when this repo changes. No `release: published`
# trigger either: release-please creates the release before the manifest lists
# are pushed, so such a run races the build and fails on `manifest unknown`.
# Use workflow_dispatch to scan a specific release once its build is green.
name: Trivy Scan

on:
schedule:
# 14:00 UTC daily, after the Trivy DB's usual refresh window.
- cron: "0 14 * * *"
workflow_dispatch:
inputs:
tag:
description: "Image version to scan (e.g. 1.17.0). Blank scans latest."
required: false
type: string

# Only the SARIF upload needs a write scope.
permissions:
contents: read
security-events: write

concurrency:
group: trivy-scan-${{ github.ref }}-${{ inputs.tag || 'latest' }}
cancel-in-progress: true

env:
IMAGE_NAME: chrisleekr/github-app

jobs:
scan:
name: Trivy scan (${{ matrix.variant }} / ${{ matrix.arch }})
runs-on: ubuntu-24.04
timeout-minutes: 20
strategy:
# Every (variant, arch) leg reports independently: one red orchestrator
# arch must not hide the other three legs' findings.
fail-fast: false
matrix:
arch: [amd64, arm64]
variant: [orchestrator, daemon]
steps:
- name: Checkout (for .trivyignore.yaml)
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

# Authenticated pulls only: anonymous Docker Hub rate limits are shared
# per runner egress IP and would flake this matrix.
- name: Login to Docker Hub
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

- name: Compute image tag
id: imgtag
env:
INPUT_TAG: ${{ inputs.tag }}
VARIANT: ${{ matrix.variant }}
run: |
set -euo pipefail
# Accept both `1.17.0` and `v1.17.0`; the published tags are unprefixed.
BASE="${INPUT_TAG#v}"
if [ -z "$BASE" ]; then
BASE="latest"
elif ! [[ "$BASE" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z][a-zA-Z0-9.-]*)?$ ]]; then
echo "ERROR: tag input does not match the expected version format." >&2
exit 1
fi
echo "tag=${BASE}-${VARIANT}" >> "$GITHUB_OUTPUT"

# Regression gate: fail the workflow if the published image is missing
# its Sigstore-signed SLSA provenance + SBOM attestations. Re-enabling
# also needs `attestations: read` in the permissions block above.
# Note: As SBOM file size is over 16MB, temporary disable SBOM attestations.
# - name: Verify image attestations
# env:
# GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# IMAGE_REF: oci://${{ env.IMAGE_NAME }}:${{ steps.imgtag.outputs.tag }}
# REPO: ${{ github.repository }}
# run: |
# # Two separate calls: each fails if the requested predicate type is
# # absent, so this catches a regression that drops EITHER attestation.
# gh attestation verify "$IMAGE_REF" --repo "$REPO" \
# --predicate-type https://slsa.dev/provenance/v1
# gh attestation verify "$IMAGE_REF" --repo "$REPO" \
# --predicate-type https://cyclonedx.org/bom

- name: Scan image with Trivy
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
env:
TRIVY_PLATFORM: "linux/${{ matrix.arch }}"
with:
image-ref: "${{ env.IMAGE_NAME }}:${{ steps.imgtag.outputs.tag }}"
format: sarif
output: "trivy-results-${{ matrix.variant }}-${{ matrix.arch }}.sarif"
severity: CRITICAL,HIGH
limit-severities-for-sarif: true
ignore-unfixed: true
# Daemon image ships a full toolchain (nvm, Docker CLI, apt, etc.) so
# CVEs are a permanent fact of life: scan it for visibility in the
# GitHub Security tab but don't fail the run. Orchestrator is lean
# enough that a HIGH there is worth a red run and a fix.
exit-code: ${{ matrix.variant == 'daemon' && '0' || '1' }}
trivyignores: .trivyignore.yaml

# `if: always()` so the SARIF still lands when the orchestrator leg exits
# non-zero. That upload is the whole point of a red run.
- name: Upload Trivy SARIF to GitHub Security
uses: github/codeql-action/upload-sarif@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
if: always()
with:
sarif_file: "trivy-results-${{ matrix.variant }}-${{ matrix.arch }}.sarif"
category: "trivy-container-scan-${{ matrix.variant }}-${{ matrix.arch }}"
8 changes: 5 additions & 3 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,17 +109,19 @@ GitHub-side auth defaults to the App installation token minted on demand from `G

## CI/CD Pipeline

Five pipeline files; each owns one responsibility.
Six pipeline files; each owns one responsibility.

| Workflow | Trigger | Owns |
| -------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `.github/workflows/ci.yml` | `pull_request` + `push: main` + `workflow_call` | Quality gates only: typecheck, lint, format, audit:ci, test, build |
| `.github/workflows/secrets-scan.yml` | `push: branches-ignore: [gh-pages]` + `workflow_dispatch` | Standalone gitleaks secret scan, decoupled from ci.yml so every push (incl. chore/docs/ci/test branches) is gated |
| `.github/workflows/release-please.yml` | `push: [main, beta]` | release-please maintains a Release PR per branch; merging it cuts the release (main = stable `v<x.y.z>` + `latest` image; beta = `v<x.y.z>-beta` prerelease, no `latest`) then calls `docker-build.yml`. On a stable `main` release it also dispatches `github-app-released` to `chrisleekr/helm-charts` (`notify-helm-charts` job, after `docker`) so that repo opens the chart-sync PR |
| `.github/workflows/docker-build.yml` | `workflow_call` + `workflow_dispatch` | Reusable image builder: matrix split-and-merge (amd64 on `ubuntu-24.04` + arm64 on `ubuntu-24.04-arm`), Trivy scan. SLSA v1 provenance + SBOM attestations and the `gh attestation verify` gate are **currently commented out** (SBOM exceeded 16MB); the code is retained in place for re-enablement |
| `.github/workflows/docker-build.yml` | `workflow_call` + `workflow_dispatch` | Reusable image builder: matrix split-and-merge (amd64 on `ubuntu-24.04` + arm64 on `ubuntu-24.04-arm`). **No CVE scanning** (see `trivy-scan.yml`). SLSA v1 provenance + SBOM attestations and the `gh attestation verify` gate are **currently commented out** (SBOM exceeded 16MB); the code is retained in place for re-enablement |
| `.github/workflows/trivy-scan.yml` | `schedule` (daily 14:00 UTC) + `workflow_dispatch` | Trivy CVE scan of the **published** Docker Hub images, SARIF uploaded to the GitHub Security tab. Orchestrator legs gate on CRITICAL/HIGH (`exit-code: 1`); daemon legs report only (`exit-code: 0`, full toolchain image) |
| `.gitlab-ci.yml` | every branch (gates) + `main` (images) | GitLab CI: the same quality gates on **every** branch, then on `main` publishes `latest-orchestrator` (multi-arch) and `latest-daemon` (amd64-only) to the GitLab container registry |

- **Two image paths.** release-please → Docker Hub `<version>-orchestrator` / `<version>-daemon` plus the mutable `latest-orchestrator` / `latest-daemon` aliases (prod releases only), versioned and Trivy-scanned. GitLab `main` → GitLab registry `latest-orchestrator` / `latest-daemon`, mutable and unscanned. Neither path feeds the other. **Neither currently produces attestations**: the `provenance`/`sbom` inputs and the `gh attestation verify` gate in `docker-build.yml` are commented out ("SBOM file size is over 16MB, temporary disable"), and the GitLab builds pass `--provenance false`.
- **CVE scanning is decoupled from the release path.** `trivy-scan.yml` scans the images Docker Hub already serves, on its own daily schedule, and is never a `needs:` of any release job. The old arrangement put `scan` inside `docker-build.yml` behind `needs: merge`: it ran after the manifest lists were pushed, so `exit-code: 1` could not stop a vulnerable image from shipping, and its only real effect was to fail the reusable workflow's conclusion and skip `notify-helm-charts`, withholding the chart-sync PR for an image already live (v1.17.0). Do NOT reintroduce a CVE verdict as a dependency of a publish or dispatch job. If a genuine pre-publication gate is ever wanted, it has to run **before** push, not after.
- **Two image paths.** release-please → Docker Hub `<version>-orchestrator` / `<version>-daemon` plus the mutable `latest-orchestrator` / `latest-daemon` aliases (prod releases only), versioned and Trivy-scanned out of band. GitLab `main` → GitLab registry `latest-orchestrator` / `latest-daemon`, mutable and unscanned. Neither path feeds the other. **Neither currently produces attestations**: the `provenance`/`sbom` inputs and the `gh attestation verify` gate in `docker-build.yml` are commented out ("SBOM file size is over 16MB, temporary disable"), and the GitLab builds pass `--provenance false`.
- **Bun version is single-sourced** via `.tool-versions` (`bun 1.3.14`). All GitHub workflows use `oven-sh/setup-bun` with `bun-version-file: .tool-versions`; `.gitlab-ci.yml` pins `image: oven/bun:<ver>-alpine` by hand, so a `.tool-versions` bump must be applied there too (`check:docs-versions` does not cover it).
- **`audit:ci` (`scripts/audit-ci.ts`)** wraps `bun audit --json` to gate on severity: blocks on high+critical, warns on moderate+low, with an inline `IGNORED` GHSA allowlist (each entry must carry an `expires` date). Required because `bun audit` exits 1 on **any** finding regardless of `--audit-level`.
- **Releases run on `release-please`** (Google), not semantic-release. Two source-controlled config + manifest pairs, selected by branch at runtime: `release-please-config.json` + `.release-please-manifest.json` on `main` (stable), and `release-please-config.beta.json` + `.release-please-manifest.beta.json` on `beta` (`versioning-strategy: prerelease` + `prerelease: true` + `prerelease-type: beta`). Both pairs live on both branches so the branches never share release-please state. Only `feat`/`fix`/`!`(breaking) commits bump the version; `refactor`/`perf`/`revert` alone do not cut a release (unlike the retired semantic-release rules).
Expand Down
Loading
Loading