ci: configure Renovate, skip Rekor tlog, fix prometheus build

.github/workflows/reusable-build-image.yml

@@ -185,9 +185,15 @@ jobs:
       - name: Sign container image
         if: github.event_name != 'pull_request' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
         run: |
+          # Skip the public Rekor transparency log upload. The signature is
+          # stored in the OCI registry alongside the image, where it can be
+          # verified manually with `cosign verify --insecure-ignore-tlog=true`.
+          # Rekor (rekor.sigstore.dev) is a shared, intermittently-flaky
+          # service whose outages were failing this step; it isn't needed for
+          # the key-based signer.
           IMAGE_FULL="${{ steps.registry_case.outputs.lowercase }}/${{ steps.image_case.outputs.lowercase }}"
           for tag in ${{ steps.metadata.outputs.tags }}; do
-            cosign sign -y --key env://COSIGN_PRIVATE_KEY $IMAGE_FULL:$tag
+            cosign sign -y --tlog-upload=false --key env://COSIGN_PRIVATE_KEY $IMAGE_FULL:$tag
           done
         env:
           TAGS: ${{ steps.push.outputs.digest }}
@@ -201,7 +207,14 @@ jobs:
           COSIGN_PRIVATE_KEY: ${{ secrets.SIGNING_SECRET }}
           COSIGN_TIMEOUT: 5m0s
         run: |
+          # Skip the public Rekor transparency log upload. The attestation is
+          # stored in the OCI registry alongside the image, where it can be
+          # verified manually with `cosign verify-attestation
+          # --insecure-ignore-tlog=true`. Rekor (rekor.sigstore.dev) is a
+          # shared, intermittently-flaky service whose outages were failing
+          # this step; it isn't needed for the key-based signer.
           cosign attest -y \
+            --tlog-upload=false \
             --predicate ./sbom.json \
             --type spdxjson \
             --key env://COSIGN_PRIVATE_KEY \

AGENTS.md

@@ -0,0 +1,23 @@
+# AGENTS.md — repository knowledge
+
+## CI/CD overview
+- Three GitHub Actions workflows in `.github/workflows/`:
+  - `build-images.yml` — top-level; triggers on PR/push/schedule (Sun 10:05 UTC) and fans out to per-image reusable workflow jobs: base, nginx, k3s, prometheus. (nas is commented out.)
+  - `reusable-build-image.yml` — the actual build/push/sign/attest logic. Buildah builds each target from `Containerfile`, pushes to GHCR (`ghcr.io/<owner>/<repo>/<image>`), generates an SBOM (anchore), scans it (anchore scan-action), uploads SARIF, signs with cosign (key-based, `SIGNING_SECRET`), then attaches an SBOM attestation with cosign.
+  - `build-artifact.yml` — manual/PR-triggered; turns a built bootc image into a disk artifact (ami/qcow2/vmdk/...) via bootc-image-builder.
+- There is also a `.gitlab-ci.yml`, but the active pipelines are GitHub Actions.
+- Containerfile build targets: `base`, `observability`, `prometheus`, `loki`, `nginx`, `k3s`, `nas`. See `Containerfile` for the stage graph.
+
+## Dependency management
+- **Renovate** is the dependency manager (see `renovate.json`). Configured to manage `github-actions` and `dockerfile` dependencies, grouped, with a Monday schedule and a dependency dashboard. Major docker base-image bumps require dashboard approval.
+- Dependabot was removed (its config had an empty `package-ecosystem` and would have conflicted with Renovate).
+- GitHub Actions are pinned by digest; Renovate keeps both digest and version tag up to date.
+
+## Known issue: Rekor (sigstore transparency log) flakiness
+- The public Rekor instance (`rekor.sigstore.dev`) is a shared service with a 99.5% SLO and is intermittently unavailable. Since cosign 2.0, `cosign sign` and `cosign attest` upload to it by default even for key-based signing, so a Rekor outage fails the pipeline with `Post "https://rekor.sigstore.dev/api/v1/log/entries": ... giving up after 4 attempt(s)`.
+- This is widely reported (sigstore/cosign#3602, GitHub community discussions#144833) and transient.
+- Fix applied in `reusable-build-image.yml`: both the `Sign container image` and `Add SBOM Attestation` steps always run with `--tlog-upload=false`. The signature/attestation is stored in the OCI registry alongside the image (where it can be verified manually); the public Rekor transparency-log entry is skipped entirely. This trades off the public audit record for a reliable build.
+- Verify images signed this way with `cosign verify --key cosign.pub --insecure-ignore-tlog=true ...` (and `cosign verify-attestation --insecure-ignore-tlog=true ...` for attestations).
+
+## Known issue: prometheus build target fails dnf install of loki/grafana
+- The `prometheus` Containerfile stage pulls the grafana yum repo and runs `dnf install loki grafana -y`, which failed with a Curl SSL CA cert error reaching `https://rpm.grafana.com/repodata/repomd.xml`. The grafana repo config points `sslcacert` at `/etc/pki/tls/certs/ca-bundle.crt`, which must exist/populated before dnf runs.

Containerfile

@@ -42,6 +42,15 @@ RUN tar -xvf prometheus-3.4.0.linux-amd64.tar.gz && \
     COPY observability/prometheus/prometheus.service /etc/systemd/system
     RUN systemctl enable prometheus
 # Install Loki
+# Refresh base packages and (re)extract the TLS CA bundle before adding the
+# grafana repo. dnf's librepo fails with curl error 77 ("error adding trust
+# anchors from file: /etc/pki/tls/certs/ca-bundle.crt") when that bundle is
+# missing/empty in the bootc build container. Updating ca-certificates runs its
+# %post (update-ca-trust), and we call update-ca-trust explicitly too so the
+# bundle is repopulated even when ca-certificates is already current. This must
+# run before the grafana repo is configured, otherwise dnf tries to refresh
+# grafana's repo metadata against the broken bundle and fails the same way.
+RUN dnf update -y && update-ca-trust
 RUN curl -LO https://rpm.grafana.com/gpg.key && \
     rpm --import gpg.key && \
     echo -e '[grafana]\nname=grafana\nbaseurl=https://rpm.grafana.com\nrepo_gpgcheck=0\nenabled=1\ngpgcheck=1\ngpgkey=https://rpm.grafana.com/gpg.key\nsslverify=1\nsslcacert=/etc/pki/tls/certs/ca-bundle.crt' | sudo tee /etc/yum.repos.d/grafana.repo

renovate.json

@@ -1,3 +1,43 @@
 {
-  "$schema": "https://docs.renovatebot.com/renovate-schema.json"
+  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+  "extends": [
+    "config:recommended",
+    ":dependencyDashboard",
+    "helpers:pinGitHubActionDigests"
+  ],
+  "schedule": [
+    "before 6am on Monday"
+  ],
+  "timezone": "America/Chicago",
+  "labels": [
+    "dependencies"
+  ],
+  "github-actions": {
+    "enabled": true,
+    "commitMessageAction": "Update",
+    "groupName": "github-actions"
+  },
+  "dockerfile": {
+    "enabled": true
+  },
+  "docker-compose": {
+    "enabled": true
+  },
+  "packageRules": [
+    {
+      "matchManagers": [
+        "github-actions"
+      ],
+      "groupName": "GitHub Actions"
+    },
+    {
+      "matchDatasources": [
+        "docker"
+      ],
+      "matchUpdateTypes": [
+        "major"
+      ],
+      "dependencyDashboardApproval": true
+    }
+  ]
 }