Something broke the Nix CI, you need a quick and dirty fix to unblock you as fast as possible, follow these guides.
You can adopt the nuclear approach and generate the AMI image regardless of the test outcome. To do that, remove the three conditions checking the nix-build-checks result in the if clause of the run-testinfra step in the .github/workflows/nix-build.yml file.
IE. remove the following conditions:
(needs.nix-build-checks-aarch64-linux.result == 'skipped' || needs.nix-build-checks-aarch64-linux.result == 'success')
(needs.nix-build-checks-aarch64-darwin.result == 'skipped' || needs.nix-build-checks-aarch64-darwin.result == 'success')
(needs.nix-build-checks-x86_64-linux.result == 'skipped' || needs.nix-build-checks-x86_64-linux.result == 'success')
Note: the merge queue check will block the PR from getting merged to develop.
A: Edit the BUILD_RUNNER_MAP dictionary in the github_matrix.py script and change the labels entry to match one of the still functional GitHub runners.
You can see the available runners and their associated labels on this page. Note: the blacksmith runners are considered as "self-hosted" by GitHub.
A: The evaluation can be quite costly memory-wise. nix-eval-jobs is spinning up multiple nix evaluation in parallel to speed things up. The tradeoff is an increased memory consumption compared to a single-process eval.
There are two ways to reduce memory consumption, both configurable from the github_matrix call in github/workflows/nix-eval.yml.
Reduce the number of parallel workers by overriding --nb-eval-jobs-workers. By default, github_matrix.py spins up one eval instance per CPU. For a blacksmith-32vcpu-ubuntu-2404 worker, that means 32 nix eval instances.
nix run --accept-flake-config .\#github-matrix -- --nb-eval-jobs-workers 16 checks legacyPackages
Reduce the per-worker memory limit by overriding --max-memory-size (in MiB). The default is 3072 (3 GiB). Lowering this value causes nix-eval-jobs to restart workers that exceed the threshold, trading evaluation speed for lower peak memory usage.
nix run --accept-flake-config .\#github-matrix -- --max-memory-size 2048 checks legacyPackages
Both flags can be combined for tighter control over total memory consumption.
The Nix artifacts are built from the Nix CI workflow defined in the .github/workflows/nix-build.yml file.
It's performed in 4 steps. Each step depending on the previous one.
Conceptually, this workflow evaluates the legacyPackages and checks flake outputs using nix-eval-jobs. This step produces a json map containing the jobs to build/check for each architecture. That json map is later consumed by the subsequent build and check steps.
Implementation-wise, most of the code lives in the /nix/packages/github-matrix/github_matrix.py python script. The script starts an instance of nix-eval-jobs and parses its output. Each parsed job is associated with a builder tag using the following order:
- KVM packages -> self-hosted runners
- Large packages on Linux -> 32vcpu ephemeral runners
- Darwin packages -> self-hosted runners
- Default -> ephemeral runners
KVM packages and large packages are determined respectively by the kvm and big-parallel Nix attributes.
GHA-wise, .github/workflows/nix-eval.yml is called by the nix-build.yml workflow. github_matrix.py is instantiated in the Generate Nix Matrix step through a nix run call. The resulting json map is stored in the workflow output and later used by the subsequent steps.
This step is in charge of building the various Nix packages. Build matrices are instantiated for each system architecture.
Implementation-wise, this step is less complex than the eval one. Most of the magic lies in the machine selection. The previous step attached an instance label to each job on which kind of GitHub runner it should be executed.
The actual build step is a simple nix build ${job} invocation. The result of this build is pushed to the nix-postgres-artifacts s3 cache. This step is instantiated 3 times, once for each of the supported architectures: aarch64 darwin, aarch64 linux and x86_64 linux.
This step uses again the JSON generated by the evaluation step to run various automated tests. Some of those require virtualization and are run on the self hosted runners able to perform KVM virtualization.
Implementation-wise, this step is very similar to the previous one. A matrix job instantiated once per target architecture. It's "just" running on a different set of Nix jobs. These tests do assume the various plugins have been built and are part of the Nix cache.
The last step builds AMI images using the artifacts generated during step 2 and uses the nix/packages/build-ami.nix script to generate a AMI image based on ubuntu noble. The generation of the image is done in two steps.