ci

Pre-Alpha. This page describes behavior that may change.

Ze's CI runs the same Make targets that you run locally, so that "works locally" and "passes CI" are the same statement. CI runs on GitHub Actions, and the workflows live in the repository under .github/workflows/. Validation previously ran on Codeberg's shared Woodpecker runners; that pipeline is gone.

This page covers what actually runs when you push a branch, what to expect, and what to do when it fails.

The workflows

Workflow	Trigger	What it runs
verify.yml	Every push and pull request	make ze-precommit-verify. The merge gate.
govulncheck.yml	Nightly, plus manual	make ze-dependency-vulnerability-check. Dependency vulnerability scan.
evidence-nightly.yml	Nightly, plus manual	Fuzz, kernel integration, BGP interop, and IPsec interop. Advisory.
qemu-nightly.yml	Nightly, plus manual	The Linux-only functional surface, inside a QEMU VM. Advisory.
perf-nightly.yml	Nightly, plus manual	Performance benchmarks.
codeql.yml	Scheduled and on push	CodeQL static analysis across the shipped build-tag combinations.

What the verify runner needs

Two of the merge gate's requirements are easy to miss when reading the workflow.

A full clone. actions/checkout is pinned to fetch-depth: 0. A shallow clone attributes every file to the graft commit, so the test-health collector cannot derive package age and answers unknown, which never matches the ok committed in the tree. The regeneration gate was then red on every run with a diagnostic that could not fix it.

A second loopback address. The runner adds fd00::2/128 to lo before make ze-precommit-verify. A BGP session needs a different address at each end, and a host is given exactly ::1 in IPv6, so the IPv6 fixtures have nowhere to put the far end. The IPv4 fixtures spend 127.0.0.0/8, which Linux already routes to lo. make ze-dev-setup adds the same address on a developer machine.

What gates a merge

verify.yml runs make ze-precommit-verify and nothing else. The stage list is deliberately not duplicated in the workflow: it lives in stagesForMode (scripts/status/verify_run.go) and nowhere else, so a gate missing from that function runs nowhere. Read the live list with make ze-precommit-verify-list.

That single-source arrangement is the point. Adding a check to the Makefile is not enough; it has to be in the stage list to actually gate anything.

What runs on a schedule instead

Three kinds of work are kept off the merge path.

Vulnerability scanning. make ze-dependency-vulnerability-check runs govulncheck against the Go vulnerability database and reports issues reachable from Ze's own call graph. It is scheduled rather than blocking on purpose: the scan fetches the vulnerability database over the network, and neither a transient fetch failure nor a newly published advisory should wedge the merge gate. The always-on supply-chain guard that does gate merges is the vendored-updater fix-marker test in the unit suite.

Heavy evidence. Fuzz targets and the kernel integration suites need root and CAP_NET_ADMIN, so they run nightly on a runner that has them, and each job is advisory. evidence-nightly.yml also runs the BGP interop suite and the IPsec interop suite, which had a runner and no automated caller.

The Linux-only functional surface. qemu-nightly.yml is its own scheduled, advisory workflow with three jobs: needs-linux (make ze-qemu-needs-linux-test), protocol-labs (LDP and IS-IS against FRR, VRRP against keepalived), and runtime-kernel-labs (L2TP, PPPoE against accel-ppp, traffic usage). It is not a merge gate: one job boots a VM, cross-compiles three binaries, and runs every functional suite inside it, on a budget of 5400 seconds.

Two things make it work on a hosted runner. A udev rule grants the runner user /dev/kvm when the device is present, and QEMU is started with -machine accel=hvf:kvm:tcg, so an unusable /dev/kvm costs time rather than stopping the run. Every job that runs a kernel-guarded target stages the kernel first with make ze-kernel-vmlinuz-stage KERNEL_ARCH=amd64; a wiring test refuses a job that does not.

Everything else on demand.

• make ze-mutation-test / make ze-mutation-test-changed. Mutation testing via gomu. Advisory only. See Testing.
• make ze-interop-test. Docker-based interop against FRR, BIRD, GoBGP.
• make ze-live-test. Live BGP tests against real RPKI caches and looking glasses.
• make ze-stress-test. Long-duration stress runs.

Ask the maintainer to run these if your change touches the relevant area.

How to read a failing CI run

The first thing to look at is whether the failure happened before or after make ze-lint. If lint failed, the run did not get far enough to run tests, and the fix is in the lint output.

If the failure is in a unit test, the test output names the package and the test. go test output is not the friendliest, but the structure is predictable. Look for the --- FAIL: lines.

If the failure is in a functional test, the runner prints a diff between the expected output and the actual output. Look at the diff first: most functional failures are either "encoder produced different bytes" (real bug) or "expected output is stale" (the .ci file needs updating).

If the failure is in a fuzz test, the fuzzer saved a reproducer under testdata/fuzz/<target>/. Add it to your local corpus, reproduce it, fix the parser, and commit the reproducer alongside the fix.

Reproducing CI failures locally

The whole pipeline runs from Make. The same Make target will reproduce the same failure locally.

make ze-lint          # Reproduce the lint failure
make ze-unit-test     # Reproduce the unit failure
make ze-functional-test
make ze-fuzz-test

If something passes locally and fails in CI, the usual culprit is a flaky test. Rerun the specific test with --count 10 to make the flake visible.

ze-test bgp encode --count 10 <test-nick>

A test that passes nine out of ten times is a bug, not a flake. Find the race, fix it, do not mark the test as "known flaky".

make ze-precommit-verify vs make ze-ci-verify

These are close but not identical. ze-precommit-verify is the thorough target: lint, changed-file wiring and doc gates, unit tests, functional tests, and ExaBGP. It is what the merge gate runs and what you should run before submitting. ze-ci-verify is lighter (lint, unit tests, build) and is useful as a quick local sanity check.

make ze-precommit-verify        # What CI runs: lint, static gates, unit, functional, ExaBGP
make ze-precommit-verify-list   # The live stage list
make ze-ci-verify            # Quick local check: lint, unit tests, build

make ze-precommit-verify-changed is the finest-grained iteration target: it only runs ze-precommit-verify on the packages your branch has touched. Use it on every save during development.

What neither the gate nor the schedule runs

ze-precommit-verify runs 28 stages and the nightly workflows run about a dozen more. The repository has more than forty check and census targets, so a large set was in neither and was run by nobody. make ze-cadence-daily-run, make ze-cadence-weekly-run, and make ze-cadence-monthly-run collect that set, and they are deliberately not stages of ze-precommit-verify: the merge gate must be fast and decisive, while most of what a cadence run surfaces is a census with no verdict to give. See Building.

One check belongs to neither list for a different reason. make ze-repository-tracked-build-check compiles what git holds rather than the working tree, so it judges HEAD after a commit lands. It is a stage of ze-precommit-verify in both modes and a structural gate on the commit path.

Doc drift

make ze-doc-drift-check

ze-doc-drift-check scans the in-tree documentation for claims that no longer match the code: a flag that was removed, a command name that changed, a plugin that is no longer shipped. If your change renames a flag or moves a command, run this target locally and fix whatever it surfaces.

When the build fails for no obvious reason

If a build fails and the failure does not come from your changes:

1. Pull the latest main and rebase.
2. make clean and rebuild from scratch.
3. Check if the vendored third-party code is out of sync (make ze-vendor-web-check).
4. Check if there is a known issue in the maintainer's status.

If none of those fix it, ask the maintainer. It is often faster than debugging CI flakes alone.

See also

• Building for the Make targets.
• Testing for what the test targets actually do.
• Contributing for the pre-submission checklist.

Adapted from main/docs/ci-test-coverage.md and main/Makefile.