Nix reproducible build: satd binary intermittently non-deterministic across hosts #321

Summary

The Nix reproducibility check (nix.yml → nix repro / compare, which asserts two independent runners produce byte-identical binaries of the same commit) intermittently fails for the satd binary. sat-cli was reproducible in the same runs — only satd diverged.

This is a real (if intermittent) break of the reproducible-build guarantee documented in the Operator Manual's packaging chapter (§"Determinism hazards addressed"). It is not a build failure — both replicas build successfully; their outputs just differ.

Evidence

First observed on PR #320 (a docs-only PR that incidentally triggered nix.yml by touching contrib/repro/):

• Run 1 — FAIL (compare job):

  MISMATCH for satd: 44c060f11c8c533f69d007e658b363aaffc77cb8da369fb3c230c6c76c5cfabf
                 vs  2233b69430da428f8ed35ba00c690242180c562bb46c3812e7bdf15d7a6b716a
  

  Both nix repro / x86_64-linux pair (a) and (b) built successfully (~11m each); sat-cli matched.
• Run 2 — PASS (same PR, next commit, fresh pair rebuild): nix repro / compare green in 4s.

So the divergence is intermittent, not chronic. Historically nix.yml passed on the v0.1.0, v0.2.0, and v0.2.1 tag builds — but those are only three data points, all from before the large post-v0.2.1 churn (streaming API, unified auth, API-runtime split, invalidateblock, etc.).

Why this matters

• The reproducible-build claim (docs/manual/src/packaging.md, "Reproducible build via Nix") is the basis packagers use to verify our binaries. An intermittent mismatch means an independent rebuild can legitimately fail to match a release, undermining that trust.
• Because it's intermittent, a tagged release could ship a satd whose hash a third party can't reproduce on the first try.

Hypotheses (only satd diverges, sat-cli is clean)

satd links a much larger graph than sat-cli — RocksDB (librocksdb-sys / bindgen), jemalloc, tonic/prost protobuf codegen (events), rayon. Likely sources of host- or schedule-dependent nondeterminism:

1. librocksdb-sys bindgen output varying with the runner's libclang / system-header set (the flake's bindgenHook is "deterministic for a fixed libclang version" — are both runners pinned to the exact same one?).
2. Parallel codegen-unit ordering leaking into the binary (symbol/section ordering) despite -C link-arg=-Wl,--build-id=none + strip=symbols.
3. An embedded path / timestamp / env value present only in satd's dependency tree (e.g. an OUT_DIR-baked string, a proto-gen artifact).

Suggested next steps

• Reproduce deterministically: build satd twice on two hosts (or contrib/repro/diff-build.sh) and run diffoscope on the divergent outputs to localize the differing bytes/section.
• If RocksDB bindgen is the culprit, pin libclang and the rocksdb build inputs identically across runners.
• Once root-caused, add a regression guard (the repro check already exists — consider making it non-skippable / running on more commits, not just flake-touching PRs + tags).

Notes

• Surfaced via PR docs: consolidate operator docs into an mdbook manual; retire ECOSYSTEM.md #320 (operator-manual mdbook consolidation); that PR is doc-only and is not the cause.
• nix.yml only runs on flake-touching PRs, workflow_dispatch, and v* tags, so this won't be caught on most PRs.