feat: add Nix crane build system with Cachix caching by radupopa369 · Pull Request #270 · movement-network/aptos-core

radupopa369 · 2025-12-15T17:01:22Z

Summary

This PR transitions the aptos-core build system from nix develop -c cargo build to proper nix build .#<package> pattern using crane, enabling:

Reproducible builds via Nix flakes
Cachix caching for sharing build artifacts across machines
Dual platform support: Apple Silicon (aarch64-darwin) and Linux (x86_64-linux)

Problem

The current build system uses nix develop -c cargo build which:

Does NOT cache build artifacts in Nix store
Does NOT benefit from Cachix for sharing built artifacts
Requires full rebuilds on each machine

Solution

The new system uses nix build .#<package> which:

Runs builds inside Nix's sandbox
Produces deterministic outputs (/nix/store/...)
Can be cached and shared via Cachix
Enables instant rebuilds when pulling from cache

Changes

Files Modified/Created

File	Description
`nix/flake.nix`	Added crane, 5 package outputs, 2 container outputs
`nix/flake.lock`	Added crane input, updated all dependencies
`Justfile`	Added `build-nix`, `build-all-nix`, `container-nix`, `cache-*` recipes
`docs/nix-build-setup.md`	Comprehensive Nix installation and usage guide
`docs/nix-cachix-setup.md`	Cachix setup documentation
`.github/workflows/nix-build.yaml`	CI workflow for automated builds and cache population

New Build Targets

Target	Command	Description
`aptos-node`	`just build-nix aptos-node`	Main Aptos node binary
`movement`	`just build-nix movement`	Movement CLI (renamed from aptos)
`l1-migration`	`just build-nix l1-migration`	L1 migration tool
`aptos-faucet-service`	`just build-nix aptos-faucet-service`	Faucet service
`aptos-transaction-emitter`	`just build-nix aptos-transaction-emitter`	Transaction testing tool
`all-binaries`	`just build-all-nix`	All 5 binaries

Container Targets (Linux only)

Target	Command	Description
`container-aptos-node`	`just container-nix aptos-node`	Node container with aptos-node, movement, l1-migration
`container-aptos-faucet-service`	`just container-nix aptos-faucet-service`	Faucet service container

How to Test

Prerequisites

Install Determinate Nix (recommended):

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

Fix "public key is not valid" Error (required for Determinate Nix):

Determinate Nix adds FlakeHub cache keys that can cause validation errors. Fix by editing /etc/nix/nix.conf:

sudo nano /etc/nix/nix.conf

Find and comment out or remove the line starting with:

extra-trusted-public-keys = cache.flakehub.com-3:... cache.flakehub.com-4:...

Then restart the daemon:

macOS:

sudo launchctl stop org.nixos.nix-daemon && sudo launchctl start org.nixos.nix-daemon

Linux:

sudo systemctl restart nix-daemon

Configure Movement Labs Cachix Cache (optional, for faster builds):

macOS:

sudo tee /etc/nix/nix.custom.conf << 'EOF'
trusted-users = root @admin @staff
trusted-substituters = https://movementlabs.cachix.org
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= movementlabs.cachix.org-1:qqCkWyzFSZCH2Tcy/HPRXVOOlYR3Sv+4GKMXSZtyN8s=
accept-flake-config = true
EOF

sudo launchctl stop org.nixos.nix-daemon && sudo launchctl start org.nixos.nix-daemon

Install Cachix CLI (optional, for pushing builds):

nix profile add nixpkgs#cachix
cachix authtoken YOUR_AUTH_TOKEN

Test Build Commands

cd nix && nix flake show

just build-nix aptos-node

cd nix && nix build .#aptos-node -L

just build-all-nix

Automated Cache Population

Both CI and local builds push to the same Cachix cache:

CI (GitHub Actions)

The .github/workflows/nix-build.yaml workflow automatically:

Builds all 5 binaries on push to main/develop
Pushes all artifacts to movementlabs Cachix cache

Required Secret: Add CACHIX_AUTH_TOKEN to repository secrets.

Local Development

Option 1: Auto-push while developing (recommended)

Run in a separate terminal - automatically pushes ALL nix builds to cache:

just cache-watch

Option 2: Build and push specific binary

just cache-build aptos-node

just cache-build all-binaries

Option 3: Manual push after building

nix build .#aptos-node -L
cachix push movementlabs ./result

nix build .#all-binaries -L
cachix push movementlabs ./result

Verify Cache Hits

After pushing, subsequent builds should pull from cache:

rm -f result
nix build .#aptos-node -L

You'll see cache hits in the output:

copying path '/nix/store/xxx-aptos-node-0.1.0' from 'https://movementlabs.cachix.org'...

Related Issues

Closes #227

Out of Scope (Future Work)

Cross-compilation (building x86_64-linux on ARM Mac)
Windows platform support

- Add crane input for reproducible Rust builds via Nix - Add package outputs for all 5 binaries: - aptos-node, movement, l1-migration - aptos-faucet-service, aptos-transaction-emitter - Add all-binaries aggregate package - Add container outputs using dockerTools (Linux only) - Add Cachix substituter configuration for movementlabsxyz - Add Justfile recipes: build-nix, build-all-nix, container-nix, container-load, container-push - Add comprehensive documentation in docs/nix-build-setup.md and docs/nix-cachix-setup.md - Support dual platform builds: aarch64-darwin and x86_64-linux - Platform-specific RUSTFLAGS (SSE4.2 only on x86_64)

- Update nixpkgs from pinned commit to nixos-unstable (required by crane 25.11+) - Migrate from deprecated darwin.apple_sdk.frameworks to new apple-sdk_15 pattern - Remove libelf dependency (now provided by SDK) - Reference: https://discourse.nixos.org/t/the-darwin-sdks-have-been-updated/55295

- Focus on trusted-users setup (the actual fix for warnings) - Remove cachix CLI instructions (cache not yet created) - Add Determinate Nix specific instructions - Clarify current cache status

- Add cache-push, cache-push-all, cache-status recipes to Justfile - Rewrite docs/nix-cachix-setup.md with complete workflow: - One-time admin setup (create cache, generate tokens) - Developer setup (trusted-users, install cachix CLI) - PR review workflow (author pushes, reviewer pulls) - CI/CD integration example Enables team members to share builds during PR review to avoid redundant compilation.

Update from 'movementlabsxyz' to 'movementlabs' which is the existing cache. Update public key to: movementlabs.cachix.org-1:qqCkWyzFSZCH2TcyHPRXVOOlYR3Sv+4GKMXSZtyN8s=

Use group-based trust (@admin @staff) instead of individual usernames for Determinate Systems' Nix on macOS. Individual usernames don't work reliably even when configured correctly. Also include full nix.custom.conf with: - trusted-substituters for Cachix - trusted-public-keys for cache verification - accept-flake-config for automatic flake nixConfig acceptance

…y error - Add Nix build section to README with quick start and feature comparison - Update nix-build-setup.md to strongly recommend Determinate Nix - Add feature comparison table (parallel eval, lazy trees, native Linux builder) - Document upgrade command: sudo determinate-nixd upgrade - Add workaround for 'public key is not valid' error in cachix setup

- Add .github/workflows/nix-build.yaml for CI-based builds - Add cache-watch and cache-build Justfile recipes for local builds - Both CI and local push to same movementlabs Cachix cache - Update cache-status to check config file existence

…rrors The 'public key is not valid' error occurs with some Nix versions when the Cachix key is specified in flake.nix. Moving configuration to system config (/etc/nix/nix.custom.conf) avoids this issue. Cachix still works via: - System config for local builds - cachix-action for CI builds

Determinate Nix adds FlakeHub cache keys to /etc/nix/nix.conf which can trigger a key validation error. Document the fix: comment out the extra-trusted-public-keys line containing FlakeHub keys. Updated: - README.md: Quick troubleshooting note - docs/nix-build-setup.md: Detailed fix instructions - docs/nix-cachix-setup.md: Troubleshooting section and updated flake config docs

Add patterns for files embedded via include_str!/include_bytes! macros: - .json files (JWKS test keys) - .pem files (cryptographic keys) - .bpl files (Boogie prover prelude) - .mv files (Move bytecode) These files were being filtered out by craneLib.filterCargoSources, causing build failures for l1-migration and other packages.

Move stdlib uses include_bytes! for error_description.errmap file.

Config crate uses include_str! for test_data YAML files.

The movementlabs Cachix public key was missing a '/' character. Correct key: movementlabs.cachix.org-1:qqCkWyzFSZCH2Tcy/HPRXVOOlYR3Sv+4GKMXSZtyN8s=

The crate was renamed from 'aptos' to 'movement' in Cargo.toml

These files are embedded at compile time via include_str!: - test_proofs_for_localnet_txn_emitter.txt - doc/spec.html (API documentation) - doc/.version (version info)

On macOS, system jemalloc doesn't have the _rjem_ prefixed symbols that jemallocator crate expects. Let jemalloc-sys build from source on macOS to get the correct prefixed symbols.

radupopa369 added 20 commits December 15, 2025 18:00

docs: simplify binary cache setup guide

27dc6d2

- Focus on trusted-users setup (the actual fix for warnings) - Remove cachix CLI instructions (cache not yet created) - Add Determinate Nix specific instructions - Clarify current cache status

fix: use correct Cachix cache name 'movementlabs'

468790b

Update from 'movementlabsxyz' to 'movementlabs' which is the existing cache. Update public key to: movementlabs.cachix.org-1:qqCkWyzFSZCH2TcyHPRXVOOlYR3Sv+4GKMXSZtyN8s=

docs: add nix shell alternative for cachix install

6006a23

fix: add meta.description to apps to suppress flake check warnings

4e479c9

fix: remove invalid cachix authtoken --check command

3851f07

fix: add .errmap pattern to crane source filter

826f984

Move stdlib uses include_bytes! for error_description.errmap file.

fix: add .yaml pattern to crane source filter

a18f9fe

Config crate uses include_str! for test_data YAML files.

fix: correct Cachix public key in documentation

6be8d08

The movementlabs Cachix public key was missing a '/' character. Correct key: movementlabs.cachix.org-1:qqCkWyzFSZCH2Tcy/HPRXVOOlYR3Sv+4GKMXSZtyN8s=

fix: correct cargoPackage name for movement binary

0a476c9

The crate was renamed from 'aptos' to 'movement' in Cargo.toml

fix: add .txt, .html, .version patterns to crane source filter

af375ad

These files are embedded at compile time via include_str!: - test_proofs_for_localnet_txn_emitter.txt - doc/spec.html (API documentation) - doc/.version (version info)

fix: only use JEMALLOC_OVERRIDE on Linux for correct symbol prefixes

094be3d

On macOS, system jemalloc doesn't have the _rjem_ prefixed symbols that jemallocator crate expects. Let jemalloc-sys build from source on macOS to get the correct prefixed symbols.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add Nix crane build system with Cachix caching - #270

feat: add Nix crane build system with Cachix caching#270
radupopa369 wants to merge 20 commits into
m1from
feat/nix-crane-build

radupopa369 commented Dec 15, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

radupopa369 commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Changes

Files Modified/Created

New Build Targets

Container Targets (Linux only)

How to Test

Prerequisites

Test Build Commands

Automated Cache Population

CI (GitHub Actions)

Local Development

Verify Cache Hits

Related Issues

Out of Scope (Future Work)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

radupopa369 commented Dec 15, 2025 •

edited

Loading