fix(index): checkpoint after building the FTS index, or the next boot… #743

Workflow file for this run

	name: CI

	# Re-enabled at v1.0.0 (M6). During bootstrap CI was paused
	# (`workflow_dispatch` only) and the local four-command gate
	# (`cargo fmt` / `clippy -D warnings` / `test` / `build --release`)
	# was the merge gate per CLAUDE.md "How we work". With the workspace
	# stabilised at v1 the GitHub Actions safety net is back on for
	# every push to main and every PR.
	#
	# This is the correctness gate: fmt + clippy + test + build on Linux, on
	# every PR, every push to main, and every release tag. The cross-platform
	# RELEASE binaries (Windows + macOS + Linux packaging + GitHub Release)
	# live in `release.yml`, which fires on `v*` tags only — PRs stay
	# Linux-only.
	on:
	workflow_dispatch:
	push:
	branches: [main]
	tags: ["v*"]
	pull_request:

	concurrency:
	group: ci-${{ github.ref }}
	cancel-in-progress: true

	env:
	CARGO_TERM_COLOR: always
	RUSTFLAGS: "-D warnings"
	RUST_BACKTRACE: short
	# The workspace statically links libduckdb into its fat test binaries.
	# Linking them with full debuginfo, in parallel, blows past the 7 GB on a
	# GitHub-hosted runner — the first re-enabled CI run got SIGTERM'd (exit
	# 143) mid-link before any test ran. Drop test/dev debuginfo (duckdb's is
	# enormous). CI-only; the committed profiles and local dev builds are
	# untouched.
	#
	# `CARGO_BUILD_JOBS: "1"` used to sit here too, job-wide, as the other half
	# of that memory cap. Job-wide was too broad: it also serialised `cargo
	# clippy`, which runs no linker at all and so was paying the cap without
	# ever having posed the risk. It now sits on the steps that link (the
	# `cargo test` build below, and `cargo build --release` in its own job).
	CARGO_PROFILE_DEV_DEBUG: "0"
	CARGO_PROFILE_TEST_DEBUG: "0"

	jobs:
	ci:
	name: fmt + clippy + test
	runs-on: ubuntu-latest
	# DuckDB is no longer compiled from source — libduckdb-sys downloads
	# the precompiled libduckdb (see .cargo/config.toml: DUCKDB_DOWNLOAD_LIB).
	# `CARGO_BUILD_JOBS=1` (the link-memory cap) still serialises linking on
	# the test step, so keep generous headroom.
	timeout-minutes: 90
	steps:
	- uses: actions/checkout@v4

	# rust-toolchain.toml drives the channel + components; rustup
	# honours it automatically on first invocation.
	- name: Install Rust toolchain
	run: \|
	rustup show active-toolchain \|\| rustup toolchain install
	rustup component add rustfmt clippy

	- uses: Swatinem/rust-cache@v2
	with:
	# Bumped from `workspace` to invalidate caches that hold the
	# libduckdb-sys build-script fingerprint but NOT the downloaded
	# libduckdb (the old caches link-fail with `cannot find -lduckdb`).
	shared-key: workspace-duckdb-dl
	# The build script downloads libduckdb into this custom dir and
	# bakes its path into the cached link-search output. It is NOT a
	# standard cargo artifact, so cache it explicitly — otherwise a
	# cache hit restores the fingerprint (script does not re-run) but
	# not the .so, and the final test/release link fails.
	cache-directories: target/duckdb-download
	# Save the cache even when a later step fails — without this, a
	# single test failure throws away the warm cache and every push
	# re-pays the cold dependency build.
	cache-on-failure: true

	# Safety net for cache restores that lack the downloaded libduckdb
	# (e.g. an older cache, or a partial restore): the libduckdb-sys
	# build script only re-downloads when it actually re-runs, so force
	# that by cleaning just that crate when the .so is absent.
	# `cargo clean -p <pkg>` is per-PROFILE: bare, it cleans target/debug
	# only. This job builds dev/test artifacts, so bare is correct here —
	# but see the `build --release` job below, where the missing `--release`
	# silently made the identical step a no-op and cost that job its cache.
	- name: Ensure libduckdb is present (cache-restore safety)
	run: \|
	if ! ls target/duckdb-download///libduckdb.so >/dev/null 2>&1; then
	echo "libduckdb not in restored cache — forcing libduckdb-sys rebuild"
	cargo clean -p libduckdb-sys \|\| true
	fi

	- name: cargo fmt --check
	run: cargo fmt --all -- --check

	- name: cargo clippy
	run: cargo clippy --workspace --all-targets -- -D warnings

	# Still plain `cargo test`, and that is a measured decision rather than
	# inertia — `cargo nextest` was tried here and is SLOWER on this runner.
	#
	# nextest schedules tests across binaries where libtest runs the
	# binaries one after another, which is worth a lot on a workstation:
	# 100s -> 50s on 32 cores. But it runs each test in its OWN process,
	# and this suite's per-process cost is real (an RSA keypair for the
	# test issuer, which `Keys::shared` can only amortise within a
	# process). That trades CPU for scheduling, and a 2-core runner has no
	# spare CPU to trade. Pinned to two cores, measured on this workspace:
	#
	# cargo test --workspace --all-targets 4m49s wall, 6m41s CPU
	# cargo nextest run --workspace 5m46s wall, 8m41s CPU
	#
	# So nextest stays a local convenience (see .config/nextest.toml) and
	# CI keeps libtest. Revisit if this job ever moves to a bigger runner —
	# the ordering flips as soon as cores stop being the binding constraint.
	#
	# This step links the fat binaries, so it keeps the `CARGO_BUILD_JOBS:
	# 1` memory cap that used to apply to the whole job. That cap is on
	# cargo's build parallelism only; it does not throttle libtest's test
	# threads.
	- name: cargo test
	run: cargo test --workspace --all-targets
	env:
	CARGO_BUILD_JOBS: "1"

	# `cargo build --release` in its OWN job.
	#
	# It was the last step of `ci` and the first thing to die: exit 143
	# mid-link on a cold cache. One `target/` was holding check, test AND
	# release artifacts for a workspace that statically links libduckdb —
	# which is why this file already carries CARGO_BUILD_JOBS=1 and
	# *_DEBUG=0. Its own runner gets its own memory, disk and cache key.
	#
	# This matters more than a slow build: a cancelled job SKIPS the
	# cache-save step, so a run that dies leaves no warm cache and the next
	# run starts cold and dies in the same place. Splitting breaks that
	# loop.
	build:
	name: build --release
	runs-on: ubuntu-latest
	timeout-minutes: 90
	steps:
	- uses: actions/checkout@v4

	- name: Install Rust toolchain
	run: \|
	rustup show active-toolchain \|\| rustup toolchain install
	rustup component add rustfmt clippy

	# This job used to run with NO cache at all, and the comment here said
	# that was deliberate: libduckdb-sys (DUCKDB_DOWNLOAD_LIB, see
	# .cargo/config.toml) downloads libduckdb into target/duckdb-download,
	# Swatinem/rust-cache prunes it on save, and a restored cache then
	# link-fails with `unable to find library -lduckdb` — the build-script
	# fingerprint comes back, so the script does not re-run and re-download,
	# but the .so it points at is gone. All of that is true. The conclusion
	# drawn from it — that the `cache-directories` + `cargo clean -p
	# libduckdb-sys` safety step "did NOT close this" — was not.
	#
	# The safety step never ran against this job's artifacts. `cargo clean
	# -p <pkg>` cleans the DEV profile only; it needs `--release` to touch
	# target/release. Measured on this workspace with both profiles built:
	#
	# cargo clean -p libduckdb-sys -> 58 files, all target/debug
	# cargo clean -p libduckdb-sys --release -> 48 files, all target/release
	#
	# So in the `ci` job (dev profile) the safety net worked and the cache
	# was fine; here it was a guaranteed no-op, the stale fingerprint
	# survived, and the link failed. The fix is the missing flag, not the
	# missing cache — and dropping the cache cost a full cold dependency
	# build on every PR (23m36s, the longest job in the run).
	- uses: Swatinem/rust-cache@v2
	with:
	# Its own key: this job's target/ holds release artifacts, the `ci`
	# job's holds dev+test ones, and they must not overwrite each other.
	shared-key: release-build
	cache-directories: target/duckdb-download
	cache-on-failure: true

	# The safety net, with the flag that makes it apply to THIS job's
	# profile. rust-cache prunes the downloaded libduckdb even though it is
	# listed in `cache-directories` (verified in run 33258823279, where this
	# step fired on a warm-cache `ci` job), so assume the .so is absent and
	# force the build script to re-run whenever it is.
	- name: Ensure libduckdb is present (cache-restore safety)
	run: \|
	if ! ls target/duckdb-download///libduckdb.so >/dev/null 2>&1; then
	echo "libduckdb not in restored cache — forcing libduckdb-sys rebuild"
	cargo clean -p libduckdb-sys --release \|\| true
	fi

	- name: cargo build --release
	run: cargo build --workspace --release
	env:
	# Same link-memory cap as the test build; this job links every
	# binary in the workspace.
	CARGO_BUILD_JOBS: "1"

	# The SHIPPED feature set, in its OWN job.
	#
	# Without this, `s3.rs` and `gcs.rs` are never compiled on the merge
	# path — they sit behind optional features and the job above passes no
	# `--features`. That is how the S3 backend reached production
	# unbuildable: the Dockerfile selected a feature CI had never once
	# compiled.
	#
	# Why a separate job rather than another step in `ci`: the AWS and
	# Google SDK dependency trees are large, and the main job already runs
	# `CARGO_BUILD_JOBS=1` with a 90-minute budget because this workspace
	# statically links libduckdb and sits near the runner's memory ceiling.
	# Adding those trees to the same `target/` pushed `cargo build
	# --release` into a SIGTERM (exit 143). Its own runner keeps that
	# budget intact.
	#
	# clippy without `--all-targets`: the backends' own integration tests
	# need real containers and run in `live.yml`. This job's job is to
	# prove the library code still compiles and lints under the flags the
	# release artifacts actually use.
	features:
	name: clippy (shipped features)
	runs-on: ubuntu-latest
	timeout-minutes: 60
	steps:
	- uses: actions/checkout@v4

	- name: Install Rust toolchain
	run: \|
	rustup show active-toolchain \|\| rustup toolchain install
	rustup component add clippy

	- uses: Swatinem/rust-cache@v2
	with:
	shared-key: features-s3-gcs
	cache-directories: target/duckdb-download
	cache-on-failure: true

	- name: Ensure libduckdb is present (cache-restore safety)
	run: \|
	if ! ls target/duckdb-download///libduckdb.so >/dev/null 2>&1; then
	echo "libduckdb not in restored cache — forcing libduckdb-sys rebuild"
	cargo clean -p libduckdb-sys \|\| true
	fi

	- name: cargo clippy --features s3,gcs
	run: cargo clippy -p escurel-server -p escurel-storage --features s3,gcs -- -D warnings

Navigation Menu

fix(index): checkpoint after building the FTS index, or the next boot… #743

Workflow file

fix(index): checkpoint after building the FTS index, or the next boot… #743

Uh oh!

Workflow file for this run