[fcostaoliveira]

Follow-up to #369 (now merged; rebased onto master). Adds native .deb and .rpm packages for a release, built, install-tested and published the way memtier-benchmark's are, so vector-db-benchmark can be installed from the Redis APT and YUM repositories.

Staging rehearsed; production off until vars.PACKAGES_PRODUCTION=true. Publishing goes to the GitHub environment named by the run. staging is our own rehearsal bucket, vdb-package-staging (us-east-2, AWS account 726902207197), with the production layout at its root (deb/, rpm/, and the throwaway signing key's public half as /gpg), so a rehearsal needs nothing from any other team. build is packages.redis.io. The publish job pins the bucket to the environment (staging must be a staging bucket, build must not be), so a repository-level secret inherited by staging cannot send a rehearsal to production. The Redis-wide staging bucket (redis-test-package-repository) is accepted only under a repo_prefix: its deb/ and rpm/ are signed with the production key, and publishing there would re-sign indexes other products rely on.

What's here

• debian/ (control, rules, copyright, docs, source/format) and rpm/vector-db-benchmark.spec. Both build with --features release-static from Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369, so a package depends on nothing but that distribution's glibc (dpkg-shlibdeps / rpm auto-requires confirm it). Installed layout:
  • /usr/bin/{vector-db-benchmark,prepare-msmarco,generate-dataset}
  • /usr/lib/vector-db-benchmark/{bench_npy,bench_jsonl,bench_hdf5} (/usr/libexec/... on RPM)
  • /usr/share/vector-db-benchmark/{datasets/datasets.json,datasets/random-100,experiments/configurations}
• scripts/release/build-deb.sh --dist <codename>, build-rpm.sh: run on the target distribution (a container of it in CI; make release-deb DIST=… INSTALL_DEPS=1 / make release-rpm INSTALL_DEPS=1 locally). --install-deps apt-gets/dnfs the build dependencies and a current stable Rust via rustup; the distros' own cargo/rustc are too old. Versions follow the memtier convention: deb <crate version>~<codename>, rpm <crate version>-1.<dist>. The git commit is passed through (VDB_BUILD_GIT_SHA / --define vdb_git_sha) so --version reports it even from an RPM source tarball.
• scripts/release/test-package.sh <pkg> --expect-sha: the install test, run on a bare container of every codename/distro on both architectures. Installs from the file with dependencies resolved by the package manager, checks every CLI's --version is exactly <name> <version> (git <sha>), runs --describe datasets and bench_jsonl / bench_hdf5 against the packaged data, then removes the package and installs it again from a local repository laid out like packages.redis.io (Packages index for APT; createrepo_c + the README's rpm/el$releasever/$basearch / rpm/amzn2023/$basearch baseurl for YUM).
• scripts/release/publish-apt.sh: deb-s3 upload per codename into deb/ (--origin packages.redis.io, public objects, --preserve-versions, --fail-if-exists, always signed; before uploading it checks that each codename's existing InRelease is signed by the same key, or a subkey of it).
• scripts/release/publish-yum.sh: rpm/<el8|el9|el10|amzn2023>/<x86_64|aarch64>/ with createrepo_c --update. The rpm/ prefix is shared with other Redis products (prod: redis, redis-stack; staging: redis-rpm's rockylinux*/), so it only touches the directories it has an RPM for, downloads each in full before regenerating metadata, never deletes an RPM, and uploads RPMs before repodata/.
• .github/workflows/release-packages.yml: release: published, workflow_dispatch (tag, release_type internal/public/none, overwrite for staging re-runs, codename/distro overrides) and pull_request on the packaging paths (build + test, never publish). Matrix: focal, jammy, noble, resolute, bullseye, bookworm, trixie × amd64/arm64; el8, el9, el10, amzn2023 × x86_64/aarch64 (repo vars DEB_DISTS / RPM_DISTROS override, as memtier's BUILD_DISTS). Publishing always signs, refuses production unless the key matches the environment's APT_SIGNING_FPR pin, and verifies the published indexes serve the packages. The packages are not attached to the GitHub release; Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369's archives own the release assets.
• README install section (APT, YUM, AL2023), marked coming soon until production publishing is switched on, and AGENTS.md layout entries.

Staging setup (done, self-serve)

• AWS 726902207197: bucket vdb-package-staging (ACLs enabled, public reads, like the real buckets), GitHub OIDC provider, role vdb-package-staging-publisher trusting only the staging environment of this repository with read/write + ACL on that bucket. The repository uses GitHub's immutable OIDC subject, so the trust condition names repo:redis@1529926/vector-db-benchmark@662205136:environment:staging (the plain-name form is listed too). The production role behind build needs the same immutable form, or configure-aws-credentials fails with "Not authorized to perform sts:AssumeRoleWithWebIdentity".
• GitHub environment staging: APT_S3_BUCKET=vdb-package-staging, APT_S3_REGION=us-east-2, APT_S3_IAM_ARN=arn:aws:iam::726902207197:role/vdb-package-staging-publisher, APT_SIGNING_KEY = throwaway RSA-4096 key, fingerprint FD183D88E9814786987424DAE26208135FFAAD25, uid "vector-db-benchmark staging rehearsal (throwaway, NOT the Redis package key)".
• Rehearse with gh workflow run release-packages.yml -f release_type=internal (optionally -f tag=<release tag>; -f overwrite=true to replace the version already there); install from https://s3.us-east-2.amazonaws.com/vdb-package-staging/{deb,rpm,gpg} with the README steps and that base URL.
• Production (build environment) needs the same five secrets as memtier-benchmark's, from the packaging team: bucket packages.redis.io (us-east-2), the real signing key, credentials. It must be protected before those secrets go in (see "Still to do").

Verified locally

• Worst-case builds in Docker: ubuntu:focal .deb (oldest cmake/debhelper; Depends libc6 (>= 2.30), libgcc-s1, asserted by the build script) and rockylinux:8 + rockylinux:9 .rpm (oldest glibc/gcc; the curl-minimal path; Requires glibc/libgcc only), each then installed and run with test-package.sh on a bare container of that distribution, from the file and from a local repository in the served layout. Builds used RELEASE_RUST_TOOLCHAIN=1.99.1 (the pin from Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369) with a deliberately set HDF5_DIR: rustc 1.99.1 used, no libhdf5 dependency.
• make release-selftest-packages: 85 checks of publish-apt.sh / publish-yum.sh against fake aws / deb-s3 (codename grouping, --sign=<fpr>, refuse-unsigned, dist tag → directory, same-build skip / different-build refuse / --overwrite, RPMs before repodata, another product's RPM kept and indexed, another product's RPM landing mid-publish in either race window, path-style S3 URLs for dotted buckets).
• actionlint, shellcheck -x, make release-selftest, cargo test --test harness_invariants (INV-A5: both release workflows pin the same toolchain, no inline builds anywhere in the workflow, self-test + dry run wired in, PR paths cover the package-specific inputs: packaging sources, src/bin/** and the shipped data; packaged binaries = cargo's bin targets = REQUIRED_BINS).
• Nine independent adversarial reviews (debian, rpm, test script, APT publish, YUM publish, workflow security, consistency with Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369, release-process parity, maintainability); every blocker and should-fix is in the second commit. Two of them are worth knowing: a bare --sign in deb-s3 is a string option and would have swallowed the first .deb as the key id, and an unsigned deb-s3 run strips Release.gpg off a codename every Redis product shares, so signing is now mandatory for any publish.

Staging rehearsal (done)

Run 40863645017, tag-less release_type=internal at d030b2e: all 14 deb and 8 rpm cells built and passed test-package.sh on bare containers, then the publish job signed and published every codename and rpm/<distro>/<arch> directory to vdb-package-staging and its verify step passed (InRelease signature, Packages hash, primary.xml listing).

Then the README install steps, with the staging base URL and signature checking on, on bare containers: Ubuntu focal/noble/resolute and Debian bullseye/trixie (amd64), Ubuntu jammy and Debian bookworm (arm64), Rocky 8/9/10 and Amazon Linux 2023 (x86_64), Rocky 9 and Amazon Linux 2023 (aarch64). Each installed 0.1.0~<codename> or 0.1.0-1.<dist>, printed vector-db-benchmark 0.1.0 (git d030b2e…) and listed the packaged datasets.

The rehearsal found three workflow bugs, fixed here: container steps ran under dash on Debian/Ubuntu (every deb build failed; the workflow now defaults to bash and INV-A5 asserts it), dnf install curl in the el9/AL2023 install-test setup conflicted with curl-minimal, and Debian 11's pool is gone from deb.debian.org since LTS ended on 2026-08-31 (bullseye containers now use archive.debian.org; memtier and packages.redis.io still ship bullseye).

Review follow-ups (fcf40fd)

From the automated review:

• Verify step works against a dotted bucket. It built https://$BUCKET.s3.$REGION.amazonaws.com. S3's certificate is a one-label wildcard, so for packages.redis.io that fails TLS (curl exit 60, confirmed against the real bucket) after the publish has already written. The URL now comes from s3_origin_url in lib.sh, which is path-style: https://s3.<region>.amazonaws.com/<bucket>[/<prefix>]. The regional host exists for us-east-1 too. The path-style URL returns 200 for packages.redis.io. INV-A5 refuses the virtual-hosted form in the workflow and in every release script, and the selftest covers a dotted bucket, a prefix and us-east-1. No other script builds an S3 URL: deb-s3 and the aws CLI handle dotted buckets themselves.
• YUM read-modify-write race. No lock covers rpm/. publish-yum.sh now lists each directory again just before uploading repodata/, and re-indexes any RPM that appeared since the sync, so the window shrinks to the repodata upload itself. After the upload, every *.rpm in the directory must appear in the primary.xml.gz the directory now serves. If one is missing, the run fails and names it, telling the operator to re-run the publish, which re-syncs the directory and rebuilds the index. The selftest's fake aws drops another product's RPM into both windows: the first is caught by the re-list, the second fails with the hint, and the re-run then indexes everything.
• Same-second builds. A published build was identified by NEVRA plus build time, in whole seconds. So a different build made within the same second counted as "already published". The selftest hit this intermittently (1 run in 3). The identity now also includes SIGMD5, the header and payload digest, which signing leaves unchanged. The fixtures pin SOURCE_DATE_EPOCH, so that case is now deterministic.
• The installed-from-repo check now proves what its comment says. For apt: before the install, the package must not be installed and the local repository must be its only source. A second source carrying the package now fails the check (fault-injected locally on jammy). For dnf: dnf list installed must record @vdb-local-test. The same format holds on dnf 4.7 (el8), 4.14 (el9, AL2023) and 4.20 (el10). Before, an empty from or @System was accepted.
• build environment protection is documented in the workflow header. It needs required reviewers, and deployment branches limited to master plus the release tag forms. This matters because workflow_dispatch runs the dispatched ref's copy of the workflow, guards included. Applying it is listed below.
• Release assets. The body and the workflow agree: the packages are served by the repositories and are not attached to the GitHub release.
• Size of a production YUM publish, measured from production's public indexes on 2026-10-29: each of the 8 rpm/<distro>/<arch>/ directories a release touches indexes 27 RPMs, about 20 MB, so roughly 160 MB of downloads per release. This is noted in the workflow header. Objects that no index lists can't be seen without listing rights.
• actionlint: the dnf package list is now an array, and ~/.gnupg is created with mkdir+chmod.

Gate: make agent-check, make release-selftest-packages (61/61), shellcheck -x, actionlint. Each new guard was mutation-checked: removing it, or reverting to the virtual-hosted URL, turns the selftest or INV-A5 red.

Full-matrix staging rehearsal at fcf40fd: run 40873011651, green. It covered all 7 codenames × amd64/arm64 and el8/el9/el10/amzn2023 × x86_64/aarch64. All 22 cells built and passed test-package.sh, including the tightened installed-from-repo checks. The publish signed and published every codename and directory to vdb-package-staging. The new index check passed in all 8 RPM directories, and the path-style verify step passed: InRelease signature, Packages digest, primary.xml listing.

Review round 2 (0fd1f3a)

From the seven-reviewer pass on fcf40fd. Each new guard was mutation-checked.

• Signing key vs. the shared repository. deb-s3 re-signs a whole codename. So publish-apt.sh now requires each target codename's existing InRelease to verify as signed by our key, and refuses otherwise; a new codename with no InRelease is accepted. It also test-signs before deb-s3 starts. The publish job pins the key to the environment variable APT_SIGNING_FPR, which is required for build: 54318FA4052D1E61A6B6F7BB5F4349D6BF53AA0C, from packages.redis.io/gpg.
• YUM metadata. No --delete on the repodata/ sync: it could only ever delete another product's freshly uploaded metadata. repomd.xml is uploaded last. Old generations accumulate. The one-directional residual race is documented.
• Production gate. publish_to=build happens only when vars.PACKAGES_PRODUCTION == 'true'. Otherwise a final release builds, tests and publishes nothing (with a notice), and release_type=public is refused. Production also takes only final X.Y.Z / vX.Y.Z tags (check_production_tag), whatever the pre-release flag says.
• APT verify matches our stanza (package, version and arch) by the .deb's SHA256, and checks that its pool file is served.
• Publish job. Tooling is installed before any credential or key is in the job. The deb-s3 gem checksum and ruby/setup-ruby are pinned to a commit. The publish scripts come from the workflow's own commit (github.sha, the same as the guards and the selftest); the packages come from the planned commit.
• Inputs. The PR trigger and INV-A5 now cover src/bin/**, README.md, datasets/datasets.json, datasets/random-100/** and experiments/configurations/**. The rest of src/ is compiled release-static by release-binaries' PR run. INV-A5 requires debian/rules, the spec, REQUIRED_BINS and cargo's bin targets to name the same binaries, and bench_hdf5 was added to REQUIRED_BINS.
• Docs. AGENTS.md "Cutting a release" has a Packages runbook. The workflow header covers the gate, the setup order, the tag-rule caveat, the concurrency cancellation and the residual YUM race.

Gate: make agent-check, package selftest 85/85, archive selftest 95/95, shellcheck -x, actionlint (with ShellCheck 0.11.0).

Full-matrix staging rehearsal at 0fd1f3a: run 40879384558, green. 47 jobs passed and the dry run was skipped. In the publish job:

• each of the 7 codenames' existing InRelease was checked as signed by the staging key before deb-s3 ran;
• all 14 deb cells passed the stanza and SHA256 verify;
• all 8 RPM directories passed the index check;
• the gem checksum matched.

Automated-review follow-up (03fa177):

• publish-yum.sh refuses a directory whose metadata is signed (repodata/repomd.xml.*) before writing anything; none is signed in production today.
• configure-aws-credentials is pinned to a commit.
• Doc wording fixes in the publish-yum.sh header and AGENTS.md.
• Full-matrix staging rehearsal at 03fa177: run 40881976162, green.

Still to do

• Merge gate: a green full-matrix staging rehearsal of the final head (gh workflow run release-packages.yml --ref release-packages-deb-rpm -f release_type=internal -f overwrite=true). PR runs cover only jammy and el9. Latest: run 40881976162 at 03fa177, green.

• build environment and switching production on. build does not exist yet; this repository has only pypi and staging. Production stays off until vars.PACKAGES_PRODUCTION is true, so merging can't reach it. In order:

  1. Settings → Environments → New environment build → Required reviewers: fcostaoliveira.
  2. Deployment branches and tags: Selected branches and tags → branch master, tags [0-9]*.[0-9]*.[0-9]* and v[0-9]*.[0-9]*.[0-9]*.
  3. Add a tag ruleset restricting who can create those tags.
  4. Add the five secrets, and the environment variable APT_SIGNING_FPR=54318FA4052D1E61A6B6F7BB5F4349D6BF53AA0C.
  5. Then set the repository variable PACKAGES_PRODUCTION=true.

  The same steps are in AGENTS.md.

• Before the first production publish: have the packaging team confirm that every .rpm object in the 8 production rpm/<distro>/<arch>/ directories is meant to be indexed, because createrepo_c --update indexes them all. The --pkglist scoping and an automatic post-check retry are follow-ups.

🤖 Generated with Claude Code

[github-actions]

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

Scope note: this is release and packaging infrastructure. It doesn't touch engine, filter, harness or dataset logic, so most of this repo's usual checklist doesn't apply (per-config isolation, the query_filter.rs hard-error choke point, corpus-identity checks). I read the whole diff for this repo's main bug class instead: an artifact that builds and passes, but carries the wrong contents.

The PR body already covers the things I would otherwise have asked about. It reports two full-matrix staging rehearsals (40873011651 at fcf40fd, 40879384558 at 0fd1f3a). It says each new guard was mutation-checked against selftest-packages.sh / INV-A5, and it lists what the two earlier automated rounds found and how each was fixed: path-style S3 URLs for the dotted production bucket, the YUM re-list and post-upload index check, SIGMD5 in the build identity, the signer check on InRelease, and refusing signed repomd.xml. I'm not repeating any of that.

Worth fixing before merge: nothing found.

Optional / follow-up:

1. debian/rules ignores CARGO_TARGET_DIR, which the RPM side already handles. override_dh_auto_build runs cargo build, which writes to $CARGO_TARGET_DIR when that is set. But override_dh_auto_install (debian/rules, the two install -m 0755 target/release/$$b loops) always copies from ./target/release. override_dh_auto_clean deliberately keeps target/ between builds.

   • Effect: run make release-deb with CARGO_TARGET_DIR exported (package.sh:62 and build-rpm.sh both honour it, so this is a supported setup) on a tree with an older ./target/release. The .deb then ships those older binaries, labelled <current version>~<codename>. build-deb.sh only checks Depends, so it passes. With an empty ./target/release the build fails loudly instead, which is fine.
   • Who catches it: only test-package.sh --expect-sha, and only if someone runs it. CI is not affected, because nothing in the workflow sets CARGO_TARGET_DIR.
   • Context: commit 88f58e8 fixed exactly this on the RPM side (bindir="${CARGO_TARGET_DIR:-target}/release" in the spec).
   • Fix (two lines): in build-deb.sh, export CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$repo/target}", as build-rpm.sh does. In debian/rules, install from $(or $(CARGO_TARGET_DIR),target)/release. Optionally add a check after the build that the packaged vector-db-benchmark --version reports $sha. That would cover the local path without relying on someone running the install test.

2. Minor, cosmetic: the RPM %changelog entry is fixed at 0.0.0-1 while the package version is %{vdb_version}-1%{?dist}. Debian generates its changelog per build, so the two differ. rpmlint will flag the mismatch, but it changes nothing at install time. Not a blocker.

Everything else I checked held up:

• the bucket ↔ environment pinning, and the PACKAGES_PRODUCTION gate with check_production_tag;
• --sign=<fpr> in = form;
• the order of RPM upload → hashed metadata → repomd.xml, with no --delete;
• the stanza match in the verify step, which checks package, version, arch and SHA256 in a Packages index shared with other products.

The residual one-directional YUM race is real but is already documented in the workflow header and in publish-yum.sh. Closing it needs a lock shared across products, which is outside what this repo controls.

Net: careful, heavily self-reviewed and rehearsed release plumbing. The only gap I found is local-only: a .deb built with CARGO_TARGET_DIR set can carry stale binaries, and it can be filed separately. Whether to merge, and the build-environment setup in "Still to do", are for the maintainer and the packaging team.

[fcostaoliveira]

Replying to the automated review, point by point. Everything is in fcf40fd.

1. Verify-step URL breaks TLS for dotted buckets. Fixed. I confirmed it against the real bucket: https://packages.redis.io.s3.us-east-2.amazonaws.com/... fails with curl exit 60, and the path-style https://s3.us-east-2.amazonaws.com/packages.redis.io/... returns 200. The base URL now comes from s3_origin_url in lib.sh, which is path-style and uses the regional host for every region, us-east-1 included.

• INV-A5 now refuses .s3. / .s3- on any code line that mentions amazonaws.com, in the workflow and in all release scripts. Re-adding a virtual-hosted URL turns it red (checked).
• The selftest covers a dotted bucket, a prefix and us-east-1. Reverting s3_origin_url to the virtual-hosted form fails 6 checks.
• Nothing else builds an S3 URL. deb-s3 and the aws CLI pick path-style for dotted buckets themselves, which is how memtier already publishes to packages.redis.io.

2. YUM read-modify-write race. Fixed in both ways you suggested.

• Just before uploading repodata/, the directory is listed again. RPMs that appeared since the sync are downloaded, and createrepo_c --update runs once more.
• After the upload, every *.rpm in the directory must be in the served primary.xml.gz. If any is missing, the run fails, names the files, and says to re-run the publish, which re-syncs the directory and rebuilds the index.
• The selftest's fake aws drops another product's RPM into each window. Removing the re-list fails 3 checks, and so does removing the post-check.
• While doing this, the selftest exposed an intermittent bug. NEVRA plus whole-second BUILDTIME let a different build made in the same second count as "already published". The identity now includes SIGMD5, which signing leaves unchanged, and the fixtures pin SOURCE_DATE_EPOCH. Dropping SIGMD5 now fails deterministically.

3. build environment protection. The workflow header now documents it: required reviewers, plus deployment branches limited to master and the release tag forms [0-9]*.[0-9]*.[0-9]* / v[0-9]*.[0-9]*.[0-9]*. It also explains why: workflow_dispatch runs the dispatched ref's copy of the guards. I couldn't apply it, because build doesn't exist yet; the repository has only pypi and staging. I didn't create it. The exact clicks are under "Still to do" in the PR body. If it is left missing, GitHub creates it on first use with no protection, but also without any APT_* secrets, and none are set at repository level. So the configuration check fails before anything is written.

Full-matrix rehearsal as a merge gate. It's now listed as one in the PR body. Run 40873011651 at fcf40fd is green:

• all 7 codenames × amd64/arm64 and el8/el9/el10/amzn2023 × x86_64/aarch64 built and passed test-package.sh;
• the staging publish passed the new index check in all 8 RPM directories and the path-style verify step.

YUM download size. I measured it from production's public primary.xml.gz files on 2026-10-29: each of the 8 directories a release touches indexes 27 RPMs, about 20 MB, so roughly 160 MB of downloads per release. This is in the workflow header. Objects that no index lists can't be seen without listing rights.

PR body vs. release assets. The body had already been corrected before this review ran. It now says the packages are not attached to the GitHub release, which matches the workflow header.

README. The section is marked "Coming soon". It says production publishing isn't switched on yet, so apt-get / dnf install won't find the package today, and points to the release archives instead.

Installed-from-repo check. I tightened it rather than softening the comment.

• apt: before installing, apt-cache policy must show the package not installed, with the local repository as its only source. Fault-injecting a second source carrying the package makes it fail.
• dnf: dnf list installed must record @vdb-local-test. That format holds on dnf 4.7, 4.14 and 4.20, and the full rehearsal passed with it on every RPM cell.

The comment now describes exactly these two checks.

[fcostaoliveira]

Second review round (seven reviewers on fcf40fd). Everything below is in 0fd1f3a. Each new guard was mutation-checked: breaking it turns the package selftest, INV-A5, or a local run of the changed workflow step red. Gate: make agent-check, make release-selftest-packages (85/85), make release-selftest (95/95), shellcheck -x and actionlint. actionlint ran with ShellCheck 0.11.0 bundled and is clean, so the SC2153 info is gone.

Blocking

p2: the APT signing key was never checked against the key the shared repo is signed with. Fixed.

• Before deb-s3 writes anything, publish-apt.sh reads each target codename's existing dists/<codename>/InRelease. It then requires gpg --status-fd 1 --verify to report VALIDSIG <our fpr>, and otherwise refuses, naming both keys. A codename with no InRelease is new and accepted, so a fresh staging tree or prefix still works.

• I read InRelease with aws s3 (the job's credentials, the exact object deb-s3 is about to replace), not over HTTPS through s3_origin_url. That way the check doesn't depend on public-read ACLs or the CDN, and the selftest can exercise it against the fake bucket.

• From your optional Update.redisearch hybrid benchmarks #2: the script also test-signs first, because deb-s3 uploads Release before it signs.

• Pin: the import step compares the key with the environment variable APT_SIGNING_FPR. It is required for build, so production refuses to run without it, and optional for staging. Production's value is 54318FA4052D1E61A6B6F7BB5F4349D6BF53AA0C, from packages.redis.io/gpg; it is in the workflow header and in AGENTS.md.

• RPM: the same imported key signs the RPMs, so the pin covers YUM as well. YUM's repodata/ is unsigned, and other products' RPMs keep their own signatures, so a wrong key there would only affect our own package. I added no existing-signature check on that side.

• Selftest: the fake deb-s3 now really clearsigns InRelease. Two new cases fail closed before any deb-s3 call:

  • a codename whose InRelease has a valid signature from another key (its public half is imported, so the signature verifies; it just isn't ours);
  • a passphrase-protected key.

  Removing the signer check fails 3 checks; removing the test signature fails 1. Locally, I ran the import step with a generated key in 5 cases: build without a pin, build with the wrong pin, build with a matching pin written in lowercase with spaces, staging without a pin, and staging with the wrong pin. All behaved as intended.

p3 #1: --delete on the repodata/ sync. Removed. The header now says that old generations accumulate, and that pruning, if it is ever needed, goes by age and never by a delete-sync.

• From optional Update.redisearch hybrid benchmarks #2: repodata/ is uploaded with --exclude repomd.xml, and repomd.xml is uploaded on its own, last.
• Selftest: another product's new hashed primary.xml.gz lands during our repodata upload, and it must still be there afterwards. A second check confirms repomd.xml goes after the rest and that no --delete is used. Putting --delete back fails 2 checks; uploading repomd.xml first fails 1.

p7 #1: an rc tag could publish final-version packages to production. Fixed. Whenever publish_to=build, the plan job calls check_production_tag from lib.sh, which accepts only ^v?[0-9]+\.[0-9]+\.[0-9]+$, whatever the pre-release flag says.

• The plan job now checks out scripts/release at the workflow's own commit to get lib.sh.
• The selftest covers 11 tags. INV-A5 requires the call.
• I also ran the plan step's script itself in 11 event/variable combinations, with a stub gh. Removing the guard makes the two unflagged-rc cases publish.

p6 #1, p1 #7, p3 #6, p2 #4: AGENTS.md "Cutting a release". There is a new "Packages (release-packages.yml)" subsection. It covers:

• the production variable gate and the setup order (create build, add reviewers, add branch/tag rules plus a tag ruleset, then add the secrets and APT_SIGNING_FPR);
• the crate version staying X.Y.Z, also for an rc;
• a pre-release promoted to final needing release_type=public tag=…;
• "Re-run failed jobs" versus "Re-run all jobs". The latter is refused as a different build, so after the artifacts expire the fix is a dispatch limited to the missing deb_dists/rpm_distros;
• recovering from a stale deb-s3 lockfile;
• the staging rehearsal to run before tagging;
• removing "Coming soon" after the first production publish.

One deviation: you asked for "final releases go red until production is configured". With the variable gate below, they no longer go red. The run stays green and shows a notice.

p6 #2: the PR trigger didn't cover every packaging input. I took the preferred route, with one deliberate narrowing.

• The trigger and INV-A5 now include src/bin/**, README.md, datasets/datasets.json, datasets/random-100/** and experiments/configurations/**.

• I used src/bin/** rather than all of src/**. The rest of src/ is already compiled --features release-static by release-binaries.yml's PR run (INV-A4), so adding it would only add four cold static package builds to every Rust PR.

• Binary set: INV-A5 now requires the packaged binaries to equal cargo's bin targets and lib.sh's REQUIRED_BINS:

  • debian/rules BINS+LIBEX;
  • the spec's %install loops;
  • cargo's bin targets, i.e. the [[bin]] entries plus anything auto-discovered under src/bin.

  So a renamed bench_hdf5.rs fails in CI even on a PR where no package job runs. Mutation-checked: removing src/bin/** from the trigger, removing bench_hdf5 from REQUIRED_BINS, adding a new src/bin/*.rs, and dropping bench_hdf5 from the spec each turn INV-A5 red.

• bench_hdf5 is now in REQUIRED_BINS, and verify.sh checks it unconditionally. No platform excluded it. The archive selftest's stand-in binary was updated to match.

• I corrected the INV-A5 comment to say which inputs it covers and why.

Optional items done

• p1 Add 1M,10M,20M,40M,100M,200M laion datasets #3, p7 Add laion 400M dataset to datasets.json file #4: production is gated behind the repository variable vars.PACKAGES_PRODUCTION == 'true'. Without it:

  • a final release builds and tests, publishes nothing, and shows a ::notice::; no build environment is auto-created;
  • release_type=public is refused.

  This is documented in the header and in AGENTS.md. Locally, removing the gate makes a final release publish while the variable is off.

• p1 Update.redisearch hybrid benchmarks #2, p2 adding the capability to retry a download.  #5: the APT verify now finds our stanza (package, version and architecture) with deb_index_entry in lib.sh. It requires that stanza's SHA256 to equal the run's .deb, and checks with curl -fsSI that the pool Filename is served. The selftest uses a shared index where another package carries the same version string; ignoring Package turns it red.

• p1 adding the capability to retry a download.  #5: createrepo, ruby and deb-s3 are now installed before configure-aws-credentials and the key import. ruby/setup-ruby is pinned to 651ff0d… (v1.329.0, released 2026-10-12). The .gem is checked against sha256 b052d7f0…, recorded today from the deb-s3 GitHub release. deb-s3's own gem dependencies still come from rubygems unpinned; that's noted as a follow-up.

• p1 Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: publish now checks out scripts/release at github.sha, the same commit as the workflow's guards and the selftest job, while the packages come from the planned commit (the tag). This way a publish-tooling bug can be fixed on master and an existing tag can still be finished with release_type=public tag=…, and the scripts that run are always the ones that were just selftested. On a release event, the two commits are the same.

• p1 Add REDIS_KEEP_DOCUMENTS and --only-configure flag #8: PRERELEASE was renamed to EVENT_PRERELEASE.

• p3 Added optimization for OpenSearch engine which were missing  #7: rpm --addsign now gets --define "__gpg …" --define "_gpg_name $fpr", and nothing is written to ~/.rpmmacros.

• p3 Add laion 400M dataset to datasets.json file #4: the one-directional residual race is documented in the publish-yum.sh header and the workflow header.

• p1 Add laion 400M dataset to datasets.json file #4 / Add laion-img-emb-768-1G-cosine to datasets #6: the header now says that the tag deployment rule alone doesn't stop a modified workflow (it recommends a tag ruleset, with required reviewers as the real gate), and that a third queued publish cancels the pending one.

• p7 Add 1M,10M,20M,40M,100M,200M laion datasets #3: INV-A5 also refuses https://$BUCKET / https://${BUCKET anywhere in the workflow, so a two-line URL can't get past the per-line scan.

Deferred (follow-ups, not in this PR)

• p3 Add 1M,10M,20M,40M,100M,200M laion datasets #3: automatic retry of the YUM post-check. For now it fails loudly with the re-run instruction.
• p3 adding the capability to retry a download.  #5 / p7 Update.redisearch hybrid benchmarks #2: the --pkglist scope. createrepo_c --update indexes every .rpm object in the directory. Before the first production publish, the packaging team should confirm that every .rpm in the 8 production rpm/<distro>/<arch>/ directories is meant to be indexed. This is listed in the PR body.
• p2 Add 1M,10M,20M,40M,100M,200M laion datasets #3: the fake deb-s3 should accept an identical re-upload and refuse different bytes under the same name, which is what real deb-s3 does.
• p4: third-party licence notices (HDF5, OpenSSL, zlib), shared with the Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369 archives; an RPM Requires allow-list in build-rpm.sh; a packaging-revision scheme; the bullseye/bookworm sort order; the spec changelog version and maintainer email, plus the SPDX licence name; shipping the .py files under experiments/configurations.
• p5: pre-installed curl/ca-certificates in the deb test container; Recommends: ca-certificates and an HTTPS probe; --expect-version on release runs; an RPM dist-tag check; choosing the libdir by package type.
• p6 Add 1M,10M,20M,40M,100M,200M laion datasets #3– adding the capability to retry a download.  #5: the Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369 refactors (release-binaries path filter, toolchain pin in lib.sh, a shared clean-tree helper); the global safe.directory '*' in the build scripts; the spec changelog.
• p7 adding the capability to retry a download.  #5: INV-A4's narrow run: scan, which comes from Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369.

The staging rehearsal at 0fd1f3a is in the PR body.

[fcostaoliveira]

Replying to the automated review's pass on 0fd1f3a. The fixes are in 03fa177.

1. A stale repomd.xml.asc. I checked production first. rpm/{el8,el9,el10,amzn2023}/…/repodata/repomd.xml.asc returns 403 both through the CDN and from the S3 origin. With public-read objects and no listing rights, that means the object does not exist, so no directory's metadata is signed today.
   • I went with the refusal. publish-yum.sh now stops, before writing anything, if the synced repodata/ holds any repomd.xml.* signature. The error says it must learn to re-sign repomd.xml before it can publish there. The header documents this.
   • A selftest case puts a repomd.xml.asc in the fake bucket and checks three things: the run refuses, the error names the file, and no cp/sync to S3 happened. Removing the guard fails all 3 checks.
2. Pinning. aws-actions/configure-aws-credentials is now pinned to the v4.3.1 commit (7474bc4…), and I agree it matters most there. The rustup bootstrap (curl sh.rustup.rs | sh) is left as a follow-up, as you suggest: rustup verifies the toolchain it installs, and pinning the installer touches Publish prebuilt Linux (x86_64, aarch64) and macOS arm64 binaries as release assets #369's shared ensure_release_toolchain.
3. AGENTS.md wording. Fixed. It now says the reused artifacts are unsigned, that the publish job signs them each time, and that the identity check ignores signatures.
4. --retain-old-md 1 vs. the header. Fixed. The header now says it prunes only the local copy, and that old generations accumulate in S3 because nothing deletes them remotely.

The full-matrix staging rehearsal at 03fa177 is in the PR body.

[fcostaoliveira]

Thanks — the automated pass on 03fa177 finds nothing pre-merge. I'm leaving both optional items as follow-ups:

1. debian/rules copies from ./target/release even when CARGO_TARGET_DIR is set. This only affects local builds, because CI never sets it. The fix is to export it in build-deb.sh and install from $(or $(CARGO_TARGET_DIR),target)/release, mirroring the spec, plus an optional post-build --version sha check. I'm not doing it on this branch because it changes debian/rules and would need another full-matrix rehearsal; it can be filed separately.
2. The RPM %changelog version is already in the round-2 deferred list (p4/p6).

[fcostaoliveira]

Delta review at 03fa177: addressed in 20808ec. Each new guard was mutation-checked: I broke it and confirmed the selftest or INV-A5 turns red for that reason.

Blocking

d6 #1: ci.yml didn't rerun INV-A5 when its inputs change. Fixed.

• .github/workflows/release-packages.yml, debian/** and rpm/** are now in both the push and pull_request path lists.
• The self-enforcing list in harness_invariants.rs requires each of them exactly twice.
• Mutations:
  • dropping rpm/** from the pull_request list fails with "ci.yml's push and pull_request paths must both list - 'rpm/**'";
  • dropping release-packages.yml from the push list fails the same way (left: 1).

d1 (workflow)

• Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: a master dispatch can't finish a tag whose APT side is already published. I took the narrowing route, not reproducible builds.
  • Why not reproducible builds: byte-identical rebuilds would need more than the changelog date. Cargo and the linker would have to be deterministic across runner image updates, which a selftest can't prove. Proving it takes two real builds days apart.
  • What changed:
    • The comment above the publish checkout now says a master dispatch rebuilds, and can only publish targets nothing of this version has reached yet.
    • The AGENTS.md re-run entry now spells out what works:
      • Re-run failed jobs works, but uses the run's own scripts, so it can't pick up a fix to them.
      • A new dispatch can only cover untouched targets: both lists must be non-empty, and APT publishes before YUM.
      • APT-complete, or one RPM arch published: once the run can't be re-run, this needs a manual publish with the packaging team.
  • Follow-up: reproducible deb/rpm rebuilds. Take the changelog date and SOURCE_DATE_EPOCH from the tag commit, then prove with a rebuild days later.
• Update.redisearch hybrid benchmarks #2: dry-run ran the tag's publish scripts. Fixed. The ref: is dropped, so it checks out github.sha, as publish does.
• Add 1M,10M,20M,40M,100M,200M laion datasets #3: deb-s3's gem dependencies float. Deferred to a follow-up issue: a Gemfile.lock with checksums, installed with bundle install --frozen.

d2 (APT)

• Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: a signing-subkey key would be refused. Fixed.
  • The new lib.sh function gpg_status_signed_by accepts a VALIDSIG whose signing key or primary key (the last field) is the pinned fingerprint. GOODSIG alone, or a fingerprint prefix, doesn't count.
  • Both publish-apt.sh and the verify step use it.
  • The selftest generates a certify-only primary with a signing subkey, publishes twice, and asserts the signature really came from the subkey.
  • Mutation: matching the first field only gives 2 red.
• Update.redisearch hybrid benchmarks #2: Release.gpg without InRelease counted as new. Fixed. publish-apt.sh refuses that codename before deb-s3 runs. Selftest covered; mutation gives 2 red.

d3 (YUM)

• Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: "nothing was written" was false after a partial publish. Fixed both ways.
  • Preflight: before any upload, publish-yum.sh now lists every target directory's repodata/ and refuses if any repomd.xml.* is there.
  • In-loop check kept: it still runs after the sync, for a signature that appears later, and now says "directories before it in this run may already be published".
  • Test: the selftest signs only el9/x86_64, which sorts last, and asserts nothing reached amzn2023/.
  • Errors: a repodata/ listing error fails closed.
• Update.redisearch hybrid benchmarks #2: the --retain-old-md comment was wrong. Corrected: old generations accumulate both in S3 and in each publish's download. The include-only sync is not done; it's listed below as deferred.

d4 (packaging)

• Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: test-package.sh's binary lists were outside INV-A5. Fixed. Its for b in …; do loops are now one more set in the binary-set equality check. Mutation: dropping bench_hdf5 from the loop fails INV-A5.

d5 (tests)

• Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: same as d2 Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1.
• Update.redisearch hybrid benchmarks #2: no test for an aws s3 ls error. Fixed.
  • The fake aws takes a new FAKE_LS_FAIL regex that returns AccessDenied with exit 254.
  • Covered for:
    • APT's InRelease probe (mutation: rc != 0 treated as absent gives red);
    • YUM's preflight (mutation gives red).
• Add 1M,10M,20M,40M,100M,200M laion datasets #3: the production gates were only checked for the presence of a string. Fixed.
  • Moved to lib.sh: the plan gate (production_gate) and the signing pin (check_signing_fpr) are now functions there, with selftest tables:
    • for the gate: on/off × release/dispatch, and yes ≠ true;
    • for the pin: whitespace and case, a missing pin on build, a wrong key, and a short key id.
  • INV-A5 call sites: INV-A5 now requires each workflow call site verbatim on a non-comment line, including its || { …; exit 1; }. It does the same for publish-apt.sh's signer check.
  • Mutations:
    • commenting out the gate fails INV-A5;
    • appending || true to the pin fails INV-A5;
    • removing the publish-apt signer check fails INV-A5;
    • opening the gate gives 2 red in the selftest;
    • making the pin optional on build gives red.

d6 (docs)

• Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: see Blocking.
• Update.redisearch hybrid benchmarks #2: the runbook dispatch can't finish a YUM-only remainder. Fixed by rewording the runbook, as in d1 Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1. The suggested none option for either list is deferred.
• Add 1M,10M,20M,40M,100M,200M laion datasets #3: unmarked rc. The runbook now says an rc must be marked as a pre-release. Once production is on, one that isn't fails at the plan job, before anything is built.
• Add laion 400M dataset to datasets.json file #4: stale PR body. Updated:
  • "Staging rehearsed; production off until vars.PACKAGES_PRODUCTION=true";
  • always signed, with the InRelease signer check;
  • the APT_SIGNING_FPR pin.
• adding the capability to retry a download.  #5: lock error text. AGENTS.md and the publish-apt.sh header now quote both the log line and the actual failure, uncaught throw "Unable to obtain a lock after 60, giving up.".

d7 (maintainer)

• Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1: leading zeros. Fixed. check_production_tag now uses ^v?(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$, and 00.2.0 and 0.02.0 are in the refuse list. Mutation gives 2 red.

Deferred (follow-ups)

• d1 Add 1M,10M,20M,40M,100M,200M laion datasets #3: lockfile with checksums for deb-s3's dependencies.
• d1 Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1 (preferred route): reproducible rebuilds.
• d6 Update.redisearch hybrid benchmarks #2 (alt): explicit none for deb_dists / rpm_distros.
• d3 Update.redisearch hybrid benchmarks #2 (optional): sync only the metadata files that repomd.xml names.
• Carried over from earlier rounds: the YUM post-check auto-retry, the --pkglist scope, and the debian/rules CARGO_TARGET_DIR and spec changelog points.

Gate at 20808ec

• shellcheck -x clean on scripts/release/*.sh; actionlint clean on release-packages.yml and ci.yml.
• Package selftest: 116 passed, 0 failed (88 before).
• Archive selftest: 95 passed, 0 failed.
• make agent-check passes; harness_invariants 44/44.
• Staging rehearsal, full matrix, because the publish scripts changed: https://github.com/redis/vector-db-benchmark/actions/runs/40888657747 at 20808ec, green (47 success, 1 skipped). The YUM preflight and the new signer check ran against the existing staging codenames ("existing InRelease is signed by FD18…AD25").

Merged master (#325)

origin/master (6970fda) is merged into the branch as 5e1345a, a merge commit. It merged without conflicts.

• No new binaries: the src/bin changes are inside the existing vector_db_benchmark bin.
• No debian, rpm, release-script or workflow changes. The Makefile only gains integration-test-redis-sq8.
• datasets/datasets.json changed; it is already in the package trigger paths, so the PR's package jobs rebuilt and installed with it.
• CI on 5e1345a: 35 success, 2 skipped (production publish and release upload, as expected). That includes make agent-check-lint/-test (harness_invariants), "Self-test the publish scripts", and the jammy/el9 build and install cells. The build and publish scripts are unchanged, so the rehearsal above still covers them.