Skip to content

Retain previous versions in the published pool - #17

Merged
kn4oqw-clint merged 2 commits into
mainfrom
fix/apt-pool-retention
Aug 2, 2026
Merged

Retain previous versions in the published pool#17
kn4oqw-clint merged 2 commits into
mainfrom
fix/apt-pool-retention

Conversation

@kn4oqw-clint

@kn4oqw-clint kn4oqw-clint commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The apt-side half of KN4OQW/waypoint#221. The engine-side half is KN4OQW/waypoint#224.

The problem

waypointd's stack updater is confirm-or-revert: on a failed health gate it rolls back with apt-get install <the previously installed versions>. That only works if those versions are still downloadable — and they were not. This script did rm -rf "$OUT" and rebuilt the tree from only the current build's .debs, so the published repo carried exactly one version of each package.

Observed live on the bench Pi: an update installed fine, failed its health gate, and could not roll back —

stackupdate: REVERT FAILED (...): E: Version '0.1.0' for 'waypoint-stack' was not found

leaving the node stranded on the new version.

Retention

The pool now keeps the current version plus three previous ones, per package per architecture (KEEP_VERSIONS, default 4). ~3 MiB per version set across both arches, so size is not a constraint.

Pages deploys from a workflow artifact, not a branch, so the previously published .debs exist in exactly one place: the live site. Each publish reads the live Packages indices, fetches what they list, and merges the new build over it. A failed fetch is fatal — degrading quietly to a one-version pool would look like a successful publish while silently restoring the bug. ALLOW_EMPTY_POOL=1 is the explicit opt-out for a genuine first publish.

The defect retention exposed

Having exactly one version was hiding a second problem. Apt's candidate is the highest-sorting version available, and daemon versions were 0~git<sha>+wp1 — a git SHA is not ordered:

$ dpkg --compare-versions '0~gitfd4a6a4+wp1' gt '0~git5527546+wp1'; echo $?   # 0 — "greater"
$ dpkg --compare-versions '0~git2b480aa+wp1' gt '0~git9751c6e+wp1'; echo $?   # 1 — "less"

With one candidate there is nothing to order. With several, apt would pick the highest-sorting SHA rather than the newest build, and apt list --upgradable — which drives the updater's entire plan — could present an older build as an update. Turning on retention without fixing this would have broken the upgrade path.

Versions now carry the pinned commit's date: 0~git<YYYYMMDD>.<sha>+wp<n>.

Why the epoch

Adding the date is not by itself an increase. dpkg compares the leading non-digit run first, so a date-prefixed version sorts below the old form whenever the SHA began with a letter:

$ dpkg --compare-versions '0~git20250709.c72b989+wp1' gt '0~gitc72b989+wp1'; echo $?
1

waypoint-mmdvmhost and waypoint-m17gateway were both in that position and could never have upgraded. Epoch 1: clears it unconditionally. The metapackage's exact-version relations name the epoch too, since a relation that omits it matches nothing.

Two corrections that come with rewriting the versions

  • waypoint-mmdvmhost was mislabelled. It declared 0~gitfd4a6a4+wp1 while pins.env has pinned 71e598c since 306f304 bumped it — that commit updated the pin but not the nfpm config. CI has been building 71e598c and publishing it under the old SHA, so the version string never changed and no node was ever offered the deferred station identification. Its version now names the commit it is actually built from. Worth a look, since this is a behaviour change beyond #221.
  • README pin table had drifted the same way, and listed DAPNETGateway as pending though Pin, build, and package DAPNETGateway #15 pinned and packaged it.

Verification

Run locally against a throwaway signing key and a local HTTP server standing in for Pages:

  • Bootstrap publish with no pool to inherit (ALLOW_EMPTY_POOL=1) — succeeds.
  • Second publish inheriting the first — pool carries both generations, and the Packages index lists two versions of the package. This is the thing that was missing.
  • Rolled forward six more generations — pool holds at exactly 4, keeping the newest.
  • Unreachable BASE_URL without the opt-out — exits 1 and writes no dists/ tree.
  • Pruning drives ordering through dpkg --compare-versions, verified to keep 0.10.0 over 0.9.0 (a filename sort gets this backwards) and to prune the old sha-only version in favour of the four date-scheme ones. Per-arch groups prune independently.
  • All 13 packages build under the pinned nfpm 2.43.0, and every metapackage dependency pin resolves exactly against the built versions.
  • shellcheck clean.

CI has not run the real publish path — the first merge to main will be the first live inherit, and it will inherit the 13 currently-published sha-only .debs, which is what gives nodes on today's versions a rollback target.

waypointd's stack updater is confirm-or-revert: it health-gates an update and,
on failure, rolls back with `apt-get install <the previously installed
versions>`. That only works if those versions are still downloadable, and they
were not. publish-apt.sh rebuilt the tree from only the current build's .debs, so
the published repo carried exactly one version of each package. A node whose
update installed cleanly and then failed its health gate had nowhere to go:

    stackupdate: REVERT FAILED (...): E: Version '0.1.0' for 'waypoint-stack'
    was not found

and was stranded on the new version (KN4OQW/waypoint#221).

The pool now keeps the current version plus three previous ones, per package per
architecture. Three covers a node that sat out a couple of updates and still
needs a way back; the cost is trivial, since a full version set of every package
is about 3 MiB across both arches.

Pages deploys from a workflow artifact rather than a branch, so the previously
published .debs exist in exactly one place: the live site. Each publish therefore
reads the live Packages indices, fetches what they list, and merges the new build
over it. Going through the indices rather than guessing filenames means this
follows whatever is actually published. A failed fetch is fatal: degrading
quietly to a one-version pool would look like a successful publish while silently
restoring the bug, so only an explicit ALLOW_EMPTY_POOL=1 permits it, for the
genuine first publish of a new repository.

Retention exposes a second defect that having one version had been hiding. Apt's
candidate is the highest-sorting version available, and the daemon versions were
`0~git<sha>+wp1` — a git SHA is not ordered, so with more than one version in the
pool apt would pick the highest-sorting SHA rather than the newest build, and
`apt list --upgradable` (which drives the updater's entire plan) could present an
older build as an update. The versions now carry the pinned commit's date,
`0~git<YYYYMMDD>.<sha>+wp<n>`, which makes "newest" and "highest" the same thing.

That change needs an epoch to land. dpkg compares the leading non-digit run
first, so a date-prefixed version sorts BELOW the old form whenever the SHA began
with a letter:

    $ dpkg --compare-versions '0~git20250709.c72b989+wp1' gt '0~gitc72b989+wp1'
    $ echo $?
    1

waypoint-mmdvmhost and waypoint-m17gateway were both in that position and could
never have upgraded. Epoch 1 clears it unconditionally, and the metapackage's
exact-version relations name it too, since a relation that omits the epoch
matches nothing.

Two corrections come along with rewriting these version strings. waypoint-
mmdvmhost was labelled 0~gitfd4a6a4+wp1 while pins.env has pinned 71e598c since
306f304 bumped it — CI has been building 71e598c and publishing it under the old
SHA, so the version never changed and no node was ever offered the deferred
station identification. Its version now names the commit it is actually built
from. The README's pin table had drifted the same way and listed DAPNETGateway as
pending, though #15 pinned and packaged it.

Refs KN4OQW/waypoint#221
@kn4oqw-clint
kn4oqw-clint merged commit 2cda6c6 into main Aug 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant