Retain previous versions in the published pool - #17
Merged
Conversation
waypointd's stack updater is confirm-or-revert: it health-gates an update and,
on failure, rolls back with `apt-get install <the previously installed
versions>`. That only works if those versions are still downloadable, and they
were not. publish-apt.sh rebuilt the tree from only the current build's .debs, so
the published repo carried exactly one version of each package. A node whose
update installed cleanly and then failed its health gate had nowhere to go:
stackupdate: REVERT FAILED (...): E: Version '0.1.0' for 'waypoint-stack'
was not found
and was stranded on the new version (KN4OQW/waypoint#221).
The pool now keeps the current version plus three previous ones, per package per
architecture. Three covers a node that sat out a couple of updates and still
needs a way back; the cost is trivial, since a full version set of every package
is about 3 MiB across both arches.
Pages deploys from a workflow artifact rather than a branch, so the previously
published .debs exist in exactly one place: the live site. Each publish therefore
reads the live Packages indices, fetches what they list, and merges the new build
over it. Going through the indices rather than guessing filenames means this
follows whatever is actually published. A failed fetch is fatal: degrading
quietly to a one-version pool would look like a successful publish while silently
restoring the bug, so only an explicit ALLOW_EMPTY_POOL=1 permits it, for the
genuine first publish of a new repository.
Retention exposes a second defect that having one version had been hiding. Apt's
candidate is the highest-sorting version available, and the daemon versions were
`0~git<sha>+wp1` — a git SHA is not ordered, so with more than one version in the
pool apt would pick the highest-sorting SHA rather than the newest build, and
`apt list --upgradable` (which drives the updater's entire plan) could present an
older build as an update. The versions now carry the pinned commit's date,
`0~git<YYYYMMDD>.<sha>+wp<n>`, which makes "newest" and "highest" the same thing.
That change needs an epoch to land. dpkg compares the leading non-digit run
first, so a date-prefixed version sorts BELOW the old form whenever the SHA began
with a letter:
$ dpkg --compare-versions '0~git20250709.c72b989+wp1' gt '0~gitc72b989+wp1'
$ echo $?
1
waypoint-mmdvmhost and waypoint-m17gateway were both in that position and could
never have upgraded. Epoch 1 clears it unconditionally, and the metapackage's
exact-version relations name it too, since a relation that omits the epoch
matches nothing.
Two corrections come along with rewriting these version strings. waypoint-
mmdvmhost was labelled 0~gitfd4a6a4+wp1 while pins.env has pinned 71e598c since
306f304 bumped it — CI has been building 71e598c and publishing it under the old
SHA, so the version never changed and no node was ever offered the deferred
station identification. Its version now names the commit it is actually built
from. The README's pin table had drifted the same way and listed DAPNETGateway as
pending, though #15 pinned and packaged it.
Refs KN4OQW/waypoint#221
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The apt-side half of KN4OQW/waypoint#221. The engine-side half is KN4OQW/waypoint#224.
The problem
waypointd's stack updater is confirm-or-revert: on a failed health gate it rolls back withapt-get install <the previously installed versions>. That only works if those versions are still downloadable — and they were not. This script didrm -rf "$OUT"and rebuilt the tree from only the current build's.debs, so the published repo carried exactly one version of each package.Observed live on the bench Pi: an update installed fine, failed its health gate, and could not roll back —
leaving the node stranded on the new version.
Retention
The pool now keeps the current version plus three previous ones, per package per architecture (
KEEP_VERSIONS, default 4). ~3 MiB per version set across both arches, so size is not a constraint.Pages deploys from a workflow artifact, not a branch, so the previously published
.debs exist in exactly one place: the live site. Each publish reads the livePackagesindices, fetches what they list, and merges the new build over it. A failed fetch is fatal — degrading quietly to a one-version pool would look like a successful publish while silently restoring the bug.ALLOW_EMPTY_POOL=1is the explicit opt-out for a genuine first publish.The defect retention exposed
Having exactly one version was hiding a second problem. Apt's candidate is the highest-sorting version available, and daemon versions were
0~git<sha>+wp1— a git SHA is not ordered:With one candidate there is nothing to order. With several, apt would pick the highest-sorting SHA rather than the newest build, and
apt list --upgradable— which drives the updater's entire plan — could present an older build as an update. Turning on retention without fixing this would have broken the upgrade path.Versions now carry the pinned commit's date:
0~git<YYYYMMDD>.<sha>+wp<n>.Why the epoch
Adding the date is not by itself an increase. dpkg compares the leading non-digit run first, so a date-prefixed version sorts below the old form whenever the SHA began with a letter:
waypoint-mmdvmhostandwaypoint-m17gatewaywere both in that position and could never have upgraded. Epoch1:clears it unconditionally. The metapackage's exact-version relations name the epoch too, since a relation that omits it matches nothing.Two corrections that come with rewriting the versions
waypoint-mmdvmhostwas mislabelled. It declared0~gitfd4a6a4+wp1whilepins.envhas pinned71e598csince 306f304 bumped it — that commit updated the pin but not the nfpm config. CI has been building71e598cand publishing it under the old SHA, so the version string never changed and no node was ever offered the deferred station identification. Its version now names the commit it is actually built from. Worth a look, since this is a behaviour change beyond #221.pendingthough Pin, build, and package DAPNETGateway #15 pinned and packaged it.Verification
Run locally against a throwaway signing key and a local HTTP server standing in for Pages:
ALLOW_EMPTY_POOL=1) — succeeds.Packagesindex lists two versions of the package. This is the thing that was missing.BASE_URLwithout the opt-out — exits 1 and writes nodists/tree.dpkg --compare-versions, verified to keep0.10.0over0.9.0(a filename sort gets this backwards) and to prune the old sha-only version in favour of the four date-scheme ones. Per-arch groups prune independently.shellcheckclean.CI has not run the real publish path — the first merge to
mainwill be the first live inherit, and it will inherit the 13 currently-published sha-only.debs, which is what gives nodes on today's versions a rollback target.