feat(e2e): add the ha and upgrade scenarios on Vultr, the exit of M3

CLAUDE.md

@@ -12,9 +12,10 @@ Providers, in order:
 2. **Hetzner Cloud** is second.
 3. **AWS** comes later. It must remain possible without changes to the core.
 
-**Status:** M0 Foundation (2026-09-27), M1 Vultr infrastructure (2026-09-28) and M2 Nomad bootstrap (2026-10-07) are
-complete. v0.1.0, the first release that builds a Nomad cluster, was tagged on 2026-10-07: archives, `.deb` and `.rpm`
-packages, a Homebrew cask (`ingvarch/tap/tent`), and the tent-node binaries that its nodes download.
+**Status:** M0 Foundation (2026-09-27), M1 Vultr infrastructure (2026-09-28), M2 Nomad bootstrap (2026-10-07) and M3
+Day-2 operations (2026-10-14) are complete. v0.1.0, the first release that builds a Nomad cluster, was tagged on
+2026-10-07: archives, `.deb` and `.rpm` packages, a Homebrew cask (`ingvarch/tap/tent`), and the tent-node binaries
+that its nodes download.
 - The skeleton is in place: Go module, `tent version`, Makefile, lint rules, CI on Linux, macOS and Windows, and a
   GoReleaser release pipeline. The repository is public, the release secrets are set, and the archives and packages
   ship third-party licence notices (ADR-0020). Renovate updates the Go modules and GitHub Actions.
@@ -86,14 +87,15 @@ packages, a Homebrew cask (`ingvarch/tap/tent`), and the tent-node binaries that
     Vultr on 2026-10-07 (spike v12, run `9pxbqn`); its one unexpected row was a fault of the script, which is fixed.
   - M2.9 added the E2E suite on Vultr (`test/e2e`, `test/e2e/vultrapi`, `test/e2e/janitor`, `hack/e2e-janitor`,
     ADR-0034): the scenario `smoke` builds a cluster per image (ubuntu-24.04 and ubuntu-26.04) and is built to check
-    the four exit criteria of M2, and a janitor deletes the objects of `e2e-` clusters that are older than 3 hours.
+    the four exit criteria of M2, and a janitor deletes the objects of `e2e-` clusters that are older than 4 hours.
     `make e2e` runs it from the maintainer's machine, not in CI. The labels `tent/e2e` and `tent/e2e-run` are removed.
     The runs of 2026-10-07 found four faults, all fixed: a node refuses a tent-node of another version than its
     tent's (`make e2e` builds both with one version); `update` asked for an intro token before Nomad's keyring had a
     key (the Nomad step now waits for it); the suite's `validate` lacked `--allow-single-server`; and the probe's one
     control site, `archive.ubuntu.com`, did not answer over IPv4 (it tries three now). The next three runs passed on
     both images (`r7l48w`, `58sglh`, `g57k85`): that is the exit of M2.
-- M3 Day-2 operations is in progress, in parts M3.1 to M3.9 (`docs/roadmap.md`).
+- M3 Day-2 operations was built in parts M3.1 to M3.9 (`docs/roadmap.md`) and is complete since 2026-10-14 (runs
+  `qkmdrs` and `hik34s`).
   - M3.1 added the rollout decisions (`internal/rollout`, ADR-0035; maintainer decisions 33 to 35): `Next(State,
     Mode)` returns one step of a rolling update (`Roll`) or of a removal of surplus nodes (`Shrink`) from what the
     cloud and Nomad report, and reaches nothing itself. A simulator with golden step sequences, invariants and a
@@ -210,6 +212,20 @@ packages, a Homebrew cask (`ingvarch/tap/tent`), and the tent-node binaries that
       `latest` refuses when the earlier backups are of more than one rebuild.
     - `nomadops` has `Jobs`, `TokenAccessor` and the class `ErrDenied`. `delete cluster` deletes the backups and the
       record. The run on Vultr gave a rebuilt server its old private address (platform notes §3.16).
+  - M3.9 added the E2E scenarios `ha` and `upgrade` and a probe (`test/e2e` `ha_test.go`, `upgrade_test.go`,
+    `cluster_test.go`, `probe_test.go`, `scenario_text_test.go`; ADR-0043; maintainer decision 51): `make e2e` runs
+    `smoke`, then `ha` and `upgrade` in parallel (at most 10 machines), and `E2E_RUN` picks scenarios.
+    - `ha` changes the specs, rolls the servers and then the workers, restarts an old worker between the rolls so that
+      it finds the new servers through `refresh-join`, and backs up Nomad, deletes the servers, rebuilds them and
+      restores the backup. `upgrade` rolls a combined cluster from Nomad 2.0.6 (`E2E_UPGRADE_FROM`) to the channel's
+      version.
+    - A probe polls the servers for a leader and the service for a healthy allocation during each roll and fails the
+      step on a breach of its limits (ADR-0043). The timeout of `make e2e` is 180 minutes, and a step ends 20 minutes
+      before it so the cleanups still delete the clusters.
+    - Decision 51: no command takes an upgrade back once the roll's first new server has joined (ADR-0041).
+    - Runs on Vultr (platform notes §3.16): `w28exz` found two faults of the suite (a creation date without a zone,
+      and checks that cannot be read after a hard stop of a server), `ae5dhs` passed `ha` and `upgrade`, and
+      `qkmdrs` and `hik34s` passed the full run. The channel lists 2.0.6 as tested since `ae5dhs`.
 
 ## Read before changing anything
 

Makefile

@@ -36,9 +36,11 @@ test:
 
 # e2e runs the E2E suite on Vultr from this machine (test/e2e/README.md). It runs dev-upload, which needs the R2 keys
 # and TENT_DEV_S3_URL, so the nodes get the tent-node of the same build as bin/tent, and it needs VULTR_API_KEY.
+# E2E_RUN, when set, is the -run pattern, for example make e2e E2E_RUN='TestHA|TestUpgrade'.
 e2e:
 	@lines="$$(SHELL=/bin/sh $(MAKE) -s --no-print-directory dev-upload)" && eval "$$lines" && \
-		E2E_TENT="$(CURDIR)/bin/tent" go test -tags e2e -count=1 -v -timeout 90m ./test/e2e
+		E2E_TENT="$(CURDIR)/bin/tent" go test -tags e2e -count=1 -v -timeout 180m \
+			$(if $(E2E_RUN),-run "$(E2E_RUN)") ./test/e2e
 
 golangci-lint-version:
 	@command -v golangci-lint >/dev/null 2>&1 || { \

docs/adr/0034-e2e-suite-on-vultr.md

@@ -1,6 +1,8 @@
 # ADR-0034: The E2E suite on Vultr
 
-- **Status:** Accepted
+- **Status:** Accepted; amended by [ADR-0043](0043-e2e-scenarios-ha-and-upgrade.md) (the scenarios `ha` and `upgrade`, a
+  probe of the leader and the service during a roll, `E2E_RUN`, the timeout of 180 minutes, the janitor's age of 4
+  hours)
 - **Date:** 2026-10-07
 - **Deciders:** ingvarch
 - **Related:** amends [ADR-0012](0012-testing-strategy.md), [ADR-0014](0014-vultr-first-provider-and-e2e.md),

docs/adr/0041-upgrading-a-clusters-nomad-version.md

@@ -95,7 +95,8 @@ A local lab on 2026-10-10 ran Nomad 2.0.6 and 2.0.7 together in both directions
 14. **A combined group is the one exception.** A new combined node runs a client of the new version while the other
     servers are old, and its create is not held back, since a combined group has no other order. The guide advises
     against a client ahead of its servers because new client features need new servers. The local run covered a newer
-    client with older servers and not a combined agent. M3.9's `upgrade` scenario runs it on Vultr.
+    client with older servers and not a combined agent. M3.9's `upgrade` scenario ran it on Vultr
+    (`w28exz`, `ae5dhs`): a 2.0.7 combined node joined 2.0.6 ones and the roll ended.
 15. **`rollout.CheckNewNode` holds the version rules of a new node** for `Next`, `serversHold` and `update`. A refusal
     says `the cluster's version is`, because the pin can differ from the version that a check holds. `validate cluster`
     takes the same phrase and warns that `update` has not applied the specs while the completed spec differs.
@@ -126,19 +127,21 @@ A local lab on 2026-10-10 ran Nomad 2.0.6 and 2.0.7 together in both directions
   lowered joins later ahead of the pin, and `rolling-update` refuses until the version is raised again or `update`
   removes the machine.
 - **No command takes an upgrade back once the roll's first new server has joined** (Q14 of
-  [architecture §18](../architecture.md#18-open-questions), open). The way back is by hand: delete that machine in the
-  cloud, wait until its peer is gone, and run `tent update cluster`.
+  [architecture §18](../architecture.md#18-open-questions), answered (a) on 2026-10-13: decision 51, with no lab for
+  (b)). The way back is by hand: delete that machine in the cloud, wait until its peer is gone, and run `tent update
+  cluster`.
 - **`validate cluster` exits with 2 from `upgrade --yes` until the roll has ended,** since the nodes differ from the
   specs.
-- **Not verified on a real cluster:** whether autopilot's report holds a version for every Raft peer at the read
-  before the first client of a first build. M3.9's run on Vultr shows it.
+- **Autopilot's report held a version for every Raft peer** at the read before the first client of a first build, in
+  the create of the `ha` cluster on Vultr, in both runs `w28exz` and `ae5dhs` (2026-10-14).
 - **The tests know two Nomad versions, one patch apart.** A move to the next minor version is tested as a rule and has
   never run.
 
 ### Follow-ups
 
-- **M3.9** runs the `upgrade` scenario on Vultr from 2.0.6 to 2.0.7 with a combined cluster, and may then add 2.0.6 to
-  the channel's `tested` list.
+- **M3.9** ran the `upgrade` scenario on Vultr from 2.0.6 to 2.0.7 with a combined cluster (runs `w28exz` and
+  `ae5dhs`, [ADR-0043](0043-e2e-scenarios-ha-and-upgrade.md)), and 2.0.6 is in the channel's `tested` list since
+  `ae5dhs`.
 - A flag for the target version, a flow with a third version and an in-place upgrade of the binary
   ([ADR-0005](0005-immutable-nodes-and-nomad-aware-rollouts.md)) are left for later.
 

docs/adr/0042-backups-in-the-state-store-and-restore-after-a-loss-of-the-servers.md

@@ -140,8 +140,10 @@
 
 ### Follow-ups
 
-- M3.9's `ha` scenario runs a backup and a restore on Vultr. `tent backup delete`, `backup create --keep N` and a
-  restore inside `update` wait for issues.
+- M3.9's `ha` scenario ran a backup and a restore on Vultr in run `ae5dhs` (2026-10-14) with three servers: Vultr gave
+  the rebuilt servers one of their old private addresses back, the workers came back on their own, `validate` passed
+  after the 6-minute wait and no forced roll was needed ([ADR-0043](0043-e2e-scenarios-ha-and-upgrade.md)).
+  `tent backup delete`, `backup create --keep N` and a restore inside `update` wait for issues.
 
 ## Alternatives considered