Skip to content

Battery green: per-machine identity, journalctl on the appliance, combined-page restore that actually restores - #975

Merged
VijitSingh97 merged 11 commits into
develop-v2from
fix/battery-green
Aug 14, 2026
Merged

Battery green: per-machine identity, journalctl on the appliance, combined-page restore that actually restores#975
VijitSingh97 merged 11 commits into
develop-v2from
fix/battery-green

Conversation

@VijitSingh97

Copy link
Copy Markdown
Collaborator

Root-causes and fixes every failing leg of the KVM battery. Closes #966, closes #967, closes #968, closes #969, closes #970.

Product fixes

Battery fixes (the asserts were wrong, the product was right)

  • media phase: pre-seed staged an invalid Tari address; sticks hot-attached without removable='on' (the RM=1 physical-presence contract); the "changed" wallet was byte-identical to the provisioning wallet.
  • install phase: the restore leg backed up an unprovisioned wizard-state machine; it now provisions first and settles the stack before the backup.
  • reset phase: the container-store probe now checks a pulled image, not the baked wizard archive the post-reset boot legitimately reloads. The machine-id inequality assert stays strict — it caught Every machine flashed from one release boots with the same machine-id #966.

Ponytail review

One finding, deliberately not applied: restore_carry_dir() could inline at its two call sites (−4 lines) — not worth re-touching code three KVM phase runs verified.

Verification

  • make lint + make test green (new tier-2: installer restore door, backup retry, machine-id empty-adopt).
  • KVM phases green in isolation on gouda: media 8/8, install 42/0 (identity carry proven end to end: original wallet AND original onion on a fresh disk), provision migration-hold release confirmed in the guest journal.
  • Full battery (all phases + media + reset) running on gouda now; merges only on green.

🤖 Generated with Claude Code

VijitSingh97 and others added 10 commits August 14, 2026 11:24
… through the /etc overlay

Two root causes from the KVM battery's reset and migration legs:

- dbus's postinst bakes /var/lib/dbus/machine-id into the image, and
  systemd's first-boot logic prefers that file — every machine flashed
  from one release booted with the SAME machine-id (one DHCP DUID per
  fleet, shared identity across handovers). The reset leg's inequality
  assert caught it. Standard symlink to /etc/machine-id instead, plus a
  verify-image check so it stays a symlink.

- pithead-machine-id answered PID 1's read-only transient bind with a
  bind mount of its own; provision_console_login() later mounts the
  /run-backed /etc overlay OVER /etc, and an overlay lower never shows
  a file bind mounted beneath it — /etc/machine-id read empty from
  that moment and journalctl found no journal (the migration leg's
  'deadlock' was this: the release line was in the journal, unreadable).
  The restore now mounts that same shared /etc overlay itself and
  writes through it.

Also: reset-leg store assert probes a pulled image (monerod), not the
baked wizard archive the post-reset boot legitimately reloads; adopt
path refuses loudly when there is no id to adopt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rightly rejected the dummy

The product did its job: consume_preseed_config validates the staged
config and 'harness-dummy-tari-address' is not a Tari address, so every
media leg fell back to the wizard before its first assert. Use the same
HARNESS_TARI fixture the wizard-submit legs pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… filter keys on RM=1

virsh attach-disk cannot express removable='on', so the guest saw a
fixed USB disk and pithead-media-config's lsblk RM==1 filter (the
physical-presence contract, same attribute real sticks report) rightly
ignored it — every apply/abort assert starved. attach-device with the
same removable declaration the install phase's virt-install disks use.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
new_wallet was byte-identical to HARNESS_WALLET (copy-paste), so the
'changed wallet' leg changed nothing: the diff rightly showed no wallet
line and the console assert could never match. Swap in the Monero
project's donation address — public, checksum-valid, verifiably not
the provisioning wallet.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…backing it up

The leg assumed a fully-provisioned machine, but every reinstall — keep
included — returns to the wizard, and 'pithead backup' rightly refuses
there (no .env, no onion keys; the #964 log-tail named it). Drive the
same wizard HTTP flow a human would, ack the credentials card only if
one appears (a keep-machine keeps its login and may show none), wait
for the stack, then take the backup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…all shortcut

The keep-everything shortcut fired on 'install-request=keep + no config
candidate' BEFORE the restore consume ran, so a restore uploaded with
keep as its erase policy (the disaster-recovery combo: config from the
archive, synced chains kept) installed bare and silently dropped the
operator's backup — the battery's restore leg watched the machine
'keep'-install a blank disk. Three guards:

- the shortcut also requires no staged restore-archive
- a rejected restore hands the install-request back with the form (a
  typo'd passphrase must not quietly install without the restore)
- a failed remote-node preflight does the same for the typed path

Proven at tier 4 by the install phase's restore leg (restore+keep) and
keep leg (bare keep). Harness: settle the stack before backing it up —
containers still being created slip past the backup's stop and churn
files mid-tar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t, not just its config

The docs promise the same Tor onion and dashboard history on the
restored machine, and the direct setup-page restore delivers it — but
the installer path extracted the archive onto the USB machine, staged
only config.json to the target's ESP, and the identity never crossed
(the battery watched the restored box mint a fresh onion). Restructure:

- restore_apply(): the shared acceptance (cap, decrypt verify, tar
  integrity, path audit, extract-validate-commit), one set of checks
  behind both doors; a config-only mode for the installer, where the
  restored tree belongs to the TARGET and decrypted keys must never
  rest on the stick
- installer boots park the accepted archive + passphrase in root-only
  tmpfs, stage them on the ESP instead of a bare config, and
  pithead-install carries them across; both are scrubbed from the
  stick after the install either way
- consume_preseed_restore(): the target's first boot decrypts and
  restores itself — config, onion keys, dashboard database — then
  spends the pair (the passphrase beside the archive is
  plaintext-equivalent), same hygiene as the config pre-seed

Tier 2 pins the installer door (config surfaces, stick untouched,
carry parked); the battery's restore leg proves the identity crossing
end to end. Harness: wait for the carried restore to land instead of
racing firstboot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Even with the stack stopped, tar can catch a teardown's last flush —
'file changed as we read it' is exit 1, pipefail fails the pipeline,
and the KVM bench watched a legitimate backup die on it. One loud,
bounded retry on a quiet tree; failing twice stays fatal. The partial
archive is removed between attempts exactly as on the way out (#551),
and tar's stderr stays on the terminal — the failing member IS the
diagnosis. Tier 2 pins it with a sudo fixture that fails the first tar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…naries

CVE-2026-39821 and CVE-2026-46600 flag both docker-compose v5.3.1 and
cosign v3.1.2 — fixed only in go1.25.13/1.26.6, and no upstream rebuild
exists yet (compose v5.4.0 ships go1.26.5, byte-verified). Same clearing
condition as the batch above: pin-watch flags the rebuilds, bump the
pins, drop these.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VijitSingh97

Copy link
Copy Markdown
Collaborator Author

⚠️ Active session lock (2026-08-14): an interactive Claude session is driving this PR to completion — full KVM battery running detached on gouda (/srv/code/pithead-rc-v2/kvm-battery-green.log), merge + bench deploy follow on green. Other sessions: do not merge, rebase, close, or push to fix/battery-green until this comment is superseded by a completion comment.

…irst boot mints

With the dbus bake gone (#966), systemd's next first-boot source in a
VM is the DMI product UUID — deterministic per VM, so the bare
inequality can never pass here even with the product right (real
hardware falls through to random). Assert the regenerated id is the
platform's or changed; only an id that is neither is carried owner
state.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VijitSingh97

Copy link
Copy Markdown
Collaborator Author

✅ Complete — supersedes the session-lock comment above.

This branch is done — no further work planned here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant