release: harden installation and lifecycle journeys - #208
Closed
Zariel wants to merge 18 commits into
Closed
Conversation
Derive the local kube-apiserver endpoint from its static pod manifest and use it while inspecting peers. This keeps control-plane upgrades progressing after Katl intentionally withdraws the node from the shared BGP endpoint.
Set KUBECONFIG from the immutable runtime profile only for root shells that did not make an explicit selection. This makes on-node kubectl useful after bootstrap without writing into immutable root state.
Document kubeadm API and kubeconfig availability as the default bootstrap success boundary. Explain that NotReady nodes and pending CoreDNS are expected until the user installs and operates a CNI.
Carry every inherited extension into each Kubernetes candidate so activation and reboot never depend on a previous generation. Require the local control-plane pods and node to become Ready before reporting an online upgrade healthy.
Keep the previous boot-proven generation as the default while a successful live candidate is armed through the standard one-shot trial path. Derive active-live state from the completed operation so later planning remains correct without claiming an unbooted generation is healthy.
Recognize an explicit boot of the recorded previous-known-good entry as recovery, make its loader entry durable, and retire the bypassed default. Expose selected-generation boot health so node status cannot present stale durable state as healthy.
Permit a running failed generation to become good again only after it passes boot health. Restore the selected fallback loader entry and remove the bypassed default from rollback candidates after transient recovery failures.
Detect and remove the checkout-owned domain's libvirt managed-save image before undefining it. This keeps persistent VM stop and reset composable without weakening ownership checks.
Add an explicit pre-install access handoff that applies the selected node public keys and starts ephemeral root SSH without accepting an install or touching disk. Expose it through katlctl and document the recovery workflow so ISO and PXE installers have a predictable, key-only debugging surface.
Replace recursive systemctl calls during endpoint activation shutdown with native PartOf relationships. This prevents systemd shutdown transactions from deadlocking and lets reboot, shutdown, and destructive reset reach firmware poweroff predictably.
Use the validated target member client endpoint when moving leadership before removal. This lets a surviving coordinator safely remove a remote leader while retaining the existing identity, quorum, and post-removal checks.
Make control-plane wipe resumable after an already-completed etcd removal, route Kubernetes deletion through the healthy coordinator, and expose cleanup diagnostics. Complete replacement joins with a bounded reboot and health trial, including interrupted-apply recovery, without inventing unchanged kubeadm rollouts.
Complete operation-backed bootstrap by rebooting each pending candidate and verifying committed boot health, kubelet activity, and local control-plane components before reporting success. Deliberately exclude Kubernetes Node Ready so CNI installation remains operator-owned, and derive repeat completion from current generation health to avoid redundant reboots when historical operation status is stale.
Wait for configured networking before URL-backed installation and keep the installer as a blocking initrd oneshot until reboot or operator action. Add a Matchbox-friendly halt-if-installed guard that recognizes Katl's target layout before mutation, and configure ephemeral root SSH from the selected bundle node so guarded and failed PXE boots remain debuggable.
Use a timer-backed hold while automatic installs await their scheduled reboot or preserve an installed target for SSH diagnostics, avoiding Go runtime deadlock detection when the parent context has no cancellation channel. Reactivate the test-only CNI fixture after bootstrap-controlled generation reboots so the lifecycle journey continues to model CNI as user-owned state.
Treat bootHealthPending on the terminal kubeadm operation as historical evidence that its live candidate still required a trial boot. Assert current health through the generation and boot-selection records produced by the separate boot-health phase instead of expecting retroactive operation mutation.
Greptile SummaryThe PR hardens installation, Kubernetes bootstrap and replacement, upgrade, recovery, and lifecycle workflows.
Confidence Score: 3/5The PR does not appear safe to merge while the waiting installer still permits unauthenticated provisioning of root SSH access. The current SSH-access handler accepts request-controlled key material without authenticating the caller, and its callback installs those keys for root and starts sshd. Files Needing Attention: internal/installer/handoff/handoff.go; cmd/katlos-install/main.go
|
| Filename | Overview |
|---|---|
| internal/installer/handoff/handoff.go | Adds installer SSH handoff and retry-state behavior; the previously reported unauthenticated root-access path remains. |
| cmd/katlos-install/main.go | Connects installer SSH handoff to root authorized-key provisioning and sshd startup without an authentication boundary. |
| cmd/katlctl/install.go | Adds the operator-facing command that submits configured SSH keys to a waiting installer. |
| cmd/katlctl/kubeadm_control_plane_config.go | Adds durable reboot and boot-health validation for joined replacement nodes. |
| cmd/katlctl/etcd.go | Makes node-wipe etcd cleanup resumable when the target member was already removed. |
Reviews (3): Last reviewed commit: "kubernetes: preserve identity on reprovi..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+242
to
+244
| func (s *HandoffServer) handleSSHAccess(w http.ResponseWriter, r *http.Request) { | ||
| nodeName := strings.TrimSpace(firstNonEmpty(r.URL.Query().Get("node"), r.Header.Get("X-Katl-Node-Name"))) | ||
| expectedDigest := strings.TrimSpace(firstNonEmpty(r.URL.Query().Get("digest"), r.Header.Get("X-Katl-Bundle-Digest"))) |
There was a problem hiding this comment.
When a network peer can reach a waiting installer, /v1/ssh-access accepts that peer's bundle and caller-controlled or omitted digest, then installs its keys for root and starts SSH, giving the peer unauthorized root access. How this was verified: The request-controlled bundle flows through no authentication or independent trust check to the callback that overwrites root's authorized_keys and starts sshd.service.
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/installer/handoff/handoff.go
Line: 242-244
Comment:
**Unauthenticated root SSH provisioning**
When a network peer can reach a waiting installer, `/v1/ssh-access` accepts that peer's bundle and caller-controlled or omitted digest, then installs its keys for root and starts SSH, giving the peer unauthorized root access. **How this was verified:** The request-controlled bundle flows through no authentication or independent trust check to the callback that overwrites root's `authorized_keys` and starts `sshd.service`.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Add a Markdown-only documentation home and task-oriented ISO, PXE/Matchbox, Kubernetes handoff, ownership, command, and node-membership guides. Reconcile the existing reference with the tested beta surface, and enforce navigation, local links, anchors, and documented katlctl commands in the Go baseline.
Add an operator-held Kubernetes identity artifact for kubeadm CA and signing material, with explicit create, import, inspect, and bootstrap flows. Validate and stage the secret outside durable operation metadata, install it idempotently before kubeadm mutation, and document its ownership, transport, backup, and recovery boundaries. Prove that a wiped and freshly installed cluster can retain its public Kubernetes CA while generation zero remains correctly NotReady until the operator installs a CNI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Release journey testing exposed failures that unit-level success could not show: PXE raced DHCP and initrd handoff, live generations were treated as boot-proven, inherited extensions were missing after reboot, replacement and etcd workflows could not safely resume, and several recovery paths lacked usable operator access.
The changes move those contracts onto durable generation, systemd, kubeadm, and etcd state and keep the supported operator experience explicit and recoverable. They were validated with the current release artifacts through ISO, local Matchbox PXE, fresh two-node bootstrap, persistent katldev, and destructive three-node PVE lifecycle journeys.