Skip to content

Commit c64cb72

Browse files
Varashiclaude
andauthored
rename to vsphere-passthrough-node-controller + add crash-fence controller (#7)
* feat: optional crash-fence controller (out-of-service taint on host crash) Adds fence.py — a second, optional Deployment (fence.enabled, off by default) sharing this image and reusing the vCenter client + node↔VM map. Automates non-graceful node shutdown for passthrough-GPU workers that vSphere HA can't restart elsewhere during a host crash: applies the node.kubernetes.io/out-of-service taint to a node confirmed dead by BOTH k8s (NotReady) and vCenter (VM connectionState disconnected/inaccessible/ orphaned), sustained graceSeconds, so RWO volumes force-detach and pods reschedule; removes it on recovery (connected + Ready). Disjoint from the maintenance controller (clean power-off stays 'connected', only a crash goes 'disconnected') — no coordination needed. Taint/un-taint only; power-on stays with vSphere HA. Own SA + least-privilege ClusterRole (nodes only) + kill switch + dryRun. test_fence.py covers the two-gate guard, grace hold, no-double-apply, un-fence on recovery, and the partition/notfound guards (8/8). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: rename to vsphere-passthrough-node-controller The product now covers two passthrough-node host-lifecycle concerns — planned maintenance and (new) crash fencing — so "maintenance-controller" no longer fits. Renames repo/image/chart/templates/helpers/docs to vsphere-passthrough-node-controller (image + chart at ghcr.io/varashi/[charts/]vsphere-passthrough-node-controller). GitHub repo renamed (redirects cover old git/web refs). The internal maintenance-state annotation domain (vsphere-maintenance.boeye.net/*) is left unchanged — it's controller state, not user-facing, and no nodes carry it currently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: fix ruff format + hadolint DL3021 ruff format controller.py (line-length wrapping); Dockerfile COPY multi-arg destination must end with / (DL3021). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CodeRabbit review on #7 - controller.py: match out-of-service taint by full (key,value,effect) identity, not key-only, so a same-key taint with a different value/effect isn't mistaken for ours on fence/un-fence - chart fence.yaml: drop redundant `and` in SA conditional; remove `update` verb from fence ClusterRole (patch suffices for taint ops) - CHANGELOG: add 0.5.0/0.4.4 link defs, advance Unreleased baseline - README: bump chart examples 0.4.3 -> 0.5.0; note fence.py in layout Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent d69ef1d commit c64cb72

20 files changed

Lines changed: 715 additions & 108 deletions

.github/workflows/ci.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,4 +75,4 @@ jobs:
7575
context: .
7676
platforms: linux/amd64,linux/arm64
7777
push: false
78-
tags: ci/gpu-node-vsphere-maintenance-controller:ci
78+
tags: ci/vsphere-passthrough-node-controller:ci

.github/workflows/release.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ jobs:
148148
run: |
149149
version="${{ steps.ver.outputs.version }}"
150150
# helm push prints "Digest: sha256:..." to stderr; tee to capture.
151-
helm push "gpu-node-vsphere-maintenance-controller-${version}.tgz" \
151+
helm push "vsphere-passthrough-node-controller-${version}.tgz" \
152152
"oci://${{ env.CHART_REPO }}" 2>&1 | tee push.log
153153
digest=$(awk '/^Digest: /{print $2}' push.log)
154154
if [ -z "$digest" ]; then
@@ -160,7 +160,7 @@ jobs:
160160
- name: Cosign keyless sign (chart)
161161
env:
162162
DIGEST: ${{ steps.chart_push.outputs.digest }}
163-
CHART_REF: ${{ env.CHART_REPO }}/gpu-node-vsphere-maintenance-controller
163+
CHART_REF: ${{ env.CHART_REPO }}/vsphere-passthrough-node-controller
164164
run: cosign sign --yes "${CHART_REF}@${DIGEST}"
165165

166166
- name: Create GitHub Release
@@ -173,4 +173,4 @@ jobs:
173173
prerelease: false
174174
files: |
175175
sbom.spdx.json
176-
gpu-node-vsphere-maintenance-controller-${{ steps.ver.outputs.version }}.tgz
176+
vsphere-passthrough-node-controller-${{ steps.ver.outputs.version }}.tgz

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
__pycache__/

CHANGELOG.md

Lines changed: 38 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,29 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
77

88
## [Unreleased]
99

10+
## [0.5.0] — 2026-06-03
11+
12+
### Added
13+
- **Crash-fence controller** (`fence.py`) — a second, optional Deployment
14+
(`fence.enabled`, **off by default**) that shares this image and reuses the
15+
vCenter client + node↔VM mapping. It automates non-graceful node shutdown for
16+
passthrough-GPU workers that vSphere HA can't restart elsewhere during a host
17+
crash: it applies the `node.kubernetes.io/out-of-service` taint to a node
18+
confirmed dead by **both** gates — k8s `NotReady` **and** vCenter VM
19+
`runtime.connectionState` in `{disconnected, inaccessible, orphaned}`
20+
sustained for `fence.graceSeconds`, so RWO volumes force-detach and stateful
21+
pods reschedule. The taint is removed on recovery (VM `connected` + node
22+
`Ready`).
23+
- **Disjoint from the maintenance controller**: a clean (maintenance)
24+
power-off leaves the VM `connected`; only a real host loss makes it
25+
`disconnected`. The two controllers trigger on different vCenter facts and
26+
never collide — no coordination contract needed.
27+
- **Taint/un-taint only.** Power-on is owned by vSphere HA (it restarts
28+
passthrough VMs on the original host once it returns); eviction is handled
29+
by `tolerationSeconds` + the taint.
30+
- Own ServiceAccount + least-privilege ClusterRole (`nodes` get/list/watch/
31+
patch only) + kill switch (`fence.enabled`) + independent `fence.dryRun`.
32+
1033
## [0.4.4] — 2026-05-01
1134

1235
### Fixed
@@ -84,7 +107,7 @@ No controller code change. Supply-chain and CI polish only.
84107
now consults the map instead of making a per-node `get_vm_host` round-trip
85108
to vCenter on every poll.
86109
- Minimal Helm chart under `chart/`, published as OCI to
87-
`ghcr.io/varashi/charts/gpu-node-vsphere-maintenance-controller`.
110+
`ghcr.io/varashi/charts/vsphere-passthrough-node-controller`.
88111
- GitHub Actions: `ci.yaml` (ruff, hadolint, helm lint, buildx smoke build)
89112
on pull requests; `release.yaml` on `v*.*.*` tag push builds multi-arch
90113
images (amd64, arm64), cosign-signs keyless via OIDC, attaches SBOM and
@@ -163,15 +186,17 @@ No controller code change. Supply-chain and CI polish only.
163186
- Initial release: drain → power-off → wait-for-exit → power-on →
164187
uncordon, driven by edge-triggered `HostSystem.recentTask` polling.
165188

166-
[Unreleased]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.4.3...HEAD
167-
[0.4.3]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.4.2...v0.4.3
168-
[0.4.2]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.4.1...v0.4.2
169-
[0.4.1]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.4.0...v0.4.1
170-
[0.4.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.3.0...v0.4.0
171-
[0.3.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.3...v0.3.0
172-
[0.2.3]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.2...v0.2.3
173-
[0.2.2]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.1...v0.2.2
174-
[0.2.1]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.0...v0.2.1
175-
[0.2.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.1.1...v0.2.0
176-
[0.1.1]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.1.0...v0.1.1
177-
[0.1.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/releases/tag/v0.1.0
189+
[Unreleased]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.5.0...HEAD
190+
[0.5.0]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.4.4...v0.5.0
191+
[0.4.4]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.4.3...v0.4.4
192+
[0.4.3]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.4.2...v0.4.3
193+
[0.4.2]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.4.1...v0.4.2
194+
[0.4.1]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.4.0...v0.4.1
195+
[0.4.0]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.3.0...v0.4.0
196+
[0.3.0]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.2.3...v0.3.0
197+
[0.2.3]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.2.2...v0.2.3
198+
[0.2.2]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.2.1...v0.2.2
199+
[0.2.1]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.2.0...v0.2.1
200+
[0.2.0]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.1.1...v0.2.0
201+
[0.1.1]: https://github.com/Varashi/vsphere-passthrough-node-controller/compare/v0.1.0...v0.1.1
202+
[0.1.0]: https://github.com/Varashi/vsphere-passthrough-node-controller/releases/tag/v0.1.0

Dockerfile

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,18 @@
11
FROM python:3.13-slim
22

3-
LABEL org.opencontainers.image.title="gpu-node-vsphere-maintenance-controller"
3+
LABEL org.opencontainers.image.title="vsphere-passthrough-node-controller"
44
LABEL org.opencontainers.image.description="Kubernetes controller that automates ESXi maintenance mode for worker nodes with PCI passthrough (GPU or otherwise)."
5-
LABEL org.opencontainers.image.source="https://github.com/Varashi/gpu-node-vsphere-maintenance-controller"
6-
LABEL org.opencontainers.image.documentation="https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/blob/main/README.md"
5+
LABEL org.opencontainers.image.source="https://github.com/Varashi/vsphere-passthrough-node-controller"
6+
LABEL org.opencontainers.image.documentation="https://github.com/Varashi/vsphere-passthrough-node-controller/blob/main/README.md"
77
LABEL org.opencontainers.image.licenses="MIT"
88

99
WORKDIR /app
1010

1111
RUN pip install --no-cache-dir --disable-pip-version-check \
1212
pyVmomi==9.1.0.0 kubernetes==36.0.1
1313

14-
COPY controller.py .
14+
COPY controller.py fence.py ./
1515

16+
# Default entrypoint = maintenance controller. The fence controller (fence.py)
17+
# is the same image with the command overridden to `python -u fence.py`.
1618
CMD ["python", "-u", "controller.py"]

0 commit comments

Comments
 (0)