Skip to content

Commit 2c9ba39

Browse files
Varashiclaude
Varashi
and
claude
committed
v0.4.0: TLS opt-in, pending-drain reconciler, Helm chart, CI/release workflows
Controller: - Optional `VCENTER_CA_BUNDLE` enables TLS verification against vCenter. - `reconcile_pending_drains` runs every poll so hosts skipped by a saturated `MAX_CONCURRENT_DRAINS` cap are retried instead of silently dropped after their `prev_active → now_active` edge. - `get_inventory_snapshot` returns both host states and a vm→host map from a single view walk; `reconcile_powered_off` consults the cached map instead of a per-node vCenter round-trip each tick. - `power_off_vm` fallbacks (Tools-unavailable, VimFault, timeout) route through `_hard_power_off`, which treats `InvalidPowerState` ("Powered off") as success — symmetric to the existing concurrent power-on handling and compatible with the guest-shutdown flow added in v0.3.0. - Dockerfile pinned to `python:3.13-slim` (pyVmomi 8.0.3.0.1 predates 3.14). Packaging / supply chain: - Helm chart under `chart/` (v0.4.0) published as OCI to `ghcr.io/varashi/charts/gpu-node-vsphere-maintenance-controller`. - `.github/workflows/ci.yaml`: ruff, hadolint, helm lint, multi-arch buildx smoke. - `.github/workflows/release.yaml` on `v*.*.*` tag: multi-arch image push, cosign keyless sign, SBOM, build-provenance attestation, chart package + push, GitHub Release with body from CHANGELOG.md. - `strategy.type: Recreate` on the example Deployment closes the brief double-run window during rollouts at `replicas: 1`. - `policy/poddisruptionbudgets` RBAC verb removed — unused by the code. - CHANGELOG.md backfills the v0.3.0 entry for the graceful-shutdown + narrowed-migration-logging commit that was pushed directly to main. See CHANGELOG.md for the full entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 36a7ab6 commit 2c9ba39

17 files changed

Lines changed: 1021 additions & 105 deletions

‎.github/workflows/ci.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
name: ci
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- main
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
python:
14+
name: python / ruff
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v4
18+
- uses: actions/setup-python@v5
19+
with:
20+
python-version: "3.13"
21+
- name: Install ruff
22+
run: pip install --no-cache-dir ruff==0.7.4
23+
- name: ruff check
24+
run: ruff check controller.py
25+
- name: ruff format --check
26+
run: ruff format --check controller.py
27+
28+
dockerfile:
29+
name: dockerfile / hadolint
30+
runs-on: ubuntu-latest
31+
steps:
32+
- uses: actions/checkout@v4
33+
- uses: hadolint/hadolint-action@v3.1.0
34+
with:
35+
dockerfile: Dockerfile
36+
37+
chart:
38+
name: helm / lint + template
39+
runs-on: ubuntu-latest
40+
steps:
41+
- uses: actions/checkout@v4
42+
- uses: azure/setup-helm@v4
43+
with:
44+
version: v3.16.3
45+
- name: helm lint
46+
run: helm lint chart/
47+
- name: helm template (defaults + CA bundle + ESO path)
48+
run: |
49+
helm template ci chart/ > /dev/null
50+
helm template ci chart/ \
51+
--set vcenter.host=vc.example.com \
52+
--set vcenter.user=u --set vcenter.password=p > /dev/null
53+
helm template ci chart/ \
54+
--set vcenter.existingSecret=my-eso-secret \
55+
--set vcenter.caBundle.configMapName=vcenter-ca > /dev/null
56+
57+
image:
58+
name: image / buildx smoke
59+
runs-on: ubuntu-latest
60+
steps:
61+
- uses: actions/checkout@v4
62+
- uses: docker/setup-qemu-action@v3
63+
- uses: docker/setup-buildx-action@v3
64+
- name: docker build (no push, linux/amd64,linux/arm64)
65+
uses: docker/build-push-action@v6
66+
with:
67+
context: .
68+
platforms: linux/amd64,linux/arm64
69+
push: false
70+
tags: ci/gpu-node-vsphere-maintenance-controller:ci

‎.github/workflows/release.yaml‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
name: release
2+
3+
on:
4+
push:
5+
tags:
6+
- "v*.*.*"
7+
8+
permissions:
9+
contents: write # create GitHub Release
10+
packages: write # push to ghcr.io
11+
id-token: write # cosign keyless + build provenance
12+
attestations: write # build provenance attestation
13+
14+
env:
15+
REGISTRY: ghcr.io
16+
IMAGE_NAME: ${{ github.repository }}
17+
CHART_REPO: ghcr.io/${{ github.repository_owner }}/charts
18+
19+
jobs:
20+
release:
21+
runs-on: ubuntu-latest
22+
steps:
23+
- uses: actions/checkout@v4
24+
with:
25+
fetch-depth: 0
26+
27+
- name: Resolve version
28+
id: ver
29+
run: |
30+
tag="${GITHUB_REF_NAME}"
31+
version="${tag#v}"
32+
echo "tag=${tag}" >> "$GITHUB_OUTPUT"
33+
echo "version=${version}" >> "$GITHUB_OUTPUT"
34+
35+
- name: Extract CHANGELOG section
36+
id: changelog
37+
run: |
38+
version="${{ steps.ver.outputs.version }}"
39+
awk -v v="$version" '
40+
$0 ~ "^## \\[" v "\\]" { flag=1; next }
41+
flag && $0 ~ "^## \\[" { exit }
42+
flag { print }
43+
' CHANGELOG.md > RELEASE_BODY.md
44+
if [ ! -s RELEASE_BODY.md ]; then
45+
echo "See [CHANGELOG.md](./CHANGELOG.md)." > RELEASE_BODY.md
46+
fi
47+
48+
- uses: docker/setup-qemu-action@v3
49+
- uses: docker/setup-buildx-action@v3
50+
51+
- name: Log in to ghcr.io
52+
uses: docker/login-action@v3
53+
with:
54+
registry: ${{ env.REGISTRY }}
55+
username: ${{ github.actor }}
56+
password: ${{ secrets.GITHUB_TOKEN }}
57+
58+
- name: Docker metadata
59+
id: meta
60+
uses: docker/metadata-action@v5
61+
with:
62+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
63+
tags: |
64+
type=semver,pattern={{version}}
65+
type=semver,pattern={{major}}.{{minor}}
66+
type=semver,pattern={{major}}
67+
type=raw,value=latest
68+
69+
- name: Build and push (multi-arch)
70+
id: build
71+
uses: docker/build-push-action@v6
72+
with:
73+
context: .
74+
platforms: linux/amd64,linux/arm64
75+
push: true
76+
tags: ${{ steps.meta.outputs.tags }}
77+
labels: ${{ steps.meta.outputs.labels }}
78+
provenance: true
79+
sbom: true
80+
cache-from: type=gha
81+
cache-to: type=gha,mode=max
82+
83+
- name: Install cosign
84+
uses: sigstore/cosign-installer@v3
85+
86+
- name: Cosign keyless sign
87+
env:
88+
DIGEST: ${{ steps.build.outputs.digest }}
89+
TAGS: ${{ steps.meta.outputs.tags }}
90+
run: |
91+
for tag in $TAGS; do
92+
cosign sign --yes "${tag}@${DIGEST}"
93+
done
94+
95+
- name: Generate SBOM (SPDX)
96+
uses: anchore/sbom-action@v0
97+
with:
98+
image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}
99+
format: spdx-json
100+
output-file: sbom.spdx.json
101+
upload-release-assets: false
102+
upload-artifact: false
103+
104+
- name: Attest build provenance
105+
uses: actions/attest-build-provenance@v2
106+
with:
107+
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
108+
subject-digest: ${{ steps.build.outputs.digest }}
109+
push-to-registry: true
110+
111+
- name: Attach SBOM to image
112+
env:
113+
DIGEST: ${{ steps.build.outputs.digest }}
114+
IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
115+
run: |
116+
cosign attest --yes --predicate sbom.spdx.json \
117+
--type spdxjson \
118+
"${IMAGE}@${DIGEST}"
119+
120+
- uses: azure/setup-helm@v4
121+
with:
122+
version: v3.16.3
123+
124+
- name: Package Helm chart
125+
run: |
126+
version="${{ steps.ver.outputs.version }}"
127+
helm package chart/ \
128+
--version "${version}" \
129+
--app-version "${version}" \
130+
--destination .
131+
132+
- name: Helm registry login
133+
run: |
134+
echo "${{ secrets.GITHUB_TOKEN }}" \
135+
| helm registry login ${{ env.REGISTRY }} \
136+
--username ${{ github.actor }} --password-stdin
137+
138+
- name: Push Helm chart (OCI)
139+
run: |
140+
version="${{ steps.ver.outputs.version }}"
141+
helm push "gpu-node-vsphere-maintenance-controller-${version}.tgz" \
142+
"oci://${{ env.CHART_REPO }}"
143+
144+
- name: Create GitHub Release
145+
uses: softprops/action-gh-release@v2
146+
with:
147+
tag_name: ${{ steps.ver.outputs.tag }}
148+
name: ${{ steps.ver.outputs.tag }}
149+
body_path: RELEASE_BODY.md
150+
draft: false
151+
prerelease: false
152+
files: |
153+
sbom.spdx.json
154+
gpu-node-vsphere-maintenance-controller-${{ steps.ver.outputs.version }}.tgz

‎CHANGELOG.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Changelog
2+
3+
All notable changes to this project are documented here.
4+
5+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
6+
this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [Unreleased]
9+
10+
### Planned
11+
- `VCENTER_TLS_VERIFY` flag (bool) for vCenter certs issued by a public
12+
CA. Enabling it uses `ssl.create_default_context()` with no `cafile`,
13+
which falls back to OpenSSL's system trust store. Chart exposes it as
14+
`vcenter.tlsVerify`. Existing `VCENTER_CA_BUNDLE` continues to cover
15+
the self-signed and self-hosted-CA cases unchanged.
16+
17+
## [0.4.0] — 2026-04-21
18+
19+
### Added
20+
- Optional TLS verification against vCenter via `VCENTER_CA_BUNDLE`.
21+
When set, uses `ssl.create_default_context(cafile=...)`; otherwise falls
22+
back to the previous unverified behaviour and logs a warning.
23+
- `reconcile_pending_drains(host_states)` runs every poll. Picks up GPU
24+
nodes on in/entering-maintenance hosts that still have no state
25+
annotation — covers the case where `MAX_CONCURRENT_DRAINS` throttled the
26+
edge-trigger and the skipped host would otherwise never be retried.
27+
- `get_inventory_snapshot()` emits both `host_states` and a
28+
`vm_host_map` from a single `HostSystem` view walk. `reconcile_powered_off`
29+
now consults the map instead of making a per-node `get_vm_host` round-trip
30+
to vCenter on every poll.
31+
- Minimal Helm chart under `chart/`, published as OCI to
32+
`ghcr.io/varashi/charts/gpu-node-vsphere-maintenance-controller`.
33+
- GitHub Actions: `ci.yaml` (ruff, hadolint, helm lint, buildx smoke build)
34+
on pull requests; `release.yaml` on `v*.*.*` tag push builds multi-arch
35+
images (amd64, arm64), cosign-signs keyless via OIDC, attaches SBOM and
36+
build-provenance attestations, packages and pushes the Helm chart, and
37+
creates a GitHub Release with the body extracted from this file.
38+
- This `CHANGELOG.md`, seeded from the previous README "Version history".
39+
40+
### Changed
41+
- Dockerfile pinned to `python:3.13-slim`. `pyVmomi==8.0.3.0.1` predates
42+
Python 3.14 and has not been tested against it upstream.
43+
- `startup_reconcile` delegates its "host already in maintenance at boot"
44+
branch to `reconcile_pending_drains` so the two paths share one
45+
implementation.
46+
- Example Deployment in the README sets `strategy.type: Recreate`. With
47+
`replicas: 1` this closes the brief double-run window that was previously
48+
only partially mitigated by idempotency at the state-machine level.
49+
50+
### Fixed
51+
- Concurrent power-off race: a `PowerOff()` landing on an already-off VM
52+
previously bubbled an `InvalidPowerState` error through the generic
53+
exception catch and aborted the cycle mid-transition. Now treated as
54+
success, symmetric to the existing power-on handling.
55+
56+
### Removed
57+
- `policy/poddisruptionbudgets` verb from the example ClusterRole. The
58+
controller never calls the PDB API — PDB-blocked evictions are handled
59+
via the 429 response on `pods/eviction`.
60+
61+
## [0.3.0] — 2026-04-19
62+
63+
### Added
64+
- `GUEST_SHUTDOWN_TIMEOUT_SECONDS` (default `120`). `power_off_vm` now
65+
requests a graceful guest shutdown via VMware Tools and falls back to
66+
a hard `PowerOff()` on `ToolsUnavailable`, on any `VimFault` rejecting
67+
the shutdown, or after the timeout elapses.
68+
69+
### Changed
70+
- `_try_migrate` distinguishes expected vSphere failures from bugs:
71+
`RuntimeError` (surfaced from `_wait_task`) and `vim.fault.VimFault`
72+
(e.g. `NoCompatibleHost`, `InsufficientResources`) log as one-line
73+
WARNINGs. Unexpected exceptions still log with a full traceback.
74+
75+
## [0.2.3] — 2026-04
76+
77+
### Changed
78+
- Reconcile loop classifies transient kube-apiserver errors (408, 429,
79+
5xx) and urllib3 transport blips as WARNING instead of ERROR +
80+
traceback. Genuine exceptions still log with the full traceback.
81+
82+
## [0.2.2]
83+
84+
### Added
85+
- OCI image labels. No code change; extracted to a dedicated GitHub repo.
86+
87+
## [0.2.1]
88+
89+
### Fixed
90+
- Concurrent power-on race with DRS handled explicitly.
91+
- Stale `powered-off` annotation: a VM already running elsewhere now
92+
transitions to `migrated` on the next poll.
93+
94+
## [0.2.0]
95+
96+
### Added
97+
- Cold-migrate to a free GPU-capable host after power-off, either via
98+
DRS full-automation `PowerOn()` or manual `RelocateVM`.
99+
100+
## [0.1.1]
101+
102+
### Fixed
103+
- `reconcile_powered_off` checked host state before uncordoning.
104+
105+
## [0.1.0]
106+
107+
### Added
108+
- Initial release: drain → power-off → wait-for-exit → power-on →
109+
uncordon, driven by edge-triggered `HostSystem.recentTask` polling.
110+
111+
[Unreleased]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.4.0...HEAD
112+
[0.4.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.3.0...v0.4.0
113+
[0.3.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.3...v0.3.0
114+
[0.2.3]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.2...v0.2.3
115+
[0.2.2]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.1...v0.2.2
116+
[0.2.1]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.2.0...v0.2.1
117+
[0.2.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.1.1...v0.2.0
118+
[0.1.1]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/compare/v0.1.0...v0.1.1
119+
[0.1.0]: https://github.com/Varashi/gpu-node-vsphere-maintenance-controller/releases/tag/v0.1.0

‎Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM python:3.14-slim
1+
FROM python:3.13-slim
22

33
LABEL org.opencontainers.image.title="gpu-node-vsphere-maintenance-controller"
44
LABEL org.opencontainers.image.description="Kubernetes controller that automates ESXi maintenance mode for worker nodes with PCI passthrough (GPU or otherwise)."

0 commit comments

Comments
 (0)