This runbook covers the supported single-operator deployment model. Keep installation values, chart version, controller image, workspace image, and CLI version aligned.
helm status devboxes -n devboxes
kubectl get deployment,service,pvc,pod -n devboxes -o wide
kubectl rollout status deployment/devboxes -n devboxes
kubectl logs deployment/devboxes -n devboxes --tail=200
./scripts/verify-install.sh/health proves that the HTTP process responds. /ready performs a namespaced Deployment list against the Kubernetes API and, when enabled, verifies the Insights database. It returns 503 when either required dependency is unavailable. Neither endpoint creates resources.
/metrics exposes devboxes_total{state=...} for starting, ready, stopped, and degraded boxes. When Insights is enabled it also exposes low-cardinality ingest outcomes, dropped points by safe provider label, database bytes, store readiness, and last successful rollup. Enable the chart ServiceMonitor only when Prometheus Operator CRDs already exist:
serviceMonitor:
enabled: true
interval: 30s
labels:
release: monitoringController logs record configuration loading, automatic TTL stops, and cleanup errors. Kubernetes events remain the primary source for scheduling, volume, image, and Service allocation failures.
Recommended alerts include controller readiness failure, controller crash loops, degraded boxes, workspace restart growth, PVC capacity pressure, boxes that remain starting beyond the normal image and volume provisioning window, stale Insights collectors, reported outbox loss, and an Insights database beyond its configured warning threshold.
Add the CPU and memory requests of concurrently running presets, then reserve capacity for the controller and cluster services. CPU has no workspace limit, so boxes can burst when the node has spare capacity. Memory is limited per preset and exceeding it causes container termination.
Stopped boxes consume PVC capacity but no workspace CPU or memory. Include retained volumes in storage forecasts. Track provisioned and used bytes through the CSI provider, because Kubernetes PVC requests do not report filesystem utilization.
Use node selectors, affinity, tolerations, and an existing PriorityClass only when the cluster scheduling policy requires them. A priority class can improve scheduling under pressure, but it can also preempt lower-priority workloads.
For GPU capacity, count the extended-resource units requested by active profile allocations and interpret them according to the vendor plugin's dedicated, partitioned, or sharing mode. count: 1 means one advertised unit, which is not always one physical board. Monitor allocatable resources and Pending events on every GPU pool. Keep CPU-only headroom because GPU workspaces still request the selected CPU and memory preset.
For custom image profiles, add each sidecar's configured CPU and memory request to the selected workspace preset when forecasting node pressure. A catalog profile is trusted supply-chain policy, not an ad hoc developer image setting. Pin or regularly review image references, validate every supported architecture, verify registry credentials from each eligible node, and record the intended pod-local port and resource envelope.
Profile selectors and tolerations are policy, not capacity detection. Before publishing a profile, prove that at least one node matches all selectors, tolerates the intended taints, advertises the exact resource, and can pull the profile image. Document whether each profile is dedicated or shared.
The home PVC is the durable state. Back up important PVCs with a CSI snapshot system or storage-provider backup that is compatible with the active StorageClass.
Before relying on a backup process:
- Stop the devbox to quiesce processes and filesystem writes.
- Snapshot or back up the PVC named
devbox-NAME-home. - Restore into a non-production namespace or isolated cluster.
- Verify ownership, SSH host keys, repositories, tool state, and provider credentials.
- Record recovery time and the exact restore procedure.
Devboxes does not create VolumeSnapshots and does not automatically back up or delete PVCs.
When Insights is enabled, create a consistent database snapshot with the authenticated online-backup endpoint:
curl -fsS \
-H "Authorization: Bearer $DEVBOX_TOKEN" \
-o devboxes-insights.db \
"$DEVBOX_URL/api/v1/insights/export?format=sqlite"Do not copy only the live database file while SQLite WAL mode is active. Restore with the controller stopped, replace the database on the Insights PVC using an administrative pod, preserve ownership for uid and gid 10001, then verify /ready, /insights, and devbox metrics status.
Read CHANGELOG.md, back up important volumes, and render the new chart before applying it.
helm template devboxes oci://ghcr.io/vicotrbb/charts/devboxes \
--version NEW_VERSION \
--namespace devboxes \
--values values.yaml > /tmp/devboxes-new.yaml
helm upgrade devboxes oci://ghcr.io/vicotrbb/charts/devboxes \
--version NEW_VERSION \
--namespace devboxes \
--values values.yaml \
--waitThen run scripts/verify-install.sh, confirm /ready, list existing boxes, create a disposable smoke box, connect over SSH, stop it, start it, and delete it with purge.
When GPU configuration or images change, render the profile JSON before applying, run devbox gpu profiles after rollout, and create a disposable box for every changed profile. Verify the vendor diagnostic inside the box, stop and start it, and confirm the same profile contract remains. Stopping releases live device capacity, so Kubernetes may select a different physical device on start. Existing GPU Deployments retain their resolved snapshots; a Helm profile edit affects only later creations. Plan migrations as explicit delete and recreate operations, with a separate PVC retention decision.
When custom image profiles change, render the catalog before applying, run devbox image profiles, and create a disposable test box for every changed profile. For a sidecar, verify a declared non-root image user, a port from 1024 through 65535, the container list, no unexpected Secret or PVC mount, application health over pod loopback, SSH tunneling, stop and start, and cleanup. For a workspace-mode profile, also verify SSH readiness, persistent-home reuse, repository bootstrap, and Insights behavior if enabled. Existing Deployments retain their creation-time resolved snapshot; changing or disabling a Helm profile rejects new requests but does not rewrite existing boxes. Retire a profile only after inventorying devbox list --json and making an explicit delete or migration plan.
Enabling Insights does not force-restart an active legacy workspace. devbox metrics status reports restart_required until the box goes through a normal stop and start. Stopped workspaces are updated without starting compute. Confirm the expected state before and after the rollout.
Prefer a reviewed values file over --reuse-values. It makes removed defaults and configuration drift visible.
Inspect release history and changelog compatibility first:
helm history devboxes -n devboxes
helm rollback devboxes REVISION -n devboxes --waitA Helm rollback changes controller resources, not the contents of workspace PVCs. Do not roll back across an explicitly incompatible data or resource migration without following that release's instructions.
Disabling GPU support or rolling back the catalog rejects new GPU requests but does not rewrite existing GPU Deployments. This is intentional. Inventory allocations with devbox list --json before retiring drivers, runtimes, images, or node pools that those boxes still require.
Disabling custom image support or removing a profile similarly rejects new requests but does not remove a running sidecar or change an existing workspace image. Inventory image allocations before retiring a registry image or pull credential, then delete or migrate each affected box deliberately.
Replace the configured controller Secret value, then restart the controller:
kubectl rollout restart deployment/devboxes -n devboxes
kubectl rollout status deployment/devboxes -n devboxesRotation invalidates browser sessions and saved CLI tokens. Log in again after the rollout. Rotate immediately after suspected disclosure, then review controller access logs available at the ingress or network boundary.
When insights.signingKeyKey is empty, the same controller-token rotation also rotates the domain-separated ingest signing key. Active workspaces need a normal stop and start to receive a new scoped credential. With a dedicated Insights signing key, rotate that field independently and follow the same workspace reconciliation procedure.
Rotate workspace provider tokens independently through the workspace Secret. Existing files copied into persistent homes are not overwritten automatically, so revoke compromised provider sessions at the provider and update affected homes explicitly.
Uninstalling the chart removes chart-owned controller resources. Per-devbox resources created by the controller and retained PVCs require an explicit data decision.
Inventory before deletion:
kubectl get deployment,service,pvc -n devboxes \
-l app.kubernetes.io/managed-by=devboxes-controllerBack up required PVCs, delete boxes through the CLI, and use --purge only for volumes approved for permanent removal. Verify the namespace contents before deleting the namespace.
If insights.storage.retainOnDelete=true, the chart-created central Insights PVC remains after uninstall. Per-workspace outboxes remain on retained home PVCs. Remove either data layer only after a separate backup and retention decision.
Recreate the namespace, controller Secret, workspace Secret, and Helm release from versioned values. Restore home PVCs with their expected devbox-NAME-home names before recreating corresponding boxes. Reusing the name reconnects compute to the retained home and its SSH host identity.
If the original controller token is unavailable, issue a new token and authenticate clients again. Kubernetes resources and PVC contents do not depend on browser or CLI sessions.
The public repository has no standing ruleset bypass. Normal recovery uses a short-lived branch, a pull request, and the complete required check set. Before changing protection, capture the active rulesets and their identifiers with gh api repos/vicotrbb/devboxes/rulesets so the original state is auditable.
If GitHub Actions or the pull request service is unavailable and a critical security or release-integrity fix cannot wait, the repository owner may temporarily disable only the devboxes-main ruleset in Settings → Rules → Rulesets. Keep the release-tag ruleset active, make the smallest reviewed and signed recovery change, then immediately reactivate the unchanged main ruleset. Verify its active enforcement and full rule list through the API, run the omitted checks as soon as the service recovers, and record the incident, commit, reason, operator, timestamps, and validation in the relevant security advisory or issue. Never add a permanent administrator or organization-wide bypass for this procedure.