feat(den-api): add a kubernetes worker provisioner - #4680
Draft
nathanielheitsch wants to merge 6 commits into
Draft
feat(den-api): add a kubernetes worker provisioner#4680nathanielheitsch wants to merge 6 commits into
nathanielheitsch wants to merge 6 commits into
Conversation
Adds PROVISIONER_MODE=kubernetes so self-hosted Den deployments can auto-provision cloud workers as in-cluster workloads instead of relying on the Daytona or Render providers. - kubernetes-client.ts: lean Kubernetes REST client over undici (no new dependencies), resolving in-cluster ServiceAccount defaults with optional KUBERNETES_API_URL/TOKEN/CA_FILE overrides - kubernetes.ts: per-worker Deployment, ClusterIP Service, token Secret, and workspace/data PVCs named from the worker id; provisioning adopts existing objects on conflict, wake patches replicas (and a stale image), stop scales to zero preserving PVCs, deprovision deletes the full object set - env.ts: kubernetes config block mirroring the daytona block, with KUBERNETES_WORKER_IMAGE required when the mode is selected - dispatch wiring in provisioner.ts, worker-access.ts, cloud-lifecycle.ts (idle stop now covers kubernetes), and routes/workers/shared.ts so the mode participates in cloud-instance routing like daytona - cloud-failure.ts: generalize the health-timeout classifier so kubernetes health deadlines map to runtime_health_timeout Signed-off-by: Nathaniel Heitsch <nathaniel@heitsch.dev>
- tolerate 409 AlreadyExists on PVC/Secret/Service creates in the fresh provision path by verifying the existing object and adopting it, so a partial create or half-completed deprovision cannot dead-end reprovision - default KUBERNETES_WORKER_APPROVAL_MODE to auto; manual mode's gated writes time out 403 with no approvals responder in a sandbox - accept the kubernetes provisioner in the remote-session capability and cloud-agent-executor runtime gates; routes/cloud stays daytona-gated - pin the worker Deployment to Recreate strategy so image patches cannot MultiAttach-stall behind a rolling surge pod on ReadWriteOnce PVCs - redact worker tokens from pod log tails embedded in health-timeout errors - drop OPENWORK_CONNECT_HOST from the container env; the entrypoint binds openwork-server with --host 0.0.0.0 and only used CONNECT_HOST for banners - substitute the DNS-safe worker name (not the raw worker id) into WORKER_URL_TEMPLATE and document the placeholder contract - patch an adopted Deployment's stale container image before reporting healthy so the DB imageVersion matches the running pod - drop the unused base64 helper and unused getPod client method; add the getSecret client method used by the conflict-adopt verify path - expand provisioning tests: 409 adopt, secret delete/recreate, stale-image adopt patch, URL template substitution, deprovision 404 tolerance, and token redaction in health-timeout diagnostics Signed-off-by: Nathaniel Heitsch <nathaniel@heitsch.dev>
…ourney, and docs - values.yaml: config.kubernetes block + workers.kubernetes RBAC/namespace config; secret.kubernetesApiToken (KUBERNETES_API_TOKEN) in the release secret - configmap.yaml: emit the KUBERNETES_* surface only in kubernetes mode, with the worker namespace pinned so rendered RBAC and API calls agree - kubernetes-workers.yaml: ServiceAccount, optional worker Namespace, and a namespaced Role/RoleBinding (no cluster-scoped objects) for the worker namespace; den-api.yaml sets serviceAccountName in kubernetes mode - tests/kubernetes-workers.sh: default/mode/custom/invalid renders, including an assert that no ClusterRole or ClusterRoleBinding is ever rendered - evals: mock-kubernetes-apiserver witness (create/get/patch/delete, strategic merge, labelSelector, logs, readiness toggles) with unit tests, and the kubernetes-worker-provisioning journey spec covering provision, idle stop, wake, stale-image recycle, deprovision, and stub-mode inertia - docs: README values section + docs/kubernetes-workers.md operator guide (architecture, RBAC, lifecycle, data-erasure semantics, troubleshooting) Signed-off-by: Nathaniel Heitsch <nathaniel@heitsch.dev>
…ontent type, fail fast on missing worker image - RBAC now matches the kubernetes client exactly: get/create/patch/delete on deployments (the client never uses update or list), get/create/delete on services/secrets/pvcs, and get+list on pods (get is required for the pods/log subresource reads used by health-timeout diagnostics) - mock apiserver only applies strategic merge semantics for PATCH requests with application/strategic-merge-patch+json, returns 415 otherwise, and records the PATCH content type for journey assertions - chart render fails loudly when config.provisioner.mode=kubernetes and config.kubernetes.workerImage is empty, mirroring the SA-name fail-fast Signed-off-by: Nathaniel Heitsch <nathaniel@heitsch.dev>
Signed-off-by: Nathaniel Heitsch <nathaniel@heitsch.dev>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
@nathanielheitsch is attempting to deploy a commit to the Different AI Team on Vercel. A member of the Team first needs to authorize it. |
Completes the deferred PR different-ai#2: routes/cloud/index.ts now dispatches getSandboxRecord / inspectSandbox / refreshSignedPreview / stopCloudWorker through the provisioner-mode-aware defaults (worker-access.ts) instead of hard-coding the Daytona implementations, and hasDaytonaProvisioner accepts kubernetes when KUBERNETES_WORKER_IMAGE is set. cloudAvailable therefore returns true for kubernetes, so the desktop's GET /v1/cloud/instance resolves and the auto-provisioned worker option appears. - worker-access.ts: export defaultGetSandboxRecord / defaultInspectSandbox / defaultRefreshSignedPreview + the RefreshSignedPreview type - cloud/index.ts: use those defaults, kubernetes-aware hasDaytonaProvisioner, cloudWorkerImage() for latestVersion / update detection tsc clean; cloud-instance-route 40/40 pass; den-api suite no worse than baseline (failures are pre-existing missing-local-MySQL). Signed-off-by: Nathaniel Heitsch <nathaniel@heitsch.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
kubernetesworker-provisioner mode to Den. Instead of requiring aDaytona/Render account, Den can provision each cloud worker as an in-cluster
Deployment+Service+ tokenSecret+ twoPVCs, using a namespacedRole/RoleBinding(no ClusterRole). This lets self-hosted Den deploymentsrun workers entirely inside their own cluster.
Mirrors the existing
daytona.tsprovider surface exactly:readiness via the Service DNS name.
replicasback to 1 and reconcile the image (adopt-imagepath for workers created on an older image).
state, and health-timeout diagnostics that redact worker-printed tokens.
No new runtime dependencies:
kubernetes-client.tsis a ~210-line RESTclient on
undici(already a direct dependency).Changes
ee/apps/den-api/src/env.ts—PROVISIONER_MODE=kubernetesenum value +KUBERNETES_*config vars (apiUrl, apiToken, apiCaFile, workerNamespace,workerImage, workerPort, workerStorageClass, workerResources, TTL, approval
mode, imagePullPolicy).
ee/apps/den-api/src/workers/kubernetes-client.ts— minimal K8s REST client(get/create/patch/delete/list, namespaced, bearer + optional CA).
ee/apps/den-api/src/workers/kubernetes.ts— the provider (744 lines),mirroring
daytona.ts.provisioner.ts,cloud-lifecycle.ts,worker-access.ts,routes/workers/shared.ts,cloud-failure.ts.kubernetesis accepted byremote-session-capabilities.tsand
cloud-agent-executor.ts.packaging/helm/openwork-ee/— chart support: worker SA + namespacedRole/RoleBinding (
kubernetes-workers.yaml), den-api SA wiring, configmaptests/kubernetes-workers.sh).evals/— a mock K8s apiserver witness(
mock-kubernetes-apiserver.ts) + a journey spec(
kubernetes-worker-provisioning.test.ts) that exercises the fullprovision/wake/stop/deprovision cycle against the mock.
docs/kubernetes-workers.md— operator-facing docs.Testing
Unit / eval / chart (all passing)
ee/apps/den-api/test/kubernetes-provisioning.test.ts— 19 tests (namemapping, fresh create, adopt, 409 tolerance, token upsert, image reconcile,
stop, deprovision, preview, inspect, health-timeout redaction).
evals/packages/labs/test/mock-kubernetes-apiserver.test.ts— 125 lines.evals/specs/kubernetes-worker-provisioning.test.ts— 264-line journey.packaging/helm/openwork-ee/tests/kubernetes-workers.sh— 159 lines.tscclean; full den-api suite green.Live cluster test (k3s,
openwork-dev/openwork-workers-dev)Deployed the fork chart against a real k3s cluster and exercised the
provisioner end-to-end:
POST /v1/workers→ 202 → den-api created Deployment + Service + Secret +2 PVCs in
openwork-workers-dev; worker pod reached/healthvia ServiceDNS; DB status →
healthy.DELETE /v1/workers→ 204; all five objects deleted.an in-cluster Bifrost gateway (
http://bifrost.bifrost.svc:80) frontingvLLM. Materialized the provider + key into a running worker and ran a real
prompt through the worker's opencode → routed to Bifrost → vLLM
qwen3.8-27b→ returned the expected response. Confirms the worker canreach an in-cluster inference backend directly (Den is not in the
inference data path).
Notes / follow-ups
routes/cloud/index.tscloud-browser-instance path stays daytona-only inthis PR (deferred to a follow-up); the worker lifecycle + gates accept
kubernetes.ee/contribution.