feat(compute): RunPod as an alternative compute provider to Modal

[lucastononro]

Closes #130

What

Adds RunPod as a full-parity alternative compute provider to Modal, selected globally via COMPUTE_PROVIDER=modal|runpod. Modal stays the default and its behavior is unchanged (the whole pre-existing suite passes untouched after the refactor commit, before any RunPod code exists).

Commit 1 — provider abstraction (pure refactor)

New backend/services/compute/ package with per-concern interfaces selected by settings.compute_provider:

Interface	Modal impl	Call-site facade
SandboxProvider (one-shot exec, streamed stdout/stderr)	wraps modal.Sandbox.create	services/sandbox.py:run_code (signature unchanged)
KernelTransport (newline-JSON cmd/event channel)	sandbox stdin/stdout	services/kernel_manager.py
StorageBackend (workspace volume, 1:1 with volume.py helpers)	Modal Volume	services/volume.py (all functions keep their names)
ServingBackend (codegen/deploy/stop/secret)	wraps _serving_app_code / modal deploy CLI helpers	services/deploy.py (orchestration + DB rows stay)

Also: billing rows now record the actual provider (record_sandbox_usage(provider=...) replaces the hardcoded "modal"), sandbox.yml gains a runpod: rate key, and deployments gains provider / provider_endpoint_id columns (ALTER-TABLE migration in db.py).

The Modal adapters deliberately resolve modal.Sandbox / get_app / get_image / get_volume through the legacy module namespaces so every existing test monkeypatch keeps working.

Commit 2 — RunPod implementation

• Code execution → serverless "code-runner" endpoints, one per canonical GPU tier (≤8, lazily created, workersMin=0, 60 s idle keeps workers warm across consecutive agent calls, FlashBoot). The worker generator-handler subprocess-runs the code and yields stdout/stderr lines; the backend polls /stream/{job} and re-exposes them as the same async-iterator handle shape Modal has. Job timeout via policy.executionTimeout; TIMED_OUT maps to a provider-neutral SandboxTimeoutError that the execute-code skill handles like Modal's.
• Notebook kernels → long-lived CPU pod running an HTTP kernel gateway (same AsyncKernelManager logic as the Modal stdin/stdout proxy): POST /cmd, cursor-based GET /events long-poll ≤25 s (the pod proxy kills connections at 100 s), per-pod token auth, SDK preamble via env. Ready timeout is provider-aware (600 s vs 120 s — first boot pulls the image). The existing idle reaper bounds pod cost.
• Storage → RunPod network volume: live filesystem mount in workers/pods + S3-compatible API for backend reads/writes (the role modal.Volume's client API plays today). Multipart above 256 MB (500 MB single-PUT cap), Modal-shaped listdir entries (.path, .type.name), FileNotFoundError parity, auto-created volume with pinnable id.
• Serving → one serverless endpoint per deployed model. create-serving-app writes a readable handler.py to the volume; a shared serving image loads it via HANDLER_PATH env (deploy = metadata, no image build — same property as Modal). Pure REST, no CLI subprocess. Per-model key enforced in-handler (input.api_key); rotation updates the template env (rolls on next cold start). endpoint_url = https://api.runpod.ai/v2/{id}/runsync.
• GPU mapping: canonical labels stay unchanged everywhere; runpod_provider/gpu.py maps each to an ordered RunPod pool (T4→RTX A4000-class, A10G→RTX A5000/A40, A100-40GB schedules and bills as 80 GB — surfaced in /deploy/compute-options notes and docs).
• Config/CLI/docs: RUNPOD_* settings with startup validation, trainable init compute-provider step (RunPod branch collects API key + S3 key pair + datacenter), .env.example block, docker/runpod-worker/ image + make runpod-image, docs/compute-providers.md.

✅ Verified automatically (no live account needed)

• Modal regression: entire pre-existing suite (296 tests) green immediately after the refactor commit and at HEAD — proves COMPUTE_PROVIDER=modal behavior is byte-for-byte preserved, including every existing modal.Sandbox.create / get_volume monkeypatch.
• 47 new unit tests (mocked RunPod client / in-memory S3 fake, no network):
  • factory: provider selection, singleton rebuild on switch, unknown-provider error, missing-credentials errors, provider-aware kernel ready timeout
  • sandbox: stdout/stderr stream routing, returncode extraction, TIMED_OUT→SandboxTimeoutError (rc 124), FAILED→nonzero rc, cancel-on-terminate, policy.executionTimeout + workdir in the run payload, lazy endpoint creation, oversized-payload guard, and a full run_code() pass on the runpod provider asserting usage is recorded with provider="runpod" + the actual GPU
  • storage: key mapping, write/read round-trip (str + bytes), FileNotFoundError parity for files and dirs, recursive + delimiter listings with synthesized directory entries, prefix-recursive remove, session-workspace bootstrap, live-reload no-op
  • serving: handler codegen ast.parse + api-key gate presence/absence, no /data/data double-prefix, xgboost-native loader selection, first-deploy creates template+endpoint (env, volume, DC, GPU pool asserted), CPU deploys use computeType=CPU, redeploy PATCHes instead of re-creating, stop scales to 0 then deletes, missing-endpoint-id error
  • kernel: cursor advance across polls, boot-error retry (502 → retry → events), /cmd POST shape, pod-delete on terminate, pod-create payload (mount path, ports, env, dockerStartCmd), worker scripts parse + long-poll stays under the 100 s proxy cap
  • db: ALTER-TABLE migration on a legacy-shaped database (columns added, existing rows default provider='modal', idempotent)
  • billing: every canonical GPU label resolves a nonzero runpod rate
• CLI wizard logic smoke-tested: _has_compute_creds truth table, .env write→read round-trip for the RunPod block, modal-only configs emit no RunPod keys.
• entrypoint.sh shell syntax (sh -n); ruff check + ruff format clean; frontend production build green; full suite at HEAD: 343 passed, 8 skipped.

🧑‍💻 Requires a human with a live RunPod account

These touch real RunPod infrastructure and cannot be exercised by CI. In order:

1. Publish the worker image (one-time prerequisite): make runpod-image RUNPOD_IMAGE=<your-registry>/trainable-runpod-worker:latest — needs docker buildx + a registry; verify the image is pullable by RunPod (public, or registry creds configured in the RunPod console). Nothing below works without this.
2. Wizard + boot: trainable init → pick RunPod → enter API key, S3 key pair, datacenter → trainable up. Confirm backend starts with no COMPUTE_PROVIDER validation errors and the network volume is auto-created (check the log line, then pin RUNPOD_NETWORK_VOLUME_ID in .env).
3. Code execution: start a session, ask the agent to run some Python. Verify stdout streams incrementally into the chat (not one blob at the end), created files appear in the file tree, and trainable.log(...) metrics render on the dashboard. Expect minutes of cold start on the very first call per GPU tier; a second call within ~60 s should be near-instant (warm worker).
4. Timeout path: run something like time.sleep(9999) with a short project timeout — the agent should receive the sandbox-timeout tool error, not a crash.
5. Notebook kernel: run a notebook cell; verify state transitions (starting→idle→busy), streamed output, interrupt works, and the pod is deleted after ~15 idle minutes (check RunPod console — this is a billing leak if it fails).
6. Storage round-trips: upload a dataset (>256 MB if possible, to exercise multipart), browse the file tree, open a file preview, delete a file.
7. Serving: create-serving-app → Deploy from /models → curl -X POST <endpoint_url> -H "Authorization: Bearer $RUNPOD_API_KEY" -d '{"input": {"records": [...], "api_key": "<model key>"}}' → predictions returned; wrong api_key → 401-shaped error; Rotate key → old key rejected after next cold start; Stop → endpoint gone from the RunPod console.
8. Billing: after 3–7, the Usage page shows sandbox rows priced with runpod rates and provider=runpod.
9. GPU tier (cost ~cents): set the project training profile to T4, run a heavy=true call, confirm nvidia-smi/torch.cuda.is_available() sees a GPU and the endpoint created in the console targets the RTX A4000 pool.
10. Modal regression smoke: flip .env back to COMPUTE_PROVIDER=modal, restart, rerun 3 + 7 briefly.

Known limitations / follow-ups

• Serving clients need a RunPod API key (no anonymous URLs on RunPod); request/response envelope differs ({"input": ...} / {"output": ...}) — documented with curl examples.
• Network volume is datacenter-bound; GPU availability varies per DC (ordered gpuTypeIds fallbacks mitigate).
• Kernel pods bill while alive (reaper caps at 15 idle minutes).
• RunPod-only SKUs (RTX 4090, L40S, H200…) not exposed as canonical labels yet.
• Follow-up: agent-selectable GPU per execute-code call with a per-project allowance (Agent-selectable GPU/CPU per execution with per-project allowance (presented in the system prompt) #146 / feat(agent): agent-selectable GPU/CPU per execute-code call with project allowance #160).

🤖 Generated with Claude Code

https://claude.ai/code/session_012GrGFBtdZv7qtgn8KQEQEB

Greptile Summary

This PR adds RunPod as a full-parity alternative compute provider to Modal, selected via COMPUTE_PROVIDER=modal|runpod. It introduces a clean four-interface abstraction (SandboxProvider, KernelTransport, StorageBackend, ServingBackend) with Modal and RunPod implementations, while keeping all existing Modal call-sites and test patch points unchanged.

• Refactor (commit 1): New backend/services/compute/ package with provider factory; all services/volume.py, sandbox.py, kernel_manager.py, and deploy.py call sites route through the abstraction with no signature changes. DB migration adds provider/provider_endpoint_id to deployments; billing rows now record the actual provider.
• RunPod implementation (commit 2): Serverless code-runner endpoints per GPU tier, long-lived CPU pods for notebook kernels via an HTTP gateway, S3-backed network volume storage, and one serverless endpoint per deployed model — all with 44 new unit tests (340 total passing).

Confidence Score: 3/5

RunPod code execution fails for scripts exceeding ~2 MB (Linux ARG_MAX) despite the backend accepting up to 5 MB, and large model artifacts written via write_to_volume bypass multipart and hit RunPod's 500 MB single-PUT cap.

Two concrete defects on the RunPod hot path: the worker subprocess rejects code via -c above ~2 MB while the backend permits 5 MB; and RunPodStorage.write() uses put_object without multipart, capping in-memory writes at RunPod's documented 500 MB S3 limit. The Modal path is untouched and its full suite passes. The abstraction is well-designed.

docker/runpod-worker/worker/runner_handler.py (subprocess ARG_MAX), backend/services/compute/runpod_provider/storage.py (write() missing multipart), backend/services/compute/runpod_provider/bootstrap.py (template/lock issues).

Important Files Changed

Filename	Overview
docker/runpod-worker/worker/runner_handler.py	Passes code as a -c argument to Python; fails for strings above Linux ARG_MAX (~2 MB) while the backend allows 5 MB.
backend/services/compute/runpod_provider/storage.py	write() uses put_object directly, bypassing multipart and RunPod's 500 MB single-PUT cap.
backend/services/compute/runpod_provider/bootstrap.py	Known reentrant-lock deadlock; image-match in template lookup causes duplicate templates on image updates.
backend/services/compute/runpod_provider/serving.py	rotate_key may not propagate the new API key if RunPod caches template env at endpoint-creation time.
backend/services/compute/runpod_provider/kernel.py	Well-structured pod + HTTP gateway transport; token auth and cursor-based polling are clean.
docker/runpod-worker/worker/kernel_gateway.py	FastAPI HTTP gateway wrapping AsyncKernelManager with ring-buffer event store.
backend/services/compute/init.py	Clean provider factory with startup validation and lazy singletons.
backend/services/compute/base.py	Well-defined abstract interfaces with clear contracts and provider-neutral SandboxTimeoutError.
backend/services/deploy.py	Provider-neutral orchestration cleanly routes through the backend interface; Modal paths preserved.
backend/services/volume.py	Stable facade routing all async helpers through StorageBackend correctly.
backend/db.py	Safe ALTER TABLE migration adding provider columns with proper existence checks.
backend/services/compute/runpod_provider/client.py	Thin async httpx wrapper with consistent error surfacing via RunPodAPIError.
backend/services/kernel_manager.py	Clean KernelTransport abstraction with provider-aware readiness timeout.

_{Reviews (2): Last reviewed commit: "test(db): cover the deployments provider..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

read_volume_file is Modal-only but not visibly guarded at call sites

The function still calls get_volume().read_file(path) (a raw modal.Volume method) instead of routing through _storage(). Any call site that invokes this path when COMPUTE_PROVIDER=runpod will either fail if Modal is not configured or silently read from the wrong volume. The async twin read_volume_file_async was correctly migrated to _storage().read_file(), but the sync version was deliberately left behind. Consider raising NotImplementedError or logging a clear warning when the active provider is not Modal.

[greptile-apps]

Too many files changed for review. (192 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

[lucastononro]

Merged into integration branch staging-v0.0.5 (see the STAGING-v0.0.5.md ledger on that branch for merge order, Greptile follow-ups and test results). These changes will land on main via the staging merge — closing to clear the queue.