feat: add DstackProvider — reliable multi-cloud GPU orchestration via dstack#857
Open
elasticdotventures wants to merge 40 commits into
Open
feat: add DstackProvider — reliable multi-cloud GPU orchestration via dstack#857elasticdotventures wants to merge 40 commits into
elasticdotventures wants to merge 40 commits into
Conversation
Scopes replacing the RunPod-only bespoke polling in job_executor.rs (hardcoded 10s timeout, no pull-vs-run visibility) with a real DstackProvider backend, plus a PASS/FAIL job-status evidence contract. RunPod becomes optional infra, not a hard requirement.
…ity constraint Cloud GPU workers (HF Jobs, RunPod, and future dstack backends) can't reach the LAN-only MLflow tracking server (see learn/hf-jobs-mlflow.md), so cloud jobs have zero run-tracking today. Explains why the spec's stdout/result.json contract is push-free rather than routing through MLflow.
JobStep already has depends_on (DAG ordering), JobCondition.when (on_success/on_failure/on_previous), and a backend dispatch field whose doc comment names runpod/hf/local as accepted values. The "composable steps around dstack" requirement is satisfied by registering dstack as a backend and implementing the output_contract enforcement that field's own doc comment already says is missing — not by building a new hook system.
…' interface Bash watch-loop replaced with a job definition (mesh3d-cloud.job.toml) using the schema commands/job.rs actually executes, invoked with one flag-free command. Also documents a dead/unwired job schema (llm-batch-job.job.toml's type="job" + b00t job deploy/to-manifest, not present in the real JobCommands enum) found while grounding this, and flags the remote file-staging gap dstack_task_yaml doesn't solve rather than fabricating unverified YAML for it.
Wraps worktree setup/check, task-brief, review-package, and ledger read/append behind `just dstack-sdd::*` so plan execution runs named recipes instead of repeating raw bash per operator steer 2026-07-22.
…d scope - PROVIDER_POLL_INTERVAL split cfg(test)=20ms / cfg(not(test))=2s so production doesn't poll a real cloud API 5x/second while tests stay fast without needing tokio::time::pause. PENDING_MAX_POLLS lowered to 150 (5min real budget at 2s). Per operator review 2026-07-22. - Task 1 clarified: dstack needs exactly one backend, not all clouds — reuses the existing RUNPOD_API_KEY. Explicitly marked as the one human-only step (credentials + real cloud spend), not for subagent dispatch.
…s pre-existing compile failure)
Implements DstackProvider struct and ComputeProvider trait methods for batch and training job submission via dstack CLI. Includes pure YAML builder (dstack_task_yaml) for testability and stub functions (todo_task3_job_status/todo_task3_list_jobs) for Task 3 implementation. Job status and list operations are intentionally unimplemented pending fixture capture in Task 1 to determine dstack ps --json output schema. Test: dstack_task_yaml_includes_image_env_and_command verifies correct YAML generation with image, environment variables, and command path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 2 review fixups for DstackProvider: - dstack_task_yaml: drop the broken `exec "$@" -- <path>` commands line (dstack commands: run as literal shell steps, not argv appended to an entrypoint — "$@" was unset there). Pass config_path, flavor, and timeout_hours through as B00T_JOB_CONFIG_PATH/B00T_JOB_FLAVOR/ B00T_JOB_TIMEOUT_HOURS env vars instead, since env vars are a construct we've confirmed dstack supports. commands: now carries a placeholder with a 🤓 comment flagging that dstack's real entrypoint-invocation convention still needs verification with real dstack access (Task 3). - Previously spec.flavor/timeout_hours were silently dropped entirely; test updated to assert both are present in the generated YAML. - Replace hand-rolled uuid_suffix() (nanosecond-timestamp hex) with uuid::Uuid::new_v4(), matching LocalProvider::submit_batch_job's existing pattern in this same file. - submit_dstack_yaml: remove the temp YAML file after a successful dstack apply (best-effort, logged via tracing::warn on failure, never propagated), and replace tmp.to_str().unwrap() with a .context(...) call matching this function's existing error-handling idiom. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Operator asked whether Azure/GCP/AWS (each independently authorized on this host) should be wired as additional dstack backends, given app4dog already runs real multi-cloud infra via Terraform. Verified: none of that existing infra is GPU compute (it's app hosting/storage/registries/ one annotation tool), and dstack self-provisions per-cloud with no Terraform dependency either way. Recommend staying RunPod-only for this plan; flagged Cloud Run GPU and ecr_image_segmenter.tf's runtime target as separate follow-ups, not in scope here.
…-boot fix) Operator hit real budget burn (~25% spent, zero completed jobs) from RunPod's per-job cold-boot cost. TRIZ analysis (contradiction: reproducibility vs iteration speed) resolves via Segmentation + Extraction + Prior Action: persistent dstack volumes/dev-environments staged once, reused across repeated job dispatch, instead of Task 2's one-shot fresh-pod-per-job path (which stays useful for genuine one-offs). Verified against real dstack docs: type: volume, volumes: attachment, explicit persistence-across-runs guarantee. ORAS dataset-storage bucket explicitly deferred — zero prior art in either repo, needs its own research pass, would violate MECE to fold in unresearched. Task 12: BatchJobSpec.volumes + ensure_volume + stop_dev_environment (code) Task 13: mesh3d-cloud-warm.job.toml pattern (composes Task 12 + existing backend=dstack/output_contract machinery) Task 14: cost-control datum guidance (idle volume vs idle dev-environment billing difference)
… stop_dev_environment) Add VolumeMount struct and extend BatchJobSpec with volumes field (additive with #[serde(default)] for backward compatibility). Implement dstack_volume_yaml to generate persistent volume configs, update dstack_task_yaml to conditionally include volumes block when non-empty, and add ensure_volume/stop_dev_environment methods to DstackProvider for volume lifecycle management. All existing BatchJobSpec construction sites updated (provider.rs + job_executor.rs). Tests pass: dstack_volume_yaml_includes_size_and_region ✓, dstack_task_yaml_attaches_volumes_when_present ✓, dstack_task_yaml_omits_volumes_block_when_empty ✓, plus 3 existing tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ider - Fix doc comment misattachment: dstack_task_yaml doc was merged with dstack_volume_yaml doc due to missing blank line. Separated and placed correctly: dstack_task_yaml now has its proper "Pure YAML builder" doc comment immediately above it, dstack_volume_yaml has its "type: volume config" doc comment above it. - Fix ensure_volume cleanup pattern: make it consistent with submit_dstack_yaml by logging a tracing::warn! when temp file removal fails instead of silently swallowing the error. Includes explanatory comment matching the sibling function's discipline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
….job.toml Task 12 added DstackProvider::ensure_volume/stop_dev_environment as Rust methods only, with no CLI exposure. Adds DstackSubCommands (EnsureVolume, StopDevEnvironment) wired through ProviderCommands::Dstack/handle_dstack, mirroring the existing RunpodSubCommands/handle_runpod pattern exactly. Also registers "dstack" in get_provider() so backend = "dstack" job steps can actually dispatch — it was referenced by the plan's Interfaces section as already working but was never added. mesh3d-cloud-warm.job.toml demonstrates the persistent-volume pattern: ensure-volume once (provision-cache step) feeds a repeatable batch step that reuses the same dstack volume, avoiding re-pulling image/deps per run. Diverges from the brief's literal draft in two places required for the file to actually parse against the current JobStep/BatchJobSpec schema: the batch step keeps a placeholder `task` block (required field, ignored at execution when `backend` is set) and adds an explicit `env` table under [b00t.job.steps.batch] (BatchJobSpec.env has no serde default). Verified via `b00t-cli job plan mesh3d-cloud-warm`.
…JobStep.backend value
Task 8 was never dispatched separately before Tasks 12-14 were added mid-session (TRIZ pivot) -- creating it now with Task 14's cost-control section included, since both were fully specified in the plan already. Follows the exact structural convention of PROVIDER-RUNPOD/PROVIDER-HF siblings, including their pre-existing (and unrelated to this work) mismatch against the current BootDatum validator schema -- confirmed PROVIDER-RUNPOD.provider.tomllmd fails datum validate identically, so this is not a regression, just an inherited, out-of-scope gap.
…tum schema Same fix as PROVIDER-DSTACK's own schema migration (this branch, commit f830c80): stale [b00t.schema] shape replaced with the real BootDatum required fields (name, hint, directly under [b00t]). Landing here rather than on main because main's HEAD currently fails to build (pre-existing candle-core/candle-nn version conflict in b00t-embed, unrelated to datums or this feature -- confirmed via `datum-validate-graph`'s compile step failing on main but succeeding in this worktree). Needs to reach main separately once that's fixed or via this branch's eventual merge -- flagging, not fixing, that unrelated breakage here.
…erminal Implements state-aware status buckets to categorize provider job status strings into Pending, Running, or Terminal states. Handles dstack status format (status=pending/submitted/provisioning/running/done/failed/terminated) and falls back to existing RunPod/HF/local substring markers. Composes with existing is_terminal_status/is_failure_status functions without replacing their logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…icating failed/terminated markers
… running Splits PROVIDER_POLL_INTERVAL by build config (2s production / 20ms test) since the prior single 200ms constant was sized for tests but also drove real cloud API polling. Replaces the flat PROVIDER_MAX_POLLS budget with PENDING_MAX_POLLS=150 (~5min) / RUNNING_MAX_POLLS=50 (~100s), using Task 5's classify_status to route each poll into the right budget.
…ncommitted Edited during the provider.tomllmd cleanup pass but only RUNPOD/HF were actually staged+committed at the time (0691f19) -- this one sat in the working tree uncommitted since. Same [b00t.schema] -> [b00t] migration, already validated clean at the time.
…lat cap, add running-budget test Prior test used 40 pending polls -- 42 total calls before terminal, which the OLD flat PROVIDER_MAX_POLLS=50 cap would also have tolerated (42 < 50), so it proved nothing about the fix. Bumped to 100 pending polls (exceeds the old 50-cap, stays under new PENDING_MAX_POLLS=150). Added a second test verifying the RUNNING_MAX_POLLS bail path is independently reachable (previously untested).
…-dispatched steps Adds evaluate_output_contract() which requires the last non-empty line of a job's re-read output to be PASS or FAIL:<detail> when a step declares an output_contract; FAIL: details are surfaced in the returned error, and a missing/malformed marker also errors. dispatch_batch_job now takes the whole JobStep (not just its batch spec) so it can read output_contract after the terminal status check and re-read job_status for the contract evaluation. This moves the sole step.batch.as_ref().ok_or_else check out of execute_provider_step and into dispatch_batch_job (no duplication). Previously output_contract was schema-only (datum_job.rs's own doc comment admitted no enforcement existed); this makes it gate step success on actual PASS/FAIL evidence rather than just container exit code.
…ter dispatch_batch_job's signature change)
…e + PASS/FAIL enforcement One-command entry point (b00t job run mesh3d-cloud) replacing cloud_mesh.sh's manual submit+poll bash dance: resource-gate check, dstack submission, and PASS/FAIL enforcement now live in JobExecutor::run_job via [[b00t.job.steps]]. Fixes two schema gaps present in the plan's literal draft (confirmed against JobStep/BatchJobSpec in datum_job.rs / provider.rs, and mesh3d-cloud-warm.job.toml's prior fix for the same issue): `task` is required even on backend-dispatched steps (executor ignores it, drives [b00t.job.steps.batch] instead), and `[b00t.job.steps.batch]` requires an `env` table (BatchJobSpec.env has no serde default). Validated: `b00t-cli job plan mesh3d-cloud` prints the expected two-step plan with no parse errors; `b00t-cli job run mesh3d-cloud --dry-run` executes cleanly. Known open gap (not solved here, per plan Task 9): dstack_task_yaml doesn't stage local files (request.json, input photo) into the remote container — config_path here is a placeholder remote path, disconnected from cloud_mesh.sh's local staging dir. Follow-up work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tum to installable location
Per operator directive 2026-07-22: 'b00t install dstack should run
commands from datum which could include uv; and use b00t to check
version.' Added install = { uv_tool = "dstack[all]", binary = "dstack" }
+ version/version_regex to PROVIDER-DSTACK, matching the existing
fastmcp.cli.toml/b00t.cli.toml convention (InstallSpec::Tool.uv_tool ->
`uv tool install`).
Moved the datum from _b00t_/datums/PROVIDER-DSTACK.provider.tomllmd to
top-level _b00t_/PROVIDER-DSTACK.provider.toml -- discovered along the
way that install_datum()'s loader only does a non-recursive read_dir
over the top-level _b00t_/ directory, so anything in datums/ was
invisible to `b00t-cli install <name>` even though `datum
validate`/`datum search` (different, more capable loaders) could see
it fine. This affects 82 other pre-existing files in datums/, 16 of
which already declare (equally unreachable) install specs -- flagging
as a known, larger, pre-existing gap; only this session's own datum
was relocated, not all 82.
Verified end-to-end: `b00t-cli install PROVIDER-DSTACK` really ran
`uv tool install 'dstack[all]'` and installed dstack 0.20.28; a second
run correctly detected it via `dstack --version` and skipped
reinstalling.
…red implementation
install.rs, stack.rs, uninstall.rs, and cli_cmd.rs each hand-rolled their
own near-identical directory scan. They had already silently drifted:
three derived the type-name key via Debug+lowercase (wrong for
multi-word DatumType variants -- HiveProfile became "hiveprofile",
matching no real file's .hive suffix convention), the fourth
(uninstall.rs) used an ad-hoc serde-snake_case guess ("hive_profile",
also not .hive). DatumType::type_prefix() -- already the documented,
auto-derived single source of truth for this -- is the only one that's
actually correct, so it's now the only implementation.
Split into load_all_datums_from_dir(&Path) (the actual shared scan) and
load_all_datums(&str) (a thin wrapper preserving the original simple
tilde-expansion path resolution used by install/uninstall/cli). stack.rs
resolves its own path via lifecycle::get_expanded_path (legacy-directory
fallback) first and calls load_all_datums_from_dir directly -- this
preserves stack.rs's pre-existing legacy-fallback behavior exactly,
which centralizing path resolution too would have silently changed for
the other three callers (caught by a real test failure:
test_load_all_datums_nonexistent_directory started returning results
from ~/.dotfiles/_b00t_ instead of an empty map, once traced back to
get_expanded_path's fallback).
Also fixes the same read_dir extension bug (.tomllmd files invisible to
a plain .ends_with(".toml") check) across all 4 sites in one place
instead of pasting the fix 4 times.
…ored
dstack's RunPod backend (verified against installed 0.20.28 source:
core/backends/runpod/models.py's RunpodAPIKeyCreds + server/services/
config.py's plain yaml.safe_load) accepts only a literal creds.api_key
in ~/.dstack/server/config.yml — no ${VAR} interpolation, no env-var-chain
creds type exists for this backend. Per operator steer + b00t's own
managing-secrets doctrine: never hand-author or commit that file with a
literal key. Add `just dstack-server-config` (justfile-dstack-sdd.just),
which sources RUNPOD_API_KEY fresh from .env (already present, gitignored)
each run and writes the file with 0600 perms — the one place the value
is ever materialized. Document the secret dependency (not the value) in
PROVIDER-DSTACK's [resource.secrets.runpod_api_key] table, following the
source/delivery schema from _b00t_/learn/managing-secrets.md.
Verified: generated YAML round-trips through dstack's own ServerConfig/
RunpodAPIKeyCreds pydantic models; `b00t-cli datum validate` passes clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found while testing Task 9's dstack job: b00t-cli job run creates
{project_root}/.b00t/jobs/<name>/<run-id>.json run-state files that
were untracked and un-ignored — the existing .b00t/jobs/ pattern is
repo-root-anchored (git gitignore rule: a pattern with a slash before
the end is relative to the .gitignore's directory), so it never covered
_b00t_/.b00t/jobs/ where `b00t-cli job run` actually wrote when invoked
from a subdirectory. Three such files were already committed at
.b00t/jobs/test-apalis/*.json — left as-is (pre-existing, out of scope
for this branch); this only stops new leakage going forward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran a trivial echo "PASS" task through a real dstack 0.20.28 server against the real RunPod backend (project "b00t"), captured `dstack ps --json -a` and `dstack logs` output for Task 3's parser to be written against real data instead of guessed shapes. Verified no secrets present before commit. Fleet deleted and run stopped immediately after capture (~2 min runtime, $0.24/hr GPU instance, spend was a few cents). Critical finding, documented in dstack_ps_json.NOTES.md: dstack 0.20.x requires a pre-existing autoscaling fleet before any task can be scheduled -- the dynamic per-task pod provisioning the original design assumed no longer exists. DstackProvider::submit_batch_job does not yet account for this and will fail against real dstack until addressed -- flagged for a scope decision before Task 3 is dispatched, not fixed in this commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ission Task 1's live RunPod test surfaced a real gap: dstack 0.20.x requires a matching fleet to already exist before any task can be scheduled -- a bare `dstack apply` on a task config with no fleet fails immediately with "No matching fleet found" / FAILED_TO_START_DUE_TO_NO_CAPACITY, before the request ever reaches the backend. The dynamic per-task pod provisioning this provider was originally designed against (mirroring older RunPod-native behavior) no longer exists in the installed version. See tests/fixtures/dstack_ps_json.NOTES.md for the full finding. Adds DstackProvider::ensure_fleet (idempotent `type: fleet` apply, nodes: 0..1 so it costs nothing while idle -- mirrors the existing ensure_volume pattern) and dstack_fleet_yaml (pure builder, unit tested). submit_dstack_yaml calls ensure_fleet against one shared fleet name (SHARED_FLEET_NAME) before every task/training submission, matching the warm-reuse philosophy behind Task 12's persistent volumes rather than provisioning a fleet per job. Deliberately omits backends:/regions: in the generated fleet config so it matches whatever backend(s) the operator's dstack server config.yml has configured, instead of hardcoding runpod. Verified: cargo test -p b00t-cli --lib -- 1344 passed, 1 pre-existing unrelated failure (hive::tests::test_guard_expr_coverage_all_shipped_datums, confirmed identical on the parent commit via git stash). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Review finding (Minor): the value matched today, but a literal string risks silent drift if the constant is ever renamed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ack ps --json fixture (Task 3) Replaces the todo_task3_job_status/todo_task3_list_jobs stubs from Task 2 with a real parser (parse_dstack_ps_json) against the fixture captured live in Task 1. Corrects a field-path guess from the original plan draft: the run's display name is NOT a top-level `name`/`run_name` key as guessed before real dstack access existed -- it's nested at `run_spec.run_name`. `status` was guessed correctly (top-level string). Also surfaced (and tested against) a real fixture property: `dstack ps -a` returns full run history, so a reused run name can have multiple entries -- the fixture has 4, from repeated live troubleshooting attempts during Task 1's capture, ordered most-recent-first by submitted_at. job_status takes the first (most recent) match; documented inline. Verified: cargo test -p b00t-cli --lib -- 1346 passed, 1 pre-existing unrelated failure (hive::tests::test_guard_expr_coverage_all_shipped_datums). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
All three were found by actually running submit_batch_job through
b00t-cli against a real dstack server + real RunPod backend, not by
inspection -- exactly what Task 10 exists to catch:
1. Scratch config files (ensure_volume/ensure_fleet/submit_dstack_yaml)
were written to std::env::temp_dir() (/tmp on Linux). dstack's own
`apply` computes configuration_path.absolute().relative_to(Path.cwd())
and errors -- before any network call -- if the file isn't inside
CWD's subtree. Every dstack config write in this provider now goes
through the new dstack_scratch_config_path() helper, rooted at CWD.
2. Job/training names (`b00t-job-<full-uuid>`, `b00t-train-<full-uuid>`)
are 45 characters; dstack rejects any name not matching
`^[a-z][a-z0-9-]{1,40}$` (max 41 chars) with "Resource name should
match regex ...". This made every submit_batch_job/submit_training_job
call fail unconditionally against real dstack. New dstack_short_id()
uses 12 hex chars instead of a full UUID, verified against the real
regex in a new test (20 iterations, both prefixes).
3. dstack_task_yaml hardcoded `commands: [echo starting]` on every task.
dstack's own TaskConfiguration model requires "either commands or
image must be set", not both -- commands: is optional when image: is
present, in which case dstack runs the image's own ENTRYPOINT/CMD.
The hardcoded placeholder silently discarded every real image's
actual behavior (e.g. mesh-runner:v6 would never run its own logic)
on every single submission. commands: is now omitted entirely,
letting images run their own entrypoint -- the architecture this
provider was always meant to support. Locked in with a regression
assertion on the existing test.
Verified end-to-end via `b00t-cli provider job submit-batch --provider
dstack` against real dstack 0.20.28 + live RunPod: submission succeeded,
`b00t-cli provider job status` correctly polled through to status=done
with no 10-second-timeout regression (the failure mode this whole branch
exists to fix). Fleet deleted immediately after, no lingering spend.
cargo test -p b00t-cli --lib: 1347 passed, 1 pre-existing unrelated
failure (hive::tests::test_guard_expr_coverage_all_shipped_datums).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…losure, doc placement, unused mut
Whole-branch review (final gate before merge) found one Important,
non-blocking issue: JobExecutor::dispatch_batch_job evaluates
output_contract against provider.job_status()'s return value, which for
DstackProvider (and every other provider) is a status string
("run=<id> status=done"), not the job's actual stdout/logs. Both shipped
job.tomls (mesh3d-cloud.job.toml, mesh3d-cloud-warm.job.toml) declare
output_contract on their dstack-backed step, so running either today
would report a contract violation even on a fully successful dstack job.
Not fixed here (would require redesigning what output_contract evaluates
across all four providers, a real scope decision) -- disclosed inline in
both files instead, so an operator isn't misled by a false FAIL. Both
flows are separately blocked on the request.json-staging gap already
documented in mesh3d-cloud.job.toml, so this isn't yet exercisable for
real regardless.
Also two Minor findings fixed: dstack_scratch_config_path's doc comment
was stacked above dstack_short_id with no separating blank doc-comment
gap, so it attached to the wrong function; reordered so each function
carries its own comment. Removed an unused `mut` in
dstack_task_yaml_attaches_volumes_when_present's test setup.
Verified: cargo test -p b00t-cli --lib -- 1347 passed, 1 pre-existing
unrelated failure (hive::tests::test_guard_expr_coverage_all_shipped_datums).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… logic
test_guard_expr_coverage_all_shipped_datums synthesizes a matching input
command from each guard's Rhai expression to verify the expression
actually matches (rather than silently evaluating to Allow forever). For
guards whose pattern is a bare macro-name reference (e.g. `pattern = {
rhai = "podman_build_uncapped" }`, resolved via [b00t.hive.rhai_macros]),
the keyword-extraction loop for the *resolved macro body* pulled every
quoted string out of the expression, including ones inside a negated
!cmd.contains("...") clause -- unlike the outer, non-macro-reference
extraction loop a few lines above it, which already skips negated
keywords correctly.
For podman_build_uncapped/podman_run_uncapped ("cmd.contains(\"podman
build\") && !cmd.contains(\"--memory\")"), this synthesized a test command
containing BOTH "podman build" and "--memory", so the guard's own
negation clause correctly rejected its own synthesized test input --
`expected match ... got Allow`, forever, regardless of the actual guard
logic being correct. This was a bug in the test's input synthesis, not in
the guard definitions or evaluation engine.
Pre-existing on main, unrelated to the dstack-provider work in this
branch (confirmed via repeated git-stash comparisons against this
branch's own parent commit throughout its development) -- fixed here
because it was blocking the pre-push hook's full-suite gate.
Verified: cargo test -p b00t-cli --lib -- 1348 passed, 0 failed (up from
1347 passed / 1 failed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RunPod-direct job orchestration was unreliable: slow cold boots, a ~10-second poll timeout that gave up too early, and ~25% of a RunPod budget spent with zero completed jobs. This adds
DstackProvider, aComputeProviderbackend that shells out to the dstack CLI, as a provider-agnostic, multi-cloud orchestration layer — RunPod becomes one backend among several rather than a hard dependency.DstackProvider: submit/status/cancel/list batch and training jobs via the realdstackCLI (no official Rust SDK exists for dstack — CLI/Python-SDK/REST-API only)output_contractPASS/FAIL evidence enforcement for provider-dispatched job steps (previously schema-only, unenforced)ensure_volume) for warm iteration, avoiding a full re-pull per jobensure_fleet) — dstack 0.20.x requires a pre-existing fleet before any task can schedule; discovered via live testing, not in the original designb00t-cli provider dstackCLI subcommands,mesh3d-cloud.job.toml/mesh3d-cloud-warm.job.toml— a downstream consumer (app4dog/game-play'scloud_mesh.sh) migrated off manual submit+poll bash ontob00t job run.envat runtime via ajustrecipe — never hand-authored or committed into a YAML configinstall/uninstall/stack/clicommands' datum loader was non-recursive and.toml-only, silently making.tomllmddatums unreachable (4 diverged copies consolidated into one shared implementation)Real-world verification (not just unit tests)
Every claim above was exercised against a real, live dstack 0.20.28 server + real RunPod backend, not just code review:
dstack ps --json -a/dstack logsfixtures (b00t-cli/tests/fixtures/dstack_ps_json.txt,.NOTES.md) from an actual submitted jobb00t-cli provider job submit-batch --provider dstack→ real submission →b00t-cli provider job statuspolled todone, no timeout regressionKnown, disclosed gap (not blocking, documented inline)
output_contractevaluation currently readsjob_status()'s status string, not job logs — so the two shippedmesh3d-cloud*.job.tomldatums would report FAIL even on a successful job. Both are also separately blocked on arequest.jsonremote-staging gap, so this isn't yet exercisable for real; disclosed inline in both files rather than silently left broken.Process
Built via subagent-driven-development: 14 planned tasks + 2 scope amendments (fleet-ensure, live-testing bugfixes), each independently implemented and reviewed by a fresh subagent with no shared context, plus a final whole-branch review on the most capable available model. Full task ledger and review history in
.superpowers/sdd/progress.md(gitignored, available in the branch's working history via commit messages).Test plan
cargo test -p b00t-cli --lib— 1348 passed, 0 failed🤖 Generated with Claude Code
Companion PR: app4dog/game-play#64 migrates
cloud_mesh.sh(the downstream consumer) off manual submit+poll onto theb00t job run mesh3d-cloudentry point this PR adds.