Skip to content

feat(jobs): runcrate run and runcrate jobs (RUN-486) - #4

Merged
heydryft merged 4 commits into
mainfrom
feat/jobs
Oct 20, 2026
Merged

heydryft merged 4 commits into
mainfrom
feat/jobs

Conversation

@heydryft

Copy link
Copy Markdown
Member

Adds jobs to the CLI: one command on its own GPU machine, its output streamed back, and the machine stopped when the command ends (RUN-486).

Commands

  • runcrate run --gpu A4000 [--gpu-count 1] [--volume <id>] [--max-runtime 2h] [--template t] [--image img] [--env K=V ...] [--name n] [--region r] [-d/--detach] -- <command...>
    • One argument after -- is sent as is (-- 'make && ./train'); several are joined with spaces, quoting the ones that need it (-- python -c 'print(1)' runs as typed). Flags stop at the first argument, so the command's own flags pass through.
    • --max-runtime: Go duration or bare minutes, rounded up to whole minutes, 1m to 24h, default 1h (printed with a hint to change it).
    • --env KEY with no value copies it from the local shell.
    • Prints the job, then follows logs (follow=true long polls with the cursor), writing each chunk to stdout verbatim; status notes and the final line (status, exit code, runtime, cost) go to stderr.
    • Exit code: succeeded 0, failed the command's code (or 1), timed_out 124, cancelled 130.
    • Ctrl-C detaches and never cancels: prints runcrate jobs logs -f <id> and runcrate jobs cancel <id>, exits 0. --detach prints the job ID and returns.
    • Network errors, 429 and 5xx while following are retried with backoff (1s up to 15s); the client is rebuilt before each poll so long follows pick up refreshed sign-in tokens.
  • runcrate jobs list [--status running,queued] [--limit 20] [-q] [--json]
  • runcrate jobs status <id> [--json]: exit code, error, cost, instance, env keys, volume, timestamps.
  • runcrate jobs logs <id> [-f]: without -f, pages from after=0 until the cursor stops; with -f, follows like run (same detach and exit codes).
  • runcrate jobs cancel <id>

API client

  • internal/api/jobs.go: CreateJob, ListJobs, GetJob, GetJobLogs, CancelJob. Follow requests carry a 40 s deadline through the request context; other calls keep the client's timeout.
  • checkHTTPStatus now returns a typed *StatusError (same message text as before) with code and details. A 402 BUDGET_REACHED reads "this environment's monthly budget is reached; an owner or admin can raise it in the dashboard". A launch failure after creation names details.job_id.

Before merging / releasing

  • Needs the app PR deployed first: feat/jobs in aeonmindai/runcrate_app (RUN-486), which adds /api/v1/jobs.
  • Needs a release (vX.Y.Z tag) after merge so runcrate update and the installers pick it up.

Tests
go vet ./..., go test ./... (also -race -count=5 on api and commands) and go build ./... pass. New tests use httptest fakes: max-runtime parsing, command joining, the create body (gpu_type, max_runtime_minutes, env object, storage_id), the follow loop (chunks in order across responses, cursor honored, null-seq marker printed, retry on 503, stop on 404, stop on done), exit codes per status, and Ctrl-C detach aborting an open long poll without calling /cancel.

@heydryft

Copy link
Copy Markdown
Member Author

App side: aeonmindai/runcrate_app#169 (adds /api/v1/jobs). Merge and deploy that first, then release this.

@heydryft
heydryft merged commit eaa0118 into main Oct 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant