diff --git a/README.md b/README.md index 6e739b0..2fa6624 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,112 @@ -# forge — Good Ancestor model lab +# forge — an agent-operated model lab -Agent-operated toolkit: per-GPU-class training environments, NAS dataset -archive, lease-safe training runs, and model-surgery recipes. +**forge** is the toolkit the Good Ancestor Foundation uses to run its own model +work on its own hardware: preparing per-GPU-class training environments, +maintaining a governed dataset archive, and executing training, fine-tuning, +quantization, and model-surgery jobs safely on a shared GPU fleet. -- Spec: GoodAncestor HQ `docs/superpowers/specs/2026-07-10-forge-design.md` -- Operators (LLM): read `RUNBOOK.md` -- Humans: you review PRs and flip `approved:` flags; that's the whole job. +It is designed to be driven by autonomous agents but governed by humans. Agents +do the mechanical work — resolving environments, fetching approved datasets, +launching leased runs. Humans do exactly two things: **review pull requests** +and **flip `approved:` flags**. Nothing consequential happens without a human in +that loop. + +## Why it exists + +Running models on a real fleet involves a lot of fiddly, error-prone work: which +CUDA/torch stack matches which GPU generation, which datasets are cleared for +use, whether a GPU is free or serving production, whether a job checkpointed +before it was killed. forge encodes all of that as small, testable, +single-purpose commands with machine-readable output — so an agent can operate +the fleet without improvising, and so every consequential step leaves an +auditable artifact. + +## What's in the box + +- **Per-GPU-class environments.** Each hardware class (e.g. Ada, Blackwell, + datacenter) is its own isolated project with its own pinned lockfile, so the + torch/CUDA stack always matches the silicon. A weekly job re-resolves the + locks and opens a PR; hardware smoke-tests run on the target GPU before a + human merges. +- **A governed dataset archive.** A single `manifest.yaml` is the source of + truth for every dataset: its source, license, size, storage path, and an + `approved:` flag that only a human flips. Fetches are refused for anything + unapproved, oversized against a cap, or of unknown size — no silent + workarounds. +- **Lease-safe runs.** Every training/quant/surgery job runs under a fleet lease + with a heartbeat, so two agents never contend for the same GPU, and a job that + touches a production-serving node is flagged loudly. Runs are resumable and + record a signed manifest of exactly what happened. +- **Recipes.** Declarative YAML recipes for the common jobs — LoRA SFT, DPO, + GRPO, AWQ/GGUF/NVFP4 quantization, abliteration, and expert pruning — each + pinned to the environment profile it needs. ## Quick start - uv sync # CLI + dev deps - uv run forge --help - uv run pytest -q + +```sh +uv sync # install the CLI + dev deps +uv run forge --help +uv run pytest -q # run the test suite +``` + +## The CLI + +Everything is one `forge` command with a few subcommands. Add `--json` to any of +them for machine-readable output (this is how agents drive it). + +```sh +forge env update # re-resolve env lockfiles, open a PR +forge data list # what datasets exist / are approved +forge data sizes # measure sizes against the cap +forge data fetch # fetch one approved dataset (resumable) +forge run recipes/sft-lora.yaml \ # run a recipe under a fleet lease + --host --profile +forge lease status # who holds which GPU right now +``` + +## The safety model + +forge is opinionated about not doing damage: + +- **Approval gates are final.** An unapproved or refused dataset fetch exits + non-zero and stays refused until a human edits `approved:` via a reviewed PR. + Agents do not work around the gate. +- **Automated PRs are never auto-merged.** The weekly environment update opens a + PR and stops. A human reviews it, runs the hardware smoke-test, and merges. +- **Leases before GPUs.** Nothing runs on a GPU it hasn't leased, and jobs warn + before touching a node that's serving production traffic. +- **Honest exit codes.** Commands distinguish "refused" (2), "job failed" (3), + "infrastructure unreachable" (4), and "checkpointed/paused" (75), so callers — + human or agent — never mistake one failure for another. + +## Repository layout + +``` +src/forge/ # the CLI and its modules (env, data, run, lease, envupdate) +envs/ # one uv project per GPU class, each with its own lockfile +datasets/ # manifest.yaml — the governed dataset archive +recipes/ # declarative job recipes (sft, dpo, grpo, quantize, ...) +rewards/ # reward functions for preference/RL recipes +bootstrap/ # scripts that prepare a fresh GPU node +ops/ # operational notes (scheduling, etc.) +RUNBOOK.md # the operator's guide (read this if you're driving forge) +``` + +## For humans + +If you're a person looking at this repo, your job is small and well-defined: + +1. **Review PRs** — especially environment-update PRs (check the version diff and + run the smoke command the PR body gives you) and any PR that flips a dataset + `approved:` flag. +2. **Decide what's approved** — you own the `approved:` flags in + `datasets/manifest.yaml`. Agents propose; you dispose. + +That's the whole job. Everything else is mechanical, tested, and automated. + +## Status + +forge is used in production by the Good Ancestor Foundation. It is shared here in +the spirit of the Foundation's mission — helping others stand up their own local, +sovereign AI infrastructure. Interfaces may change; the safety principles above +will not. diff --git a/RUNBOOK.md b/RUNBOOK.md index 198d65a..cbbbcd3 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -4,8 +4,11 @@ You are operating the Good Ancestor model lab. Everything below is CLI + JSON. Humans only: merge PRs, flip `approved:` flags, sign off on production-node runs. ## Environment -- Aggregator/fleet state: GET $FORGE_FLEET_URL/fleet.json (default http://192.168.77.36:8096) -- Required env: FORGE_LEASE_SECRET (from jumpbox fleet secrets). Optional: +- Aggregator/fleet state: GET $FORGE_FLEET_URL/fleet.json (set FORGE_FLEET_URL + to your aggregator; default http://localhost:8096) +- FORGE_FLEET_CONFIG: path to the private fleet overlay (profiles.hosts.yaml) + that binds each profile to real hosts. Unset → profiles have no hosts. +- Required env: FORGE_LEASE_SECRET (from your fleet secrets). Optional: FORGE_DATA_ROOT (/mnt/models/datasets), FORGE_MODEL_ROOT (/mnt/models), FORGE_DATA_CAP_TB (50 — meters the dataset subtree only, NOT the models sharing the pool; the practical ceiling is pool free space — keep ZFS ~20% free), @@ -53,12 +56,18 @@ Humans only: merge PRs, flip `approved:` flags, sign off on production-node runs datasets/manifest.yaml via PR. Do not work around the gate. ## Env maintenance - forge --json env update # lock upgrade + leased smoke + PR -- Weekly cron does this. Read reports/env-update-*.json; escalate if ok=false. + forge --json env update # re-resolve lockfiles + open PR +- Weekly cron does this. It runs where the cron lives (no GPU needed): it only + re-resolves the locks and opens a PR. `ok` = locks resolved cleanly. +- Hardware smoke is deferred to PR review: the PR body carries the exact + `forge run recipes/smoke.yaml ...` command to run on each profile's + smoke_host before merging. Read reports/env-update-*.json; escalate if + ok=false (a lock failed to resolve). - NEVER merge the PR yourself. ## Safety rails you must respect -- One purposeful connection at a time to any host (UniFi IPS blocks scanners). +- One purposeful connection at a time to any host (network IPS blocks scanners). - Never modify NAS config. Never download unapproved models/datasets. -- ai-silo serves production: floor lease + human sign-off before touching it. +- Some nodes serve production inference: floor lease + human sign-off before + touching a node whose `serving` list is non-empty. - Abliterated/pruned artifacts: NAS only, never push to HF. diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 2e9f266..0000000 --- a/TODO.md +++ /dev/null @@ -1,53 +0,0 @@ -# forge TODO — path to v1 acceptance - -Deployed state as of 2026-07-10 (see ops/cron.md "Deployed reality"): /opt/forge -live on the jumpbox, cron installed, `lease status` smoke green, and a full -lease acquire→release round-trip on ai02 verified LEASE_SECRET write auth. -TrueNAS HDD pool: 94.9T free of 98.2T — 50TB cap fits with the 20%-free rule. -Base model for the acceptance demo already on NAS: talkie-lm/talkie-1930-13b-base. - -Update 2026-07-10 (Colby): downloads approved — all manifest entries flipped -`approved: true` (6913aca) and the fetch queue is RUNNING on ai02 (tmux -session `forge-fetch`, log ~/forge-fetch.log, FORGE_DATA_ROOT= -/mnt/truenas-models/datasets — NOTE: on ai02 the TrueNAS share mounts at -/mnt/truenas-models, not the forge default /mnt/models, which is a stale -fstab entry for the Alameda NAS). Queue: historical tier → small sets → -fineweb-edu + dclm-baseline (~13TB tail). Resumable: rerun -~/forge-fetch-queue.sh. Infra changes (model parking, 5090 acceptance run) -explicitly DEFERRED by Colby — do not bring anything up. - -## Human (Colby) - -1. **gh token on jumpbox.** Create a fine-grained PAT scoped to - GoodAncestor/forge (Contents: read/write, Pull requests: read/write), then - on the jumpbox: `echo | gh auth login --with-token` - (gh 2.96.0 already installed). Until then the weekly env update pushes its - branch (write deploy key) but records `pr_error` instead of opening the PR. -2. **HF token for gated datasets** — nemotron-cc (NVIDIA Data Agreement, - gated:manual) and lmsys-chat-1m (gated:auto) are approved but excluded - from the running queue; they need a token whose account accepted terms. -3. **DEFERRED until you reopen infra work: pick the acceptance 5090.** - alien01/02/03 are all `safe_to_offline: yes` but each serves a qwen3.6 - model — the RUNBOOK forbids an agent picking a serving node. Either sign - off on one (floor lease) or park its model. - -## Agent (unblocked by the above) - -4. When the historical tier lands: build corpus-1930-1980/train.jsonl per - GoodAncestor HQ knowledge/projects/historical-model-training.md (pipeline - lives outside forge). Base model already on NAS. -5. After (3+4): acceptance run - `forge --json run recipes/talkie-1930to1980-qlora.yaml --host `, - including forced pause (SIGTERM → exit 75) and `--resume` verification. -6. Consider a manifest/docs note or env default fix for the /mnt/models vs - /mnt/truenas-models path mismatch on Berkeley nodes. - -## Agent (opportunistic) - -7. Verify fleet HF token write-scope before any HF-org push — no HF token - exists in the jumpbox fleet secrets today, so this is also a "locate or - provision token" task. -8. On-node verification of REAP/heretic tool_cmd paths. -9. Known debt: gptqmodel>=5 floor caps env transformers at 4.57.2 / trl 0.24 - (consider a quantize-only env split); FSDP exit-75 pause propagation - unverified until ai-silo. diff --git a/envs/profiles.yaml b/envs/profiles.yaml index 34f082b..677218a 100644 --- a/envs/profiles.yaml +++ b/envs/profiles.yaml @@ -2,28 +2,25 @@ # status: locked = uv.lock committed and smoke-tested on real hardware # locked-untested = uv.lock maintained, no hardware yet (datacenter) # stub = package list only, locked when hardware is first offered +# +# hosts/smoke_host are NOT defined here. They are physical fleet topology and +# come from a private overlay at $FORGE_FLEET_CONFIG/profiles.hosts.yaml +# (see src/forge/envs.py:_load_overlay). Without an overlay every profile +# resolves to an empty fleet — bind it to your own hardware to run jobs. profiles: cuda-ada: description: "RTX 4090 class (sm_89, 24GB)" torch_index: "https://download.pytorch.org/whl/cu128" - hosts: [ai01, ai02] - smoke_host: ai02 status: locked cuda-blackwell: description: "RTX 5090 (32GB) + RTX PRO 6000 (96GB, multi-GPU) — sm_120" torch_index: "https://download.pytorch.org/whl/cu128" - hosts: [alien01, alien02, alien03, ai-silo] - smoke_host: alien03 status: locked cuda-datacenter: - description: "B300 NV8 / GB300 / Vera Rubin (on order, ~Q4 2026) and loaner H100/B200/B300 nodes" + description: "Datacenter GPUs — H100 / B200 / B300 / GB300 class" torch_index: "https://download.pytorch.org/whl/cu128" - hosts: [] - smoke_host: null status: locked-untested rocm: - description: "AMD MI355-class loaners (CDNA4). Stub until hardware is offered." + description: "AMD MI355-class (CDNA4). Stub until hardware is offered." torch_index: "https://download.pytorch.org/whl/rocm6.4" - hosts: [] - smoke_host: null status: stub diff --git a/ops/cron.md b/ops/cron.md index 6b9437e..d9a175a 100644 --- a/ops/cron.md +++ b/ops/cron.md @@ -1,26 +1,22 @@ -# forge scheduled operations (jumpbox) +# forge scheduled operations -Cron owns the schedule; Hermes reads the JSON reports and escalates. +Cron owns the schedule; an operator (human or agent) reads the JSON reports +and escalates. Both jobs need `FORGE_LEASE_SECRET`, a `GH_TOKEN` (or `gh auth`) +for opening PRs, and push access to the repo. - # /etc/cron.d/forge (run as the forge service user; needs gh auth + FORGE_LEASE_SECRET) # weekly env update, Mondays 06:00 UTC - 0 6 * * 1 forge cd /opt/forge && uv run forge --json env update >> /var/log/forge-env-update.log 2>&1 + 0 6 * * 1 cd && uv run forge --json env update >> env-update.log 2>&1 # monthly manifest size refresh (report-only — see note below), 1st 06:30 UTC - 30 6 1 * * forge cd /opt/forge && uv run forge --json data sizes >> /var/log/forge-data-sizes.log 2>&1 + 30 6 1 * * cd && uv run forge --json data sizes >> data-sizes.log 2>&1 -## Deployed reality (jumpbox, 2026-07-10) -The jumpbox is Alpine/busybox crond: no /etc/cron.d support and no per-line -env, so the lines above are installed in /etc/crontabs/root via -/usr/local/bin/forge-cron — a wrapper that loads FORGE_LEASE_SECRET from -/root/.fleet-secrets (LEASE_SECRET key) and runs as root, consistent with the -other jumpbox services (same adaptation the fleet-aggregator openrc unit -documents). GitHub access is a write deploy key (/root/.ssh/forge-deploy, -Host github-forge). Known gap: `gh` is not installed/authed on the jumpbox, -so the weekly env update will record pr_error instead of opening the PR -until a repo-scoped token is provisioned (human task). +Deployment specifics (which user, where secrets/keys live, any busybox-crond +wrapper) are environment-dependent and kept in your private fleet config, not +here. The env-update job runs wherever the cron lives and needs no GPU: it only +re-resolves the lockfiles and opens a PR (hardware smoke is deferred to the +reviewer — see RUNBOOK "Env maintenance"). -Reports land in /opt/forge/reports/env-update-YYYY-MM-DD.json. -Hermes check: newest report `ok: true` and younger than 8 days; else escalate. +Reports land in `reports/env-update-YYYY-MM-DD.json`. Monitor check: newest +report `ok: true` and younger than 8 days; else escalate. The monthly job runs WITHOUT `--write`: it measures and logs current sizes but never rewrites datasets/manifest.yaml unattended. This is a deliberate diff --git a/recipes/talkie-1930to1980-fsdp.yaml b/recipes/talkie-1930to1980-fsdp.yaml index f50f184..278112a 100644 --- a/recipes/talkie-1930to1980-fsdp.yaml +++ b/recipes/talkie-1930to1980-fsdp.yaml @@ -1,5 +1,5 @@ # The v1 ACCEPTANCE DEMO with FSDP. Same model/datasets as qlora variant. -# 3-GPU training on ai-silo (3× RTX PRO 6000). +# 3-GPU training on a multi-GPU node (e.g. 3× RTX PRO 6000). name: talkie-1930to1980-fsdp profile: cuda-blackwell method: cpt-qlora diff --git a/src/forge/config.py b/src/forge/config.py index 86573bd..c06d4e5 100644 --- a/src/forge/config.py +++ b/src/forge/config.py @@ -1,7 +1,7 @@ import os import socket -FLEET_URL = os.environ.get("FORGE_FLEET_URL", "http://192.168.77.36:8096") +FLEET_URL = os.environ.get("FORGE_FLEET_URL", "http://localhost:8096") LEASE_SECRET = os.environ.get("FORGE_LEASE_SECRET", "") DATA_ROOT = os.environ.get("FORGE_DATA_ROOT", "/mnt/models/datasets") MODEL_ROOT = os.environ.get("FORGE_MODEL_ROOT", "/mnt/models") diff --git a/src/forge/envs.py b/src/forge/envs.py index 5eb7492..fa490b4 100644 --- a/src/forge/envs.py +++ b/src/forge/envs.py @@ -1,3 +1,4 @@ +import os from pathlib import Path import yaml @@ -6,25 +7,50 @@ REPO_ROOT = Path(__file__).resolve().parents[2] PROFILES_PATH = REPO_ROOT / "envs" / "profiles.yaml" +OVERLAY_ENV = "FORGE_FLEET_CONFIG" VALID_STATUS = {"locked", "locked-untested", "stub"} -REQUIRED_KEYS = {"description", "torch_index", "hosts", "smoke_host", "status"} +# hosts/smoke_host are NOT in the public profiles.yaml — they are physical +# fleet topology, injected from a private overlay (see _load_overlay). +REQUIRED_KEYS = {"description", "torch_index", "status"} class ProfileError(ValueError): pass -def load_profiles(path=None): +def _load_overlay(): + """Per-profile host bindings ({hosts: [...], smoke_host: ...}) live outside + this (public) repo — in a private fleet config pointed to by + $FORGE_FLEET_CONFIG (a directory containing profiles.hosts.yaml, or the file + itself). Absent → every profile resolves to an empty fleet, which is the + correct default for a clone that has no hardware of its own.""" + raw_path = os.environ.get(OVERLAY_ENV) + if not raw_path: + return {} + path = Path(raw_path) + if path.is_dir(): + path = path / "profiles.hosts.yaml" + if not path.exists(): + return {} + return yaml.safe_load(path.read_text()) or {} + + +def load_profiles(path=None, overlay=None): raw = yaml.safe_load(Path(path or PROFILES_PATH).read_text()) profiles = (raw or {}).get("profiles") if not isinstance(profiles, dict) or not profiles: raise ProfileError("profiles.yaml must contain a non-empty 'profiles' map") + overlay = _load_overlay() if overlay is None else overlay for name, p in profiles.items(): missing = REQUIRED_KEYS - set(p) if missing: raise ProfileError(f"profile {name}: missing keys {sorted(missing)}") if p["status"] not in VALID_STATUS: raise ProfileError(f"profile {name}: status must be one of {sorted(VALID_STATUS)}") + # Merge private host bindings over the public class definition. + binding = overlay.get(name) or {} + p["hosts"] = binding.get("hosts", []) + p["smoke_host"] = binding.get("smoke_host") if not isinstance(p["hosts"], list): raise ProfileError(f"profile {name}: hosts must be a list") return profiles diff --git a/src/forge/run.py b/src/forge/run.py index 6d63b0f..6c041a8 100644 --- a/src/forge/run.py +++ b/src/forge/run.py @@ -221,7 +221,7 @@ def register(subparsers): p.add_argument("--host", required=True) p.add_argument("--target-type", default="node", choices=["node", "gpu"]) p.add_argument("--gpu-name", default=None, - help="gpu target name, e.g. 'ai-silo:2' (required with --target-type gpu)") + help="gpu target name, e.g. 'node:2' (required with --target-type gpu)") p.add_argument("--off-fleet", action="store_true") p.add_argument("--run-root", default=None) p.add_argument("--dry-run", action="store_true") diff --git a/tests/test_cli.py b/tests/test_cli.py index 1214b2f..82dfedd 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -18,9 +18,11 @@ def test_unknown_command_exits_nonzero(): def test_config_defaults(monkeypatch): monkeypatch.delenv("FORGE_DATA_CAP_TB", raising=False) + monkeypatch.delenv("FORGE_FLEET_URL", raising=False) import importlib from forge import config importlib.reload(config) assert config.DATA_CAP_TB == 50.0 assert config.DATA_ROOT == "/mnt/models/datasets" - assert config.FLEET_URL == "http://192.168.77.36:8096" + # default carries no fleet-specific address; the real URL is set via env + assert config.FLEET_URL == "http://localhost:8096" diff --git a/tests/test_envs.py b/tests/test_envs.py index ff085f9..e606f75 100644 --- a/tests/test_envs.py +++ b/tests/test_envs.py @@ -3,29 +3,48 @@ def test_load_profiles_from_repo(): - profiles = envs.load_profiles() + profiles = envs.load_profiles(overlay={}) assert set(profiles) == {"cuda-ada", "cuda-blackwell", "cuda-datacenter", "rocm"} assert profiles["cuda-ada"]["status"] == "locked" -def test_profile_for_host(): - profiles = envs.load_profiles() - assert envs.profile_for_host("alien02", profiles) == "cuda-blackwell" - assert envs.profile_for_host("ai-silo", profiles) == "cuda-blackwell" +def test_no_overlay_means_empty_fleet(): + # public clone with no private overlay: every profile resolves, no hosts + profiles = envs.load_profiles(overlay={}) + assert profiles["cuda-ada"]["hosts"] == [] + assert profiles["cuda-ada"]["smoke_host"] is None + + +def test_overlay_binds_hosts(): + overlay = {"cuda-blackwell": {"hosts": ["node2", "node3"], + "smoke_host": "node3"}} + profiles = envs.load_profiles(overlay=overlay) + assert profiles["cuda-blackwell"]["hosts"] == ["node2", "node3"] + assert profiles["cuda-blackwell"]["smoke_host"] == "node3" + assert envs.profile_for_host("node2", profiles) == "cuda-blackwell" assert envs.profile_for_host("nonexistent", profiles) is None +def test_overlay_loaded_from_file(tmp_path, monkeypatch): + overlay_file = tmp_path / "profiles.hosts.yaml" + overlay_file.write_text("cuda-ada: {hosts: [node5], smoke_host: node5}\n") + monkeypatch.setenv(envs.OVERLAY_ENV, str(tmp_path)) # dir form + profiles = envs.load_profiles() + assert profiles["cuda-ada"]["hosts"] == ["node5"] + + def test_invalid_status_rejected(tmp_path): bad = tmp_path / "p.yaml" bad.write_text("profiles:\n x:\n description: d\n torch_index: i\n" - " hosts: []\n smoke_host: null\n status: wat\n") + " status: wat\n") with pytest.raises(envs.ProfileError): - envs.load_profiles(bad) + envs.load_profiles(bad, overlay={}) -def test_env_list_cli(capsys): +def test_env_list_cli(capsys, monkeypatch): import json from forge import cli + monkeypatch.delenv(envs.OVERLAY_ENV, raising=False) rc = cli.main(["--json", "env", "list"]) assert rc == 0 out = json.loads(capsys.readouterr().out) diff --git a/tests/test_lease.py b/tests/test_lease.py index e72b6bc..5ca0e04 100644 --- a/tests/test_lease.py +++ b/tests/test_lease.py @@ -36,21 +36,21 @@ def fake_get(url, timeout=None): def test_acquire_success(monkeypatch): - lo = {"id": "abc", "holder": "h", "target": {"type": "node", "name": "alien02"}, + lo = {"id": "abc", "holder": "h", "target": {"type": "node", "name": "node2"}, "mode": "exclusive", "notes": "", "acquired": "t", "expires": "t2"} c, calls = make_client(monkeypatch, [(200, {"ok": True, "lease": lo})]) - got = c.acquire("h", "node", "alien02", ttl_sec=28800, notes="train") + got = c.acquire("h", "node", "node2", ttl_sec=28800, notes="train") assert got["id"] == "abc" assert calls[0]["url"] == "http://fleet:8096/leases/acquire" assert calls[0]["headers"]["Authorization"] == "Bearer s3cr3t" - assert calls[0]["body"]["target"] == {"type": "node", "name": "alien02"} + assert calls[0]["body"]["target"] == {"type": "node", "name": "node2"} assert calls[0]["body"]["ttl_sec"] == 28800 def test_acquire_refused_raises(monkeypatch): c, _ = make_client(monkeypatch, [(409, {"ok": False, "error": "already held by x"})]) with pytest.raises(lease.LeaseRefused, match="already held"): - c.acquire("h", "node", "alien02") + c.acquire("h", "node", "node2") def test_unreachable_raises(monkeypatch): @@ -85,11 +85,11 @@ def test_floor_target_includes_min_floor(monkeypatch): def test_release_and_node(monkeypatch): - fleet = {"nodes": [{"name": "alien02", "serving": []}, {"name": "ai-silo", "serving": [{"model": "m"}]}]} + fleet = {"nodes": [{"name": "node2", "serving": []}, {"name": "node-multi", "serving": [{"model": "m"}]}]} c, calls = make_client(monkeypatch, [(200, {"ok": True, "released": "abc", "existed": True}), (200, fleet), (200, fleet)]) assert c.release("abc") is True - assert c.node("ai-silo")["serving"] == [{"model": "m"}] + assert c.node("node-multi")["serving"] == [{"model": "m"}] assert c.node("ghost") is None assert calls[0]["url"] == "http://fleet:8096/leases/release" assert calls[0]["body"] == {"id": "abc"} diff --git a/tests/test_lease_cli.py b/tests/test_lease_cli.py index 81d7bb4..651eefd 100644 --- a/tests/test_lease_cli.py +++ b/tests/test_lease_cli.py @@ -22,7 +22,7 @@ def release(self, lease_id): return True def fleet(self): - return {"nodes": [{"name": "alien02", "held_by": "someone", + return {"nodes": [{"name": "node2", "held_by": "someone", "serving": []}]} @@ -34,7 +34,7 @@ def acquire(self, *a, **k): def test_acquire_ok(monkeypatch, capsys): monkeypatch.setattr(lease_cli, "LeaseClient", StubClient) rc = cli.main(["--json", "lease", "acquire", "--target-type", "node", - "--name", "alien02"]) + "--name", "node2"]) assert rc == 0 assert json.loads(capsys.readouterr().out)["lease"]["id"] == "L1" @@ -50,7 +50,7 @@ def test_acquire_refused_exit_2(monkeypatch, capsys): def test_unreachable_exit_4(monkeypatch, capsys): monkeypatch.setattr(lease_cli, "LeaseClient", DownClient) rc = cli.main(["--json", "lease", "acquire", "--target-type", "node", - "--name", "alien02"]) + "--name", "node2"]) assert rc == 4 @@ -64,7 +64,7 @@ def test_status(monkeypatch, capsys): class NullServingClient(StubClient): def fleet(self): - return {"nodes": [{"name": "alien02", "held_by": None, "serving": None}]} + return {"nodes": [{"name": "node2", "held_by": None, "serving": None}]} def test_status_guards_null_serving(monkeypatch, capsys): diff --git a/tests/test_run.py b/tests/test_run.py index f6d9df1..05ca2ed 100644 --- a/tests/test_run.py +++ b/tests/test_run.py @@ -114,7 +114,7 @@ def test_fleet_requires_lease_and_releases(tmp_path, monkeypatch): client = StubClient() monkeypatch.setattr(run_mod, "LeaseClient", lambda *a, **k: client) p = _write(tmp_path, RECIPE) - rc = run_mod.execute(recipe_path=p, host="alien02", off_fleet=False, + rc = run_mod.execute(recipe_path=p, host="node2", off_fleet=False, run_root=tmp_path / "runs", as_json=True, use_uv_project=False) assert rc == 0 @@ -125,7 +125,7 @@ def test_serving_node_warns(tmp_path, monkeypatch, capsys): client = StubClient() monkeypatch.setattr(run_mod, "LeaseClient", lambda *a, **k: client) p = _write(tmp_path, RECIPE) - run_mod.execute(recipe_path=p, host="ai-silo", off_fleet=False, + run_mod.execute(recipe_path=p, host="node-multi", off_fleet=False, run_root=tmp_path / "runs", as_json=False, use_uv_project=False) err = capsys.readouterr().err @@ -137,7 +137,7 @@ def test_heartbeat_lost_pauses_job(tmp_path, monkeypatch): monkeypatch.setattr(run_mod, "LeaseClient", lambda *a, **k: client) monkeypatch.setattr(run_mod, "LeaseHeartbeat", InstantLostHB) p = _write(tmp_path, SLEEPER) - rc = run_mod.execute(recipe_path=p, host="alien02", off_fleet=False, + rc = run_mod.execute(recipe_path=p, host="node2", off_fleet=False, run_root=tmp_path / "runs", as_json=True, use_uv_project=False, poll_sec=0.1) assert rc == 75 @@ -150,7 +150,7 @@ def test_launch_failure_releases_lease_and_writes_manifest(tmp_path, monkeypatch client = StubClient() monkeypatch.setattr(run_mod, "LeaseClient", lambda *a, **k: client) p = _write(tmp_path, RECIPE.replace("{python}", "/nonexistent/binary-xyz")) - rc = run_mod.execute(recipe_path=p, host="alien02", off_fleet=False, + rc = run_mod.execute(recipe_path=p, host="node2", off_fleet=False, run_root=tmp_path / "runs", as_json=True, use_uv_project=False, poll_sec=0.1) assert rc == 3 @@ -164,7 +164,7 @@ def test_sigkilled_child_reports_failed_not_paused(tmp_path, monkeypatch): monkeypatch.setattr(run_mod, "LeaseClient", lambda *a, **k: client) monkeypatch.setattr(run_mod, "LeaseHeartbeat", InstantLostHB) p = _write(tmp_path, IGNORER) - rc = run_mod.execute(recipe_path=p, host="alien02", off_fleet=False, + rc = run_mod.execute(recipe_path=p, host="node2", off_fleet=False, run_root=tmp_path / "runs", as_json=True, use_uv_project=False, poll_sec=0.1) assert rc == 3 @@ -283,7 +283,7 @@ def test_refused_lease_does_not_create_run_dir(tmp_path, monkeypatch): client = RefusingClient() monkeypatch.setattr(run_mod, "LeaseClient", lambda *a, **k: client) p = _write(tmp_path, RECIPE) - rc = run_mod.execute(recipe_path=p, host="alien02", off_fleet=False, + rc = run_mod.execute(recipe_path=p, host="node2", off_fleet=False, run_root=tmp_path / "runs", as_json=True, use_uv_project=False) assert rc == 2 @@ -305,7 +305,7 @@ def test_mkdir_failure_still_releases_lease(tmp_path, monkeypatch): p = _write(tmp_path, RECIPE) blocker = tmp_path / "runs" blocker.write_text("not a directory") - rc = run_mod.execute(recipe_path=p, host="alien02", off_fleet=False, + rc = run_mod.execute(recipe_path=p, host="node2", off_fleet=False, run_root=blocker, as_json=True, use_uv_project=False, poll_sec=0.1) assert rc == 3 @@ -346,7 +346,7 @@ def test_run_posts_paused_on_checkpoint_exit(tmp_path, monkeypatch): monkeypatch.setattr(run_mod, "LeaseClient", lambda *a, **k: client) monkeypatch.setattr(run_mod, "LeaseHeartbeat", InstantLostHB) p = _write(tmp_path, SLEEPER) - rc = run_mod.execute(recipe_path=p, host="alien02", off_fleet=False, + rc = run_mod.execute(recipe_path=p, host="node2", off_fleet=False, run_root=tmp_path / "runs", as_json=True, use_uv_project=False, poll_sec=0.1) assert rc == 75