public-prep: split fleet topology out; descriptive README

README.md

@@ -1,13 +1,112 @@
-# forge — Good Ancestor model lab
+# forge — an agent-operated model lab
 
-Agent-operated toolkit: per-GPU-class training environments, NAS dataset
-archive, lease-safe training runs, and model-surgery recipes.
+**forge** is the toolkit the Good Ancestor Foundation uses to run its own model
+work on its own hardware: preparing per-GPU-class training environments,
+maintaining a governed dataset archive, and executing training, fine-tuning,
+quantization, and model-surgery jobs safely on a shared GPU fleet.
 
-- Spec: GoodAncestor HQ `docs/superpowers/specs/2026-07-10-forge-design.md`
-- Operators (LLM): read `RUNBOOK.md`
-- Humans: you review PRs and flip `approved:` flags; that's the whole job.
+It is designed to be driven by autonomous agents but governed by humans. Agents
+do the mechanical work — resolving environments, fetching approved datasets,
+launching leased runs. Humans do exactly two things: **review pull requests**
+and **flip `approved:` flags**. Nothing consequential happens without a human in
+that loop.
+
+## Why it exists
+
+Running models on a real fleet involves a lot of fiddly, error-prone work: which
+CUDA/torch stack matches which GPU generation, which datasets are cleared for
+use, whether a GPU is free or serving production, whether a job checkpointed
+before it was killed. forge encodes all of that as small, testable,
+single-purpose commands with machine-readable output — so an agent can operate
+the fleet without improvising, and so every consequential step leaves an
+auditable artifact.
+
+## What's in the box
+
+- **Per-GPU-class environments.** Each hardware class (e.g. Ada, Blackwell,
+  datacenter) is its own isolated project with its own pinned lockfile, so the
+  torch/CUDA stack always matches the silicon. A weekly job re-resolves the
+  locks and opens a PR; hardware smoke-tests run on the target GPU before a
+  human merges.
+- **A governed dataset archive.** A single `manifest.yaml` is the source of
+  truth for every dataset: its source, license, size, storage path, and an
+  `approved:` flag that only a human flips. Fetches are refused for anything
+  unapproved, oversized against a cap, or of unknown size — no silent
+  workarounds.
+- **Lease-safe runs.** Every training/quant/surgery job runs under a fleet lease
+  with a heartbeat, so two agents never contend for the same GPU, and a job that
+  touches a production-serving node is flagged loudly. Runs are resumable and
+  record a signed manifest of exactly what happened.
+- **Recipes.** Declarative YAML recipes for the common jobs — LoRA SFT, DPO,
+  GRPO, AWQ/GGUF/NVFP4 quantization, abliteration, and expert pruning — each
+  pinned to the environment profile it needs.
 
 ## Quick start
-    uv sync                      # CLI + dev deps
-    uv run forge --help
-    uv run pytest -q
+
+```sh
+uv sync                      # install the CLI + dev deps
+uv run forge --help
+uv run pytest -q             # run the test suite
+```
+
+## The CLI
+
+Everything is one `forge` command with a few subcommands. Add `--json` to any of
+them for machine-readable output (this is how agents drive it).
+
+```sh
+forge env update                     # re-resolve env lockfiles, open a PR
+forge data list                      # what datasets exist / are approved
+forge data sizes                     # measure sizes against the cap
+forge data fetch <name>              # fetch one approved dataset (resumable)
+forge run recipes/sft-lora.yaml \    # run a recipe under a fleet lease
+    --host <gpu-host> --profile <profile>
+forge lease status                   # who holds which GPU right now
+```
+
+## The safety model
+
+forge is opinionated about not doing damage:
+
+- **Approval gates are final.** An unapproved or refused dataset fetch exits
+  non-zero and stays refused until a human edits `approved:` via a reviewed PR.
+  Agents do not work around the gate.
+- **Automated PRs are never auto-merged.** The weekly environment update opens a
+  PR and stops. A human reviews it, runs the hardware smoke-test, and merges.
+- **Leases before GPUs.** Nothing runs on a GPU it hasn't leased, and jobs warn
+  before touching a node that's serving production traffic.
+- **Honest exit codes.** Commands distinguish "refused" (2), "job failed" (3),
+  "infrastructure unreachable" (4), and "checkpointed/paused" (75), so callers —
+  human or agent — never mistake one failure for another.
+
+## Repository layout
+
+```
+src/forge/        # the CLI and its modules (env, data, run, lease, envupdate)
+envs/             # one uv project per GPU class, each with its own lockfile
+datasets/         # manifest.yaml — the governed dataset archive
+recipes/          # declarative job recipes (sft, dpo, grpo, quantize, ...)
+rewards/          # reward functions for preference/RL recipes
+bootstrap/        # scripts that prepare a fresh GPU node
+ops/              # operational notes (scheduling, etc.)
+RUNBOOK.md        # the operator's guide (read this if you're driving forge)
+```
+
+## For humans
+
+If you're a person looking at this repo, your job is small and well-defined:
+
+1. **Review PRs** — especially environment-update PRs (check the version diff and
+   run the smoke command the PR body gives you) and any PR that flips a dataset
+   `approved:` flag.
+2. **Decide what's approved** — you own the `approved:` flags in
+   `datasets/manifest.yaml`. Agents propose; you dispose.
+
+That's the whole job. Everything else is mechanical, tested, and automated.
+
+## Status
+
+forge is used in production by the Good Ancestor Foundation. It is shared here in
+the spirit of the Foundation's mission — helping others stand up their own local,
+sovereign AI infrastructure. Interfaces may change; the safety principles above
+will not.

RUNBOOK.md

@@ -4,8 +4,11 @@ You are operating the Good Ancestor model lab. Everything below is CLI + JSON.
 Humans only: merge PRs, flip `approved:` flags, sign off on production-node runs.
 
 ## Environment
-- Aggregator/fleet state: GET $FORGE_FLEET_URL/fleet.json (default http://192.168.77.36:8096)
-- Required env: FORGE_LEASE_SECRET (from jumpbox fleet secrets). Optional:
+- Aggregator/fleet state: GET $FORGE_FLEET_URL/fleet.json (set FORGE_FLEET_URL
+  to your aggregator; default http://localhost:8096)
+- FORGE_FLEET_CONFIG: path to the private fleet overlay (profiles.hosts.yaml)
+  that binds each profile to real hosts. Unset → profiles have no hosts.
+- Required env: FORGE_LEASE_SECRET (from your fleet secrets). Optional:
   FORGE_DATA_ROOT (/mnt/models/datasets), FORGE_MODEL_ROOT (/mnt/models),
   FORGE_DATA_CAP_TB (50 — meters the dataset subtree only, NOT the models sharing
   the pool; the practical ceiling is pool free space — keep ZFS ~20% free),
@@ -53,12 +56,18 @@ Humans only: merge PRs, flip `approved:` flags, sign off on production-node runs
   datasets/manifest.yaml via PR. Do not work around the gate.
 
 ## Env maintenance
-    forge --json env update           # lock upgrade + leased smoke + PR
-- Weekly cron does this. Read reports/env-update-*.json; escalate if ok=false.
+    forge --json env update           # re-resolve lockfiles + open PR
+- Weekly cron does this. It runs where the cron lives (no GPU needed): it only
+  re-resolves the locks and opens a PR. `ok` = locks resolved cleanly.
+- Hardware smoke is deferred to PR review: the PR body carries the exact
+  `forge run recipes/smoke.yaml ...` command to run on each profile's
+  smoke_host before merging. Read reports/env-update-*.json; escalate if
+  ok=false (a lock failed to resolve).
 - NEVER merge the PR yourself.
 
 ## Safety rails you must respect
-- One purposeful connection at a time to any host (UniFi IPS blocks scanners).
+- One purposeful connection at a time to any host (network IPS blocks scanners).
 - Never modify NAS config. Never download unapproved models/datasets.
-- ai-silo serves production: floor lease + human sign-off before touching it.
+- Some nodes serve production inference: floor lease + human sign-off before
+  touching a node whose `serving` list is non-empty.
 - Abliterated/pruned artifacts: NAS only, never push to HF.

envs/profiles.yaml

@@ -2,28 +2,25 @@
 # status: locked  = uv.lock committed and smoke-tested on real hardware
 #         locked-untested = uv.lock maintained, no hardware yet (datacenter)
 #         stub    = package list only, locked when hardware is first offered
+#
+# hosts/smoke_host are NOT defined here. They are physical fleet topology and
+# come from a private overlay at $FORGE_FLEET_CONFIG/profiles.hosts.yaml
+# (see src/forge/envs.py:_load_overlay). Without an overlay every profile
+# resolves to an empty fleet — bind it to your own hardware to run jobs.
 profiles:
   cuda-ada:
     description: "RTX 4090 class (sm_89, 24GB)"
     torch_index: "https://download.pytorch.org/whl/cu128"
-    hosts: [ai01, ai02]
-    smoke_host: ai02
     status: locked
   cuda-blackwell:
     description: "RTX 5090 (32GB) + RTX PRO 6000 (96GB, multi-GPU) — sm_120"
     torch_index: "https://download.pytorch.org/whl/cu128"
-    hosts: [alien01, alien02, alien03, ai-silo]
-    smoke_host: alien03
     status: locked
   cuda-datacenter:
-    description: "B300 NV8 / GB300 / Vera Rubin (on order, ~Q4 2026) and loaner H100/B200/B300 nodes"
+    description: "Datacenter GPUs — H100 / B200 / B300 / GB300 class"
     torch_index: "https://download.pytorch.org/whl/cu128"
-    hosts: []
-    smoke_host: null
     status: locked-untested
   rocm:
-    description: "AMD MI355-class loaners (CDNA4). Stub until hardware is offered."
+    description: "AMD MI355-class (CDNA4). Stub until hardware is offered."
     torch_index: "https://download.pytorch.org/whl/rocm6.4"
-    hosts: []
-    smoke_host: null
     status: stub

ops/cron.md

@@ -1,26 +1,22 @@
-# forge scheduled operations (jumpbox)
+# forge scheduled operations
 
-Cron owns the schedule; Hermes reads the JSON reports and escalates.
+Cron owns the schedule; an operator (human or agent) reads the JSON reports
+and escalates. Both jobs need `FORGE_LEASE_SECRET`, a `GH_TOKEN` (or `gh auth`)
+for opening PRs, and push access to the repo.
 
-    # /etc/cron.d/forge  (run as the forge service user; needs gh auth + FORGE_LEASE_SECRET)
     # weekly env update, Mondays 06:00 UTC
-    0 6 * * 1  forge  cd /opt/forge && uv run forge --json env update >> /var/log/forge-env-update.log 2>&1
+    0 6 * * 1  cd <forge> && uv run forge --json env update >> env-update.log 2>&1
     # monthly manifest size refresh (report-only — see note below), 1st 06:30 UTC
-    30 6 1 * * forge  cd /opt/forge && uv run forge --json data sizes >> /var/log/forge-data-sizes.log 2>&1
+    30 6 1 * *  cd <forge> && uv run forge --json data sizes >> data-sizes.log 2>&1
 
-## Deployed reality (jumpbox, 2026-07-10)
-The jumpbox is Alpine/busybox crond: no /etc/cron.d support and no per-line
-env, so the lines above are installed in /etc/crontabs/root via
-/usr/local/bin/forge-cron — a wrapper that loads FORGE_LEASE_SECRET from
-/root/.fleet-secrets (LEASE_SECRET key) and runs as root, consistent with the
-other jumpbox services (same adaptation the fleet-aggregator openrc unit
-documents). GitHub access is a write deploy key (/root/.ssh/forge-deploy,
-Host github-forge). Known gap: `gh` is not installed/authed on the jumpbox,
-so the weekly env update will record pr_error instead of opening the PR
-until a repo-scoped token is provisioned (human task).
+Deployment specifics (which user, where secrets/keys live, any busybox-crond
+wrapper) are environment-dependent and kept in your private fleet config, not
+here. The env-update job runs wherever the cron lives and needs no GPU: it only
+re-resolves the lockfiles and opens a PR (hardware smoke is deferred to the
+reviewer — see RUNBOOK "Env maintenance").
 
-Reports land in /opt/forge/reports/env-update-YYYY-MM-DD.json.
-Hermes check: newest report `ok: true` and younger than 8 days; else escalate.
+Reports land in `reports/env-update-YYYY-MM-DD.json`. Monitor check: newest
+report `ok: true` and younger than 8 days; else escalate.
 
 The monthly job runs WITHOUT `--write`: it measures and logs current sizes but
 never rewrites datasets/manifest.yaml unattended. This is a deliberate

recipes/talkie-1930to1980-fsdp.yaml

@@ -1,5 +1,5 @@
 # The v1 ACCEPTANCE DEMO with FSDP. Same model/datasets as qlora variant.
-# 3-GPU training on ai-silo (3× RTX PRO 6000).
+# 3-GPU training on a multi-GPU node (e.g. 3× RTX PRO 6000).
 name: talkie-1930to1980-fsdp
 profile: cuda-blackwell
 method: cpt-qlora

src/forge/config.py

@@ -1,7 +1,7 @@
 import os
 import socket
 
-FLEET_URL = os.environ.get("FORGE_FLEET_URL", "http://192.168.77.36:8096")
+FLEET_URL = os.environ.get("FORGE_FLEET_URL", "http://localhost:8096")
 LEASE_SECRET = os.environ.get("FORGE_LEASE_SECRET", "")
 DATA_ROOT = os.environ.get("FORGE_DATA_ROOT", "/mnt/models/datasets")
 MODEL_ROOT = os.environ.get("FORGE_MODEL_ROOT", "/mnt/models")

src/forge/envs.py

@@ -1,3 +1,4 @@
+import os
 from pathlib import Path
 
 import yaml
@@ -6,25 +7,50 @@
 
 REPO_ROOT = Path(__file__).resolve().parents[2]
 PROFILES_PATH = REPO_ROOT / "envs" / "profiles.yaml"
+OVERLAY_ENV = "FORGE_FLEET_CONFIG"
 VALID_STATUS = {"locked", "locked-untested", "stub"}
-REQUIRED_KEYS = {"description", "torch_index", "hosts", "smoke_host", "status"}
+# hosts/smoke_host are NOT in the public profiles.yaml — they are physical
+# fleet topology, injected from a private overlay (see _load_overlay).
+REQUIRED_KEYS = {"description", "torch_index", "status"}
 
 
 class ProfileError(ValueError):
     pass
 
 
-def load_profiles(path=None):
+def _load_overlay():
+    """Per-profile host bindings ({hosts: [...], smoke_host: ...}) live outside
+    this (public) repo — in a private fleet config pointed to by
+    $FORGE_FLEET_CONFIG (a directory containing profiles.hosts.yaml, or the file
+    itself). Absent → every profile resolves to an empty fleet, which is the
+    correct default for a clone that has no hardware of its own."""
+    raw_path = os.environ.get(OVERLAY_ENV)
+    if not raw_path:
+        return {}
+    path = Path(raw_path)
+    if path.is_dir():
+        path = path / "profiles.hosts.yaml"
+    if not path.exists():
+        return {}
+    return yaml.safe_load(path.read_text()) or {}
+
+
+def load_profiles(path=None, overlay=None):
     raw = yaml.safe_load(Path(path or PROFILES_PATH).read_text())
     profiles = (raw or {}).get("profiles")
     if not isinstance(profiles, dict) or not profiles:
         raise ProfileError("profiles.yaml must contain a non-empty 'profiles' map")
+    overlay = _load_overlay() if overlay is None else overlay
     for name, p in profiles.items():
         missing = REQUIRED_KEYS - set(p)
         if missing:
             raise ProfileError(f"profile {name}: missing keys {sorted(missing)}")
         if p["status"] not in VALID_STATUS:
             raise ProfileError(f"profile {name}: status must be one of {sorted(VALID_STATUS)}")
+        # Merge private host bindings over the public class definition.
+        binding = overlay.get(name) or {}
+        p["hosts"] = binding.get("hosts", [])
+        p["smoke_host"] = binding.get("smoke_host")
         if not isinstance(p["hosts"], list):
             raise ProfileError(f"profile {name}: hosts must be a list")
     return profiles

src/forge/run.py

@@ -221,7 +221,7 @@ def register(subparsers):
     p.add_argument("--host", required=True)
     p.add_argument("--target-type", default="node", choices=["node", "gpu"])
     p.add_argument("--gpu-name", default=None,
-                   help="gpu target name, e.g. 'ai-silo:2' (required with --target-type gpu)")
+                   help="gpu target name, e.g. 'node:2' (required with --target-type gpu)")
     p.add_argument("--off-fleet", action="store_true")
     p.add_argument("--run-root", default=None)
     p.add_argument("--dry-run", action="store_true")

tests/test_cli.py

@@ -18,9 +18,11 @@ def test_unknown_command_exits_nonzero():
 
 def test_config_defaults(monkeypatch):
     monkeypatch.delenv("FORGE_DATA_CAP_TB", raising=False)
+    monkeypatch.delenv("FORGE_FLEET_URL", raising=False)
     import importlib
     from forge import config
     importlib.reload(config)
     assert config.DATA_CAP_TB == 50.0
     assert config.DATA_ROOT == "/mnt/models/datasets"
-    assert config.FLEET_URL == "http://192.168.77.36:8096"
+    # default carries no fleet-specific address; the real URL is set via env
+    assert config.FLEET_URL == "http://localhost:8096"