Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,4 @@ htmlcov/

# Subagent-driven-development scratch: per-plan ledgers, briefs, review packages.
.superpowers/
\n\n# Machine-specific project roots/names; the tracked projects.yaml stays generic.\nconfig/projects.local.yaml\n
33 changes: 22 additions & 11 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,33 @@

All notable changes to jobd. Format roughly follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]\n\n### Privacy — tracked config no longer carries real paths

- `config/projects.local.yaml` (gitignored, beside `projects.yaml`) now holds
machine-specific `roots:` and private project names; each entry replaces the
same-named tracked entry wholesale. Operators must create it on deploy or lose
cwd-derived identity for those roots — see docs/projects-yaml.md.
- The tracked `projects.yaml`, the replay corpus `tests/data/project_cwd_corpus.csv`,
tests, plans and docs are pseudonymised (`/home/user/...`, greek-letter project
names). `tests/test_no_private_paths.py` fails the suite if a private path, host,
address or session trailer is ever tracked again.

## [Unreleased]

## [0.5.42] — 2026-09-01

### Added

- **The cwd-derived project identity feature is no longer inert: `agrigen`, `jepagame`,
`orchid-sdxl`, and `dreamer-chassis` now declare `roots:` in `config/projects.yaml`.**
- **The cwd-derived project identity feature is no longer inert: `alpha`, `beta`,
`gamma`, and `delta` now declare `roots:` in `config/projects.yaml`.**
The matching machinery and its acceptance gate against 3,608 rows of real job history
(`tests/test_corpus_replay.py`) landed first, but no project had ever declared a root,
so a job typed with an unregistered run label (`pillar2a1_sweep`, `arf-promoter`,
`orchid-sdxl-stage4b`, ...) still fell through to `_default` regardless. Each root was
`gamma-stage4b`, ...) still fell through to `_default` regardless. Each root was
picked by replaying `tests/data/project_cwd_corpus.csv` grouped by cwd and keeping only
directories where every project name ever typed from them reads as a variant of one
project — deliberately excluding `/home/mjarnold/trellis2` and `.../hunyuan3d`
(>=97% agrigen-typed in the corpus, trellis2 the single largest cwd in it) because both
project — deliberately excluding `/home/user/trellis2` and `.../hunyuan3d`
(>=97% alpha-typed in the corpus, trellis2 the single largest cwd in it) because both
are generic third-party tool checkouts a differently-owned future clone could silently
inherit. `docs/projects-yaml.md` gains a `roots:` section covering the schema, the
three-rule resolution order (an explicit, registered `--project` always wins over cwd —
Expand All @@ -29,7 +40,7 @@ All notable changes to jobd. Format roughly follows [Keep a Changelog](https://k
Shipping in the same release, the user-visible surface that makes a substituted identity
findable and explainable rather than merely correct:
- `job list --project NAME` now matches **either** the scheduling identity or the typed
run label. Without this a job submitted as `pillar2a1_sweep` and priced as `jepagame`
run label. Without this a job submitted as `pillar2a1_sweep` and priced as `beta`
was unfindable under the only name its submitter ever knew it by.
- `JobInfo` gains `project_label`: the name as typed, `null` when it agrees with
`project`, so the field reads as "something was substituted here".
Expand All @@ -48,8 +59,8 @@ All notable changes to jobd. Format roughly follows [Keep a Changelog](https://k
projects fold onto the same key the fold is refused and cwd decides, yet the names
still fold together. `job submit --explain` carries `matched_root` and answers it.
- Path matching now collapses `..` lexically before comparing components. It previously
did not, so `/home/mjarnold/jepagame/../../tmp` — a job actually running in `/tmp` —
matched a root of `/home/mjarnold/jepagame` and was priced at that project's `78`.
did not, so `/home/user/beta/../../tmp` — a job actually running in `/tmp` —
matched a root of `/home/user/beta` and was priced at that project's `78`.
`cwd` is free text on the wire, so this was caller-reachable. A root containing a `..`
component is now a load error, in the same raise-don't-drop style as the other root
validations. Symlinks and bind mounts remain unresolved by design: the broker cannot
Expand All @@ -76,12 +87,12 @@ All notable changes to jobd. Format roughly follows [Keep a Changelog](https://k

- **A registered project no longer loses its priority to how the name was typed.**
Project names are free text chosen at submit time and were matched with a bare
`name in projects`, so `ARFDSynInt` ran at `_default` 40 while `arfdsynint` sat
`name in projects`, so `epsilon` ran at `_default` 40 while `epsilon` sat
deliberately registered at 65 — a difference of case alone, raising no error and
visible only in a warning nothing consumed. Case and `-`/`_` are now folded when
matching, the resolved name is what `/submit` stores and `/resolve` previews, and an
exact hit always wins. Folding is deliberately **not** fuzzy: `phelipanche` is not
folded onto `phelipanche-fm`, because a suffix difference is a registration decision,
exact hit always wins. Folding is deliberately **not** fuzzy: `kappa` is not
folded onto `kappa-fm`, because a suffix difference is a registration decision,
and guessing there would run work at another project's priority — the same bug
inverted. Two registered names that fold together are reported at load and fall back
to exact matching rather than one being picked arbitrarily.
Expand Down
46 changes: 23 additions & 23 deletions config/projects.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,33 +18,33 @@ projects:
# never a shared scratch path, and never a directory the corpus also shows
# in the hands of a genuinely different project. `tests/test_corpus_replay.py`
# replays the same corpus against the resolver these roots feed.
agrigen:
alpha:
priority: 80
roots:
# 287 jobs across agrigen's own tree (backend + worktrees), all typed
# `agrigen` or an `agrigen-*`/`phase*` variant. Deliberately NOT
# including /home/mjarnold/trellis2, .../hunyuan3d or .../instantmesh —
# each is >=97% agrigen-typed in the corpus too (trellis2 alone is the
# 287 jobs across alpha's own tree (backend + worktrees), all typed
# `alpha` or an `alpha-*`/`phase*` variant. Deliberately NOT
# including /home/user/trellis2, .../hunyuan3d or .../instantmesh —
# each is >=97% alpha-typed in the corpus too (trellis2 alone is the
# single largest cwd in the whole corpus, 1209 jobs), but all three are
# generic third-party tool checkouts (TRELLIS, Hunyuan3D, InstantMesh);
# a differently-owned future clone at the same path would silently
# inherit agrigen's identity. Revisit if that risk turns out to be
# inherit alpha's identity. Revisit if that risk turns out to be
# theoretical.
- /home/mjarnold/agrigen
jepagame:
- /home/user/alpha
beta:
priority: 78
roots:
# 187 jobs directly in /home/mjarnold/jepagame plus several hundred more
# 187 jobs directly in /home/user/beta plus several hundred more
# in its .claude/worktrees/* and .worktrees/* subdirectories (matched
# via the same root, component-wise) — every typed name is `jepagame`
# via the same root, component-wise) — every typed name is `beta`
# or a `pillarN...` sweep label.
- /home/mjarnold/jepagame
- /home/user/beta
# 12 jobs, one typed label (`pillar1l-emerge`), unregistered — every one
# of them falls to _default today. A sibling of the root above (not
# nested under it, so it needs its own entry), and unlike orchid-data/
# dreamer-chassis below this one is NOT forward-looking: it recovers
# nested under it, so it needs its own entry), and unlike gamma-data/
# delta below this one is NOT forward-looking: it recovers
# real historical jobs today.
- /home/mjarnold/jepagame-1l-sweep
- /home/user/beta-1l-sweep

# Tier 2 — active development
project-c:
Expand All @@ -55,27 +55,27 @@ projects:
# grace window) — only safe for jobs that checkpoint. See docs/preemption.md.
preemptible: true
checkpoint_grace_s: 60
orchid-sdxl:
gamma:
priority: 60
roots:
# 335 jobs at /home/mjarnold/orchid-sdxl under `orchid-sdxl` plus a
# dozen `orchid-*`/`stage*` variants — real historical recovery.
- /home/mjarnold/orchid-sdxl
# 335 jobs at /home/user/gamma under `gamma` plus a
# dozen `gamma-*`/`stage*` variants — real historical recovery.
- /home/user/gamma
# Forward-looking, not evidence-backed: every one of the 17 jobs here
# was already typed `orchid-sdxl` exactly, so this root recovers ZERO
# was already typed `gamma` exactly, so this root recovers ZERO
# historical jobs (rule 1 already handled all of them). Kept because
# the directory is 17/17 clean for this project — it exists so a
# future mistyped run from here lands right, not because the corpus
# needed it.
- /home/mjarnold/orchid-data
dreamer-chassis:
- /home/user/gamma-data
delta:
priority: 60
roots:
# Forward-looking, not evidence-backed: all 18 jobs here were already
# typed `dreamer-chassis` exactly, so this root recovers ZERO
# typed `delta` exactly, so this root recovers ZERO
# historical jobs (rule 1 already handled all of them). Kept as a
# safety net for a future mistyped label from this directory.
- /home/mjarnold/dreamer-chassis
- /home/user/delta

# Tier 3 — background / best-effort
project-d: { priority: 40 }
Expand Down
9 changes: 0 additions & 9 deletions docs/plans/2026-06-30-submit-cwd-probe.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
- **Follow existing patterns:** mirror the `*_json` nullable-TEXT column style (`Worker.mount_roots_json`, `host_aliases_json`, `tags_json`) and the existing refuse-admission call site (`job_worker.py:1416`).
- **Commit footer (every commit):**
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

```
- **Deploy is out of scope for these tasks** (multi-host; gated on user go-ahead). Task 8 writes the deploy checklist only.
Expand Down Expand Up @@ -90,7 +89,6 @@ feat(matcher): WorkerSnapshot.mount_roots + _build_snapshots population
Threads the worker-reported mount_roots prefixes onto the snapshot so
submit-time routability checks can read them. Default [] = back-compat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down Expand Up @@ -245,7 +243,6 @@ Empty mount_roots = unknown (never false-reject). A shared-prefix path (/home)
is routable here by design — the worker-side isdir check catches host-local
paths under it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down Expand Up @@ -333,7 +330,6 @@ Hard-400s a pinned cwd no mount_root covers (generalizes /mnt/c); folds the
any-pin "no worker covers cwd" case into the submit warnings. /mnt/c fast-path
unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down Expand Up @@ -412,7 +408,6 @@ Per-job set of hosts that refused the job for a missing cwd, so the matcher
won't re-offer it to them. Nullable TEXT, NULL -> [] (back-compat). [Schema
init path noted for deploy: create_all vs ALTER.]

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down Expand Up @@ -575,7 +570,6 @@ A worker that finds cwd missing refuses with reason=cwd_missing; broker records
the host in excluded_workers_json and re-queues, or fails the job
cwd_unreachable when no eligible worker remains. gpu_contention path unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down Expand Up @@ -631,7 +625,6 @@ git add src/jobd/app.py tests/test_api.py
git commit -F - <<'EOF'
feat(broker): /next-job skips workers in a job's cwd-exclusion set

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down Expand Up @@ -716,7 +709,6 @@ Before the launcher check, verify os.path.isdir(cwd). If absent, POST
refuse-admission reason=cwd_missing so the broker re-routes to a host that has
the path, instead of cd-failing to exit 127.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down Expand Up @@ -753,7 +745,6 @@ git add -A
git commit -F - <<'EOF'
docs(routing): cwd-probe deploy checklist + changelog; full suite green

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

EOF
```
Expand Down
34 changes: 17 additions & 17 deletions docs/plans/2026-08-31-cwd-project-identity-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ field means every submitter has to satisfy 1 while thinking about 2, and the
failure is silent: an unregistered name falls through to `_default` and runs at
40 with no error.

v0.5.39–41 closed the _spelling_ half of this (`ARFDSynInt` now folds onto
`arfdsynint`; a write reports the name it landed on). The retyped-string
v0.5.39–41 closed the _spelling_ half of this (`epsilon` now folds onto
`epsilon`; a write reports the name it landed on). The retyped-string
mechanism itself is untouched, and it is the larger half.

### Measured, live DB `/home/mjarnold/jobd/data/jobd.db`, 2026-08-31
### Measured, live DB `/home/user/jobd/data/jobd.db`, 2026-08-31

3,608 job rows; 249 distinct `(project, cwd)` pairs; 138 distinct project
names against 148 distinct cwds.
Expand All @@ -39,11 +39,11 @@ The top entries show the mechanism plainly — these are run labels, not
projects:

```
27 pillar2a1_sweep /home/mjarnold/jepagame (jepagame is registered at 78)
26 orchid-sdxl-geo /home/mjarnold/orchid-sdxl
18 dreamer-chassis /home/mjarnold/dreamer-chassis
16 pillar4_1 /home/mjarnold/jepagame
15 orchid-stage2 /home/mjarnold/orchid-sdxl
27 pillar2a1_sweep /home/user/beta (beta is registered at 78)
26 gamma-geo /home/user/gamma
18 delta /home/user/delta
16 pillar4_1 /home/user/beta
15 gamma-stage2 /home/user/gamma
```

The relation is genuinely many-to-many — 33 projects span more than one cwd, 26
Expand Down Expand Up @@ -79,19 +79,19 @@ per-job field defaults and a root is not one of those.

```yaml
projects:
jepagame:
beta:
priority: 78
roots:
- /home/mjarnold/jepagame
orchid-sdxl:
- /home/user/beta
gamma:
priority: 60
roots:
- /home/mjarnold/orchid-sdxl
- /home/user/gamma
```

Matching is **longest root wins**, compared **path-component-wise, not as a
string prefix**. `/home/mjarnold/jepagame2` must not match root
`/home/mjarnold/jepagame`; a naive `str.startswith` says it does. This is the
string prefix**. `/home/user/beta2` must not match root
`/home/user/beta`; a naive `str.startswith` says it does. This is the
single most likely defect in the feature and gets a dedicated negative test.

Symlinks are not resolved. The broker has no access to the worker's filesystem,
Expand Down Expand Up @@ -158,7 +158,7 @@ through to `_default`" signal is not weakened by the new one.
## Testing

Unit: precedence (all three rules), longest-root selection, the
`jepagame2`/`jepagame` boundary negative, ambiguous-roots warn-and-fall-through,
`beta2`/`beta` boundary negative, ambiguous-roots warn-and-fall-through,
label preserved in every branch, bad-root load error.

The real-execution check is a **corpus replay**: run the resolver over every one
Expand Down Expand Up @@ -194,8 +194,8 @@ would read as a clean pass.
## Out of scope

Eighteen entries currently sit at priority 65, and most of them are
experiment-arm names rather than projects (`host_grammar`,
`convergent_mimicry`, `outgroup_control`, `effector_host`, `kai2d`,
experiment-arm names rather than projects (`h_grammar`,
`c_mimicry`, `og_control`, `eff_host`, `k2d`,
`arf-dimer-g1`, …) registered one at a time as a workaround for exactly this
gap. They become redundant once roots exist, but rule 1 keeps them winning and
nothing breaks while they stay. Retiring them is separate cleanup, after this
Expand Down
Loading