audit 2026-09-02: remediate all findings on the v0.5.34..v0.5.42 delta

.github/workflows/ci.yml

@@ -59,7 +59,9 @@ jobs:
 
       - name: Tests (excluding live broker)
         if: matrix.python-version != '3.12'
-        run: uv run pytest -m "not live" -q
+        # -rs: print each skip's reason. Real-execution tests that need a
+        # systemd --user scope skip on GitHub runners; silent skips hid that.
+        run: uv run pytest -m "not live" -q -rs
 
       # One matrix leg also enforces a coverage floor (term-missing report,
       # no external uploader). BRANCH coverage — line coverage alone lets an
@@ -70,7 +72,7 @@ jobs:
       # slowest step buys nothing.
       - name: Tests with coverage gate (excluding live broker)
         if: matrix.python-version == '3.12'
-        run: uv run pytest -m "not live" -q --cov=src/jobd --cov=src/job_cli --cov-branch --cov-report=term-missing --cov-fail-under=83
+        run: uv run pytest -m "not live" -q -rs --cov=src/jobd --cov=src/job_cli --cov-branch --cov-report=term-missing --cov-fail-under=83
 
       # mypy is clean and now gates the build: type regressions fail CI.
       - name: Types (mypy)

.github/workflows/codeql.yml

@@ -20,7 +20,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        language: ["python"]
+        # `actions` scans the workflows themselves (untrusted-input injection,
+        # unpinned actions, over-broad permissions) — six of them ship here.
+        language: ["python", "actions"]
     steps:
       - name: Checkout
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

.github/workflows/release.yml

@@ -15,6 +15,12 @@ on:
   push:
     tags: ["v*"]
 
+# Read-only default for every job; the three that publish (pypi, docker,
+# github-release) declare their own narrower-but-writable blocks below, which
+# REPLACE this one rather than extend it.
+permissions:
+  contents: read
+
 env:
   # See ci.yml: `uv run` re-locks a stale lockfile unless told not to, which
   # would let the test gate below test something other than the committed lock.
@@ -140,7 +146,7 @@ jobs:
     permissions:
       contents: write # create the release + upload assets
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.0 (for CHANGELOG.md)
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 (for CHANGELOG.md)
       - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
         with:
           name: dist

CITATION.cff

@@ -14,13 +14,15 @@ authors:
     given-names: Jaret
     orcid: "https://orcid.org/0009-0003-4055-5238"
 doi: "10.5281/zenodo.21636369"
+# Only the concept DOI is recorded here. Zenodo mints a version DOI AFTER the
+# GitHub release is published, so a "DOI for this specific release" in this
+# file is stale by construction on the commit that ships the release (it named
+# the v0.5.38 record beside version 0.5.42 until the 2026-09-02 audit). Each
+# release's own DOI is listed on the Zenodo record the concept DOI resolves to.
 identifiers:
   - type: doi
     value: "10.5281/zenodo.21636369"
     description: "Concept DOI — always resolves to the latest release."
-  - type: doi
-    value: "10.5281/zenodo.21831629"
-    description: "DOI for this specific release."
 repository-code: "https://github.com/musharna/jobd"
 url: "https://pypi.org/project/jobd/"
 license: MIT

README.md

@@ -227,7 +227,7 @@ Now an agent can "run this overnight," check on it next session, and route GPU w
 
 Three optional YAML files under `JOBD_CONFIG_DIR` (defaults shipped in `config/`):
 
-- **`projects.yaml`** — per-project base priority and submit defaults (preemptibility, wall/idle timeouts, host pins, capability requirements). See [docs/plans/projects-yaml.md](https://github.com/musharna/jobd/blob/main/docs/projects-yaml.md) for the full resolution model.
+- **`projects.yaml`** — per-project base priority and submit defaults (preemptibility, wall/idle timeouts, host pins, capability requirements). Entries may also declare `roots:` so a job typed with an unregistered run label is priced by the project whose directory it runs in. See [docs/projects-yaml.md](https://github.com/musharna/jobd/blob/main/docs/projects-yaml.md) for the full resolution model and [docs/events.md](https://github.com/musharna/jobd/blob/main/docs/events.md) for the event catalog.
 - **`profiles.yaml`** — named resource bundles (`--profile gpu-train-large`) the matcher uses to size a job.
 - **`classifier.yaml`** — rules that auto-suggest a profile from the command string.
 

changelog.d/audit-2026-09-02-broker.fixed.md

@@ -0,0 +1,3 @@
+- **`job list --project` and `/events?project=` now fold the name the way submit does.** Submit stores a job under its registered spelling (`BETA` prices as, and is recorded as, `beta`), but both read filters compared the raw string, so no single spelling returned all of a project's jobs and a query for `Project_A` found no event written after the fold landed. The filter now matches the folded identity, the typed spelling, and the typed run label. `project_label` gains an index (created by `migrate()` on in-place upgrades too) since the list filter ORs on it.
+- **A project declared with `roots:` but no `priority:` is a load error, not a silent drop.** `load_projects` skipped any entry lacking `priority` before the roots validation ran, so `name: {roots: [...]}` vanished with no log line — the invisible removal the roots validation promises to prevent. `_default` may no longer declare roots either (they were validated, accepted, and never consulted). A leading `//` in a root or cwd is collapsed onto `/`; previously `//home/x` was a valid root that could never match.
+- **`/reload` takes the projects mutation lock.** A `set`/`nudge` that landed between reload's re-read and its swap of the projects table was persisted from the new table and lost.

changelog.d/audit-2026-09-02-cli.fixed.md

@@ -0,0 +1 @@
+- **`job submit --explain` names the typed spelling on a pure fold, `job projects set/nudge` percent-encode the project name, and `GET /projects` / `job projects list` show each project's `roots`.** The dry run printed the typed name only when cwd had supplied the identity, so `--project PROJECT-C` rendered the registered `project-c` and never what was typed. The write commands spliced the name into the URL raw, so `job projects set ../reload 5` normalised to `POST /reload`, ran a config reload, and then reported a write that never happened. And no runtime surface listed the roots a broker had actually loaded; the only way to learn one was a dry run from inside it. The runtime overlay still persists priorities alone, so exposing roots on the read surface cannot leak them into it.

changelog.d/audit-2026-09-02-docs.changed.md

@@ -0,0 +1 @@
+- **Docs and metadata caught up with what shipped.** `CITATION.cff` records only the concept DOI: the "DOI for this specific release" entry named the v0.5.38 Zenodo record beside `version: 0.5.42`, and it is stale by construction because Zenodo mints a version DOI only after the GitHub release exists. The cwd-identity design and plan under `docs/plans/` carry "shipped in v0.5.42" banners (the plan's 44 unchecked boxes were never status), `docs/projects-yaml.md` is retitled from "Plan" and says which sections are historical, and the README's link text matches its target. New `docs/events.md` catalogs every event name in `KNOWN_EVENTS` with a one-line meaning, pinned two-way by a test. `config/projects.yaml`'s header says plainly that every name and path in it is a pseudonym, where the real ones go (`projects.local.yaml`), and what a fresh clone must replace. A real tailnet address and login that the privacy scrub missed in the plan doc, and the maintainer's database path hard-coded in the corpus export script, are gone; the script now requires `JOBD_DB`.

changelog.d/audit-2026-09-02-gitignore.security.md

@@ -0,0 +1 @@
+- **`config/projects.local.yaml` is gitignored, and a test now keeps it so.** The rule added in #129 landed as a single line containing literal `\n` sequences, so the pattern matched nothing and the file that exists to hold the operator's real project roots and names showed up as untracked, one `git add -A` away from being published. The rule is corrected in the companion privacy fix; `tests/test_no_private_paths.py` now asserts `git check-ignore` accepts the path, so it cannot silently regress.

changelog.d/audit-2026-09-02-mcp.fixed.md

@@ -0,0 +1 @@
+- **The MCP server no longer hides a substituted project identity, crashes on a missing argument, or stalls the session behind one slow call.** `jobd_list` rows and the `jobd_submit` result now carry `project_label` (the name as typed, `null` when it equals `project`), so an agent can see that the broker scheduled a job under a different identity than it asked for; previously only `jobd_status` exposed it. A missing or mis-typed argument (`jobd_status {}`) is returned as an `is_error` result of kind `invalid_arguments` with a hint, instead of a protocol-level "Internal server error" with a traceback on stderr. Tool dispatch runs off the event loop, so a `jobd_submit wait=true` (up to 270 s) no longer blocks every other request on the stdio session. The `project` schema description now describes the cwd-roots fallback rather than the old `_default`-only one.

changelog.d/audit-2026-09-02-worker.fixed.md

@@ -0,0 +1 @@
+- **A worker running two jobs at once can no longer kill the second job's fresh child during the first job's cleanup.** The reparented-orphan `/proc` sweep at job finalize gated on a point read of the in-flight count and then scanned with the lock released. Between the two, the poll loop could register and start job B, whose new child has this worker as parent and is not in job A's tracked set, so the sweep SIGTERMed it. The check and the scan now run under the same lock registration takes before any `Popen`, so B cannot appear inside the window and, if it registered first, the sweep declines. Only reachable with `max_concurrent > 1`.

config/projects.yaml

@@ -2,6 +2,18 @@
 # This file is OPTIONAL — with no projects.yaml the broker uses the global
 # default priority for every job. Edit freely for your own projects.
 #
+# WHAT IS IN HERE: every name and path in this file is a PSEUDONYM. The
+# `project-a..d` rows are placeholders, and the `alpha`/`beta`/`gamma`/`delta`
+# entries with `roots:` mirror the maintainer's fleet with names and home
+# directory rewritten, so that CI can replay 3,608 rows of real job history
+# (tests/data/project_cwd_corpus.csv, rewritten the same way) through
+# tests/test_corpus_replay.py without publishing where anything really lives.
+# The real names and roots go in `config/projects.local.yaml` beside this file
+# -- gitignored, and read by the broker as an overlay that replaces same-named
+# entries and adds new ones (docs/projects-yaml.md, "projects.local.yaml").
+# If you cloned jobd to run your own broker: put YOUR projects there, or
+# replace the entries below; `docker-compose.yml` bind-mounts ./config.
+#
 # Change priorities live (persists + reloads) with:
 #   job projects set NAME PRIORITY
 projects:
@@ -36,8 +48,9 @@ projects:
     roots:
       # 187 jobs directly in /home/user/beta plus several hundred more
       # in its .claude/worktrees/* and .worktrees/* subdirectories (matched
-      # via the same root, component-wise) — every typed name is `beta`
-      # or a `pillarN...` sweep label.
+      # via the same root, component-wise). Typed names are `beta`, the
+      # `pillarN...` sweep labels, and a few others (`synthetic-souls`,
+      # `ss-1l-emerge`, `beta-2c-*`) that all read as this one project.
       - /home/user/beta
       # 12 jobs, one typed label (`pillar1l-emerge`), unregistered — every one
       # of them falls to _default today. A sibling of the root above (not

docs/events.md

@@ -0,0 +1,72 @@
+# Event catalog
+
+Every event the broker or a worker records in `events.jsonl`, by name. The
+list is pinned to `KNOWN_EVENTS` in `src/jobd/models.py` by
+`tests/test_events_catalog_doc.py`; an event listed in one place but not the
+other fails CI. Read them with `job events`, `GET /events`, or the MCP
+`jobd_events` tool (whose `event` enum is derived from the same constant).
+
+A name that is NOT in this catalog still reaches `events.jsonl` (hooks may
+emit their own), but its Prometheus counter collapses into the `other`
+bucket, so an alert written against it never fires.
+
+## Job lifecycle (broker)
+
+| Event | Meaning |
+|---|---|
+| `job_submitted` | A job row was created (one per array member). |
+| `job_dispatched` | The dispatcher assigned the job to a worker. |
+| `job_started` | The worker reported the workload running. |
+| `job_completed` | The workload reached a terminal state the worker reported (completed / failed / exit code carried in the payload). |
+| `job_cancelled` | The job was cancelled by request or by a dependency cascade. |
+| `job_orphaned` | Its worker died or restarted while it was in flight; the job is parked for possible resurrection. |
+| `job_resurrected` | An orphaned job was re-queued after its worker came back. |
+| `job_uncancelled` | A dependency-cascade cancel was reversed because the parent came back. |
+| `scheduling_timeout` | The job waited longer than its `scheduling_timeout_s` without a matching worker and was failed. |
+| `dispatch_skip` | A dispatch pass considered the job and passed it over (payload names the reason, e.g. no worker advertises a required tag). |
+| `admission_blocked` | Admission control refused to dispatch (quota, contention, or exclusion). |
+| `auto_preempt` | A higher-priority job caused a preemptible one to be preempted. |
+| `checkpoint_complete` | The worker confirmed the workload checkpointed inside its grace window. |
+| `cwd_refused` | Submit refused the job because its `cwd` cannot be reached from any eligible worker. |
+
+## Submit-time warnings (broker)
+
+`submit_warning` fires once per job whatever warned; the per-cause events
+below fire beside it so an alert can name a single cause.
+
+| Event | Meaning |
+|---|---|
+| `submit_warning` | The submit response carried at least one warning. |
+| `unknown_project` | The typed `--project` was not registered and no `roots:` entry identified `cwd`; the job is priced at `_default`. |
+| `preflight_warning` | The preflight check on the command or environment reported something non-fatal. |
+| `cwd_route_warning` | `cwd` is reachable from some but not all otherwise-eligible workers. |
+| `serialization_warning` | A `depends_on` target was already terminal when the job was submitted. |
+| `gpu_contention_warning` | The requested GPU is currently held by another process on the pinned host. |
+| `sweep_warning` | The post-commit TOCTOU sweep found and cascaded a dependency that failed during submit. |
+| `cwd_identity_applied` | The typed name was not registered, and a project's `roots:` supplied the scheduling identity. Not a warning: payload carries `project`, `project_label`, and `matched_root`. |
+
+## Sweeper and retention (broker)
+
+| Event | Meaning |
+|---|---|
+| `reclaim_suppressed` | The sweep declined its time-based terminal phases because the broker had not been observing the interval (fresh start or a suspend/stall gap). |
+| `jobs_pruned` | Terminal job rows older than the retention window were deleted. |
+| `logs_pruned` | Job log files older than the retention window were deleted. |
+| `env_scrubbed` | Environment blobs on terminal jobs were replaced with `***` after the at-rest window. |
+
+## Worker lifecycle (broker-observed)
+
+| Event | Meaning |
+|---|---|
+| `worker_registered` | A worker registered or re-registered. |
+| `worker_offline` | A worker missed heartbeats past the offline threshold. |
+| `worker_stale` | A worker missed heartbeats past the stale threshold (still expected back). |
+| `version_drift` | A worker has run a different version than the broker for the full drift window. |
+
+## Worker-posted
+
+| Event | Meaning |
+|---|---|
+| `worker_shutdown` | The worker is draining and exiting. |
+| `watchdog_fired` | The worker's watchdog escalated a workload that ignored SIGTERM to SIGKILL. |
+| `stale_scope_sweep` | The worker found and cleaned a leftover systemd scope from a previous run. |

docs/plans/2026-08-31-cwd-project-identity-design.md

@@ -1,6 +1,9 @@
 # cwd-derived project identity
 
-Design, 2026-08-31. Status: approved, not yet planned.
+Design, 2026-08-31. **Status: shipped in v0.5.42 (2026-09-01).** Historical design
+record; the shipped behaviour diverged in places (lexical `..` collapse, the
+stderr substitution note's wording, `matched_root` in the dry-run plan). The
+current specification is `docs/projects-yaml.md` §10.
 
 ## The problem
 
@@ -21,7 +24,7 @@ v0.5.39–41 closed the _spelling_ half of this (`epsilon` now folds onto
 `epsilon`; a write reports the name it landed on). The retyped-string
 mechanism itself is untouched, and it is the larger half.
 
-### Measured, live DB `/home/user/jobd/data/jobd.db`, 2026-08-31
+### Measured, live broker DB, 2026-08-31
 
 3,608 job rows; 249 distinct `(project, cwd)` pairs; 138 distinct project
 names against 148 distinct cwds.

docs/plans/2026-08-31-cwd-project-identity-plan.md

@@ -1,5 +1,10 @@
 # cwd-derived project identity — Implementation Plan
 
+> **Historical.** Executed and shipped in v0.5.42 (2026-09-01). The checkboxes
+> below were not maintained during execution and do not reflect status. For
+> what actually shipped, read `docs/projects-yaml.md` §10 and the 0.5.42
+> section of `CHANGELOG.md`.
+
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 
 **Goal:** Let a job's scheduling identity be derived from its `cwd` when the typed `--project` name is not a registered project, so the 241 measured jobs that ran at `_default` 40 inside a registered project's directory get that project's priority.
@@ -1121,7 +1126,7 @@ signal.signal(
 )
 signal.alarm(60)
 
-DB = os.environ.get("JOBD_DB", "/home/user/jobd/data/jobd.db")
+DB = os.environ["JOBD_DB"]  # e.g. /srv/jobd/data/jobd.db on the broker host
 rows = sqlite3.connect(f"file:{DB}?mode=ro", uri=True).execute(
     "SELECT project, cwd, COUNT(*) FROM jobs GROUP BY project, cwd ORDER BY project, cwd"
 ).fetchall()

docs/projects-yaml.md

@@ -1,8 +1,13 @@
-# Plan: Per-project defaults file (`projects.yaml` enforcement)
+# `projects.yaml` — per-project priorities, defaults, and roots
 
-Design notes for the per-project `defaults:` feature. Read alongside the
-live source files (`src/jobd/app.py`, `src/jobd/config.py`,
-`src/jobd/models.py`, `src/job_cli/cli.py`).
+Sections 1–9 are the original design notes for the per-project `defaults:`
+feature (2026-07) and are kept as the rationale record: their `path:line`
+references and effort estimates are historical, and submit now lives in
+`src/jobd/broker/submit.py` with the precedence cascade in
+`src/jobd/config.py:resolve_effective_config`. Section 10 is the current
+specification for `roots:` (cwd-derived project identity, shipped v0.5.42).
+Read alongside the live source: `src/jobd/config.py`, `src/jobd/models.py`,
+`src/job_cli/cli.py`.
 
 ---
 
@@ -12,7 +17,7 @@ live source files (`src/jobd/app.py`, `src/jobd/config.py`,
 
 `projects.yaml` already lives on the broker host at the path passed to
 `build_app(projects_path=...)`. In production this resolves to
-`/app/config/projects.yaml` (controlled by `JOBD_CONFIG_DIR` in `main.py:13`).
+`/app/config/projects.yaml` (controlled by `JOBD_CONFIG_DIR`, read in `main.py`).
 In the Docker stack this maps to `/srv/jobd/config/projects.yaml`
 on the broker host, bind-mounted into the container.
 
@@ -756,11 +761,13 @@ debugging:
   the job that was priced as `beta` — though the rendered table column
   shows `project`, the scheduling identity.
 - `matched_root` — the root that supplied the identity — is **not** on the
-  Job row. It is computed at resolution time and surfaced in exactly two
+  Job row. It is computed at resolution time and surfaced in three
   places: the `POST /resolve` response (so `job submit --explain` prints
-  it, alongside the typed label) and the `cwd_identity_applied` event
-  recorded when rule 2 fires. It is `None`/absent whenever cwd was not
-  consulted.
+  it, alongside the typed label), the `validation.effective_matched_root`
+  field of a `job submit --dry-run` plan, and the `cwd_identity_applied`
+  event recorded when rule 2 fires. It is `None`/absent whenever cwd was
+  not consulted. The roots a broker has loaded are listed by
+  `GET /projects` / `job projects list`.
 
 ### The roots shipped in this file