fdev host hygiene: browser-run and /tmp holds, disk-backed TMPDIR, dead-run temp cleanup (Refs #5240)

.agents/skills/atlas/assets/seed/records/58901d008c36a777dc138fa6b6d198030e0e843bc889cbad3acb954c826b3608.json

@@ -0,0 +1 @@
+{"citations":[{"path":"scripts/fdev-lane-capacity.py","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"3657c8432792a461dc26f97a0375c3a7f8ead038408ada4b3b1bc8e3cd650467"},{"path":"scripts/fdev-tmp-cleanup.py","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"6fcb5fe6d16fc56f042c152502d1024aae5b5e7f0214b54fa26147fe7f08a083"},{"path":"deployment/fdev-chromium/fdev-chromium-env","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"8cbc5e158b1619c23cd4666667e1d5c316de6229f71ccf5a888576050ab3b2a5"},{"path":"docs/coordination/fdev-host-hygiene.md","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"af791bd6cf7e83c3cc0ff28e5b725b3ea1248daf657c1c257e282379b8635e61"}],"conflictsWith":[],"id":"environment/fdev-host-hygiene","kind":"lesson","recordedAt":"2026-11-02T06:57:04Z","repository":"FS-GG/.github","schema":"fsgg.atlas/1","status":"recorded","summary":"Observed 2026-11-02 (FS-GG/.github#5240): /tmp on fdev is a 16 GB tmpfs shared by every lane; it filled (about 15k entries, 9.5 GB; fifteen browser-smoke mktemp trees of about 232 MB each and a 725 MB fsgg-gate-engine cache were the bulk) and caused 38 spurious SDD test failures while many browser smokes ran and the quiet-host probe did not admit for 60 minutes. Delivered on routine/5240-fdev-host-hygiene (cited files at the cited revision; merge and install are separate effects). (1) scripts/fdev-lane-capacity.py holds at 2 or more headless Chromium MAIN processes (argv[0] Chromium binary, --headless, --remote-debugging-pipe|port, no --type=; renderer/GPU/zygote/crashpad children and node/timeout launchers do not count; --browser-hold-at, 0 disables) and when /tmp has under 3 GiB free (--min-tmp-gib, --tmp-path), JSON browsers.count/pids and tmp_free_gib. LESSON: Chromium rewrites its process title, so on fdev /proc/<pid>/cmdline is ONE space-joined string with a trailing NUL, for the main process and every --type= child; a NUL-split parser sees one word and never counts a browser, so the probe splits a single spaced field and takes argv[0] from readlink exe; checked live: one real Playwright chrome-headless-shell -> count 1, two -> hold. (2) deployment/fdev-chromium/fdev-chromium-env is now the repository source of the wrapper (it was only ~/.local/bin; environment/fdev-headless-chromium says it is not in Git, which this supersedes in fact but not by digest) and defaults TMPDIR to /home/developer/.cache/fsgg-tmp unless set, mkdir -p then chmod 0700 (shellcheck SC2174 forbids mkdir -p -m); capped at 32 bytes because a Unix socket path is at most 107 bytes and the SDD race test binds /sdd-race-characterization-<32 hex>/inputs/socket (TMPDIR plus 73); mixed TMPDIR values may fragment MSBuild node reuse; --print-tmpdir serves .NET runs. (3) scripts/fdev-tmp-cleanup.py is dry-run unless --apply and removes only known names (fsgg-sdd-tests-<pid>-* idle 1 h; fsgg-sdd-<hex>, fsgg-ai-tests/ai-tr/resolver/cli-tests-cache-sandbox-<hex>, playwright_chromiumdev_profile-*, org.chromium.Chromium.*, and tmp.* only with driver/run-smoke.mjs, idle 6 h because they carry no PID) whose owner is dead (PID absent, and no live process references the path via cmdline, environ, cwd, fd, maps or /proc/net/unix, relative words resolved against the process cwd and symlink aliases by realpath). It refuses to delete while a same-user process cannot be inspected (non-dumpable processes have a root-owned /proc dir; allowlist systemd --user, (sd-pam), sshd), uses an iterative O_NOFOLLOW directory-fd removal that never follows a symlink or crosses a mount, and refuses euid 0. playwright-artifacts-* and fsgg-gate-* are deliberately not swept. Hourly user timer templates (minute 37) and install.py tmp-cleanup and chromium-env effects ship in source; Root installs after merge. Limits: a dry run on the real host listed 874 of 1077 matched entries (3.3 GiB) removable and removed nothing; the probe samples, it does not reserve; lanes still need TMPDIR set to the disk path for the pressure to drop at the source.","supersedes":[],"tags":["fdev","tmp","tmpfs","cleanup","capacity","browser","chromium","tmpdir","unix-socket","environment","testing"],"title":"fdev host hygiene: /tmp is a shared 16 GB tmpfs, so hold on browser count and /tmp headroom, use a short disk TMPDIR, and sweep only dead harness dirs"}

.claude/skills/atlas/assets/seed/records/58901d008c36a777dc138fa6b6d198030e0e843bc889cbad3acb954c826b3608.json

@@ -0,0 +1 @@
+{"citations":[{"path":"scripts/fdev-lane-capacity.py","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"3657c8432792a461dc26f97a0375c3a7f8ead038408ada4b3b1bc8e3cd650467"},{"path":"scripts/fdev-tmp-cleanup.py","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"6fcb5fe6d16fc56f042c152502d1024aae5b5e7f0214b54fa26147fe7f08a083"},{"path":"deployment/fdev-chromium/fdev-chromium-env","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"8cbc5e158b1619c23cd4666667e1d5c316de6229f71ccf5a888576050ab3b2a5"},{"path":"docs/coordination/fdev-host-hygiene.md","repository":"FS-GG/.github","revision":"d0571c221bc268be84271453f8c4d9f87406713f","sha256":"af791bd6cf7e83c3cc0ff28e5b725b3ea1248daf657c1c257e282379b8635e61"}],"conflictsWith":[],"id":"environment/fdev-host-hygiene","kind":"lesson","recordedAt":"2026-11-02T06:57:04Z","repository":"FS-GG/.github","schema":"fsgg.atlas/1","status":"recorded","summary":"Observed 2026-11-02 (FS-GG/.github#5240): /tmp on fdev is a 16 GB tmpfs shared by every lane; it filled (about 15k entries, 9.5 GB; fifteen browser-smoke mktemp trees of about 232 MB each and a 725 MB fsgg-gate-engine cache were the bulk) and caused 38 spurious SDD test failures while many browser smokes ran and the quiet-host probe did not admit for 60 minutes. Delivered on routine/5240-fdev-host-hygiene (cited files at the cited revision; merge and install are separate effects). (1) scripts/fdev-lane-capacity.py holds at 2 or more headless Chromium MAIN processes (argv[0] Chromium binary, --headless, --remote-debugging-pipe|port, no --type=; renderer/GPU/zygote/crashpad children and node/timeout launchers do not count; --browser-hold-at, 0 disables) and when /tmp has under 3 GiB free (--min-tmp-gib, --tmp-path), JSON browsers.count/pids and tmp_free_gib. LESSON: Chromium rewrites its process title, so on fdev /proc/<pid>/cmdline is ONE space-joined string with a trailing NUL, for the main process and every --type= child; a NUL-split parser sees one word and never counts a browser, so the probe splits a single spaced field and takes argv[0] from readlink exe; checked live: one real Playwright chrome-headless-shell -> count 1, two -> hold. (2) deployment/fdev-chromium/fdev-chromium-env is now the repository source of the wrapper (it was only ~/.local/bin; environment/fdev-headless-chromium says it is not in Git, which this supersedes in fact but not by digest) and defaults TMPDIR to /home/developer/.cache/fsgg-tmp unless set, mkdir -p then chmod 0700 (shellcheck SC2174 forbids mkdir -p -m); capped at 32 bytes because a Unix socket path is at most 107 bytes and the SDD race test binds /sdd-race-characterization-<32 hex>/inputs/socket (TMPDIR plus 73); mixed TMPDIR values may fragment MSBuild node reuse; --print-tmpdir serves .NET runs. (3) scripts/fdev-tmp-cleanup.py is dry-run unless --apply and removes only known names (fsgg-sdd-tests-<pid>-* idle 1 h; fsgg-sdd-<hex>, fsgg-ai-tests/ai-tr/resolver/cli-tests-cache-sandbox-<hex>, playwright_chromiumdev_profile-*, org.chromium.Chromium.*, and tmp.* only with driver/run-smoke.mjs, idle 6 h because they carry no PID) whose owner is dead (PID absent, and no live process references the path via cmdline, environ, cwd, fd, maps or /proc/net/unix, relative words resolved against the process cwd and symlink aliases by realpath). It refuses to delete while a same-user process cannot be inspected (non-dumpable processes have a root-owned /proc dir; allowlist systemd --user, (sd-pam), sshd), uses an iterative O_NOFOLLOW directory-fd removal that never follows a symlink or crosses a mount, and refuses euid 0. playwright-artifacts-* and fsgg-gate-* are deliberately not swept. Hourly user timer templates (minute 37) and install.py tmp-cleanup and chromium-env effects ship in source; Root installs after merge. Limits: a dry run on the real host listed 874 of 1077 matched entries (3.3 GiB) removable and removed nothing; the probe samples, it does not reserve; lanes still need TMPDIR set to the disk path for the pressure to drop at the source.","supersedes":[],"tags":["fdev","tmp","tmpfs","cleanup","capacity","browser","chromium","tmpdir","unix-socket","environment","testing"],"title":"fdev host hygiene: /tmp is a shared 16 GB tmpfs, so hold on browser count and /tmp headroom, use a short disk TMPDIR, and sweep only dead harness dirs"}

.github/workflows/fdev-board-tend-selftest.yml

@@ -9,12 +9,16 @@ on:
   pull_request:
     paths:
       - "deployment/fdev-board-tend/**"
+      - "deployment/fdev-chromium/**"
+      - "scripts/fdev-tmp-cleanup.py"
       - "tests/fdev-board-tend/**"
       - ".github/workflows/fdev-board-tend-selftest.yml"
   push:
     branches: [main]
     paths:
       - "deployment/fdev-board-tend/**"
+      - "deployment/fdev-chromium/**"
+      - "scripts/fdev-tmp-cleanup.py"
       - "tests/fdev-board-tend/**"
       - ".github/workflows/fdev-board-tend-selftest.yml"
   workflow_dispatch:

.github/workflows/fdev-host-hygiene-selftest.yml

@@ -0,0 +1,45 @@
+name: fdev-host-hygiene-selftest
+
+# fdev host hygiene (.github#5240): scripts/fdev-tmp-cleanup.py removes dead harness temp directories from the shared
+# /tmp tmpfs (dry-run unless --apply), and deployment/fdev-chromium/fdev-chromium-env defaults TMPDIR to a short
+# disk-backed path. The capacity probe's browser and /tmp holds are covered by fdev-lane-capacity-selftest. These
+# fixtures build a fake temp root and a fake procfs, offline and credential-free, and never touch the runner's /tmp
+# procfs, systemd or a real browser.
+on:
+  pull_request:
+    paths:
+      - "scripts/fdev-tmp-cleanup.py"
+      - "deployment/fdev-chromium/**"
+      - "tests/fdev-tmp-cleanup/**"
+      - "tests/fdev-chromium-env/**"
+      - ".github/workflows/fdev-host-hygiene-selftest.yml"
+  push:
+    branches: [main]
+    paths:
+      - "scripts/fdev-tmp-cleanup.py"
+      - "deployment/fdev-chromium/**"
+      - "tests/fdev-tmp-cleanup/**"
+      - "tests/fdev-chromium-env/**"
+      - ".github/workflows/fdev-host-hygiene-selftest.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+# PR-keyed concurrency (#4730 CI-load): a newer run for the same pull request cancels the superseded
+# one. Any other event keys on `github.run_id`, so two events at one ref or sha never drop a run.
+concurrency:
+  group: fdev-host-hygiene-selftest-${{ github.event.pull_request.number || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  fixture:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v7
+      - uses: ./.github/actions/setup-policy-python
+      - name: Run the cleanup fixtures
+        run: bash tests/fdev-tmp-cleanup/run.sh
+      - name: Run the chromium wrapper TMPDIR fixtures
+        run: bash tests/fdev-chromium-env/run.sh

deployment/fdev-board-tend/fsgg-tmp-cleanup.service.in

@@ -0,0 +1,20 @@
+# Rendered by deployment/fdev-board-tend/install.py from fsgg-tmp-cleanup.service.in (Refs FS-GG/.github#5240).
+# Do not edit the installed copy; change the template and run `install.py tmp-cleanup --force` again.
+# Removes dead, idle harness temp directories from /tmp, $TMPDIR and ~/.cache/fsgg-tmp (scripts/fdev-tmp-cleanup.py).
+# Only known harness names; only when the owning PID is dead (or no live process references the entry) and
+# nothing under it changed for over an hour. Never follows a symlink. Never touches a live run.
+[Unit]
+Description=FS-GG fdev dead-run temp cleanup (Refs FS-GG/.github#5240)
+Documentation=https://github.com/FS-GG/.github/blob/main/docs/coordination/fdev-host-hygiene.md
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h
+UMask=0077
+NoNewPrivileges=yes
+# Housekeeping must never compete with a lane for CPU or disk.
+Nice=15
+IOSchedulingClass=idle
+TimeoutStartSec=600
+# Deliberately no PrivateTmp: the job exists to see the real /tmp that the lanes share.
+ExecStart=/usr/bin/python3 -I @LIBEXEC@/fdev-tmp-cleanup.py --apply

deployment/fdev-board-tend/fsgg-tmp-cleanup.timer.in

@@ -0,0 +1,12 @@
+# Rendered by deployment/fdev-board-tend/install.py from fsgg-tmp-cleanup.timer.in (Refs FS-GG/.github#5240).
+[Unit]
+Description=Hourly FS-GG fdev dead-run temp cleanup (minute 37, away from board tend's :07, the drift observer's :23 and the :53 dispatch)
+
+[Timer]
+OnCalendar=*-*-* *:37:00
+Persistent=true
+RandomizedDelaySec=120
+Unit=fsgg-tmp-cleanup.service
+
+[Install]
+WantedBy=timers.target

deployment/fdev-board-tend/install.py

@@ -1,7 +1,7 @@
 #!/usr/bin/python3 -I
 """Install the fdev board-tend units (board-reconciler design section 8, #5136 PR 8). User-level only.
 
-Three subcommands, each writing one kind of thing and nothing else. Every one has --dry-run, which prints what it
+Five subcommands, each writing one kind of thing and nothing else. Every one has --dry-run, which prints what it
 would write and changes nothing. None needs or uses root; every destination must be under $HOME.
 
   units        Renders fsgg-board-tend.{service,timer} and fsgg-board-tend-daily.{service,timer} into the user
@@ -11,6 +11,11 @@
   kit-root     Writes the two lifecycle-contract.md mirrors of one released ref into the pinned kit root, after
                checking each against the expected sha256 and for exactly one fsgg:board-tend-contract/v1 marker.
   enable-file  Writes the EMPTY enable file ({"apply": []}) with mode 0600 and never replaces an existing one.
+  tmp-cleanup  Renders fsgg-tmp-cleanup.{service,timer} (hourly dead-run /tmp cleanup, .github#5240) into the user
+               systemd directory and copies scripts/fdev-tmp-cleanup.py to the libexec directory. The unit runs
+               the script with --apply. Without --enable it does not touch systemd.
+  chromium-env Copies deployment/fdev-chromium/fdev-chromium-env to ~/.local/bin (the source of truth for the
+               installed wrapper, which sets the short disk-backed TMPDIR default, .github#5240).
 
 The engine and the kit root are pinned to one release: the engine must live in .../coord-<version>/ and the kit
 root must be named kit-<version>.
@@ -32,6 +37,11 @@
 UNIT_TEMPLATES = ("fsgg-board-tend.service", "fsgg-board-tend.timer", "fsgg-board-tend-daily.service", "fsgg-board-tend-daily.timer")
 TIMERS = ("fsgg-board-tend.timer", "fsgg-board-tend-daily.timer")
 WRAPPER = "fsgg-board-tend-run.py"
+CLEANUP_UNITS = ("fsgg-tmp-cleanup.service", "fsgg-tmp-cleanup.timer")
+CLEANUP_TIMER = "fsgg-tmp-cleanup.timer"
+CLEANUP_SCRIPT_SOURCE = HERE.parent.parent / "scripts" / "fdev-tmp-cleanup.py"
+CLEANUP_SCRIPT = "fdev-tmp-cleanup.py"
+CHROMIUM_ENV_SOURCE = HERE.parent / "fdev-chromium" / "fdev-chromium-env"
 ENABLE_TEMPLATE = "board-tend.json"
 MIRRORS = (".claude/skills/work-board/references/lifecycle-contract.md", ".agents/skills/work-board/references/lifecycle-contract.md")
 MARKER_LINE = re.compile(r"^\s*<!--\s*fsgg:board-tend-contract/(\S*?)\s*-->\s*$")
@@ -181,6 +191,38 @@ def command_units(args: argparse.Namespace) -> int:
     return 0
 
 
+def command_tmp_cleanup(args: argparse.Namespace) -> int:
+    if os.geteuid() == 0:
+        raise Refusal("run as the fdev user, not root")
+    unit_dir = user_path(args.unit_dir, "--unit-dir")
+    libexec = user_path(args.libexec_dir, "--libexec-dir")
+    plan = Plan(args.dry_run)
+    for name in CLEANUP_UNITS:
+        text = render((HERE / f"{name}.in").read_text(encoding="utf-8"), {"LIBEXEC": str(libexec)})
+        plan.add(unit_dir / name, text.encode("utf-8"), 0o644, args.force)
+    plan.add(libexec / CLEANUP_SCRIPT, CLEANUP_SCRIPT_SOURCE.read_bytes(), 0o755, args.force)
+    plan.apply()
+    commands = [["systemctl", "--user", "daemon-reload"], ["systemctl", "--user", "enable", "--now", CLEANUP_TIMER]]
+    if not args.enable:
+        print("systemd not touched; to enable: " + " && ".join(" ".join(c) for c in commands))
+        return 0
+    for command in commands:
+        print(("would run  " if args.dry_run else "run        ") + " ".join(command))
+        if not args.dry_run:
+            subprocess.run(command, check=True, timeout=60)
+    return 0
+
+
+def command_chromium_env(args: argparse.Namespace) -> int:
+    if os.geteuid() == 0:
+        raise Refusal("run as the fdev user, not root")
+    dest = user_path(args.dest, "--dest")
+    plan = Plan(args.dry_run)
+    plan.add(dest, CHROMIUM_ENV_SOURCE.read_bytes(), 0o755, args.force)
+    plan.apply()
+    return 0
+
+
 def markers(text: str) -> list[str]:
     fenced, found = False, []
     for line in text.replace("\r\n", "\n").split("\n"):
@@ -259,13 +301,24 @@ def parser() -> argparse.ArgumentParser:
     enable = sub.add_parser("enable-file")
     enable.add_argument("--dest", default=str(pathlib.Path(os.environ.get("XDG_CONFIG_HOME") or h / ".config") / "fsgg/board-tend.json"))
     enable.add_argument("--dry-run", action="store_true")
+    cleanup = sub.add_parser("tmp-cleanup")
+    cleanup.add_argument("--unit-dir", default=str(pathlib.Path(os.environ.get("XDG_CONFIG_HOME") or h / ".config") / "systemd/user"))
+    cleanup.add_argument("--libexec-dir", default=str(h / ".local/libexec"))
+    cleanup.add_argument("--enable", action="store_true", help="daemon-reload and enable --now the cleanup timer")
+    cleanup.add_argument("--force", action="store_true", help="replace installed files that differ")
+    cleanup.add_argument("--dry-run", action="store_true")
+    chromium = sub.add_parser("chromium-env")
+    chromium.add_argument("--dest", default=str(h / ".local/bin/fdev-chromium-env"))
+    chromium.add_argument("--force", action="store_true", help="replace the installed wrapper when it differs")
+    chromium.add_argument("--dry-run", action="store_true")
     return root
 
 
 def main(argv: list[str] | None = None) -> int:
     args = parser().parse_args(argv)
     try:
-        return {"units": command_units, "kit-root": command_kit_root, "enable-file": command_enable_file}[args.command](args)
+        return {"units": command_units, "kit-root": command_kit_root, "enable-file": command_enable_file,
+                "tmp-cleanup": command_tmp_cleanup, "chromium-env": command_chromium_env}[args.command](args)
     except Refusal as error:
         print(f"install.py: refused: {error}", file=sys.stderr)
         return 2