[sudoTomas]

Summary

QA cache volumes are named from a hash of the deployment's instance id and the scope key, carry ownership labels, and are checked against every one of them before they are reused, removed or mounted. A per-repo volume backs only a QA run of the forge's default branch at its current head; every other run gets a per-job volume. Baking a QA image from any other branch is admin tier, a QA run selects only an image baked from the forge's default branch, and the Turbo remote cache is off by default.

Motivation — Intent + Problem

A per-repo volume is mounted writable by every QA job of its repository, and a pre-built QA image becomes the base of every QA sandbox of its repository. Each therefore has one owner and is populated only from code the repository has merged: it belongs to one deployment and one repository, is identified by labels, is checked against them before anything uses or removes it, and is trusted only when the forge itself names the run's branch and commit as the default branch's head.

How it works

• Names. volume_name_for(scope, key, instance_id=…) returns embry0-qa-vol-<scope>-<sha256(instance id, key)>. The warmer sandbox is named after the volume.
• Labels, checked at every step. A volume carries managed, instance, volume_scope, scope_key (and job_id per-job). label_mismatches requires each one to be exactly what this deployment would stamp for that scope and key, and any it would not stamp to be absent.
  • Reuse: ensure() reuses only a matching volume; otherwise a per-repo request gets a per-job volume and a per-job request raises VolumeOwnershipError.
  • Removal: recreate() and destroy_for_job() read the labels before removing anything; a volume that does not match is refused and logged (qa_volume_remove_refused, qa_job_volume_remove_refused).
  • Mount: verify() runs right before the warmer mounts a volume, before the QA fan-out hands one to the sub-tasks, and in each sub-task. A missing volume is refused, never left for the daemon to create unlabelled; volumes are created explicitly, labelled, before any mount.
  • Names from job state: mountable(name, repo=…, job_id=…) accepts only this release's name for the run's repository or job. Any other name, such as one a checkpoint written by an earlier release holds, is re-resolved to the job's per-job name and checked like any mount, so it is never mounted; the sub-tasks then clone.
  • Without an instance id no per-repo volume is used (logged once, qa_per_repo_volume_unavailable).
• Trust. In the bootstrap sandbox, forge_default_head runs git ls-remote --symref <repo URL> HEAD through sandbox_git (new step default_branch), outside the workspace, so the git-proxy answers for the repository with its own credential. parse_symref_head accepts exactly one symref line and one HEAD line; anything else is no answer. trusted_per_repo_volume is true only when the config asks for per-repo, the run's branch is that default branch and the run's resolved head (qa["head_sha"]) is its current head. A job's base_branch override and the base_branch preference choose the diff base only. No answer from the forge trusts nothing.
• Per-repo lifecycle. A trusted run recreates the per-repo volume empty when its recorded lockfile differs from the run's and reuses it as it is (warm skipped) while the lockfile matches; docs/qa-yaml-reference.md lists what carries over. The QA agent, which has a shell, runs during trusted runs, so anything it is led to write into the tree (node_modules, .git/hooks) can persist into later trusted runs until the lockfile changes. The qa.yaml reference and SECURITY.md (the data-retention row and "What does not hold") say so, and that the per-repo scope suits applications whose QA-visible content (pages, seeded data) is trusted. When the manager hands back a per-job volume instead, the warmer is given, and records, that volume's scope and key. Each round starts with no shared volume and mounts only the one it prepared.
• Images. POST /qa/images/build reads the default branch from the forge's API (get_repo / get_project) and needs admin tier for any other branch, and for any bake when the forge does not answer. Selection takes the default branch as a required argument (qa["forge_default_branch"], read from the forge by the run) and selects only a row whose recorded metadata.branch is exactly it.
• Retention. A per-repo volume is aged by the qa_volume_state row written for that volume (its volume_name). A per-repo volume under a name other than this release's is never mounted again, so it is removed once older than the retention from its creation, live jobs of its repository or not, unless a container mounts it.
• Turbo. cache.turbo_remote.enabled defaults to false.

Pattern provenance

N/A — originated in this PR.

Verification

Automated (rebased on ca9843d5, which contains #353)

• Full unit suite against a private Postgres database, -n 4 --fail-on-warnings, DOCKER_HOST=unix:///nonexistent.sock: 18129 passed, 21 skipped in 425.09s (0:07:05)
• Integration suite: 51 passed, 20 skipped (the Docker-backed ones skip, with DOCKER_HOST pointed at no daemon)
• ruff format --check . and ruff check . on the whole tree: clean. mypy embry0: Success: no issues found in 650 source files. test_sandbox_git_guard.py: 4 passed.
• Fail first. The 50 new tests (84 cases with their parameters), run inside a clean git archive of main with only the test files copied in: 84 failed, every one on an AssertionError (or pytest.fail) carrying its own message; none on an import or type error. tests/unit/cache/test_volume_manager.py reaches the newer API through _api/_params, so it collects on main and fails on assertions there.
• Mutations. 31, each run against a copy of the tree with the related test files; all 31 caught:
  • the warmer recording the requested scope instead of the volume handed out (the fallback tracking);
  • turbo_remote back on by default;
  • trust ignoring the head, ignoring the branch, or taken from the configured base (job override or preference);
  • the forge lookup run inside the workspace, or against origin; a second symref line or a second HEAD line accepted, or the last of each taken (the five new test_trusted_run.py cases for this also fail on the previous head, 7b0bd437);
  • recreate(), destroy_for_job() or ensure() skipping the label check; extra ownership labels ignored;
  • verify() skipping labels, accepting a missing volume, accepting any name; mountable() trusting a name from state; per-repo volumes without an instance id;
  • the sub-task, the fan-out or the warmer mounting without the check; a round keeping the last round's volume; a changed lockfile not recreating the volume;
  • image selection without a default branch, of any branch, or by the diff base; the build gate assuming main or reading the preference;
  • retention reading another volume's row, or keeping an older-name volume for a live job.
• Real daemon (throwaway qfix- volumes and containers, a qfix-instance id, all removed afterwards; 12/12): a first recreate() creates the labelled per-repo volume; reuse keeps it; a volume labelled for another key is never reused, and recreate() refuses to remove it; a mounted volume is left in place and the run gets a per-job one; an idle one is emptied; verify() refuses a missing volume without creating it, refuses a foreign one and passes ours; an older-format name is not mounted; destroy_for_job() removes the job's own volume and refuses one named as the job's but labelled for another; with no instance id a per-repo request gets a per-job volume.
• Real git (a qfix- container serving a bare repository over HTTP, embry0's own git configuration written by sandbox_git.configure; 6/6): forge_default_head returns the default branch and its head; the head is trusted, an older commit of the same branch and another branch are not; no answer from the forge, and a lookup the prelude refuses (its configuration made writable), read as no answer.
• gitleaks (--no-git over the changed files, live as shown by a canary): no leaks. Name denylist after staging: 0 matches.

Deploy notes

• infra/deploy.sh. No image rebuild: embry0/safety/error_codes.py and embry0/execution/sandbox_manager.py are baked into the sandbox images, but what changes there (an error-code mapping for an orchestrator-side exception, a docstring) does nothing inside a sandbox; the next routine rebuild picks it up.

• Per-repo volumes start empty under the new names: each trusted repository's first QA run after the deploy installs from scratch.

• A QA job resumed from a checkpoint written before the deploy holds a volume name in the earlier format; it is re-resolved and not mounted, and that round's sub-tasks clone their own workspace.

• Earlier-format volumes. Per-job ones keep their labels and go with their job (its cleanup or the reaper). Per-repo ones (embry0-qa-vol-<repo with / as ->) are no longer used. With QA_IMAGE_RETENTION_DAYS set, the sweep removes each once it is older than the retention, counted from its creation, unless a container mounts it. To reclaim them now, or without retention, on the sandbox daemon (with the default runtime, through embry0-dind; with AGENT_RUNTIME=rootless, against the rootless daemon). The instance filter leaves every other stack's volumes on that daemon alone:

  # this deployment's instance id, from the orchestrator's startup log
  ID=$(docker logs embry0-orchestrator 2>&1 | grep volume_instance_id_resolved | tail -1 | grep -oE '[0-9a-f]{16}' | tail -1)
  echo "instance id: $ID"   # 16 hex characters; stop here if it is empty
  docker exec embry0-dind docker volume ls -q \
    --filter label=embry0.managed=true \
    --filter "label=embry0.instance=$ID" \
    --filter label=embry0.volume_scope=per-repo \
    | grep -Ev '^embry0-qa-vol-per-repo-[0-9a-f]{64}$' \
    | xargs -r docker exec embry0-dind docker volume rm

  volume rm (no --force) refuses a volume a container still mounts.

• Pre-built images. A QA run now selects an image only when its bake recorded the forge's default branch (metadata.branch, recorded by every bake through POST /qa/images/build). A repository whose active image records another branch, or none, runs on the profile's own image until it is baked again from the default branch (operator tier). An image bake of any other branch needs an admin-tier key, and so does a bake when the forge's API does not answer for the repository.

• Turbo. A stored config that omits cache.turbo_remote no longer gets the TURBO_* variables; set cache.turbo_remote: { enabled: true } for a repository that should keep the remote cache.

• embry0_sandbox_git_refusals_total gains the step="default_branch" series, seeded at 0.

• Merging with fix(sandbox): a Maven cache per repository, not one shared #374: both edit embry0/cache/qa_image_retention.py, its tests and the QA_IMAGE_RETENTION_DAYS line of docs/configuration.md. Keep both sides; the older-name rule here applies to per-repo volumes only, so fix(sandbox): a Maven cache per repository, not one shared #374's Maven caches keep their live-job rule.