Skip to content

fix(sandbox): a Maven cache per repository, not one shared - #374

Merged
sudoTomas merged 1 commit into
mainfrom
fix/sandbox-maven-cache-per-repo
Nov 5, 2026
Merged

sudoTomas merged 1 commit into
mainfrom
fix/sandbox-maven-cache-per-repo

Conversation

@sudoTomas

@sudoTomas sudoTomas commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Summary

Each repository has its own Maven (~/.m2) cache volume in the JVM dev sandbox, named from a hash of the instance and the repository, labelled like the QA volumes and checked against its labels before it is mounted. A sandbox that cannot use it gets a labelled per-job cache removed with its job, and a cache that cannot be shown to be the job's is not mounted at all. The retention sweep removes a Maven cache older than QA_IMAGE_RETENTION_DAYS.

Motivation — Intent + Problem

The JVM dev sandbox keeps Maven's local repository in a volume so a job does not spend its turns downloading dependencies. That volume holds one repository's dependencies, so it has one owner: a repository on one deployment, identified by labels the volume reaper, the job's volume cleanup and the retention sweep all read.

How it works

  • Name and labels. maven_volume_name(instance_id, repo) returns embry0-maven-vol-<sha256(instance id, repo)>, labelled managed / instance / volume_scope=maven / scope_key=<repo>.
  • Mounting. SandboxManager._resolve_maven_volume runs before the sandbox is created. An existing cache is mounted only when every ownership label is exactly this repository's cache on this deployment (and no other ownership label is present); a missing one is created, labelled, first, so the daemon never creates one at mount time.
  • Per-job fallback. A sandbox with no repository, a deployment with no instance id (logged once, maven_cache_per_repo_unavailable), or a repository cache whose labels are not ours gets embry0-maven-vol-<sha256(instance id, job:<job id>)>, labelled as the job's per-job volume (volume_scope=per-job, job_id), so SharedVolumeManager.destroy_for_job and the volume reaper remove it with the job. The fallback is checked the same way: one whose labels are not the job's, or one that cannot be created, is not mounted, and the sandbox runs with no cache (maven_cache_not_mounted).
  • Sharing. Every JVM sandbox of a repository mounts its cache writable, whatever started the job; no cache is shared across repositories. SECURITY.md ("What does not hold") says so.
  • Retention. The sweep lists volume_scope=maven volumes with the per-repo ones. A Maven cache has no warmth row and Docker records no use, so it is aged from its creation: one older than the retention is removed unless a job of its repository is live or a container mounts it, and the repository's next JVM job downloads its dependencies again.
  • Wiring. Startup sets sandbox_mgr.instance_id from the resolved instance id, after resolving it; test_maven_cache_wiring.py pins that from the AST of lifespan.

Pattern provenance

N/A — originated in this PR.

Verification

Automated (rebased on ca9843d5)

  • Full unit suite against a private Postgres database, -n 4 --fail-on-warnings, DOCKER_HOST=unix:///nonexistent.sock: 18066 passed, 21 skipped in 384.86s (0:06:24)
  • Integration suite: 51 passed, 20 skipped (the Docker-backed ones skip, with DOCKER_HOST pointed at no daemon)
  • ruff format --check ., ruff check .: clean. mypy embry0: Success: no issues found in 649 source files.
  • Fail first. The 12 new tests (17 cases), run inside a clean git archive of main with only the test files copied in: 17 failed, each on an AssertionError with its own message.
  • Mutations. 12, all caught: the fallback labelled maven instead of per-job; the fallback not label-checked; a cache that failed to be created still mounted; a cache mounted without being created; a shared cache with no instance id; the missing instance id logged per sandbox; None mounted; extra ownership labels ignored; Maven caches aged by a QA warmth row, or not swept at all; one cache for every repository; the startup wiring dropped.
  • Removed with its job. test_the_per_job_maven_cache_is_removed_with_its_job creates per-job and repository caches through SandboxManager.create on one fake daemon, then runs SharedVolumeManager.destroy_for_job: the job's cache goes, another job's and the repository's stay.
  • Real daemon (throwaway qfix- volumes, a qfix-instance id, all removed afterwards; 6/6): a repository's cache is created labelled before any mount and reused by its next sandbox; a cache labelled for another repository is not mounted and the job gets its own; a per-job cache labelled for another job is not mounted and nothing is created in its place; with no instance id two jobs of one repository get caches of their own; destroy_for_job removes the job's cache and leaves the repository's.
  • gitleaks (--no-git over the changed files): no leaks. Name denylist after staging: 0 matches.

Deploy notes

  • infra/deploy.sh. No image rebuild: the QA image bakes embry0/execution/sandbox_manager.py, but the Maven cache is chosen where sandboxes are created, in the orchestrator.

  • Each repository's first JVM job after the deploy starts from an empty cache.

  • The earlier shared volume embry0-maven-repository is no longer mounted by this release. It lives on the sandbox daemon (with the default runtime, inside embry0-dind), which other embry0 stacks may share; one running an earlier release still mounts it by name. Check before removing it:

    # no container, of any stack, uses it (prints nothing)
    docker exec embry0-dind docker ps -a --filter volume=embry0-maven-repository --format '{{.Names}}'
    # the instance ids with volumes on this daemon; any other than this deployment's
    # (volume_instance_id_resolved in the orchestrator's startup log) is another stack
    docker exec embry0-dind docker volume ls --format '{{.Label "embry0.instance"}}' | sort -u
    docker exec embry0-dind docker volume rm embry0-maven-repository

    If another stack shares the daemon and runs an earlier release, leave the volume until it runs this one; volume rm without --force also refuses a volume a container uses.

  • With QA_IMAGE_RETENTION_DAYS set, a repository's Maven cache is removed once it is older than that from its creation, unless a job of the repository is live or a container mounts it at the sweep.

  • Merging with fix(cache): collision-free QA volumes, trusted per-repo only #372: both edit embry0/cache/qa_image_retention.py, its tests and the QA_IMAGE_RETENTION_DAYS line of docs/configuration.md. Keep both sides: the scope tuple and the per-repo-only warmth read here, and fix(cache): collision-free QA volumes, trusted per-repo only #372's own-row aging and its older-name rule, which applies to per-repo volumes only, so Maven caches keep the live-job rule.

The JVM dev sandbox keeps Maven's local repository (~/.m2) in a volume
so a job does not spend its turns downloading dependencies. Each
repository has a cache of its own, owned by this deployment and
identified by labels, so the volume reaper, the job's cleanup and the
retention sweep can account for it.

The cache is embry0-maven-vol-<sha256(instance id, repository)>,
labelled managed / instance / volume_scope=maven / scope_key=<repo>.
Every JVM sandbox of the repository mounts it, writable, whatever
started the job; SECURITY.md says so. It is created, labelled, before
the sandbox mounts it, and an existing one is mounted only when every
ownership label matches. A sandbox with no repository, a deployment
with no instance id, or a repository cache whose labels are not ours
gets a per-job cache instead, labelled as the job's per-job volume, so
the job's volume cleanup and the reaper remove it with the job. A
per-job cache whose labels are not the job's, or one that cannot be
created, is not mounted: the sandbox runs with no cache, and the daemon
never creates a cache at mount time.

The retention sweep (QA_IMAGE_RETENTION_DAYS) removes a Maven cache
once it is older than the retention, counted from its creation, unless
a job of its repository is live or a container mounts it.
@sudoTomas
sudoTomas force-pushed the fix/sandbox-maven-cache-per-repo branch from 826b2cf to 4197434 Compare November 5, 2026 17:17
@sudoTomas
sudoTomas merged commit 7a6d988 into main Nov 5, 2026
13 checks passed
@sudoTomas
sudoTomas deleted the fix/sandbox-maven-cache-per-repo branch November 5, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant