A stale runtime that cannot be deleted kills the jac process, and the error names the wrong path

[MusabMahmoodh]

Summary

Clearing a stale runtime from the jac cache is fatal. If any rmdir inside the stale directory fails, jac dies with Uncaught exception and exit 1 before running the command at all - even though the new runtime has already been extracted successfully and nothing needs the stale one.

main is red on jac-check right now for exactly this, and it is not the diff that merged.

Evidence

Two jobs, same stale generation id f4ca7e3e22fe2d5d-2796bddb91431b70, different file each time, both dying in the "Warm the runtime cache" step before reading a single source file:

main @ 9b02a4b2d9 (#8944), jac-check job 101933238973:

jac:   extracting runtime...
jac:   removing stale runtime /home/runner/_work/_temp/.cache/jac/rt/f4ca7e3e22fe2d5d-2796bddb91431b70...
Uncaught exception: rmtree: could not remove '/home/runner/_work/_temp/.cache/jac/rt/f4ca7e3e22fe2d5d-2796bddb91431b70/python/lib/python3.14/site-packages/websocket/tests'
##[error]Process completed with exit code 1.

PR #9045 @ a810161, jac-check job 101923479268:

jac:   removing stale runtime /home/runner/_work/_temp/.cache/jac/rt/f4ca7e3e22fe2d5d-2796bddb91431b70...
Uncaught exception: rmtree: could not remove '/home/runner/_work/_temp/.cache/jac/rt/f4ca7e3e22fe2d5d-2796bddb91431b70/python/lib/python3.14/site-packages/openai/types/conversations/__pycache__'

The immediately preceding commit on main (7e2ddc1f8a) was green on the same job, so this is intermittent and driven by which cache actions/cache/restore hands the job, not by any diff.

Mechanism

Two separate defects, and the second is why the error message points at the wrong file.

1. A best-effort cleanup is fatal. jac/jaclang/dist/fused/materialize.jac:213-214:

say(f"  removing stale runtime {full}...");
shutil.rmtree(full);

Unguarded. This loop only ever visits directories it has already decided are stale (name != keep_key, same path hash). The freshly extracted runtime at keep_key is complete before this runs, so failing to delete a directory nobody will read again cannot affect correctness - but it takes the whole process down.

2. The failure is reported at the wrong level. jac/jaclang/runtime/na_stdlib/shutil.jac:96-102:

if os.path.isdir(child) {
    rmtree(child);
} else {
    unlink(child);          // i32 return, discarded
}
...
if rmdir(path) != 0 {
    raise ValueError(f"rmtree: could not remove '{path}'");
}

unlink is declared -> i32 (:3) and its result is dropped. So a file that cannot be unlinked fails silently, the directory is then non-empty, and rmdir fails on the parent. That is why both messages name a directory (websocket/tests, .../__pycache__) rather than the file that actually could not be removed. Whoever debugs this from the message alone is looking one level above the real cause.

shutil.jac:70 drops an unlink return the same way inside move.

Why this is worth fixing rather than re-running

• It is not attributable to any diff, so the first reaction is always to re-run, and the second is to go looking through an innocent PR. It cost a cycle on fix(config): fail loud on unresolved env vars #9045 and on fix(serve): one asset URL law — shared root table, public/ at the web root, every server consumes it #8175 this week, and it currently has main red.
• Every jac invocation on any machine holding a stale generation is exposed, not just CI. Local dev caches accumulate generations the same way.
• The cleanup is the one part of the boot path where failure is provably harmless, and it is the only part that is fatal.

Suggested shape

1. materialize.jac:214 - a stale-runtime removal that fails should warn and continue. Leaving a stale directory on disk costs disk, not correctness, and the next run retries it.
2. shutil.jac:97 - check unlink's return and raise naming the file, so when a removal genuinely must succeed the diagnostic points at the cause. Same for :70.

I have not reproduced this on a local machine; both witnesses are CI jobs and the mechanism above is read off the source as it stands on 9b02a4b2d9. What I have not established is why the unlink fails on those particular paths (permissions from the cache restore, or a concurrent reader) - but that is the third defect, and the first two make it non-fatal either way.