[ronleizrowice-ant]

Human: @ronleizrowice-ant

Requested by Ron · project thread

On top of #584. To be retargeted to main once that has landed. The two should ship in one release: this adds a field and a directory to what #584 writes.

One commit, on top of #584's head. The base is #584's branch, so what is shown here is this PR's own. Once #584 has landed, the commit is rebased onto main and the base set to main.

On Linux an absent denyWrite path is blocked by an empty file or directory (a "mount point") that bubblewrap makes on the host. With #584 the processes of one host protect each other's, through manifests in /var/tmp. Hosts that mount one home (an NFS home on the nodes of a cluster) still see nothing of each other:

• a clean-up on one host unlinks a mount point under a sandbox that runs on another, and the denied write lands;
• or under one that is starting, which then fails with bwrap: Can't find source path;
• what a job killed on one node left in the shared home is swept only by a later command on that same node.

Now a mount point on a shared file system goes only when no sandbox on any host relies on it, and what a killed job left there goes with the next clean-up on its own host at once, and on any host after ten minutes.

What changes

• Where. Where the user's home, as the user database has it, is on a file system that is not on linux: keep the mount point manifests where the user id says, first under /var/tmp #615's list of kinds only one kernel reaches, a wrap names the mount points on such a file system in a second manifest, in <home>/.srt-mount-points. The manifest where it always was still names all of it, so a process of the same host that does not see the home reads what it did before. The one in the home names nothing but what other hosts reach, so whoever reads it can collect it whole. A command that has both binds both before anything else and puts itself on the record of each; a claim on either refuses it its start. The directory is made, bound read-only and pinned by every wrap that restricts writes, like the other places. It is a place for nothing else, and on a host whose home is its own nothing is read or written there. Such a host still keeps the directory out of its sandboxes' reach, because to the clients of a file server the disk it exports is a network home: it is bound and pinned wherever it is there, and made first where the command could make it.
• Liveness. No host can ask after another's processes. A manifest of another boot, or of another PID namespace there, is live until neither it nor its record has been changed for ten minutes. Its writer sets its time every minute, by its own clock, and says in the manifest that it does (keptYoung); one that does not say so keeps linux: keep the mount point manifests where the user id says, first under /var/tmp #615's seven days.
• Finishing at the same moment. linux: a placeholder is removed only when no running sandbox relies on it #584's <id>.finished name is what keeps two hosts from each leaving a path to the other.
• Clean-ups on record. A clean-up that may remove what other hosts reach goes on record in the home too. It comes to its name with its time set by the host's own clock. Another host's counts for a minute, like another PID namespace's. A wrap that names nothing other hosts reach does not wait for it, and one that does looks a few times a second. The two seconds a clean-up has for its removals run from before its record gets its time, and the clock is looked at after all that may have waited for the server.
• What a network file system remembers. A listing is asked of the server whenever the directory is opened, and each side has just changed the directory itself when it counts. A look-up is answered from memory for up to a minute, so every wrap on a host with such a home opens the directories on such a file system that hold what its profile relies on before its last look at the host. That also covers a mount point the plan took for the user's own file because this host still remembered it when its manifest had already gone. A clean-up does the same, directory by directory, before it removes anything there, and opens each file it is about to remove, into which another host may have written. And a wrap does it for each path it found absent before it names the path as its own to remove: a file the user has put there from another host since is then found, and the wrap plans again.
• What another host names. Only the paths count that are on such a file system here as well, going by the nearest directory above them that is there, at each reading. A manifest that names more is not collected from here for seven days, in the home or in any other directory both hosts read (a /var/tmp that diskless nodes share).
• A file server that misbehaves. ESTALE on a manifest is "gone", and the reading starts over; on a record it vouches, like any record that cannot be read. A manifest that has gone from its name between the listing and the look at its age has the clean-up judge afresh. A home over its quota says so only as the bytes are flushed, so they are, explicitly (Bun passes on neither what close says nor what the flush option finds); the manifest in the host's own place is then all there is. Where the home takes no record of a clean-up, only what other hosts reach stays. Where it cannot be written from a host at all (mounted read-only there), that host reads what the others keep and spares it, and removes as it did before. While it cannot be told whether there is a network home (an error of the moment from statfs, twice, or no answer from the user database), or whether one of the manifest directories is there (this now goes for the host's own places too), no command that restricts writes is handed out and nothing is removed.
• Round trips. A manifest there is read once by each process and followed by its name; the age of another host's is looked at again only when it could have grown old; the kind of file system is asked once a minute for a directory found on such a one; the times are set asynchronously, one request at a time.
• The home. Node asks the C library. Bun answers os.userInfo() with $HOME, so there it is /etc/passwd and then getent, at its usual places. Only that there is a network home is remembered for good; "no such user" is asked again after a minute, no answer at all after ten seconds.
• publishMountPointManifest returns a list; discardMountPointManifests takes one.

Who sees a difference

Linux, and only where the home is on a network file system: a directory .srt-mount-points in it, and a wrapped string with two records and two manifest binds when it names a mount point on a shared file system. On any other host there are two visible changes. Where a command may write the home itself, an empty .srt-mount-points is made in it and bound read-only. And for a manifest directory on a local kind of file system that is not on the list, what an earlier boot left there waits ten minutes instead of seven days.

Known limits (README, "Hosts that share a home")

Clocks have to agree, the file server's too (ten minutes for sandboxes, a minute for clean-ups). A process that does not return to its event loop for ten minutes while its sandbox runs, or that has exited and left it running, is taken for gone by other hosts. It goes by the home: hosts that share a project and not the home do not see each other. Paths have to be spelled alike. nocto defeats the fresh look. A process killed half way through a clean-up holds up other hosts' wraps for a minute. Virtual machines restored from one memory snapshot share a boot id. While the home gives errors, wraps that restrict writes are refused. A host that cannot write the home tells the others nothing. A host on an earlier release, or not on Linux, does not keep its sandboxes out of the directory. A host that exports a project from its own disk is not covered for that project.

Testing

• test/sandbox/mount-point-hosts.test.ts, 107 cases, also in the bubblewrap 0.4.1 step. Most stand a network file system in by statfs; six run two hosts for real but for the network, as processes with their own places and boot id over one shared directory.
• 73 mutations of the source, one at a time, each caught by a case.
• The whole suite locally on real bubblewrap 0.12.0: 4,601 pass, and the 24 failures are the ones main has on the same machine, which need outbound network.
• End to end, with no test seam: the built package under Node, each host a mount namespace with its own /var/tmp, /tmp, /etc/passwd and boot id, over one directory that a preloaded shim reports as NFS. Six hosts, two processes each, sixty commands each, every command trying six denied writes:

Build	Refused	Failed to start	Denied writes	Left in the project
This branch	0	0	none landed	nothing
#584 alone	3	126 of 720	landed	five paths
0.0.81	0	329 of 720	landed	five paths

• A job killed with SIGKILL on host 1, through the built CLI: it leaves 13 mount points in the shared project and one on host 1's own disk. A command on host 2 at once removes nothing. Eleven minutes later (the files' times set back) a command on host 2 sweeps the shared ones and the manifests in the home. A command on host 1 then sweeps its own disk.
• Not tested on a real NFS mount. What rests on NFS behaviour (a directory is revalidated when opened, a client's own change invalidates its listing, write errors surface at flush) is from its documentation.

Claude session #01SEZ8LU

---

Generated by Claude Code