Repository navigation
linux: walk for one wrap at a time, and let go of the listings before the scan - #679
Conversation
… the scan Wraps walk in turns, so several interleave on the one thread, and each keeps all it has listed until it ends: peak memory, and the time the event loop is held, grew with the wraps in flight. Parallel walks gain nothing there. They now take their walks first come, first served. An attempt waits before it reads anything, so it goes by what it finds afterwards. The place is given up on every way out, and a waiter whose signal is aborted leaves at once.
antdres
left a comment
There was a problem hiding this comment.
No blockers from me at 135aaaa.
I ran 711 wraps over a real tree with random denyRead/allowRead/allowWrite/denyWrite sets (300 one at a time, then 30 Promise.all batches of 5 to 20 with different configs) on the merge-base and on this branch: the bwrap command lines are identical once the random socket/port/temp names are masked. The line never got stuck: abort before start, abort while waiting, abort while walking, holder and waiter aborted in the same tick, a wrap that throws, reset() mid-queue and 20 mixed wraps with random aborts all left the next wrap able to finish. A wrap that waited while updateConfig added a deny glob and a new file appeared came out with both. The 8 new tests fail with the old sandbox-manager.ts. Peak RSS for 15 concurrent walks of a 56k-entry tree went from about 270MB to about 113MB, wall time about the same (1.3s vs 1.4s).
One thing to be aware of, not blocking: a wrap that needs no walk at all now queues behind the walks ahead of it. With five walking wraps in front, a literal-only wrap took about 450ms here against about 105ms on main (alone: 20ms). On a tree where one walk takes seconds, every quick wrap behind it waits that long.
Generated by Claude Code
antdres
left a comment
There was a problem hiding this comment.
Approving at 135aaaa. No blocking issues; what I ran is in my comment above.
Generated by Claude Code
One conflict, in the imports of test/sandbox/glob-expand.test.ts: both kept.
Since the wrap walks its read patterns in turns (#628), several wraps interleave on the one thread, and each keeps every listing of its walk (#627) until it ends. So peak memory grows with the wraps in flight. So does the time the event loop is held, because every wrap's 10 ms turn runs before timers do. Parallel walks gain nothing on one thread.
**rules over 400,150 filesmain, Node / BunOne wrap alone is unchanged. At 10 wraps: 1.4 GB / 1.8 GB against 0.5 GB / 0.4 GB.
Change (+82/-2)
waitToWalk(), a first-come-first-served line of resolvers, private to the manager.movedOn()check is as it was.reset(),initialize()andupdateConfig()do not touch the line: a waiter then behaves like a wrap called when its place comes.27 of the lines are the parameter lists of a function split, which keeps about 150 lines of the attempt from being indented under a
try.The line is no safety mechanism. Every wrap stays self-contained, so a fault in it can hang a wrap or cost memory, not lose a deny.
What it costs
setImmediatethat never fires, fake timers for instance.getFsReadConfig()is synchronous and cannot wait, so during such a call two sets of listings can be alive. Two, not N.Testing
main's source: no interleaving, order, an aborted waiter, a wrap that throws, a configuration or achdirduring the wait,reset()during the wait, the listings let go before the scan. 14 mutations, each failing a test.chdir,reset()andinitialize()with the same object), also with the wrap under test waiting when the host acts: all refused, on Node and Bun.