Skip to content

linux: walk for one wrap at a time, and let go of the listings before the scan - #679

Merged
dylan-conway merged 2 commits into
mainfrom
perf/linux-one-walk-at-a-time
Oct 10, 2026
Merged

dylan-conway merged 2 commits into
mainfrom
perf/linux-one-walk-at-a-time

Conversation

@ronleizrowice-ant

Copy link
Copy Markdown
Collaborator

Since the wrap walks its read patterns in turns (#628), several wraps interleave on the one thread, and each keeps every listing of its walk (#627) until it ends. So peak memory grows with the wraps in flight. So does the time the event loop is held, because every wrap's 10 ms turn runs before timers do. Parallel walks gain nothing on one thread.

37 wraps at once, six ** rules over 400,150 files main, Node / Bun Here, Node / Bun
Peak memory 5.5 GB / 5.3 GB 0.48 GB / 0.43 GB
First wrap ready 39 s / 24 s 1.2 s / 0.7 s
Median wrap ready 43 s / 25 s 20 s / 14 s
Longest event-loop hold 790 ms / 749 ms 28 ms / 28 ms
All ready 44.5 s / 25.6 s 38.1 s / 26.7 s

One wrap alone is unchanged. At 10 wraps: 1.4 GB / 1.8 GB against 0.5 GB / 0.4 GB.

Change (+82/-2)

  • One walk at a time. waitToWalk(), a first-come-first-served line of resolvers, private to the manager.
  • An attempt waits before it reads anything, so it goes by the configuration, the working directory and the disk it finds afterwards. A wait costs none of the restarts, and every movedOn() check is as it was.
  • The place covers the two walks only, not the network's start or the scan. It is given up on every way out: return, throw, abort, and each of the four restarts, which go to the back of the line.
  • A waiter whose signal is aborted leaves at once and rejects with the signal's reason. reset(), initialize() and updateConfig() do not touch the line: a waiter then behaves like a wrap called when its place comes.
  • The deny listings are emptied before the scan. Bun kept them for as long as the function was suspended there: 44 MB a wrap on this tree.

27 of the lines are the parameter lists of a function split, which keeps about 150 lines of the attempt from being indented under a try.

The line is no safety mechanism. Every wrap stays self-contained, so a fault in it can hang a wrap or cost memory, not lose a deny.

What it costs

  • A wrap with nothing to walk now waits behind the walkers on Linux, where it used to return at once.
  • A holder that never resumes blocks every later wrap in the process. That takes a setImmediate that never fires, fake timers for instance.
  • Bun at 10 wraps: all ready about 5 to 10% later.
  • getFsReadConfig() is synchronous and cannot wait, so during such a call two sets of listings can be alive. Two, not N.

Testing

… the scan

Wraps walk in turns, so several interleave on the one thread, and each keeps
all it has listed until it ends: peak memory, and the time the event loop is
held, grew with the wraps in flight. Parallel walks gain nothing there.

They now take their walks first come, first served. An attempt waits before
it reads anything, so it goes by what it finds afterwards. The place is given
up on every way out, and a waiter whose signal is aborted leaves at once.

@antdres antdres left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers from me at 135aaaa.

I ran 711 wraps over a real tree with random denyRead/allowRead/allowWrite/denyWrite sets (300 one at a time, then 30 Promise.all batches of 5 to 20 with different configs) on the merge-base and on this branch: the bwrap command lines are identical once the random socket/port/temp names are masked. The line never got stuck: abort before start, abort while waiting, abort while walking, holder and waiter aborted in the same tick, a wrap that throws, reset() mid-queue and 20 mixed wraps with random aborts all left the next wrap able to finish. A wrap that waited while updateConfig added a deny glob and a new file appeared came out with both. The 8 new tests fail with the old sandbox-manager.ts. Peak RSS for 15 concurrent walks of a 56k-entry tree went from about 270MB to about 113MB, wall time about the same (1.3s vs 1.4s).

One thing to be aware of, not blocking: a wrap that needs no walk at all now queues behind the walks ahead of it. With five walking wraps in front, a literal-only wrap took about 450ms here against about 105ms on main (alone: 20ms). On a tree where one walk takes seconds, every quick wrap behind it waits that long.


Generated by Claude Code

antdres
antdres previously approved these changes Oct 9, 2026

@antdres antdres left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 135aaaa. No blocking issues; what I ran is in my comment above.


Generated by Claude Code

One conflict, in the imports of test/sandbox/glob-expand.test.ts: both kept.
@dylan-conway
dylan-conway merged commit 78413b1 into main Oct 10, 2026
23 of 24 checks passed
@dylan-conway
dylan-conway deleted the perf/linux-one-walk-at-a-time branch October 10, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants