Skip to content

perf(builder): watch bounded source-tree roots instead of one watcher per file - #115

Closed
arifsisman wants to merge 1 commit into
native-federation:mainfrom
arifsisman:fix/bounded-federation-watch-roots
Closed

perf(builder): watch bounded source-tree roots instead of one watcher per file#115
arifsisman wants to merge 1 commit into
native-federation:mainfrom
arifsisman:fix/bounded-federation-watch-roots

Conversation

@arifsisman

Copy link
Copy Markdown
Contributor

Follow-up to #96.

Problem

The widened watch list from #96 registers one fs watcher per tracked federation source — ~1,900 watchers on the same real Nx workspace the original fix was measured on. Per-file watchers don't scale:

  • macOS FSEvents replays and throttles under thousands of streams (this is the same OS behaviour family that motivated the stale-event filter in fix(builder): watch the files the federation build actually tracked #96 — fewer streams attacks it at the root).
  • Every platform pays a file descriptor per watcher.
  • Each syncNfFileWatcher pass re-diffs the full file list.

Change

federationWatchPaths(files, workspaceRoot) collapses the tracked files into the few top-level workspace source trees that contain them (libs, apps, ...), and both builders watch those roots recursively instead. On the measured workspace: files=~1900 collapses to roots=2 (logged at verbose level).

Because a directory watch surfaces every event under its root, relevance moves to where events arrive:

  • build builder: an event only reaches the dirty buffer if the path is a tracked federation source or under a linked dir, and its mtime actually advanced (the fix(builder): watch the files the federation build actually tracked #96 stale-event filter). Irrelevant events return immediately.
  • remote builder: the same predicate (project root ∪ linked dirs ∪ tracked sources, then mtime) is passed into createDebouncedChangeWatcher, so irrelevant events never enter pendingPaths.

Safety rails: files outside the workspace, at the workspace root itself, or under dot dirs / node_modules / build outputs stay as single-file watches; nested candidates deduplicate to their parent tree.

Verification

  • Five new unit tests for federationWatchPaths (collapse, out-of-workspace, excluded top-levels, root-level file, parent/child dedup); typecheck, lint (0 errors, warning count unchanged) and vitest (16 files / 116 tests) pass.
  • Validated in daily use on the same Nx + Angular 22 workspace as fix(builder): watch the files the federation build actually tracked #96 (applied via patch-package): shared-mapping edits still rebuild within ~1s, idle dev server stays quiet, watcher count flat as the workspace grows.

… per file

Follow-up to native-federation#96. The widened watch list registered one fs watcher per
tracked federation source — ~1,900 watchers on a real Nx workspace.
Per-file watchers do not scale: macOS FSEvents replays and throttles
under that many streams, and every platform pays a file descriptor per
watcher.

federationWatchPaths() collapses the tracked files into the few
top-level workspace source trees that contain them (libs, apps, ...)
and both builders watch those roots recursively instead. Because a
directory watch surfaces every event under the root, relevance is now
filtered where events arrive: only tracked federation sources,
project-root files and linked-dir edits pass, and only when their
mtime actually advanced (the native-federation#96 stale-event filter). Irrelevant
events no longer reach the dirty buffer at all.

Files outside the workspace, at the workspace root, or under dot dirs,
node_modules and build outputs stay as single-file watches; nested
candidates deduplicate to their parent tree. Covered by five new unit
tests; validated on the same Nx + Angular 22 workspace as native-federation#96
(files=~1900 collapses to roots=2).
@arifsisman

Copy link
Copy Markdown
Contributor Author

Note on the red verify check: the failing step is pnpm audit, and it is not caused by this PR — this branch touches four source files and no dependency or lockfile.

Two advisories (brace-expansion GHSA-rgw5-rvv9-x895 and fast-uri GHSA-7p8r-x3mc-p8w7) were published after main's last green run, so main and every open PR fail the same step right now. Reproduced on a clean main checkout with no changes applied:

2 vulnerabilities found
Severity: 2 high

#116 fixes it via the existing overrides block (one patch bump each, no direct dependency changes). Once that lands, re-running this check should go green.

@Aukevanoost

Copy link
Copy Markdown
Contributor

Might be superseded by #112

@Aukevanoost

Copy link
Copy Markdown
Contributor

I'd love to hear your feedback on how it performs: https://github.com/native-federation/angular-adapter/releases/tag/v22.1.0-RC1

@arifsisman

Copy link
Copy Markdown
Contributor Author

Measured RC1 against this PR on the workspace this PR was written for. Short version: you're right, #112 supersedes it — the per-file watcher problem is solved in core, and I'm happy to close this.

Setup

Nx + Angular 22 workspace, one host shell and three remotes, all four dev servers running concurrently (macOS). Two installs compared:

Counts come from instrumenting addPaths in core's file-watcher.js to log watchers.size / fileDirWatchers.size after each call, so they are actual handle counts rather than inferred ones.

Watcher handles

Dev server tracked files A: this PR B: RC1 22.0.6 unpatched*
host shell 248 3 58 (20 dir + 38 file-dir) 249
remote 1 244 3 64 (17 + 47) 245
remote 2 331 3 92 (17 + 75) 332
remote 3 286 3 85 (13 + 72) 287
total 1109 12 299 1113

* not measured, it is the per-file count each server would register on plain 22.0.6.

The first group on RC1 (13–20 per server) is sharedMappingDirs + the tsconfig dir, watched recursively — new in #112. The second is the parent directories of tracked files. Their parent dirs number 383 in total, but RC1 stops at 299 because covers() skips a file whose directory already sits under one of the recursive shared-mapping roots.

Behaviour on RC1

  • Touching a shared lib source tracked by all four servers rebuilt all four. touch → bundle generation complete: 0.37s / 0.42s / 0.44s / 0.47s.
  • 60s idle with all four running and overlapping recursive watches on the same source tree: zero extra rebuilds. The mtime+size+grace dedupe holds where the mtime-only filter this PR carries would have been the weaker of the two.
  • Watch resync is idempotent — handle counts unchanged on the second sync.

Conclusion

1113 → 299 is the number that mattered, and core 4.4.0 delivers it without the adapter needing a relevance filter or a stale-event filter — both of which this PR carries and #112 correctly deletes. What this PR would add on top is 299 → 12, and on this machine that is noise: ulimit -n is 1048576 and kern.maxfiles 245760, so 299 handles is nowhere near a constraint. Rebuild latency and idle quiet are already fine on RC1.

Rebasing it would mean re-applying only federationWatchPaths onto the new watch-decisions structure for a benefit I can't measure, so closing seems right. If someone hits a real ceiling on a workspace an order of magnitude larger, top-level-root collapse is easy to bring back as a small standalone change against core 4.4.0.

One install note, unrelated to the watcher work: RC1 declares @angular/build ~22.1.0 and this workspace is on 22.0.8. Installing with --force (not --legacy-peer-deps, which prunes auto-installed peers) worked and all four apps built and served without errors.

@arifsisman arifsisman closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants