diff --git a/README.md b/README.md index edc1846..c50af17 100644 --- a/README.md +++ b/README.md @@ -345,8 +345,8 @@ that changed: at every attach (`up`/`claude`/`shell --macos`), and again every * the VM runs, which is the only thing that makes a host edit reach an agent that is *already running*. See [`docs/decisions/0016-shared-file-cache-refresh.md`](docs/decisions/0016-shared-file-cache-refresh.md). -The cost scales with the number of **changed** files (~0.36 ms each, host + guest, paid serially), and -nothing caps that number — past roughly 14,000 changed files one sweep outlasts the 5 s interval and +The cost scales with the number of **changed** files (~0.29 ms each, host + guest, paid serially), and +nothing caps that number — past roughly 17,000 changed files one sweep outlasts the 5 s interval and the loop starts running most of the time on one core. An `npm install`, a big `git checkout` or a full build can get there. Two run-scoped flags, written after the command: diff --git a/augur b/augur index 42dc7e0..c0426b7 100644 --- a/augur +++ b/augur @@ -1026,7 +1026,7 @@ require_safe_workspace() { # # One file per project, for the settings whose right value is a property of THE REPO rather than of # the run — today the two macOS share-refresh knobs, whose cost scales with the project's -# changed-file count (~0.36 ms each, so past roughly 14,000 changed files one sweep outlasts the +# changed-file count (~0.29 ms each, so past roughly 17,000 changed files one sweep outlasts the # default 5 s interval; see _MACOS_REFRESH_MODE). Without this, an operator whose repo is big enough # to need `--share-refresh attach` has to type it on every command, forever. # @@ -3581,14 +3581,21 @@ _MACOS_SWEEP_NOWORK=4 # nothing had changed since the last sweep, so the gues # WHY THERE IS A DIAL AT ALL. The "no cap" paragraph above is right about the SWEEP and wrong about # the whole. It measures the guest-side invalidation (0.23 ms/file) and the host-side `find` (0.31 s # for 12,626 files) and concludes "bounded at seconds" — but it never measured the host-side loop -# that ACCUMULATES the list (see _refresh_macos_shares_locked, which appends once per changed file -# and reopens the list file every iteration). The 2026-07-28 security snapshot did: ~130 µs per -# changed file, i.e. ~99% of the host-side cost. Added to the 0.23 ms the guest pays for the same -# file, and both are paid serially, one sweep costs ~0.36 ms per changed file end to end. Past -# roughly 14,000 changed files a single sweep therefore exceeds the default 5 s interval, and the -# loop — which sleeps THEN sweeps — degrades into a rising duty cycle on one host core for the -# lifetime of the VM. 14,000 needs no adversary: `npm install`, a large `git checkout` or a full -# build reach it, and augur's own repo is 12,626 files. +# that ACCUMULATES the list. The 2026-07-28 security snapshot did: ~130 µs per changed file, i.e. +# ~99% of the host-side cost — because that loop reopened the list file once per changed file. IT NO +# LONGER DOES: the redirect now sits on the loop rather than on the printf (see +# _refresh_macos_shares_locked), and the host-side term re-measured through that same function is +# 64.7 µs per changed file, down from 122.0. Added to the 0.23 ms the guest pays for the same file, +# both still paid serially, one sweep now costs ~0.29 ms per changed file end to end, down from +# ~0.35 ms. Past roughly 17,000 changed files a single sweep therefore exceeds the default 5 s +# interval — it was ~14,200 before that fix — and the loop, which sleeps THEN sweeps, degrades into +# a rising duty cycle on one host core for the lifetime of the VM. +# +# THE FIX DID NOT RETIRE THIS DIAL and the numbers are quoted above so that stays checkable: the +# guest's 0.23 ms per file is untouched by anything host-side and is now ~78% of the total, so +# halving the host half bought ~20% on the crossing point and nothing more. 17,000 still needs no +# adversary: `npm install`, a large `git checkout` or a full build reach it, and augur's own repo is +# 12,626 files. # # THREE MECHANISMS, and an operator turning the cost down has to be able to say WHICH: # @@ -3851,11 +3858,32 @@ _refresh_macos_shares_locked() { roots=$((roots+1)) [[ -d "$root" ]] || continue scanned=$((scanned+1)) + # THE REDIRECT IS ON THE LOOP, NOT ON THE `printf`, so `$list` is opened once per ROOT + # instead of once per CHANGED FILE. That reopen was most of the host-side cost and it is what + # ADR-0016 §Cost's "~130 us per changed file" measured. Re-measured through this very function + # (ssh_macos stubbed, so it is the host half only), 12,628 files across four roots, bash 3.2: + # 122.0 -> 64.7 us per changed file, 1541 -> 817 ms for a full blind sweep of a repo this size. + # + # It does NOT retire the dial, and saying so here is the point: the guest's own 0.23 ms per + # file is untouched by this and is ~78% of what is left, so the crossing point where one sweep + # outlasts the 5 s interval moves only ~14,200 -> ~17,000 changed files. ADR-0016 §Cost carries + # the arithmetic and the caveat that this reading was taken in a guest, not on a real host. + # + # NOT on the OUTER loop, and that is not a style choice. `warn` writes to STDOUT (augur:123), + # so a redirect spanning the outer body would put any warning added there INTO the NUL path + # list, and the guest would be handed a path built from a sentence — the exact trap + # macos_share_roots documents at its own `error`-not-`warn` line. This body is one + # assignment, one printf and one increment, and can be checked by eye to emit nothing else. + # + # Still no subshell: a redirection on a compound command does not fork, so `n`, `roots` and + # `scanned` keep accumulating in the CURRENT shell, which is what the comment above relies on. + # Byte-equivalence with the per-printf form was checked against paths containing spaces, tabs, + # embedded newlines, quotes, non-ASCII and a leading dash: identical bytes, identical NUL count. while IFS= read -r -d '' f; do rel="${f#"$root"/}" - printf '/Volumes/My Shared Files/%s/%s\0' "$name" "$rel" >> "$list" + printf '/Volumes/My Shared Files/%s/%s\0' "$name" "$rel" n=$((n+1)) - done < <(find "$root" -type f ${find_age[@]+"${find_age[@]}"} -print0 2>/dev/null) + done < <(find "$root" -type f ${find_age[@]+"${find_age[@]}"} -print0 2>/dev/null) >> "$list" done < <(macos_share_roots "$vm") # `n == 0` used to mean two different things, and only one of them is good news: @@ -6631,7 +6659,7 @@ cmd_help() { echo -e " • macOS shared-file refresh: a macOS guest's virtiofs client serves ${BOLD}stale file data${RESET} after a" echo -e " host edit, with no timeout (issues #124/#135; still present on macOS 26.6). augur invalidates" echo -e " the changed files at each attach and every ${CYAN}5${RESET}s while the VM runs. Cost scales with the" - echo -e " CHANGED-file count (~0.36 ms each, host + guest), so a repo where one sweep outlasts the" + echo -e " CHANGED-file count (~0.29 ms each, host + guest), so a repo where one sweep outlasts the" echo -e " interval should use ${CYAN}--share-refresh attach${RESET} (sweep on attach, no loop) rather than a longer" echo -e " interval. ${CYAN}--share-refresh off${RESET} disables the mitigation entirely. See ADR-0016." echo -e " • ${BOLD}augur refresh --macos${RESET} sweeps once, on demand, against the VM that is already running —" diff --git a/docs/decisions/0016-shared-file-cache-refresh.md b/docs/decisions/0016-shared-file-cache-refresh.md index 927de7c..9bb1e70 100644 --- a/docs/decisions/0016-shared-file-cache-refresh.md +++ b/docs/decisions/0016-shared-file-cache-refresh.md @@ -97,7 +97,7 @@ implementation shipped the bug until a test caught it. | | Measured | |---|---| | Host-side detection | 0.31 s (`find -newer`, 12,626 files) | -| Host-side list accumulation | ~130 µs per **changed** file (2026-07-28 snapshot, item 37) | +| Host-side list accumulation | ~130 µs per **changed** file (2026-07-28 snapshot, item 37) — **superseded 2026-08-05, see the amendment at the end of this section** | | Guest-side invalidation | 0.23 ms per file | | Full blind sweep, all five shares | 2.87 s for 12,600 files, zero failures | | Realistic incremental sweep | ~10 ms for a few dozen files | @@ -106,7 +106,10 @@ There is deliberately **no cap** on the changed set. A legitimate one can be lar a branch switch), and silently truncating the list would be the same class of failure the surrounding work exists to remove. -The second row was **added after the fact** and it corrects this section. The original four rows did +The second row was **added after the fact** and it corrects this section. *(This paragraph and the one +after it are kept as written; the reopen they describe was removed on 2026-08-05 and their figures are +superseded by the amendment at the end of this section. They are the reasoning the amendment reversed, +and are the reason the dial exists at all.)* The original four rows did not measure the shell loop that builds the path list, which appends once per changed file and reopens the list file each time — ~99 % of the host-side cost. Adding it to the guest's 0.23 ms, both paid serially, one sweep costs **~0.36 ms per changed file end to end**, so past roughly **14,000** changed @@ -127,6 +130,39 @@ figure of 30–40k is a third thing again: it is the host-side term *alone*, wit excluded. Nothing here rests on the exact crossing; what it rests on is that a crossing exists at a count no adversary is needed to reach, which all three readings agree on. +**Amendment (2026-08-05): the accumulation row was a defect, not a floor.** The loop that builds the +path list held its `>> "$list"` redirect on the `printf`, so it reopened the list file once per +changed file. Moving the redirect onto the enclosing `while … done` — one open per **root** instead +of one per **file** — is the entire change. Re-measured through `_refresh_macos_shares_locked` itself +with `ssh_macos` stubbed, so the reading is the host half alone; 12,628 files across four roots, +bash 3.2, best of three runs: + +| | before | after | +|---|---|---| +| Host-side term (`find` + accumulation) | 122.0 µs per changed file | **64.7 µs** | +| Full blind sweep of a 12,628-file tree | 1541 ms | **817 ms** | +| End to end, with the guest's 0.23 ms | ~0.35 ms per changed file | **~0.29 ms** | +| Crossing point against the 5 s interval | ~14,200 changed files | **~17,000** | + +**It does not retire the dial**, and that is the part worth keeping in view: the guest's 0.23 ms is +untouched by any host-side change and is now ~78% of the total, so halving the host half bought ~20% +on the crossing point and nothing more. §4's `--share-refresh` argument stands unchanged. It also +does not help the common case — a realistic incremental sweep is a few dozen files, where the saving +is well under a millisecond. What it helps is the large sweep: a branch switch, a build, and the +first blind sweep on a fresh marker. + +The redirect sits on the **inner** loop, not the outer one, and that is load-bearing rather than +stylistic: `warn` writes to **stdout**, so a redirect spanning the outer body would put any warning +added there into the NUL path list and hand the guest a path built from a sentence. That is the same +trap `macos_share_roots` documents at its own `error`-not-`warn` line. Byte-equivalence with the +per-`printf` form was checked against paths containing spaces, tabs, embedded newlines, quotes, +non-ASCII and a leading dash — identical bytes, identical NUL count. + +**Measured in a Virtualization.framework guest (`hw.model VirtualMac2,1`) on its local APFS, not on a +real host.** That is the one caveat on the absolute figures; the *ratio* is a property of the shell +loop and does not depend on it. The 122.0 µs "before" reading corroborates the snapshot's ~130 µs +from a different machine and a different method, which is what makes the comparison usable at all. + ## 3. Placement, and why the order is load-bearing The refresh runs **before** the `ensure_macos_*` wiring, not after. `ensure_macos_claude_profile` diff --git a/tests/41_macos_share_refresh.sh b/tests/41_macos_share_refresh.sh index 7516328..7f6fca6 100755 --- a/tests/41_macos_share_refresh.sh +++ b/tests/41_macos_share_refresh.sh @@ -903,4 +903,35 @@ else fail "…without printing the whole help" "say what to type; the command li if grep -q 'augur ${CYAN}refresh${RESET} --macos' "$AUGUR"; then ok "…and \`--help\` lists it among the macOS commands" else fail "…and \`--help\` lists it" "a command nobody can discover does not close the gap it was written for"; fi +section "the path list is opened once per ROOT, not once per changed FILE" + +# WHY THIS IS PINNED STRUCTURALLY, when nothing else in this file is pinned on cost. A revert is +# INVISIBLE to every other arm here: the output is byte-for-byte identical (checked against paths +# containing spaces, tabs, embedded newlines, quotes, non-ASCII and a leading dash — same bytes, same +# NUL count), so the whole suite stays green while the sweep silently costs twice what it should. +# What a revert costs, re-measured through _refresh_macos_shares_locked itself with ssh_macos stubbed: +# 122.0 us per changed file against 64.7, i.e. 1541 ms against 817 ms for a full blind sweep of a +# 12,628-file tree. ADR-0016 §Cost carries the amendment and the caveat on the absolute numbers. +read -r _rs _re <<<"$(fn_range _refresh_macos_shares_locked)" +_pf="$(first_in "$_rs" "$_re" "printf '/Volumes/My Shared Files/%s/%s")" +_inner="$(first_in "$_rs" "$_re" 'done < <(find "$root" -type f')" +_outer="$(first_in "$_rs" "$_re" 'done < <(macos_share_roots "$vm")')" +if [[ -n "$_pf" && -n "$_inner" && -n "$_outer" ]]; then + ok "the accumulation loop is locatable (printf $_pf, inner done $_inner, outer done $_outer)" +else fail "the accumulation loop is locatable" "printf='$_pf' inner='$_inner' outer='$_outer' — the three arms below would assert nothing"; fi + +if ! sed -n "${_pf}p" "$AUGUR" | grep -qF '>>'; then ok "the \`printf\` carries no redirect of its own" +else fail "the \`printf\` carries no redirect of its own" "it reopens \$list once per CHANGED FILE, which is ~2x the host-side cost of every sweep (ADR-0016 §Cost)"; fi + +if sed -n "${_inner}p" "$AUGUR" | grep -qF '>> "$list"'; then ok "…because the redirect is on the inner loop, so \$list opens once per root" +else fail "…because the redirect is on the inner loop" "$(sed -n "${_inner}p" "$AUGUR")"; fi + +# THE NEGATIVE CONTROL, and it guards something worse than cost. `warn` writes to STDOUT (augur:123), +# so hoisting the redirect one level further — onto the loop over the SHARE ROOTS — would put any +# warning added to that body INTO the NUL path list, and the guest would be handed a path built from +# a sentence. macos_share_roots documents the same trap at its own `error`-not-`warn` line. One open +# per root is already almost all of the win, so the remaining three opens do not buy that risk. +if ! sed -n "${_outer}p" "$AUGUR" | grep -qF '>>'; then ok "…and NOT on the outer loop, where a \`warn\` would land in the NUL list" +else fail "…and NOT on the outer loop" "warn writes to stdout, so a warning in that body becomes a path the guest is told to msync"; fi + finish diff --git a/tests/43_macos_share_refresher.sh b/tests/43_macos_share_refresher.sh index b89ddc0..26cbfde 100755 --- a/tests/43_macos_share_refresher.sh +++ b/tests/43_macos_share_refresher.sh @@ -155,8 +155,8 @@ eval "$_saved_refresh" section "the mode gate on the loop — --share-refresh attach|off" # The loop is the UNATTENDED, repeated cost: it sweeps for the whole lifetime of the VM with nobody -# watching, and its price scales with a changed-file count nobody caps (~0.36 ms per changed file, -# host + guest, both paid serially — past ~14,000 one sweep outlasts the default 5 s interval). So it +# watching, and its price scales with a changed-file count nobody caps (~0.29 ms per changed file, +# host + guest, both paid serially — past ~17,000 one sweep outlasts the default 5 s interval). So it # is the mechanism an operator turning the refresh down is actually paying for, and both non-default # modes stop it. The attach-time sweep is tests/41's subject, not this file's. _saved_mode="$_MACOS_REFRESH_MODE"