Skip to content

perf(supervisor): memoize GetMeta within one reconcile tick (sys-yre7dj) - #6

Merged
atbrace merged 1 commit into
mainfrom
fix/sys-yre7dj-getmeta-tick-cache
Sep 4, 2026
Merged

perf(supervisor): memoize GetMeta within one reconcile tick (sys-yre7dj)#6
atbrace merged 1 commit into
mainfrom
fix/sys-yre7dj-getmeta-tick-cache

Conversation

@atbrace

@atbrace atbrace commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Summary

beadReconcileTick reaches runtime.Provider.GetMeta from seven independent
call sites per session per tick (worker-handle resolution, drain-ack check,
restart-request check, pending-create-info matching, drain-ack-queue dedup,
stale-drain-ack detection), each forking an uncached tmux show-environment -t <session> <key>. A sample capture on the live 9.5-day supervisor (PID 2160)
showed GetEnvironment as the dominant leaf (365 weighted samples), scaling
with session count — the fork cost is dominated by os/exec's env
construction for the child (60+ vars in a gc session env).

Root cause

cmd/gc/city_runtime.go's beadReconcileTickreconcileSessionBeadsTracedWithNamedDemand
fans out to seven call sites that each independently call sp.GetMeta(name, key)
for the same session within one tick, with no memoization:
workerHandleForSessionTargetWithRuntimeHintsWithConfig, providerDrainOps.isDrainAcked,
providerDrainOps.isRestartRequested, runningSessionMatchesPendingCreateInfo,
queueDrainAckAsyncStop, staleReconcilerDrainAckInfo, reconcilerDrainAckMatchesSessionInfo.
Every one of them terminates at tmux.Provider.GetMeta (internal/runtime/tmux/adapter.go),
regardless of the seam-wrapping layers above it.

Fix

Add a tick-scoped memo to tmux.Provider: ResetTickCache() starts a window in
which GetMeta fetches a session's whole environment once (GetAllEnvironment)
and serves every key from it; SetMeta/RemoveMeta invalidate the affected
session's entry so a same-tick write is never masked by a stale read.
CityRuntime.beadReconcileTick calls ResetTickCache() once at the top of each
tick via a local tickCacheResetter capability interface, so providers that
don't implement it are unaffected.

Deliberately NOT a Provider-wide decorator. cmd/gc type-asserts several
optional Provider capabilities (IdleWaitProvider, RelaunchProvider, and
others) on the exact sp value threaded through this same reconcile-tick call
graph (session_reconciler.go). A decorator wrapping the whole Provider
interface would silently fail those assertions, degrading idle-probe wake
detection and drift-relaunch — a correctness regression worse than the CPU
cost it fixes. Caching inside tmux.Provider's own GetMeta/SetMeta/
RemoveMeta leaves sp's concrete type and identity untouched, so every
existing type assertion elsewhere in the codebase is unaffected.

Tests

internal/runtime/tmux/tick_meta_cache_test.go:

  • memoization within a tick (one fork serves N call sites)
  • per-session isolation (no cross-session conflation)
  • a fresh window per ResetTickCache() call — the correctness requirement: a
    memo that outlived its tick would serve stale drain-ack/restart-request state
  • SetMeta/RemoveMeta invalidate the memo so a same-tick write is visible
  • unchanged, uncached behavior when ResetTickCache is never called (tests,
    one-off CLI invocations)
  • error-classification parity with the original per-key GetMeta
    (ErrSessionNotFound/ErrNoServer propagate; everything else → ("", nil))

go test ./internal/runtime/tmux/... and go vet ./cmd/gc/... ./internal/runtime/tmux/... pass.

Validation

Verified on the Nomad cluster (crun, golang:1.26, since this box cannot
compile cmd/gc/dolt's cgo deps locally without the icu4c toolchain and per
this repo's convention that cmd/gc compiles are offloaded):

  • Full internal/runtime/tmux suite passes (no regressions).
  • cmd/gc builds and go vets clean.
  • The reconciler/drain-ack/restart/wake/worker-handle test subset in cmd/gc
    passes (-run "Reconcile|DrainAck|Restart|Wake|WorkerHandle|PendingCreate").

Not yet dogfooded against the live jonesy supervisor (would require installing
a new binary onto the live control plane — out of scope for this branch;
flagged for the human-attended install/measurement step per sys-yre7dj).

sys-yre7dj

🤖 Generated with Claude Code

https://claude.ai/code/session_01R8V78VNFSCPbidXmWwJuHQ

beadReconcileTick reaches GetMeta from seven independent call sites per
session per tick (worker-handle resolution, drain-ack check,
restart-request check, pending-create-info matching, drain-ack-queue
dedup, stale-drain-ack detection), each forking an uncached `tmux
show-environment -t <session> <key>`. `sample` on the live 9.5-day
supervisor showed GetEnvironment as the dominant leaf (365 weighted
samples), scaling with session count.

Add a tick-scoped memo to tmux.Provider: ResetTickCache() (called once
per beadReconcileTick via the new local tickCacheResetter capability
interface, so providers that don't support it are unaffected) starts a
window in which GetMeta fetches a session's whole environment once and
serves every key from it; SetMeta/RemoveMeta invalidate the affected
session's entry so a same-tick write is never masked by a stale read.

Deliberately NOT a Provider-wide decorator: cmd/gc asserts several
optional Provider capabilities (IdleWaitProvider, RelaunchProvider, ...)
on the exact `sp` value threaded through the same reconcile-tick call
graph (session_reconciler.go), and a wrapping decorator would fail those
type assertions silently, degrading idle-probe wake detection and
drift-relaunch. Caching inside tmux.Provider's own GetMeta/SetMeta/
RemoveMeta leaves sp's concrete type and identity untouched, so every
existing type assertion elsewhere in the codebase is unaffected.

Tests (internal/runtime/tmux/tick_meta_cache_test.go): memoization
within a tick, per-session isolation, a fresh window per
ResetTickCache() call (the correctness requirement — a memo that
outlived its tick would serve stale drain-ack/restart-request state),
SetMeta/RemoveMeta invalidation, unchanged uncached behavior when
ResetTickCache is never called, and error-classification parity with
the original per-key GetMeta.

Verified on the Nomad cluster (crun, golang:1.26): full
internal/runtime/tmux suite passes, cmd/gc builds and vets clean, and
the reconciler/drain-ack/restart/wake/worker-handle test subset in
cmd/gc passes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R8V78VNFSCPbidXmWwJuHQ
@atbrace
atbrace force-pushed the fix/sys-yre7dj-getmeta-tick-cache branch from b87a904 to 2739aaa Compare September 4, 2026 20:57
@atbrace
atbrace merged commit b811d26 into main Sep 4, 2026
6 checks passed
@atbrace
atbrace deleted the fix/sys-yre7dj-getmeta-tick-cache branch September 4, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant