Skip to content

perf(xds): replace once-cache's single mutex with sync.Map - #18234

Open
sujanchalla0510 wants to merge 1 commit into
kumahq:masterfrom
sujanchalla0510:perf/once-cache-sync-map
Open

sujanchalla0510 wants to merge 1 commit into
kumahq:masterfrom
sujanchalla0510:perf/once-cache-sync-map

Conversation

@sujanchalla0510

Copy link
Copy Markdown
Contributor

Motivation

Fixes #16188. omap (backing Cache.GetOrRetrieve's miss path in
pkg/xds/cache/once) took one sync.Mutex around map lookup, *once
allocation, and insertion. That mutex is shared across every key a
Cache instance manages — every mesh for the mesh-context cache, every
CLA cache entry, etc — so it serializes concurrent misses for entirely
unrelated keys, not just repeated misses of the same key.

Under load (hundreds of concurrent DataplaneWatchdog goroutines, one
per mesh, all hitting this path together) that turned map access into a
significant bottleneck, per the issue's profiling evidence: goroutine
profiles showed hundreds parked on the lock, with
DataplaneWatchdog.syncDataplane's cumulative time dominated by
serialization rather than actual work.

Implementation information

sync.Map is built for exactly this shape — a small, mostly-stable set
of keys, read far more often than written — and lets lookups for
different keys proceed independently instead of contending on one lock.
LoadOrStore replaces the check-then-insert critical section with a
single atomic operation, preserving the "insert-if-absent, otherwise
return the existing entry" semantics Cache.GetOrRetrieve relies on. No
caller-visible API change — omap.Get/Delete keep the same
signatures.

Added BenchmarkOmapGetConcurrentKeys, simulating many goroutines
hitting a small set of distinct keys concurrently. Measured on 8 CPUs:

old (single mutex): 94.48 ns/op
new (sync.Map):      22.78 ns/op   (~4.1x)

Existing once package tests — including the 100-goroutine "should
cache concurrent Get() requests" case, which exercises the
exactly-once retrieval guarantee this change must preserve — pass
unmodified under -race.

Out of scope: the issue also flags a structurally similar single-mutex
pattern in cachedManager.mapMutex
(pkg/core/resources/manager/cache.go:143-152). That's a separate data
structure in a different subsystem; leaving it for a follow-up to keep
this PR focused on one change.

Supporting documentation

Fix #16188

omap (backing Cache.GetOrRetrieve's miss path in pkg/xds/cache/once) took
one sync.Mutex around map lookup, *once allocation, and insertion. That
mutex is shared across every key a Cache instance manages - every mesh
for the mesh-context cache, every CLA cache entry, etc - so it serializes
concurrent misses for entirely unrelated keys, not just repeated misses
of the same key.

Under load (hundreds of concurrent DataplaneWatchdog goroutines, one per
mesh, all hitting this path together) that turned map access into a
significant bottleneck: goroutine profiles showed hundreds parked on the
lock, with DataplaneWatchdog.syncDataplane's cumulative time dominated by
serialization rather than actual work.

sync.Map is built for exactly this shape - a small, mostly-stable set of
keys, read far more often than written - and lets lookups for different
keys proceed independently instead of contending on one lock.
LoadOrStore replaces the check-then-insert critical section with a
single atomic operation, preserving the "insert-if-absent, otherwise
return the existing entry" semantics the callers rely on.

Added BenchmarkOmapGetConcurrentKeys, simulating many goroutines hitting
a small set of distinct keys concurrently. Measured on 8 CPUs:
  old (single mutex): 94.48 ns/op
  new (sync.Map):      22.78 ns/op   (~4.1x)

Existing once package tests (including the 100-goroutine
"should cache concurrent Get() requests" case, which exercises the
exactly-once retrieval guarantee this change must preserve) pass
unmodified under -race.

Fixes kumahq#16188

Signed-off-by: sujan reddy <sujanchalla0510@gmail.com>
@sujanchalla0510
sujanchalla0510 requested a review from a team as a code owner August 28, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(xds): mesh-context once.Cache allocates inside a single mutex

1 participant