[Bug]: xDS snapshot race in updateSnapshotAsync causes routes to randomly go missing after startup

[tricktron]

Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/Other (Anything else which does not match above categories)

Description

updateSnapshotAsync spawns a go func() per API event. When multiple events arrive in the same poll cycle, concurrent goroutines call UpdateSnapshot without mutual exclusion. A goroutine that reads the store before all APIs are added can overwrite a correct snapshot with a stale one (higher version, fewer routes). Last writer wins.

xds.SnapshotManager is the only one of five snapshot managers without a sync.RWMutex. The other four (policyxds, subscriptionxds, lazyresourcexds, apikeyxds) are all protected.

Reliably reproducible in immutable mode (tight artifact loading loop batches all events into one poll cycle). Mutable mode is affected too if 2+ events land in the same 3s poll window.

Steps to Reproduce

1. Deploy an immutable-mode gateway with 2+ REST API artifacts
2. Wait for pod startup (or trigger a rolling restart via ConfigMap change)
3. Check each pod's Envoy route table:

kubectl exec <pod> -c gateway-runtime -- curl -s localhost:9901/config_dump \
  | jq '.configs[] | select(."@type" | contains("RoutesConfigDump"))
        | .dynamic_route_configs[].route_config.virtual_hosts[].routes[].name'

4. If one pod has fewer routes than the other, the race hit

With 2 APIs, we observed one of two pods hitting the race on a single restart. More APIs widen the stale-read window.

Severity Level of the Issue

Severity/Critical (Core functionality is broken but there is a workaround. Need urgent attention)

Environment Details (with versions)

• gateway-controller / runtime 1.1.0
• Immutable mode with 2 REST API artifacts
• Kubernetes with 2 runtime replicas and 1 controller replica
• Also affects mutable mode if 2+ API events land in the same 3s poll window (unlikely but possible)