backendRef weights are ignored when ConsistentHash load balancing is configured

[CodePrometheus]

Description:
We split traffic across two backendRefs (weights 5 / 95) and attach a BackendTrafficPolicy with loadBalancer.type: ConsistentHash on a per-user header, so that a fixed slice of users consistently lands on the canary backend.

But neither the weights nor the pinning takes effect: traffic does not follow the configured 5 / 95, and the same client is served by both backends over time, so no client is pinned.

The generated config points at the reason, the two backendRefs become two weighted localities inside a single cluster, but locality weighting is never enabled on the Maglev policy, so those weights are simply ignored.

Repro steps:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: split
  namespace: default
spec:
  parentRefs:
    - name: eg
  rules:
    - backendRefs:
        - name: backend-a
          port: 3000
          weight: 5
        - name: backend-b
          port: 3000
          weight: 95
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: split-hash
  namespace: default
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: split
  loadBalancer:
    type: ConsistentHash
    consistentHash:
      type: Headers
      headers:
        - name: x-user-id

Generated cluster (abridged config_dump):

name: httproute/default/split/rule/0
lb_policy: MAGLEV
load_balancing_policy:
  policies:
    - typed_extension_config:
        name: envoy.load_balancing_policies.maglev
        typed_config: {"@type": type.googleapis.com/envoy.extensions.load_balancing_policies.maglev.v3.Maglev}   # no locality_weighted_lb_config
load_assignment:
  endpoints:
    - locality: {region: ".../rule/0/backend/0"}
      load_balancing_weight: 5
    - locality: {region: ".../rule/0/backend/1"}
      load_balancing_weight: 95

Environment:
Envoy Gateway: main.
Envoy Proxy: v1.37.1.

[CodePrometheus]

Enabling locality weighted load balancing for these clusters would fix this, I'd like to raise a PR if this direction looks right.

[CodePrometheus]

Gentle ping~ @zirain @zhaohuabing would you mind taking a look at this when you have time?

[guanchzhou]

The direction is right, and the two paths are worth separating explicitly so this
doesn't get read as a duplicate of #9626 — they're different code, and both are broken.

Why this one isn't covered by #9629. buildXdsRouteAction only emits
weighted_clusters when NeedsClusterPerSetting() is true or there's more than one
cluster (internal/xds/translator/route.go:327-348). With two plain Services, no
per-backendRef filters, no PreferLocal/WeightedZones and no invalid backends,
none of that holds — both settings collapse into the single cluster named after the
destination, and the split is expressed as two weighted localities inside it
(buildWeightedLocalities, internal/xds/translator/cluster.go:1027). That's exactly
the config_dump above. #9629 set use_hash_policy on the route-level
weighted_clusters, so it only reaches the cluster-per-setting path — its test case
forces that path with a per-backendRef addRequestHeaders filter. Your case never
gets there.

The gate. internal/xds/translator/cluster.go:462-465:

if len(args.loadBalancer.WeightedZones) > 0 {
    consistentHash.LocalityWeightedLbConfig = &commonv3.LocalityLbConfig_LocalityWeightedLbConfig{}
}

Maglev only gets locality weighting for the explicit weighted-zones feature, while
least_request and random get buildLocalityLbConfig(args) unconditionally. So
backendRef weights are honoured under the default LB and silently dropped under
ConsistentHash — the inconsistency looks unintended rather than deliberate.

Suggested shape. Drop the WeightedZones condition and always set
LocalityWeightedLbConfig on the Maglev policy. I'd keep it as the explicit literal
rather than switching to buildLocalityLbConfig(args): that helper can return
zone_aware_lb_config when PreferLocal is set, and zone-aware routing isn't
meaningful for a thread-aware hash LB. Worth confirming what Envoy does with that
combination before reusing the helper.

Pinning is preserved either way — the Maglev table is built from host weight times
locality weight, so enabling it fixes the ratio without unpinning anyone.

On the second symptom. The weight half is clear-cut. "The same client is served by
both backends over time" I'd want to pin down separately: with locality weighting off,
all endpoints from both backends still sit in one Maglev table, so a given x-user-id
should map to a fixed host. Did the reassignments line up with endpoint churn (rollouts,
scale events, readiness flaps)? Maglev remaps a share of keys whenever the host set
changes, and with one endpoint per backend in the repro, a single host coming or going
moves a lot of keys.

Tests. Worth an xds-ir case that mirrors
http-route-weighted-backend-consistent-hash with the per-backendRef filters removed,
so both the weighted-clusters and the single-cluster/locality paths are pinned down.

@zirain @zhaohuabing — happy to review the PR when it's up.

[arkodg]

is this a regression from #5833 ?
cc @jukie

[guanchzhou]

Yes — this is a regression from #5833, first shipped in v1.5.0.

Before that PR every cluster got locality weighting unconditionally, on the cluster itself:

CommonLbConfig: &clusterv3.Cluster_CommonLbConfig{
    LocalityConfigSpecifier: &clusterv3.Cluster_CommonLbConfig_LocalityWeightedLbConfig_{
        LocalityWeightedLbConfig: &clusterv3.Cluster_CommonLbConfig_LocalityWeightedLbConfig{},
    },
},

That applied to lb_policy: MAGLEV too, so the weighted localities built by
buildWeightedLocalities were honoured under ConsistentHash.

#5833 moved locality config out of common_lb_config and into the typed
load_balancing_policy extensions. least_request, round_robin and random each
got LocalityLbConfig: localityLbConfig. The maglev branch got the Maglev message
with only TableSize — no equivalent. It's visible in the PR's own golden output,
load-balancer.clusters.yaml, fourth-route-dest:

-  commonLbConfig:
-    localityWeightedLbConfig: {}
+  commonLbConfig: {}
   lbPolicy: MAGLEV
+  loadBalancingPolicy:
+    policies:
+    - typedExtensionConfig:
+        name: envoy.load_balancing_policies.maglev
+        typedConfig:
+          '@type': type.googleapis.com/envoy.extensions.load_balancing_policies.maglev.v3.Maglev

Locality weighting removed, nothing put back. lb_policy: MAGLEV is still set but is
ignored once load_balancing_policy is present, so the old field doesn't cover for it.

The field exists on the extension — Maglev.locality_weighted_lb_config, "Enable
locality weighted load balancing for maglev lb explicitly" — it was just never set.
#7251 later added it back, but only behind len(WeightedZones) > 0
(internal/xds/translator/cluster.go:462-465), so plain backendRef weights are still
dropped on main.

So: affected v1.5.0 → main, and orthogonal to #9629 (merged) — that one fixes
route-level weighted_clusters, which this repro never reaches, for the reasons in my
earlier comment.

@CodePrometheus your proposed direction is right — dropping the WeightedZones
condition restores the pre-#5833 behaviour exactly. Still worth keeping the explicit
LocalityWeightedLbConfig literal rather than reusing buildLocalityLbConfig(args),
since maglev's field is specifically the locality-weighted one and that helper can
return zone_aware_lb_config.