Today clients load balance across ate-apiserver replicas using a headless k8s service. K8s creates A records per ready pods. Clients dial with dns:/// and use round_robin load-balancing policy.
When a Pod goes away gRPC re-resolves DNS, but when a new pod is added nothing triggers a re-resolve. Our current mitigation forces every connection to close after a given period of time so clients are forced to re-resolve. It works, but feels pretty hacky.
A probably better approach is to build a k8s-aware gRPC resolver (ala github.com/sercand/kuberesolver) that watches EndpointSlices for a given Service and handles Pod updates.
Today clients load balance across ate-apiserver replicas using a headless k8s service. K8s creates A records per ready pods. Clients dial with
dns:///and useround_robinload-balancing policy.When a Pod goes away gRPC re-resolves DNS, but when a new pod is added nothing triggers a re-resolve. Our current mitigation forces every connection to close after a given period of time so clients are forced to re-resolve. It works, but feels pretty hacky.
A probably better approach is to build a k8s-aware gRPC resolver (ala github.com/sercand/kuberesolver) that watches
EndpointSlicesfor a givenServiceand handlesPodupdates.