c-ares 1.34.8 QID reuse can permanently stall Envoy AUTO DNS refresh for a cluster

[fvallenilla]

Title: c-ares 1.34.8 QID reuse can permanently stall Envoy AUTO DNS refresh for a cluster

Description:
We are observing cases where STRICT_DNS clusters will stop resolving new addresses and effectively become stuck with endpoints that become increasingly stale.

Leading up to this issue in production we see healthy proxyd.cluster.update_attempt and proxyd.cluster.update_success metrics, then they collapse to 0 and stay there. Once the cluster update metrics fall to 0, dns.cares.pending_resolutions gets pinned with a min value of 1.

Based on these symptoms it appears to be related to the issue in the following open PR: c-ares #1256.

sequenceDiagram
    participant E as Envoy AUTO resolver
    participant V6 as Completing AAAA query
    participant Q as queries_by_qid
    participant V4 as Reentrant A query

    E->>V6: Resolve AAAA for IPv4-only hostname
    V6->>Q: QID 42 points to AAAA query
    V6-->>E: Return NODATA in callback
    E->>V4: Start A fallback inline
    V4->>Q: Randomly select the same QID 42 for A query
    E-->>V6: Callback returns
    V6->>Q: Remove QID 42 unconditionally
    Note over V4,Q: A query loses its table entry
    Note over E: Refresh loop stalls

Loading

Repro steps:
We can observe this behavior in a minimal repro:

1. Run one STRICT_DNS cluster with dns_lookup_family: AUTO, dns_refresh_rate: 0.002s, dns_jitter: 0s, respect_dns_ttl: false, qcache_max_ttl: 0, and filter_unroutable_families: false.
2. Point its typed c-ares resolver at a local UDP server for an absolute hostname. Return AAAA NOERROR/NODATA, then A 127.0.0.1.
3. Observe each AAAA→A QID pair. When natural reuse occurs, return a new 127.0.0.2 address for that A response.
4. Confirm the stuck-state counters and stale membership that does not include 127.0.0.2.

Admin and Stats Output:

clusters.txt
stats.txt

Config:

server-info-excerpt.txt
envoy.yaml

Logs:

dns-observer.txt
envoy-log-excerpt.txt

Call Stack:
N/A

[mathetake]

cc @phlax @yanavlasov @yanjunxiang-google - this is the mentioned issue in the slack. Any objection against applying the patch (c-ares/c-ares#1256) until the upstream takes it? We have tested it internally and it works

[phlax]

patch looks good - but was hoping someone with c-ares-fu would sign it off

[mathetake]

but was hoping someone with c-ares-fu would sign it off

Mind pinging them in the PR? #46880