perf(shards): stream /_cat/shards decoding to cut peak memory

[pincher95]

Summary

Streams /_cat/shards decoding one shard at a time so the Shards collector's retained heap is proportional to the cluster's node count instead of the full shard list. Same approach #1159 applied to /_all/_stats.

Motivation

Shards.Collect decoded the entire /_cat/shards?format=json response into a []ShardResponse and then aggregated STARTED shards per node. On a cluster with many thousands of shards that slice (a struct with four retained strings per shard) stays live for the whole aggregation, and in multi-target /probe mode it is allocated per concurrent request. Only the small per-node count map is actually needed.

Change

streamShards decodes the JSON array element-by-element via encoding/json's token API; streamAndAggregateShards feeds each shard straight into the per-node map through the shared fetchURL helper. Collect never materializes []ShardResponse. Decoded output is unchanged.

Benchmarks

darwin/arm64, go1.26, go test ./collector/ -bench BenchmarkShards -benchmem. Synthetic /_cat/shards payloads modeled on the real fixture (all 8 API fields per object, ~1/20 UNASSIGNED with nulls, node cardinality scaled with shard count).

Allocation / throughput (BenchmarkShardsDecode, BenchmarkShardsDecodeHTTP):

payload	bytes/op buffered → streaming	reduction	throughput
1,000 shards / 141 KB	709 KB → 107 KB	−84.9%	160 → 171 MB/s
10,000 shards / 1.4 MB	7.47 MB → 1.05 MB	−85.9%	161 → 170 MB/s
50,000 shards / 7.1 MB	35.9 MB → 5.26 MB	−85.4%	170 → 167 MB/s
10,000 over a real HTTP body	7.50 MB → 1.06 MB	−85.9%	—

Allocation count rises ~33% (more, smaller, short-lived per-element decodes); total bytes — the driver of GC pressure — drop ~85%. Throughput is unchanged.

Retained live heap (BenchmarkShardsRetainedHeap, 50,000 shards / 7.1 MB / ~500 nodes) — heap still live after decode+aggregate, i.e. what the collector holds during the emit loop:

	retained live heap
buffered (whole []ShardResponse + map)	~5.4 MB
streaming (per-node map only)	~34 KB

The buffered slice is eliminated; only the per-node map (which scales with node count, not shard count) is retained — ~160× lower for this case. This benchmark measures post-GC retained heap, a conservative lower bound on the buffered path's transient peak (which also holds the read buffer and decode garbage); sub-baseline GC jitter is clamped to zero, so the streaming figure is a small near-floor number.

raw benchmark output

BenchmarkShardsDecode/shards_1000_(141KB)/buffered_decode_slice-12        899µs   159.75 MB/s    709189 B/op     2982 allocs/op
BenchmarkShardsDecode/shards_1000_(141KB)/streaming-12                    849µs   170.59 MB/s    106968 B/op     3962 allocs/op
BenchmarkShardsDecode/shards_10000_(1424KB)/buffered_decode_slice-12      9.06ms  161.00 MB/s   7465221 B/op    29549 allocs/op
BenchmarkShardsDecode/shards_10000_(1424KB)/streaming-12                  8.58ms  169.98 MB/s   1049218 B/op    39518 allocs/op
BenchmarkShardsDecode/shards_50000_(7126KB)/buffered_decode_slice-12      43.0ms  169.81 MB/s  35900520 B/op   147564 allocs/op
BenchmarkShardsDecode/shards_50000_(7126KB)/streaming-12                  43.6ms  167.33 MB/s   5256671 B/op   197524 allocs/op
BenchmarkShardsDecodeHTTP/buffered_decode_slice-12                        9.41ms                7500154 B/op    29698 allocs/op
BenchmarkShardsDecodeHTTP/streaming-12                                    8.95ms                1060937 B/op    39598 allocs/op
BenchmarkShardsRetainedHeap/shards_50000/buffered_decode_slice-12                       5.402 retained_MB
BenchmarkShardsRetainedHeap/shards_50000/streaming-12                                   0.03334 retained_MB

Testing

• TestStreamShardsEquivalence: streaming decode produces byte-for-byte the same shards and the same per-node STARTED aggregation as the previous whole-array json.Unmarshal, over the real fixture (fixtures/shards/7.15.0.json) and a synthetic payload.
• Existing TestShards unchanged and passing.
• go test -race ./..., go vet ./..., and golangci-lint run are clean.

Notes

Follows #1159, which streamed /_all/_stats. The Shards collector remains a direct-registration prometheus.Collector; no metric names or labels change.

Fixes #1188