Improve client performance

[atanzu]

This PR introduces a series of changes aimed to improve performance of the Go client.

Overview

Area	Improvement	Trade-off
Flush (any shard count)	−78–82% time, −97–100% allocs	None
Construction (high shard count)	−38% time, −99% allocs	Memory +154% at 256 shards (one-time, ~68 KB)
Sharding (low shard count ≤4)	−45–58% time, −100% allocs	None
Sharding (high shard count ≥64)	−18–24% time, −100% allocs	None
Steady-state sampling (Tags ≤60)	−34–46% time, 3→0 allocs	None
Steady-state sampling (Tags_100)	−5% time	3 allocs unchanged (large tag sets still escape)
Per-metric-type sampling	−29–40% time, 1→0 allocs	None

---

More detailed overview

---

Change 0 - New benchmark

Because we need more numbers to judge the changes.

---

Change 1 - New shard allocation and placement logic

Improve aggregator's construction and flush. Previously each shard allocated its own map on construction and its own slice on every flush cycle. Both are now amortised/eliminated. Steady-state sampling is unchanged.

Construction — allocs/op (deterministic)

Shards	Before	After	Delta
1	24	18	−25%
32	210	18	−91%
256	1,554	18	−99%

Construction — B/op and sec/op

Shards	Before B	After B	Before t	After t
1	16.9 KB	16.8 KB	23.4 µs	23.4 µs
32	24.9 KB	20.0 KB	26.1 µs	23.6 µs
256	83.4 KB	44.4 KB	45.1 µs	25.0 µs

Flush — allocs/op (deterministic)

Benchmark	Before	After	Delta
FlushEmpty/Shards_1	6	3	−50%
FlushEmpty/Shards_32	99	3	−97%
FlushEmpty/Shards_256	771	3	−99%
FlushSparse	789	24	−97%

Flush — sec/op and B/op

Benchmark	Before t	After t	Before B	After B
FlushEmpty/Shards_1	152 ns	88 ns	288 B	144 B
FlushEmpty/Shards_32	2,494 ns	505 ns	4,752 B	144 B
FlushEmpty/Shards_256	19,643 ns	3,552 ns	37,008 B	144 B
FlushSparse	21,335 ns	4,587 ns	39.1 KB	3.2 KB

Geomean (sec/op): −5.2% · Geomean (allocs/op): −18.6%

---

Change 2 - Optimize metric contexts builds

Improves the steady-state sampling hot path a.k.a. the code executed every time a metric is recorded against an existing context. Previously this allocated on every call for most tag counts; after this commit it is zero-allocation for Tags ≤ ~60.

AggregatorSampleExisting — allocs/op (deterministic)

All shard counts, Tags_5 through Tags_60: 3 allocs → 0
Tags_100: unchanged at 3 allocs (large tag sets still escape to heap)

AggregatorSampleExistingByMetricType — allocs/op

Count / Gauge / Set: 1 alloc → 0

BufferedMetricContextsSampleExisting — allocs/op

All shapes below Tags_100: 1 alloc → 0 (Tags_100: 1 → 0 as well)

AggregatorSampleExisting — sec/op (representative: Shards_8)

Tags	Before	After	Delta
5	195 ns	112 ns	−42%
10	314 ns	185 ns	−41%
40	932 ns	569 ns	−39%
60	1,277 ns	823 ns	−36%
100	1,974 ns	1,843 ns	−7%

AggregatorSampleExisting — sec/op (Shards_16, showing tag-count scaling)

Tags	Before	After	Delta
5	192 ns	102 ns	−47%
10	322 ns	176 ns	−45%
40	945 ns	564 ns	−40%
60	1,245 ns	819 ns	−34%

AggregatorSampleExistingByMetricType — sec/op

Metric type	Before	After	Delta
Count	15.6 ns	10.9 ns	−30%
Gauge	15.4 ns	10.7 ns	−30%
Set	17.7 ns	13.7 ns	−23%

BufferedMetricContextsSampleExistingByShape — sec/op

Shape	Before	After	Delta
NoTags_Low	242 ns	230 ns	−5%
FiveTags_Low	270 ns	243 ns	−10%
Tags100_Low	430 ns	342 ns	−20%
Tags100_High	429 ns	338 ns	−21%

Geomean (sec/op): −20.2%

---

Change 3 - Pad aggregator shards to avoid false sharing

Use cache-line padding on shard structs to eliminate false sharing between goroutines on different CPUs accessing different shards.

AggregatorSharding — sec/op

Shards	Before	After	Delta
1	431 ns	281 ns	−35%
2	327 ns	139 ns	−58%
3	213 ns	98 ns	−54%
4	157 ns	79 ns	−49%
8	91 ns	62 ns	−32%
16	63 ns	49 ns	−21%
32	54 ns	44 ns	−18%
64	46 ns	43 ns	−7%
128	43 ns	40 ns	−8%
256	41 ns	40 ns	−4%

AggregatorSampleExisting at low shard counts — sec/op

False-sharing removal shows through in sampling too, especially at Shards_1–4:

Config	Before	After	Delta
Shards_1 / Tags_5	382 ns	130 ns	−66%
Shards_1 / Tags_40	397 ns	154 ns	−61%
Shards_2 / Tags_5	241 ns	119 ns	−51%
Shards_4 / Tags_5	141 ns	104 ns	−26%
Shards_8 / Tags_5	112 ns	103 ns	−8%
Shards_16+ / Tags_5	~100 ns	~98 ns	≈ flat

AggregatorSampleExistingByMetricType — sec/op

Metric type	Before	After	Delta
Count	10.9 ns	9.0 ns	−17%
Gauge	10.7 ns	9.0 ns	−16%
Set	13.7 ns	11.2 ns	−18%

Geomean (sec/op): −10.2%

Trade-offs / regressions

Downside: Padding inflates the in-memory size of each shard struct, so construction memory grows proportionally to shard count.

Shards	Before B	After B	Delta
1	16.8 KB	17.0 KB	+2%
32	20.0 KB	30.9 KB	+54%
256	44.4 KB	112.7 KB	+154%

This memory is allocated once at client construction and held for the lifetime of the
client, so the impact is negligible for long-running processes. At 256 shards the
footprint grows by ~68 KB.

---

Benchmark environment:

• Apple M4 Max, 16 cores
• go1.26.3 darwin/arm64
• canary spread 1.01×
• 3 interleaved rounds

[vickenty]

Would it make sense to add some tests to make sure we correctly aggregating metrics with different context sizes?

[atanzu]

I don't think so. In the very end we append to a slice; once we exceed the capacity, Go runtime allocates a new backing array on the heap.

[vickenty]

Code above this line is the same for all flavors. Would we loose on performance if we extracted this repeated bit into a method?

Same question for per-shard code below, which is repeated in countWithStringContext.

[atanzu]

Unfortunately yes, I tried to compact those common pieces into shared functions and got hit by a performance penalty. If I understand the problem correctly, Go compiler is limited in optimizing deep nested function calls.

[vickenty]

Are getContext and getContextAndTags dead code now?

[vickenty]

These branches (and matching ones in appendContext and appendContextAndHash) seem to be lacking test coverage. Would it be possible to add it?

[datadog-datadog-prod-us1-2]

🎯 Code Coverage (details)
• Patch Coverage: 96.90%
• Overall Coverage: 84.90% (+1.68%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6270577 | Docs | Datadog PR Page | Give us feedback!}