Skip to content

perf: skip parseMap allocation and pool dynamicpb.Message - #5

Merged
yuzone merged 1 commit into
mainfrom
perf/skip-parsemap-and-pool-dynamicpb-message
Mar 6, 2026
Merged

perf: skip parseMap allocation and pool dynamicpb.Message#5
yuzone merged 1 commit into
mainfrom
perf/skip-parsemap-and-pool-dynamicpb-message

Conversation

@yuzone

@yuzone yuzone commented Mar 5, 2026

Copy link
Copy Markdown
Owner

Summary

Eliminate two major per-row heap allocations in the Flush hot path to reduce GC pressure and CPU overhead.

Changes

Direct processing of raw records

Process the map[interface{}]interface{} returned by Fluent Bit's msgpack decoder directly into proto binary, removing the parseMap deep copy into map[string]interface{}.

  • Add rawMapToBinary / rawPopulateMessage / rawSetRepeatedField
  • Add convertTimestampFieldsRaw to convert timestamps in-place on raw records
  • Add []byte cases to all scalar converters to handle msgpack string values directly

Pool dynamicpb.Message

Reuse top-level dynamicpb.Message instances via sync.Pool keyed by proto FullName, avoiding a new allocation per row.

  • Add getPooledMessage / putPooledMessage / marshalAndRelease
  • Extract populateMessage from mapToMessage to allow filling a pre-allocated message
  • putPooledMessage clears fields via Range+Clear, preserving internal map bucket memory

Remove dead code

Remove functions and tests that are no longer called after the above changes:

  • jsonToBinary, parseMap, parseSlice
  • convertTimestampFields, convertTimestampValue
  • TestParseMap, TestParseSlice, TestConvertTimestampFields

Testing

  • TestRawMapToBinary_* — raw map conversion including []byte values, nested structs, and repeated fields
  • TestMessagePoolReuse* — verify no stale field leaks across pooled message reuse
  • TestConvertTimestampFieldsRaw — timestamp conversion on raw records
  • BenchmarkRawMapToBinary — benchmark for the raw path
  • Existing Flush tests (TestFLBPluginFlushCtx, etc.) pass without modification

@yuzone
yuzone force-pushed the perf/skip-parsemap-and-pool-dynamicpb-message branch from 13b072f to 220e669 Compare March 5, 2026 08:13
@yuzone
yuzone merged commit 1770f56 into main Mar 6, 2026
1 check passed
@yuzone
yuzone deleted the perf/skip-parsemap-and-pool-dynamicpb-message branch March 9, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant