perf/refactor: throughput improvements, memory optimizations, and code quality enhancements - #6
Merged
Merged
Conversation
…tion variable gomonkey's ApplyFunc for output.NewDecoder was unreliable on ARM64 macOS when tests ran sequentially in the same process. The patch occasionally failed to intercept calls, causing checks.createDecoder to stay at 0 and TestFLBPluginFlushCtxErrorHandling to fail non-deterministically. Fix: introduce a var newDecoder function variable in out_writeapi.go (matching the pattern already used for getClient, getWriter, getFLBPluginContext, pluginGetResult, isReady). Replace all four gomonkey.ApplyFunc(output.NewDecoder, ...) calls in tests with direct assignment to newDecoder, with deferred restore. This eliminates the ARM64 code-patching dependency entirely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…location binaryData ([][]byte) was re-allocated with make([][]byte, 0, 256) on every FLBPluginFlushCtx call. Move it to outputConfig so the underlying array is reused across flushes. Nil out retained elements before each reset so the GC can reclaim the previous batch's byte slices.
FLBPluginExitCtx called checkAllStreamResponses with waitForResponse=false, which skips any in-flight response that has not yet arrived. This could cause data loss if AppendRows results were still pending when the plugin exited. Fix: pass waitForResponse=true so pluginGetResult blocks on each queued result, fully draining the queue before finalizeCloseAllStreams runs. Also switch FLBPluginExitCtx to use the injectable getFLBPluginContext variable (consistent with FLBPluginFlushCtx) to enable unit testing without gomonkey. Add TestFLBPluginExitCtxDrainsPendingResponses to verify that responses marked as not-yet-ready are still drained during exit. Fix MockManagedStream.Close and MockManagedWriterClient.Close to use their CloseFunc fields when set, avoiding nil-pointer panics in tests that call ExitCtx.
- Annotate const block (out_writeapi.go) with explanations for chunkSizeLimit, queueRequestDefault, queueByteDefault, queueRequestScalingPercent, maxNumStreamsPerInstance, minQueueRequests, maxUnixSeconds, and flushTimeoutSecDefault. - Fix "Unsuccesful" → "Unsuccessful" in sendRequestRetries comment.
…age conversion
Add toSubMessage(val, md, cache) as the single authoritative path for
converting either map[string]interface{} or map[interface{}]interface{}
values into a populated *dynamicpb.Message.
Replace the duplicated type-switch/assertion blocks in populateMessage,
rawPopulateMessage, setRepeatedField, and rawSetRepeatedField with calls
to the new helper. rawSetRepeatedField becomes identical to
setRepeatedField after this change and is therefore deleted; its one
call site in rawPopulateMessage now calls setRepeatedField directly.
…nFlushCtx Change all internal functions (buildStream, sendRequest*, createNew- StreamDynamicScaling, finalizeCloseAllStreams) from **outputConfig to *outputConfig, eliminating triple-dereference patterns throughout. Remove the configPointer intermediate variable in FLBPluginInit. Convert buildStream to an injectable var (consistent with newDecoder, getClient, etc.) and migrate TestFLBPluginFlushCtxDynamicScaling off gomonkey.ApplyFunc to fix ARM64 reliability. Extract flushChunk (stream selection + send + offset update) and decodeAndSerializeRecords (record loop + mid-chunk flush) from FLBPluginFlushCtx, reducing it from 95 to 34 lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR bundles a series of performance improvements, bug fixes, and code quality
refactors identified through production throughput analysis of the Fluent Bit
BigQuery WriteAPI sink plugin.