Skip to content

fix: decouple AppendRows/stream context from per-flush lifecycle - #11

Merged
yuzone merged 1 commit into
mainfrom
fix/decouple-append-context-from-flush-lifecycle
Jun 30, 2026
Merged

fix: decouple AppendRows/stream context from per-flush lifecycle#11
yuzone merged 1 commit into
mainfrom
fix/decouple-append-context-from-flush-lifecycle

Conversation

@yuzone

@yuzone yuzone commented Jun 30, 2026

Copy link
Copy Markdown
Owner

Summary

In the default (at-least-once) path, AppendRows and managed stream creation were passed the per-flush context. Since appends are acknowledged asynchronously on a later flush, canceling the flush context at flush end aborted the managed writer's internal retry/reconnect (managedwriter checks pendingWrite.reqCtx in lockingAppend) and immediately terminalized dynamically-scaled streams (ms.ctx derives from the creation context). As a result, a transient connection failure escalated into a permanent failure storm (mass Appending data ... failed, latency pinned at the flush timeout) that only recovered after a pod restart.

This PR decouples the append/stream lifetime from the per-flush lifecycle by introducing an instance-scoped context.

Changes

out_writeapi.go only:

  • Add instanceCtx / instanceCancel (instance-lifetime context) to outputConfig.
  • FLBPluginInit: replace the throwaway initCtx with context.WithCancel(context.Background()); use it for client creation, getDescriptors, and the initial buildStream. Call instanceCancel() on every error return path to avoid context leaks.
  • sendRequestDefault: pass config.instanceCtx to AppendRows (primary fix).
  • createNewStreamDynamicScaling: pass config.instanceCtx to buildStream so scaled streams are not terminalized at flush end.
  • sendRequestRetries: use config.instanceCtx for stream rebuild as well.
  • FLBPluginExitCtx: call instanceCancel() only after pending responses are drained and streams/client are closed, to avoid aborting in-flight appends.

@yuzone
yuzone force-pushed the fix/decouple-append-context-from-flush-lifecycle branch from 737aa34 to 2623c5d Compare June 30, 2026 05:12
@yuzone
yuzone merged commit 306fcd0 into main Jun 30, 2026
1 check passed
@yuzone
yuzone deleted the fix/decouple-append-context-from-flush-lifecycle branch June 30, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant