Skip to content

robust: handle unconsumed ChatCompleteResponse without goroutine leak #211

Description

@maragubot

Problem

robust.ChatCompleter.ChatComplete uses iter.Pull2 in commitOnFirstPart to peek the first streamed part before committing. That spawns a goroutine running the underlying provider's push iterator; stop() releases it.

The wrapped response's defer stop() only fires if the caller actually invokes Parts(). If the caller receives the response and drops it (handles an error, times out, takes a branch that never iterates), the goroutine is parked forever at the provider's yield call, holding whatever that partsFunc closed over (HTTP response body, TCP socket, SDK state).

For now PR #209 documents the contract ("caller must invoke Parts()"). This issue tracks a real fix.

Directions to consider

  • Buffer the first part and close the Pull2 iterator synchronously before returning, trading a bit of memory for a self-contained response. Requires materialising the rest of the stream too, which defeats streaming.
  • Spawn a watchdog goroutine with a short deadline; if the wrapped iterator hasn't been started, call stop() on its behalf. Introduces its own leak vector (the watchdog itself) and arbitrary timeout policy.
  • Expose an explicit Close() method on the returned response and require callers to call it on all paths; requires changing gai.ChatCompleteResponse to support closing, which is a top-level API change.
  • Finalizer as last-resort cleanup. Indeterminate timing and can mask leaks in tests.
  • Widen gai.ChatCompleteResponse to carry a Close() / context-like lifecycle, so this pattern works uniformly for all wrappers.

Acceptance

  • The goroutine is released regardless of whether the caller iterates Parts().
  • Documented contract is clear and consistent with what the provider clients themselves do.

Reference

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions