Singleflight + streaming — can multiple consumers attach to a single stream?

[aprilkobayashi]

The singleflight dedup makes sense for non-streaming requests: collapse N identical requests into 1, share the response. But for streaming, the semantics are trickier.

If two callers request the same stream simultaneously:

# Caller A
async for chunk in llmix.dispatch_stream(config, messages):
    send_to_client_a(chunk)

# Caller B (same config + messages, arrives 100ms later)
async for chunk in llmix.dispatch_stream(config, messages):
    send_to_client_b(chunk)

Does singleflight:

1. Multicast the stream — both callers get chunks in real-time from a single upstream connection? (complex but ideal)
2. Wait and replay — caller B waits for caller A's stream to complete, then gets the cached full response as a synthetic stream? (simpler but adds latency for B)
3. No dedup for streams — each caller gets its own upstream connection? (simplest but wasteful)

Option 1 requires a broadcast/fan-out mechanism. Something like:

class StreamBroadcast:
    def __init__(self, upstream):
        self.subscribers = []
        self.buffer = []  # for late joiners

    async def subscribe(self):
        # Replay buffer + live chunks
        ...

Which approach does LLMix take? And if it's option 3, is option 1 planned?