[Feature]: Add streaming request support for cross-provider fallback — currently streaming is explicitly unsupported and breaks silently

[divyanshim27]

Summary

TokenFirewall's cross-provider fallback feature explicitly documents that "Streaming not supported." However, the current implementation does not gracefully degrade or warn when a streaming request (stream: true) is intercepted with cross-provider fallback enabled. A developer who has enabled cross-provider routing and uses streaming will either receive an empty/broken response or silent failure — with no error message explaining why.

Problem

• The LIMITATIONS section in README.md documents: "Streaming not supported — Cross-provider fallback only works with non-streaming requests."
• However, there is no guard in the fetch interceptor that detects stream: true in the request body and either warns or gracefully falls back to non-streaming.
• A developer enabling enableCrossProvider: true and making a streaming request will get undefined behavior — the interceptor will attempt to transform what is actually an SSE stream, producing a corrupted response.
• The error message budget (TokenFirewall Router: Max routing retries exceeded) does not distinguish between a streaming incompatibility and an actual provider failure.

Impact

• Developers using streaming (which is the default for many OpenAI SDK integrations) silently receive broken responses when cross-provider fallback is active.
• Debugging this is extremely difficult since the error looks like a provider failure, not a library limitation.
• The library's production-readiness claim is undermined by a silent failure mode on a common use case.

Proposed Solution

I would like to add a streaming detection guard in the fetch interceptor:

// In the fetch patch / interceptor
const bodyObj = safeJsonParse(requestBody);

if (bodyObj?.stream === true && isCrossProviderEnabled()) {
  console.warn(
    '[TokenFirewall] Warning: Cross-provider fallback is enabled but the request uses streaming (stream: true). ' +
    'Cross-provider fallback does not support streaming. The request will be sent to the primary provider only, ' +
    'with no fallback on failure. Set stream: false or disable cross-provider fallback to enable resilient routing.'
  );
  // Bypass fallback logic entirely for streaming requests
  return originalFetch(url, options);
}

Additionally, I would like to add a strict mode option that throws an error (rather than warning) when streaming + cross-provider is detected, for teams that want to enforce correctness at startup.

I would be happy to implement this. Could you assign this issue to me?

Labels: bug, enhancement, streaming, GSSoC 2026

[divyanshim27]

Hey @Ruthwik000 ,
Please assign it to me under GSSoC 2026.
/assign /claim /claim-assignment