[BUG] Load balancer doesn't avoid quota‑exhausted keys, keeps routing to failed key

[MingxiYe]

Version

v0.260531.0-rc6

System Information

Ubuntu 24 LTS

Category

Protocol

Scenario

Description

When two API keys are configured under the same rule, I observe two problems:

With no external usage, requests are not evenly split 50/50 as expected (round robin seems broken).

More critically: when one key reaches its quota limit (returning quota‑exhausted errors) while the other still has quota, TB continues to route new requests to the exhausted key, instead of falling back to the healthy key.

Expected Behaviors

• Both keys used only via TB → requests should be distributed 50/50 (true round robin).
• Keys have asymmetric quota or external usage → TB should implement smart load balancing (e.g., avoid keys that are failing, prefer keys with remaining quota, or use least‑request / weighted balancing).

The current behavior fails both expectations, especially the second one. TB does not seem to detect or react to a key being exhausted.

Proposed Solution

No response

[FFengIll]

#1014 Fixed the strategy issue, so that load will be more evenly distributed thereafter; furthermore, #953 will provide a more elegant way to smoothly transition to the next after token exhaustion.
In the further future, affinity will be added to ensure effective cache hits (continuous adsorption) while guaranteeing load balance. #1076