Key pool rotation with per-org rate limits — isolation question

[rosaliewakefield57]

We're running KeyPool with keys from three different OpenAI organizations. Each org has its own TPM/RPM limits. The pool rotates keys on 429s and auto-evicts dead keys — that part works well.

The question: when a key from org-A gets a 429, does the pool try the next key in order (which might be org-A again), or does it have any awareness of key "groups"?

In our case, a 429 from org-A means all org-A keys are likely rate-limited. Ideally the pool would skip remaining org-A keys and jump to org-B. Without that, we burn through 429 retries on keys that are all going to fail.

We're considering wrapping KeyPool per-org and dispatching at the application level, but that defeats the purpose of having a unified pool. Any recommendations?

[OscarDoh]

Good question — you're right that the current KeyPool has no concept of key groups or org-awareness. It's a flat round-robin: rotate() advances to the next alive key in list order, regardless of which org it belongs to.

So if your pool is constructed as [org-A-1, org-A-2, org-A-3, org-B-1, org-B-2, org-C-1], a 429 on org-A-1 rotates to org-A-2, which will also be rate-limited — you burn through retries on keys that are guaranteed to fail.

Workaround that works today: interleave keys by org when constructing the pool:

pool = KeyPool([
    "sk-org-A-key-1",
    "sk-org-B-key-1",
    "sk-org-C-key-1",
    "sk-org-A-key-2",
    "sk-org-B-key-2",
    # ...
])

This way, rotate() naturally hits a different org on each advance. Not perfect — it doesn't guarantee org-level skipping — but in practice it avoids the worst case of burning through all org-A keys sequentially.

What a proper fix would look like: a KeyPoolGroup or a group tag on each key, where rotate() skips to the next group on 429, not just the next key. Something like:

pool = KeyPool([
    ("sk-A-1", "org-a"),
    ("sk-A-2", "org-a"),
    ("sk-B-1", "org-b"),
], group_aware=True)

This is a reasonable enhancement. The main design question is whether the pool should track per-group cooldowns (so it knows when an org's rate limit window resets) or just rotate past the group and let the retry loop handle timing. The simpler version (skip the group, let retries handle timing) is probably the right starting point.

If you want to prototype this, the relevant code is key_pool.py — rotate() is ~10 lines and the change would be straightforward. Would welcome a PR if you want to take a crack at it.