Rate-limit circuit breaker for spawned agent families - #90
Merged
Conversation
When a spawned agent hits a hard provider rate limit (429 / rate_limit_event rejected), record the tripped family+model and its reset time, and short-circuit further spawns of that family until the window passes — instead of re-launching a subprocess that fails every iteration (observed: ~9 failed claude spawns over 4 iterations of one program). Adds run.json `rate_limited_families` (additive; existing usage fields untouched), config knobs rate_limit_circuit_breaker (default on) + rate_limit_default_cooldown_s (900), and a rate_limits module with detection from the captured JSONL stream. Implemented by codex (gpt-5.6-sol xhigh); pending Grok + human review. Co-Authored-By: Codex (gpt-5.6-sol) <noreply@openai.com> Claude-Session: https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc
… failure Addresses adversarial review (Grok) of the circuit breaker: 1. A family's reset window is never shortened — a later bare-429 with no resetsAt keeps the existing far-future provider reset via a later-of merge (max), instead of overwriting it with the 15-min default and regressing to the thrash the feature prevents. 2. Only trip on a true terminal rate-limit failure: a rejected rate_limit_event followed by a successful terminal result does NOT trip; is_error=false cannot trip even if a 429 field is present. 3. The stdout-checked flag is set only after a successful scan; an OSError leaves it retry-eligible so a real limit is not missed. Plus: a Gemini terminal-error fixture, float 429.0 / millisecond resetsAt handling, and an unambiguous polymorphic spawn-result parser. Note: test_config::test_prompt_file_permission_errors_raise_system_exit is a PRE-EXISTING order-dependent flake (passes in isolation on main and this branch) — unrelated to this change. Co-Authored-By: Codex (gpt-5.6-sol) <noreply@openai.com> Claude-Session: https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a spawned agent hits a hard provider rate limit (a JSONL
rate_limit_event/resultwithapi_error_status==429oroverageStatus=="rejected", optionally carryingresetsAt), record the tripped family+model+reset and short-circuit further spawns of that family until the window passes — instead of re-launching a subprocess that fails every iteration and eats failover latency each time.Motivation (observed in a live program): a Claude family hit its 7-day cap and the loop re-attempted it ~9 times across 4 iterations, silently degrading for hours. The breaker converts that into an immediate "family rate-limited until T" that the coordinator can route around.
rate_limit_circuit_breaker(default on),rate_limit_default_cooldown_s(900).run.jsongains an additiverate_limited_familiesfield (existing usage fields untouched — loopy-loop's token parsing is unaffected).rate_limitsmodule: detection from the captured JSONL stream, a family-keyed breaker, and a short-circuit spawn path.Review
Implemented by Codex (gpt-5.6-sol xhigh); adversarially reviewed by Grok (grok-4.5) and by me. Grok's first pass was NEEDS-WORK and found three real bugs — all fixed and re-confirmed CLOSED in a second Grok pass:
max) per family; a bare-429 can never shorten an active provider reset.rejectedevent on a worker that ultimately succeeded still tripped. Now only a terminal rate-limit failure trips;is_error=falsecannot trip.Plus Gemini fixture, float/ms
resetsAt, and a polymorphic spawn-result parser.Known nit (Grok, non-blocking): successive rate-limit events within one JSONL scan don't max-merge; family-level re-trip thrash — the real failure mode — is closed.
Test plan
601 tests pass; ruff + pyright clean. New tests: never-shorten across bare-429; reject-then-success = no trip; OSError retry-eligibility; non-Claude fixture; numeric-reset handling; spawn-result polymorphism. (One pre-existing order-flake,
test_prompt_file_permission_errors_raise_system_exit, passes in isolation on main and this branch — unrelated.)🤖 Generated with Claude Code — implemented by Codex, reviewed by Grok + Claude.
https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc