Fix tier tactic health filter bypass for fast circuit breaker recovery - #1134
Open
0x0079 wants to merge 7 commits into
Open
Fix tier tactic health filter bypass for fast circuit breaker recovery#11340x0079 wants to merge 7 commits into
0x0079 wants to merge 7 commits into
Conversation
…very The HealthMonitor/HealthFilter (5-min recovery window) was pre-filtering services before the TierTactic could see them, defeating the circuit breaker's 30-second recovery. Tier tactic manages its own health via breakers, so the health filter is now bypassed for tier-based rules. https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4
Tests cover: - Tier tactic bypasses health filter (T0 reachable despite HealthMonitor) - Non-tier tactics still respect health filter - Breaker fallback works while health filter would block - 3-tier waterfall with all services HealthMonitor-unhealthy - Within-tier load sharing with partially unhealthy services - Rate limit on T0 does not stick for 5 min - All services down (both HealthMonitor + breaker open) still returns T0 https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4
The initial fix bypassed the health filter entirely for tier rules. This left a gap: auth errors (401/403) are permanent — a revoked API key never self-heals — so the breaker would cycle through half-open every 30 s, wasting a request each time. Refined approach: for tier rules, only filter out services with auth errors (permanent). Transient states (rate limits, consecutive errors) pass through to the tier tactic's circuit breaker for fast recovery. Changes: - HealthMonitor.HasAuthError(): exposes auth-error state for a service - HealthFilter.FilterAuthErrors(): removes only auth-error services - LoadBalancer.SelectService(): uses FilterAuthErrors for tier rules - 3 new tests for auth-error edge cases https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4
Without backoff, a sustained outage causes the breaker to probe the broken service every 30s indefinitely — wasting one user's round-trip each time and potentially disrupting provider-side caches. Now each consecutive half-open probe failure doubles the open window: 30s → 60s → 120s → 240s → cap (5 min). A successful probe resets the backoff to the base duration. This gives fast recovery for transient issues while reducing overhead during extended outages. https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4
30s was too aggressive for LLM API providers — rate-limit windows are typically 60s+, so the first probe was almost guaranteed to fail, wasting a user's round-trip latency for nothing. 60s aligns better with real-world provider rate-limit recovery while still being fast enough for transient server errors. Backoff sequence is now: 60s → 120s → 240s → 5 min cap. https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4
FilterAuthErrors is called per-request for tier rules. In the common case (no auth errors), it now returns the input slice directly with zero allocation. Only allocates when an auth-error service is actually encountered. Also fixes a stale "30s" reference in the backoff comment. https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4
Use windForward() to move openedAt backwards instead of sleeping through real wall-clock time. Reduces TestBreakerExponentialBackoff from 0.53s to <1ms and TestBreakerBackoffResetsOnSuccess from 0.10s to <1ms. https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The tier tactic now bypasses the full health filter to enable fast circuit breaker recovery (30s), while still filtering permanent auth errors (401/403). Previously, transient failures like rate limits would hide services for the health monitor's 5-minute recovery window, defeating the tier tactic's quick failover mechanism.
Key Changes
Health Filter
FilterAuthErrors()method to filter only permanent auth errors, leaving transient failures visibleLoad Balancer
SelectService()to useFilterAuthErrors()for tier tactics instead of the fullFilter()Circuit Breaker
DefaultBreakerOpenDurationfrom 30s to 60s for more conservative initial backoffDefaultBreakerMaxOpenDuration(5 minutes) and exponential backoff with caphalfOpenFailscounter to track consecutive half-open probe failurescurrentOpenDuration()to calculate backoff-adjusted wait times (doubles on each half-open failure, capped at max)RecordSuccess()Health Monitor
HasAuthError()method to check if a service has a permanent auth errorTests
Implementation Details
https://claude.ai/code/session_01GjvoWzPe8qvowivo7P1oG4