diff --git a/.env.example b/.env.example index 76e8219..c270b8b 100644 --- a/.env.example +++ b/.env.example @@ -10,6 +10,8 @@ UNIFIED_LLM_PROVIDER=primary # Optional resource and retry bounds UNIFIED_LLM_TIMEOUT=30 +UNIFIED_LLM_HEALTH_COOLDOWN=30 +UNIFIED_LLM_HEALTH_FAILURE_THRESHOLD=3 UNIFIED_LLM_MAX_ATTEMPTS_PER_ROUTE=2 UNIFIED_LLM_MAX_TOTAL_ATTEMPTS=4 UNIFIED_LLM_MAX_CONCURRENCY=10 diff --git a/CHANGELOG.md b/CHANGELOG.md index c53b5bb..f9a55ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ All notable changes to this project will be documented here. The project follows - Configurable raw and normalized response byte limits, response character limits, and tool-call count limits. - Runtime validation for normalized responses returned by every provider adapter. - A stateless-by-default OpenAI Responses API adapter with text, refusal, function-call, and tool-result normalization. +- Content-free sync/async attempt observation and inspectable provider health snapshots. +- Cross-request transient-failure cooldown that deprioritizes unhealthy routes without removing last-resort fallback. ### Changed diff --git a/README.md b/README.md index c8fb3dd..cd70d9f 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ async def main() -> None: asyncio.run(main()) ``` -Automatic routing uses every route in order. Passing `provider="backup"` selects one route and disables automatic fallback. A model override with multiple routes also requires an explicit provider, preventing accidental cross-provider model dispatch. +Automatic routing starts in configured route order; routes in an active health cooldown move behind healthy routes. Passing `provider="backup"` selects one route, bypasses health reordering, and disables automatic fallback. A model override with multiple routes also requires an explicit provider, preventing accidental cross-provider model dispatch. ## Failure contract @@ -129,6 +129,10 @@ The SDK raises instead of returning an ambiguous empty string: `asyncio.CancelledError` is propagated immediately and is never retried. Each provider attempt is available as sanitized `Attempt` metadata on successful responses or failure exceptions. +For metrics and tracing, pass `on_attempt=` a sync or async callback. It receives only the sanitized `Attempt` record—never messages, response content, credentials, headers, or endpoint URLs. Callback failures are isolated from generation; cancellation still propagates. + +After three consecutive transient failures by default, a provider enters a 30-second cooldown and is moved behind healthy routes. It remains a last-resort fallback, and an explicit `provider=` selection can probe it immediately. Successful probes reset its health. Inspect content-free state with `get_provider_health()`. + ## Configuration `UnifiedLLM.from_env()` supports: @@ -143,9 +147,11 @@ The SDK raises instead of returning an ambiguous empty string: | `UNIFIED_LLM_MAX_ATTEMPTS_PER_ROUTE` | No | `2` | Attempts per route, 1–5. | | `UNIFIED_LLM_MAX_TOTAL_ATTEMPTS` | No | `4` | Total request attempts, 1–16. | | `UNIFIED_LLM_MAX_CONCURRENCY` | No | `10` | In-flight requests, 1–1000. | +| `UNIFIED_LLM_HEALTH_FAILURE_THRESHOLD` | No | `3` | Consecutive transient failures before route cooldown; `0` disables. | +| `UNIFIED_LLM_HEALTH_COOLDOWN` | No | `30` | Seconds to deprioritize an unhealthy route, maximum 3600. | | `UNIFIED_LLM_ALLOW_INSECURE_HTTP` | No | `false` | Explicitly allow trusted remote plain HTTP. | -See [.env.example](.env.example). Programmatic construction additionally controls input/output limits, retry delay, and custom non-auth headers. +See [.env.example](.env.example). Programmatic construction additionally controls input/output limits, retry delay, the sanitized attempt hook, and custom non-auth headers. ## Development and verification diff --git a/ROADMAP.md b/ROADMAP.md index 14901fe..1b8d977 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -16,6 +16,7 @@ Current disposition: Already integrated; roadmap changes should use an ordinary - The productization SHA is already the remote default and its hosted checks pass. - Completed in the next hardening increment: account for the exact outbound body, cap inbound response bytes and normalized structures, and validate custom adapters. - Completed in the following API increment: add a stateless-by-default OpenAI Responses adapter without weakening the shared bounds or fallback contract. +- Completed in the health increment: expose content-free attempt observation and deprioritize repeatedly failing routes during a bounded cooldown. - Next: prove one canonical consumer and a capped live endpoint before publication. - Review priority: prove one consumer, green wheel CI, one capped live endpoint, and trusted publication. diff --git a/docs/API.md b/docs/API.md index f10c7ae..a829183 100644 --- a/docs/API.md +++ b/docs/API.md @@ -62,11 +62,16 @@ UnifiedLLM( backoff_base=0.25, max_retry_delay=5.0, retry_jitter=0.1, + health_failure_threshold=3, + health_cooldown=30.0, + on_attempt=None, ) ``` Routes are tried in order when neither `provider` nor `model` selects a single destination. +Repeated transient failures increment provider-local health state. At `health_failure_threshold`, that provider is moved behind healthy routes for `health_cooldown` seconds. Set the threshold to `0` to disable health-aware ordering. Explicit provider selection preserves operator control and bypasses reordering. + ### `UnifiedLLM.from_env(prefix="UNIFIED_LLM", *, environ=None)` Builds one route from the variables documented in the README. Passing an explicit mapping as `environ` makes configuration deterministic in tests. @@ -81,7 +86,7 @@ All methods are async. - `chat_with_metadata(..., tools=None) -> UnifiedLLMResponse` - `chat_with_tools(messages, *, tools, ...) -> UnifiedLLMResponse` -Messages are mappings with a supported `role` (`assistant`, `developer`, `system`, `tool`, or `user`) and non-empty string `content`. Additional JSON-serializable message fields are preserved for compatible endpoints. +Messages are mappings with a supported `role` (`assistant`, `developer`, `system`, `tool`, or `user`) and non-empty string `content`. An assistant turn may instead contain a non-empty `tool_calls` list for function continuation. Additional JSON-serializable message fields are preserved for compatible endpoints. Convenience methods raise the same typed errors as metadata methods; they never convert errors to empty strings. @@ -102,6 +107,8 @@ Convenience methods raise the same typed errors as metadata methods; they never Contains `provider`, `model`, one-based route-attempt `number`, `latency_ms`, sanitized `error`, and `retryable`. It intentionally excludes endpoints, headers, prompts, response bodies, and exception text from unexpected adapters. +`on_attempt` accepts a sync or async callback receiving this value after every completed attempt. Callback exceptions are ignored, except cancellation. `get_provider_health() -> dict[str, ProviderHealth]` returns provider name, consecutive transient failures, cooldown state, and remaining cooldown seconds without application content. + ## Errors All package errors derive from `UnifiedLLMError`. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2bc8e75..a4a9459 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -30,7 +30,9 @@ The Responses adapter shares the same bounded HTTP transport and error classific The router validates the exact provider payload before network I/O, including the resolved model and generation fields, then holds one concurrency permit for the logical request. It validates and bounds normalized results from built-in and custom adapters before returning them. Each route receives at most `max_attempts_per_route`; the request receives at most `max_total_attempts`. Retry delay uses bounded exponential backoff, optional jitter, and a capped `Retry-After` value. -Permanent provider failures stop immediately. Transient failures can retry and move to the next route. Cancellation propagates immediately. Unexpected custom-adapter exceptions are wrapped without copying their text. +Permanent provider failures stop immediately. Transient failures can retry and move to the next route. Repeated transient failures maintain provider-local health state; cooling providers move behind healthy routes while remaining last-resort fallbacks. Explicit route selection is never silently overridden. Cancellation propagates immediately. Unexpected custom-adapter exceptions are wrapped without copying their text. + +The optional attempt observer receives only immutable sanitized metadata. Observer failures cannot turn a successful provider response into an application failure, and the core performs no logging or exporter I/O itself. ## Trust boundaries @@ -55,7 +57,7 @@ application data and configuration (trusted host) ## Resource and cost bounds -Defaults are 30 seconds per attempt, two attempts per route, four attempts total, ten concurrent logical requests, 200,000 message-content characters, 1,000,000 exact serialized request bytes, 2,000,000 normalized response bytes, 1,000,000 response characters, 128 tool calls, 32,768 output tokens, and a five-second maximum retry delay. The built-in HTTP adapter also caps raw success bodies at 2,000,000 bytes. Constructor validation sets absolute safety ceilings. +Defaults are 30 seconds per attempt, two attempts per route, four attempts total, ten concurrent logical requests, 200,000 message-content characters, 1,000,000 exact serialized request bytes, 2,000,000 normalized response bytes, 1,000,000 response characters, 128 tool calls, 32,768 output tokens, a five-second maximum retry delay, and a 30-second health cooldown after three consecutive transient failures. The built-in HTTP adapter also caps raw success bodies at 2,000,000 bytes. Constructor validation sets absolute safety ceilings. Retries are not perfectly idempotent: a provider can finish generation while the response is lost. The bounded attempt count limits amplification; applications with stricter budgets should use one attempt per route and lower token limits. diff --git a/examples/observability.py b/examples/observability.py new file mode 100644 index 0000000..3b6df07 --- /dev/null +++ b/examples/observability.py @@ -0,0 +1,34 @@ +"""Export sanitized attempt metrics without exposing application content.""" + +from __future__ import annotations + +import asyncio +from collections import Counter + +from unified_llm import Attempt, Route, UnifiedLLM, UnifiedLLMResponse + + +class DemoProvider: + name = "demo" + + async def complete(self, **_kwargs: object) -> UnifiedLLMResponse: + return UnifiedLLMResponse(content="ok", model="demo-model", provider=self.name) + + +async def main() -> None: + counters: Counter[tuple[str, str]] = Counter() + + def observe(attempt: Attempt) -> None: + outcome = attempt.error or "success" + counters[(attempt.provider, outcome)] += 1 + + client = UnifiedLLM([Route(DemoProvider(), "demo-model")], on_attempt=observe) + async with client: + await client.generate("This text is never passed to observe().") + + print(counters) + print(client.get_provider_health()) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/tests/test_configuration.py b/tests/test_configuration.py index ca94188..20d0d58 100644 --- a/tests/test_configuration.py +++ b/tests/test_configuration.py @@ -29,11 +29,15 @@ def test_from_env_builds_a_local_route_without_a_key() -> None: "UNIFIED_LLM_MAX_ATTEMPTS_PER_ROUTE": "1", "UNIFIED_LLM_MAX_TOTAL_ATTEMPTS": "1", "UNIFIED_LLM_MAX_CONCURRENCY": "2", + "UNIFIED_LLM_HEALTH_FAILURE_THRESHOLD": "4", + "UNIFIED_LLM_HEALTH_COOLDOWN": "12.5", } ) assert client.get_available_providers() == ["local"] assert client.routes[0].model == "local-model" assert client.request_timeout == 5 + assert client.health_failure_threshold == 4 + assert client.health_cooldown == 12.5 def test_from_env_requires_model_and_default_endpoint_key() -> None: @@ -51,6 +55,10 @@ def test_from_env_requires_model_and_default_endpoint_key() -> None: {"UNIFIED_LLM_MODEL": "m", "UNIFIED_LLM_API_KEY": "k", "UNIFIED_LLM_MAX_CONCURRENCY": "x"}, "integer", ), + ( + {"UNIFIED_LLM_MODEL": "m", "UNIFIED_LLM_API_KEY": "k", "UNIFIED_LLM_HEALTH_COOLDOWN": "x"}, + "number", + ), ( {"UNIFIED_LLM_MODEL": "m", "UNIFIED_LLM_API_KEY": "k", "UNIFIED_LLM_ALLOW_INSECURE_HTTP": "x"}, "boolean", diff --git a/tests/test_health_observability.py b/tests/test_health_observability.py new file mode 100644 index 0000000..8d899e0 --- /dev/null +++ b/tests/test_health_observability.py @@ -0,0 +1,123 @@ +from __future__ import annotations + +import pytest + +from unified_llm import Attempt, ProviderError, Route, UnifiedLLM + +from .conftest import ScriptedProvider, response + + +class FakeClock: + def __init__(self) -> None: + self.now = 100.0 + + def __call__(self) -> float: + return self.now + + +async def test_cooling_provider_is_deprioritized_then_recovers() -> None: + clock = FakeClock() + primary = ScriptedProvider( + "primary", + [ + ProviderError("primary", "temporary", retryable=True), + ProviderError("primary", "temporary", retryable=True), + response("recovered"), + ], + ) + backup = ScriptedProvider("backup", [response("backup-1"), response("backup-2"), response("backup-3")]) + client = UnifiedLLM( + [Route(primary, "model-a"), Route(backup, "model-b")], + max_attempts_per_route=1, + health_failure_threshold=2, + health_cooldown=30, + _clock=clock, + ) + + assert await client.generate("one") == "backup-1" + assert await client.generate("two") == "backup-2" + health = client.get_provider_health() + assert health["primary"].consecutive_failures == 2 + assert health["primary"].cooling_down + assert health["primary"].cooldown_remaining == 30 + + assert await client.generate("three") == "backup-3" + assert len(primary.calls) == 2 + + clock.now += 31 + assert await client.generate("four") == "recovered" + assert len(primary.calls) == 3 + assert client.get_provider_health()["primary"].consecutive_failures == 0 + assert not client.get_provider_health()["primary"].cooling_down + + +async def test_opening_cooldown_stops_same_route_retries() -> None: + primary = ScriptedProvider( + "primary", + [ProviderError("primary", "temporary", retryable=True), response("should-not-run")], + ) + backup = ScriptedProvider("backup", [response("backup")]) + client = UnifiedLLM( + [Route(primary, "model-a"), Route(backup, "model-b")], + max_attempts_per_route=2, + health_failure_threshold=1, + health_cooldown=30, + ) + assert await client.generate("hello") == "backup" + assert len(primary.calls) == 1 + + +async def test_explicit_provider_selection_can_probe_cooling_route() -> None: + primary = ScriptedProvider( + "primary", + [ProviderError("primary", "temporary", retryable=True), response("probe-ok")], + ) + backup = ScriptedProvider("backup", [response("backup")]) + client = UnifiedLLM( + [Route(primary, "model-a"), Route(backup, "model-b")], + max_attempts_per_route=1, + health_failure_threshold=1, + ) + assert await client.generate("implicit") == "backup" + assert client.get_provider_health()["primary"].cooling_down + assert await client.generate("explicit", provider="primary") == "probe-ok" + assert not client.get_provider_health()["primary"].cooling_down + + +async def test_attempt_hook_receives_only_sanitized_attempts() -> None: + observed: list[Attempt] = [] + + async def observe(attempt: Attempt) -> None: + observed.append(attempt) + + primary = ScriptedProvider("primary", [ProviderError("primary", "secret", retryable=True)]) + backup = ScriptedProvider("backup", [response("safe")]) + client = UnifiedLLM( + [Route(primary, "model-a"), Route(backup, "model-b")], + max_attempts_per_route=1, + on_attempt=observe, + ) + assert await client.generate("secret prompt") == "safe" + assert [item.provider for item in observed] == ["primary", "backup"] + assert observed[0].error == "provider_error" + assert observed[0].retryable + assert observed[1].error is None + assert "secret" not in repr(observed) + + +async def test_attempt_hook_failure_does_not_break_generation() -> None: + def broken_observer(_attempt: Attempt) -> None: + raise RuntimeError("observer secret") + + provider = ScriptedProvider("primary", [response("ok")]) + client = UnifiedLLM([Route(provider, "model")], on_attempt=broken_observer) + assert await client.generate("hello") == "ok" + + +async def test_permanent_failure_does_not_open_health_cooldown() -> None: + provider = ScriptedProvider("primary", [ProviderError("primary", "bad request", retryable=False)]) + client = UnifiedLLM([Route(provider, "model")], health_failure_threshold=1) + with pytest.raises(ProviderError): + await client.generate("hello") + assert client.get_provider_health()["primary"].consecutive_failures == 0 + assert not client.get_provider_health()["primary"].cooling_down diff --git a/tests/test_router.py b/tests/test_router.py index a0507de..b40048e 100644 --- a/tests/test_router.py +++ b/tests/test_router.py @@ -251,6 +251,9 @@ def test_response_derives_total_tokens() -> None: {"max_response_bytes": True}, {"max_response_chars": 1.5}, {"max_tool_calls": False}, + {"health_failure_threshold": True}, + {"health_cooldown": -1}, + {"on_attempt": "not-callable"}, {"max_output_tokens": 0}, {"backoff_base": -1}, {"retry_jitter": 2}, diff --git a/unified_llm/__init__.py b/unified_llm/__init__.py index a1c4904..2a2eadf 100644 --- a/unified_llm/__init__.py +++ b/unified_llm/__init__.py @@ -2,6 +2,7 @@ from .unified_llm import ( Attempt, + AttemptHook, ConfigurationError, FallbackExhausted, Message, @@ -9,6 +10,7 @@ OpenAIResponsesProvider, Provider, ProviderError, + ProviderHealth, RequestValidationError, Route, ToolDefinition, @@ -22,6 +24,7 @@ __all__ = [ "Attempt", + "AttemptHook", "ConfigurationError", "FallbackExhausted", "Message", @@ -29,6 +32,7 @@ "OpenAIResponsesProvider", "Provider", "ProviderError", + "ProviderHealth", "RequestValidationError", "Route", "ToolDefinition", diff --git a/unified_llm/unified_llm.py b/unified_llm/unified_llm.py index c645f53..202f703 100644 --- a/unified_llm/unified_llm.py +++ b/unified_llm/unified_llm.py @@ -52,6 +52,25 @@ class Attempt: retryable: bool = False +@dataclass(frozen=True, slots=True) +class ProviderHealth: + """Content-free provider health snapshot maintained by the router.""" + + provider: str + consecutive_failures: int + cooling_down: bool + cooldown_remaining: float + + +@dataclass(slots=True) +class _ProviderHealthState: + consecutive_failures: int = 0 + open_until: float = 0.0 + + +AttemptHook: TypeAlias = Callable[[Attempt], Awaitable[None] | None] + + class ProviderError(UnifiedLLMError): """A sanitized provider or transport failure. @@ -634,6 +653,7 @@ def _translate_tool(tool: ToolDefinition) -> dict[str, Any]: Sleep: TypeAlias = Callable[[float], Awaitable[None]] +Clock: TypeAlias = Callable[[], float] class UnifiedLLM: @@ -656,7 +676,11 @@ def __init__( backoff_base: float = 0.25, max_retry_delay: float = 5.0, retry_jitter: float = 0.1, + health_failure_threshold: int = 3, + health_cooldown: float = 30.0, + on_attempt: AttemptHook | None = None, _sleep: Sleep = asyncio.sleep, + _clock: Clock = time.monotonic, ) -> None: self._routes = tuple(routes) if not self._routes: @@ -698,6 +722,20 @@ def __init__( raise ConfigurationError("Retry delays cannot be negative.") if not 0 <= retry_jitter <= 1: raise ConfigurationError("retry_jitter must be between 0 and 1.") + if ( + isinstance(health_failure_threshold, bool) + or not isinstance(health_failure_threshold, int) + or not 0 <= health_failure_threshold <= 100 + ): + raise ConfigurationError("health_failure_threshold must be between 0 and 100.") + if ( + isinstance(health_cooldown, bool) + or not isinstance(health_cooldown, (int, float)) + or not 0 <= health_cooldown <= 3_600 + ): + raise ConfigurationError("health_cooldown must be between 0 and 3600 seconds.") + if on_attempt is not None and not callable(on_attempt): + raise ConfigurationError("on_attempt must be callable when provided.") self.request_timeout = float(request_timeout) self.max_attempts_per_route = max_attempts_per_route @@ -711,8 +749,13 @@ def __init__( self.backoff_base = float(backoff_base) self.max_retry_delay = float(max_retry_delay) self.retry_jitter = float(retry_jitter) + self.health_failure_threshold = health_failure_threshold + self.health_cooldown = float(health_cooldown) self._semaphore = asyncio.Semaphore(max_concurrency) self._sleep = _sleep + self._clock = _clock + self._on_attempt = on_attempt + self._health = {name: _ProviderHealthState() for name in names} @classmethod def from_env( @@ -757,6 +800,10 @@ def read(name: str) -> str | None: ), max_total_attempts=_parse_env_int(read("MAX_TOTAL_ATTEMPTS"), f"{prefix}_MAX_TOTAL_ATTEMPTS", 4), max_concurrency=_parse_env_int(read("MAX_CONCURRENCY"), f"{prefix}_MAX_CONCURRENCY", 10), + health_failure_threshold=_parse_env_int( + read("HEALTH_FAILURE_THRESHOLD"), f"{prefix}_HEALTH_FAILURE_THRESHOLD", 3 + ), + health_cooldown=_parse_env_float(read("HEALTH_COOLDOWN"), f"{prefix}_HEALTH_COOLDOWN", 30.0), ) @property @@ -776,6 +823,20 @@ def get_available_providers(self) -> list[str]: return [route.provider.name for route in self._routes] + def get_provider_health(self) -> dict[str, ProviderHealth]: + """Return content-free health snapshots keyed by provider name.""" + + now = self._clock() + return { + provider: ProviderHealth( + provider=provider, + consecutive_failures=state.consecutive_failures, + cooling_down=state.open_until > now, + cooldown_remaining=max(0.0, state.open_until - now), + ) + for provider, state in self._health.items() + } + async def generate( self, prompt: str, @@ -887,6 +948,8 @@ async def chat_with_metadata( total_attempts = 0 async with self._semaphore: + if provider is None: + selected = self._prioritize_healthy_routes(selected) for route, resolved_model in selected: for route_attempt in range(1, self.max_attempts_per_route + 1): if total_attempts >= self.max_total_attempts: @@ -938,16 +1001,16 @@ async def chat_with_metadata( continue break except Exception as exc: - attempts.append( - Attempt( - route.provider.name, - resolved_model, - route_attempt, - _elapsed_ms(started), - "adapter_error", - False, - ) + attempt = Attempt( + route.provider.name, + resolved_model, + route_attempt, + _elapsed_ms(started), + "adapter_error", + False, ) + attempts.append(attempt) + await self._notify_attempt(attempt) raise ProviderError( route.provider.name, f"Provider adapter {route.provider.name!r} failed unexpectedly.", @@ -955,14 +1018,15 @@ async def chat_with_metadata( attempts=attempts, ) from exc else: - attempts.append( - Attempt( - route.provider.name, - resolved_model, - route_attempt, - _elapsed_ms(started), - ) + attempt = Attempt( + route.provider.name, + resolved_model, + route_attempt, + _elapsed_ms(started), ) + attempts.append(attempt) + self._record_success(route.provider.name) + await self._notify_attempt(attempt) return replace( response, provider=route.provider.name, @@ -981,16 +1045,16 @@ async def _handle_provider_error( started: float, attempts: list[Attempt], ) -> bool: - attempts.append( - Attempt( - route.provider.name, - model, - route_attempt, - _elapsed_ms(started), - f"http_{error.status_code}" if error.status_code is not None else "provider_error", - error.retryable, - ) + attempt = Attempt( + route.provider.name, + model, + route_attempt, + _elapsed_ms(started), + f"http_{error.status_code}" if error.status_code is not None else "provider_error", + error.retryable, ) + attempts.append(attempt) + await self._notify_attempt(attempt) if not error.retryable: message = ( f"Provider {route.provider.name!r} returned HTTP {error.status_code}." @@ -1004,6 +1068,9 @@ async def _handle_provider_error( retryable=False, attempts=attempts, ) from error + cooling_down = self._record_failure(route.provider.name) + if cooling_down: + return False if route_attempt >= self.max_attempts_per_route: return False @@ -1017,6 +1084,43 @@ async def _handle_provider_error( await self._sleep(delay) return True + def _prioritize_healthy_routes(self, selected: Sequence[tuple[Route, str]]) -> tuple[tuple[Route, str], ...]: + now = self._clock() + + def priority(item: tuple[Route, str]) -> tuple[bool, float]: + state = self._health[item[0].provider.name] + cooling_down = state.open_until > now + return cooling_down, state.open_until if cooling_down else 0.0 + + return tuple(sorted(selected, key=priority)) + + def _record_failure(self, provider: str) -> bool: + if self.health_failure_threshold == 0: + return False + state = self._health[provider] + state.consecutive_failures += 1 + if state.consecutive_failures < self.health_failure_threshold: + return False + state.open_until = self._clock() + self.health_cooldown + return self.health_cooldown > 0 + + def _record_success(self, provider: str) -> None: + state = self._health[provider] + state.consecutive_failures = 0 + state.open_until = 0.0 + + async def _notify_attempt(self, attempt: Attempt) -> None: + if self._on_attempt is None: + return + try: + result = self._on_attempt(attempt) + if inspect.isawaitable(result): + await result + except asyncio.CancelledError: + raise + except Exception: + pass + def _validate_request( self, messages: Sequence[Message], @@ -1246,6 +1350,7 @@ def _elapsed_ms(started: float) -> float: __all__ = [ "Attempt", + "AttemptHook", "ConfigurationError", "FallbackExhausted", "Message", @@ -1253,6 +1358,7 @@ def _elapsed_ms(started: float) -> float: "OpenAIResponsesProvider", "Provider", "ProviderError", + "ProviderHealth", "RequestValidationError", "Route", "ToolDefinition",