Add health-aware routing and content-safe observability

.env.example

@@ -10,6 +10,8 @@ UNIFIED_LLM_PROVIDER=primary
 
 # Optional resource and retry bounds
 UNIFIED_LLM_TIMEOUT=30
+UNIFIED_LLM_HEALTH_COOLDOWN=30
+UNIFIED_LLM_HEALTH_FAILURE_THRESHOLD=3
 UNIFIED_LLM_MAX_ATTEMPTS_PER_ROUTE=2
 UNIFIED_LLM_MAX_TOTAL_ATTEMPTS=4
 UNIFIED_LLM_MAX_CONCURRENCY=10

CHANGELOG.md

@@ -9,6 +9,8 @@ All notable changes to this project will be documented here. The project follows
 - Configurable raw and normalized response byte limits, response character limits, and tool-call count limits.
 - Runtime validation for normalized responses returned by every provider adapter.
 - A stateless-by-default OpenAI Responses API adapter with text, refusal, function-call, and tool-result normalization.
+- Content-free sync/async attempt observation and inspectable provider health snapshots.
+- Cross-request transient-failure cooldown that deprioritizes unhealthy routes without removing last-resort fallback.
 
 ### Changed
 

README.md

@@ -116,7 +116,7 @@ async def main() -> None:
 asyncio.run(main())
 ```
 
-Automatic routing uses every route in order. Passing `provider="backup"` selects one route and disables automatic fallback. A model override with multiple routes also requires an explicit provider, preventing accidental cross-provider model dispatch.
+Automatic routing starts in configured route order; routes in an active health cooldown move behind healthy routes. Passing `provider="backup"` selects one route, bypasses health reordering, and disables automatic fallback. A model override with multiple routes also requires an explicit provider, preventing accidental cross-provider model dispatch.
 
 ## Failure contract
 
@@ -129,6 +129,10 @@ The SDK raises instead of returning an ambiguous empty string:
 
 `asyncio.CancelledError` is propagated immediately and is never retried. Each provider attempt is available as sanitized `Attempt` metadata on successful responses or failure exceptions.
 
+For metrics and tracing, pass `on_attempt=` a sync or async callback. It receives only the sanitized `Attempt` record—never messages, response content, credentials, headers, or endpoint URLs. Callback failures are isolated from generation; cancellation still propagates.
+
+After three consecutive transient failures by default, a provider enters a 30-second cooldown and is moved behind healthy routes. It remains a last-resort fallback, and an explicit `provider=` selection can probe it immediately. Successful probes reset its health. Inspect content-free state with `get_provider_health()`.
+
 ## Configuration
 
 `UnifiedLLM.from_env()` supports:
@@ -143,9 +147,11 @@ The SDK raises instead of returning an ambiguous empty string:
 | `UNIFIED_LLM_MAX_ATTEMPTS_PER_ROUTE` | No | `2` | Attempts per route, 1–5. |
 | `UNIFIED_LLM_MAX_TOTAL_ATTEMPTS` | No | `4` | Total request attempts, 1–16. |
 | `UNIFIED_LLM_MAX_CONCURRENCY` | No | `10` | In-flight requests, 1–1000. |
+| `UNIFIED_LLM_HEALTH_FAILURE_THRESHOLD` | No | `3` | Consecutive transient failures before route cooldown; `0` disables. |
+| `UNIFIED_LLM_HEALTH_COOLDOWN` | No | `30` | Seconds to deprioritize an unhealthy route, maximum 3600. |
 | `UNIFIED_LLM_ALLOW_INSECURE_HTTP` | No | `false` | Explicitly allow trusted remote plain HTTP. |
 
-See [.env.example](.env.example). Programmatic construction additionally controls input/output limits, retry delay, and custom non-auth headers.
+See [.env.example](.env.example). Programmatic construction additionally controls input/output limits, retry delay, the sanitized attempt hook, and custom non-auth headers.
 
 ## Development and verification
 

ROADMAP.md

@@ -16,6 +16,7 @@ Current disposition: Already integrated; roadmap changes should use an ordinary
 - The productization SHA is already the remote default and its hosted checks pass.
 - Completed in the next hardening increment: account for the exact outbound body, cap inbound response bytes and normalized structures, and validate custom adapters.
 - Completed in the following API increment: add a stateless-by-default OpenAI Responses adapter without weakening the shared bounds or fallback contract.
+- Completed in the health increment: expose content-free attempt observation and deprioritize repeatedly failing routes during a bounded cooldown.
 - Next: prove one canonical consumer and a capped live endpoint before publication.
 - Review priority: prove one consumer, green wheel CI, one capped live endpoint, and trusted publication.
 

docs/API.md

@@ -62,11 +62,16 @@ UnifiedLLM(
     backoff_base=0.25,
     max_retry_delay=5.0,
     retry_jitter=0.1,
+    health_failure_threshold=3,
+    health_cooldown=30.0,
+    on_attempt=None,
 )
 ```
 
 Routes are tried in order when neither `provider` nor `model` selects a single destination.
 
+Repeated transient failures increment provider-local health state. At `health_failure_threshold`, that provider is moved behind healthy routes for `health_cooldown` seconds. Set the threshold to `0` to disable health-aware ordering. Explicit provider selection preserves operator control and bypasses reordering.
+
 ### `UnifiedLLM.from_env(prefix="UNIFIED_LLM", *, environ=None)`
 
 Builds one route from the variables documented in the README. Passing an explicit mapping as `environ` makes configuration deterministic in tests.
@@ -81,7 +86,7 @@ All methods are async.
 - `chat_with_metadata(..., tools=None) -> UnifiedLLMResponse`
 - `chat_with_tools(messages, *, tools, ...) -> UnifiedLLMResponse`
 
-Messages are mappings with a supported `role` (`assistant`, `developer`, `system`, `tool`, or `user`) and non-empty string `content`. Additional JSON-serializable message fields are preserved for compatible endpoints.
+Messages are mappings with a supported `role` (`assistant`, `developer`, `system`, `tool`, or `user`) and non-empty string `content`. An assistant turn may instead contain a non-empty `tool_calls` list for function continuation. Additional JSON-serializable message fields are preserved for compatible endpoints.
 
 Convenience methods raise the same typed errors as metadata methods; they never convert errors to empty strings.
 
@@ -102,6 +107,8 @@ Convenience methods raise the same typed errors as metadata methods; they never
 
 Contains `provider`, `model`, one-based route-attempt `number`, `latency_ms`, sanitized `error`, and `retryable`. It intentionally excludes endpoints, headers, prompts, response bodies, and exception text from unexpected adapters.
 
+`on_attempt` accepts a sync or async callback receiving this value after every completed attempt. Callback exceptions are ignored, except cancellation. `get_provider_health() -> dict[str, ProviderHealth]` returns provider name, consecutive transient failures, cooldown state, and remaining cooldown seconds without application content.
+
 ## Errors
 
 All package errors derive from `UnifiedLLMError`.

docs/ARCHITECTURE.md

@@ -30,7 +30,9 @@ The Responses adapter shares the same bounded HTTP transport and error classific
 
 The router validates the exact provider payload before network I/O, including the resolved model and generation fields, then holds one concurrency permit for the logical request. It validates and bounds normalized results from built-in and custom adapters before returning them. Each route receives at most `max_attempts_per_route`; the request receives at most `max_total_attempts`. Retry delay uses bounded exponential backoff, optional jitter, and a capped `Retry-After` value.
 
-Permanent provider failures stop immediately. Transient failures can retry and move to the next route. Cancellation propagates immediately. Unexpected custom-adapter exceptions are wrapped without copying their text.
+Permanent provider failures stop immediately. Transient failures can retry and move to the next route. Repeated transient failures maintain provider-local health state; cooling providers move behind healthy routes while remaining last-resort fallbacks. Explicit route selection is never silently overridden. Cancellation propagates immediately. Unexpected custom-adapter exceptions are wrapped without copying their text.
+
+The optional attempt observer receives only immutable sanitized metadata. Observer failures cannot turn a successful provider response into an application failure, and the core performs no logging or exporter I/O itself.
 
 ## Trust boundaries
 
@@ -55,7 +57,7 @@ application data and configuration (trusted host)
 
 ## Resource and cost bounds
 
-Defaults are 30 seconds per attempt, two attempts per route, four attempts total, ten concurrent logical requests, 200,000 message-content characters, 1,000,000 exact serialized request bytes, 2,000,000 normalized response bytes, 1,000,000 response characters, 128 tool calls, 32,768 output tokens, and a five-second maximum retry delay. The built-in HTTP adapter also caps raw success bodies at 2,000,000 bytes. Constructor validation sets absolute safety ceilings.
+Defaults are 30 seconds per attempt, two attempts per route, four attempts total, ten concurrent logical requests, 200,000 message-content characters, 1,000,000 exact serialized request bytes, 2,000,000 normalized response bytes, 1,000,000 response characters, 128 tool calls, 32,768 output tokens, a five-second maximum retry delay, and a 30-second health cooldown after three consecutive transient failures. The built-in HTTP adapter also caps raw success bodies at 2,000,000 bytes. Constructor validation sets absolute safety ceilings.
 
 Retries are not perfectly idempotent: a provider can finish generation while the response is lost. The bounded attempt count limits amplification; applications with stricter budgets should use one attempt per route and lower token limits.
 

examples/observability.py

@@ -0,0 +1,34 @@
+"""Export sanitized attempt metrics without exposing application content."""
+
+from __future__ import annotations
+
+import asyncio
+from collections import Counter
+
+from unified_llm import Attempt, Route, UnifiedLLM, UnifiedLLMResponse
+
+
+class DemoProvider:
+    name = "demo"
+
+    async def complete(self, **_kwargs: object) -> UnifiedLLMResponse:
+        return UnifiedLLMResponse(content="ok", model="demo-model", provider=self.name)
+
+
+async def main() -> None:
+    counters: Counter[tuple[str, str]] = Counter()
+
+    def observe(attempt: Attempt) -> None:
+        outcome = attempt.error or "success"
+        counters[(attempt.provider, outcome)] += 1
+
+    client = UnifiedLLM([Route(DemoProvider(), "demo-model")], on_attempt=observe)
+    async with client:
+        await client.generate("This text is never passed to observe().")
+
+    print(counters)
+    print(client.get_provider_health())
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

tests/test_configuration.py

@@ -29,11 +29,15 @@ def test_from_env_builds_a_local_route_without_a_key() -> None:
             "UNIFIED_LLM_MAX_ATTEMPTS_PER_ROUTE": "1",
             "UNIFIED_LLM_MAX_TOTAL_ATTEMPTS": "1",
             "UNIFIED_LLM_MAX_CONCURRENCY": "2",
+            "UNIFIED_LLM_HEALTH_FAILURE_THRESHOLD": "4",
+            "UNIFIED_LLM_HEALTH_COOLDOWN": "12.5",
         }
     )
     assert client.get_available_providers() == ["local"]
     assert client.routes[0].model == "local-model"
     assert client.request_timeout == 5
+    assert client.health_failure_threshold == 4
+    assert client.health_cooldown == 12.5
 
 
 def test_from_env_requires_model_and_default_endpoint_key() -> None:
@@ -51,6 +55,10 @@ def test_from_env_requires_model_and_default_endpoint_key() -> None:
             {"UNIFIED_LLM_MODEL": "m", "UNIFIED_LLM_API_KEY": "k", "UNIFIED_LLM_MAX_CONCURRENCY": "x"},
             "integer",
         ),
+        (
+            {"UNIFIED_LLM_MODEL": "m", "UNIFIED_LLM_API_KEY": "k", "UNIFIED_LLM_HEALTH_COOLDOWN": "x"},
+            "number",
+        ),
         (
             {"UNIFIED_LLM_MODEL": "m", "UNIFIED_LLM_API_KEY": "k", "UNIFIED_LLM_ALLOW_INSECURE_HTTP": "x"},
             "boolean",

tests/test_health_observability.py

@@ -0,0 +1,123 @@
+from __future__ import annotations
+
+import pytest
+
+from unified_llm import Attempt, ProviderError, Route, UnifiedLLM
+
+from .conftest import ScriptedProvider, response
+
+
+class FakeClock:
+    def __init__(self) -> None:
+        self.now = 100.0
+
+    def __call__(self) -> float:
+        return self.now
+
+
+async def test_cooling_provider_is_deprioritized_then_recovers() -> None:
+    clock = FakeClock()
+    primary = ScriptedProvider(
+        "primary",
+        [
+            ProviderError("primary", "temporary", retryable=True),
+            ProviderError("primary", "temporary", retryable=True),
+            response("recovered"),
+        ],
+    )
+    backup = ScriptedProvider("backup", [response("backup-1"), response("backup-2"), response("backup-3")])
+    client = UnifiedLLM(
+        [Route(primary, "model-a"), Route(backup, "model-b")],
+        max_attempts_per_route=1,
+        health_failure_threshold=2,
+        health_cooldown=30,
+        _clock=clock,
+    )
+
+    assert await client.generate("one") == "backup-1"
+    assert await client.generate("two") == "backup-2"
+    health = client.get_provider_health()
+    assert health["primary"].consecutive_failures == 2
+    assert health["primary"].cooling_down
+    assert health["primary"].cooldown_remaining == 30
+
+    assert await client.generate("three") == "backup-3"
+    assert len(primary.calls) == 2
+
+    clock.now += 31
+    assert await client.generate("four") == "recovered"
+    assert len(primary.calls) == 3
+    assert client.get_provider_health()["primary"].consecutive_failures == 0
+    assert not client.get_provider_health()["primary"].cooling_down
+
+
+async def test_opening_cooldown_stops_same_route_retries() -> None:
+    primary = ScriptedProvider(
+        "primary",
+        [ProviderError("primary", "temporary", retryable=True), response("should-not-run")],
+    )
+    backup = ScriptedProvider("backup", [response("backup")])
+    client = UnifiedLLM(
+        [Route(primary, "model-a"), Route(backup, "model-b")],
+        max_attempts_per_route=2,
+        health_failure_threshold=1,
+        health_cooldown=30,
+    )
+    assert await client.generate("hello") == "backup"
+    assert len(primary.calls) == 1
+
+
+async def test_explicit_provider_selection_can_probe_cooling_route() -> None:
+    primary = ScriptedProvider(
+        "primary",
+        [ProviderError("primary", "temporary", retryable=True), response("probe-ok")],
+    )
+    backup = ScriptedProvider("backup", [response("backup")])
+    client = UnifiedLLM(
+        [Route(primary, "model-a"), Route(backup, "model-b")],
+        max_attempts_per_route=1,
+        health_failure_threshold=1,
+    )
+    assert await client.generate("implicit") == "backup"
+    assert client.get_provider_health()["primary"].cooling_down
+    assert await client.generate("explicit", provider="primary") == "probe-ok"
+    assert not client.get_provider_health()["primary"].cooling_down
+
+
+async def test_attempt_hook_receives_only_sanitized_attempts() -> None:
+    observed: list[Attempt] = []
+
+    async def observe(attempt: Attempt) -> None:
+        observed.append(attempt)
+
+    primary = ScriptedProvider("primary", [ProviderError("primary", "secret", retryable=True)])
+    backup = ScriptedProvider("backup", [response("safe")])
+    client = UnifiedLLM(
+        [Route(primary, "model-a"), Route(backup, "model-b")],
+        max_attempts_per_route=1,
+        on_attempt=observe,
+    )
+    assert await client.generate("secret prompt") == "safe"
+    assert [item.provider for item in observed] == ["primary", "backup"]
+    assert observed[0].error == "provider_error"
+    assert observed[0].retryable
+    assert observed[1].error is None
+    assert "secret" not in repr(observed)
+
+
+async def test_attempt_hook_failure_does_not_break_generation() -> None:
+    def broken_observer(_attempt: Attempt) -> None:
+        raise RuntimeError("observer secret")
+
+    provider = ScriptedProvider("primary", [response("ok")])
+    client = UnifiedLLM([Route(provider, "model")], on_attempt=broken_observer)
+    assert await client.generate("hello") == "ok"
+
+
+async def test_permanent_failure_does_not_open_health_cooldown() -> None:
+    provider = ScriptedProvider("primary", [ProviderError("primary", "bad request", retryable=False)])
+    client = UnifiedLLM([Route(provider, "model")], health_failure_threshold=1)
+    with pytest.raises(ProviderError):
+        await client.generate("hello")
+    assert client.get_provider_health()["primary"].consecutive_failures == 0
+    assert not client.get_provider_health()["primary"].cooling_down

tests/test_router.py

@@ -251,6 +251,9 @@ def test_response_derives_total_tokens() -> None:
         {"max_response_bytes": True},
         {"max_response_chars": 1.5},
         {"max_tool_calls": False},
+        {"health_failure_threshold": True},
+        {"health_cooldown": -1},
+        {"on_attempt": "not-callable"},
         {"max_output_tokens": 0},
         {"backoff_base": -1},
         {"retry_jitter": 2},

unified_llm/__init__.py

@@ -2,13 +2,15 @@
 
 from .unified_llm import (
     Attempt,
+    AttemptHook,
     ConfigurationError,
     FallbackExhausted,
     Message,
     OpenAICompatibleProvider,
     OpenAIResponsesProvider,
     Provider,
     ProviderError,
+    ProviderHealth,
     RequestValidationError,
     Route,
     ToolDefinition,
@@ -22,13 +24,15 @@
 
 __all__ = [
     "Attempt",
+    "AttemptHook",
     "ConfigurationError",
     "FallbackExhausted",
     "Message",
     "OpenAICompatibleProvider",
     "OpenAIResponsesProvider",
     "Provider",
     "ProviderError",
+    "ProviderHealth",
     "RequestValidationError",
     "Route",
     "ToolDefinition",