feat: add per-tool circuit breakers

.github/workflows/ci.yml

@@ -39,8 +39,8 @@ jobs:
           cache: pip
       - run: python -m pip install --upgrade pip
       - run: python -m pip install -e ".[dev]"
-      - run: python -m black --check src tests examples benchmarks
-      - run: python -m ruff check src tests examples benchmarks
+      - run: python -m black --check src tests examples benchmarks scripts
+      - run: python -m ruff check src tests examples benchmarks scripts
       - run: python -m mypy
 
   package:
@@ -57,5 +57,4 @@ jobs:
       - run: python -m twine check --strict dist/*
       - run: python -m venv .smoke
       - run: .smoke/bin/python -m pip install --no-deps dist/*.whl
-      - run: .smoke/bin/python -c "from samsarix_core import MCPServer, ToolPolicyContext, ToolPolicyDecision, ToolRateLimit, ToolRuntime, serve_stdio, __version__; assert ToolRateLimit(calls=1, period_seconds=1).burst_capacity == 1; print(__version__)"
-      - run: .smoke/bin/python -c "from helix_core import MCPServer, ToolPolicyContext, ToolPolicyDecision, ToolRateLimit, serve_stdio, __version__ as legacy_version; from samsarix_core import __version__; assert legacy_version == __version__, f'{legacy_version} != {__version__}'"
+      - run: .smoke/bin/python scripts/smoke_check.py

.github/workflows/release.yml

@@ -43,8 +43,7 @@ jobs:
         run: |
           python -m venv .release-smoke
           .release-smoke/bin/python -m pip install --no-deps dist/*.whl
-          .release-smoke/bin/python -c "from samsarix_core import MCPServer, ToolPolicyContext, ToolPolicyDecision, ToolRateLimit, ToolRuntime, serve_stdio, __version__; assert ToolRateLimit(calls=1, period_seconds=1).burst_capacity == 1; print(__version__)"
-          .release-smoke/bin/python -c "from helix_core import MCPServer, ToolPolicyContext, ToolPolicyDecision, ToolRateLimit, serve_stdio, __version__ as legacy_version; from samsarix_core import __version__; assert legacy_version == __version__, f'{legacy_version} != {__version__}'"
+          .release-smoke/bin/python scripts/smoke_check.py
       - name: Require the tag to match the package version
         if: startsWith(github.ref, 'refs/tags/v')
         env:

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## Unreleased
 
+### Added
+
+- opt-in process-local per-tool consecutive-failure circuit breakers with safe
+  fail-fast results, one half-open recovery probe, queued-permit invalidation,
+  manual inspection/reset, content-free metrics/lifecycle events, and consistent
+  direct, batch, MCP, and task behavior.
+
 ### Changed
 
 - recorded immutable `v2.0.0a6` release checksums, provenance, and clean installed-wheel

MANIFEST.in

@@ -10,5 +10,6 @@ include TRADEMARKS.md
 recursive-include docs *.md
 recursive-include examples *.py
 recursive-include benchmarks *.py
+recursive-include scripts *.py
 prune legacy
 global-exclude __pycache__ *.py[cod]

README.md

@@ -18,11 +18,11 @@ persistence, or an untrusted-code sandbox.
 - turns annotated sync or async functions into inspectable tool contracts;
 - emits JSON Schema Draft 2020-12 input and output schemas;
 - validates arguments and outputs without surprising scalar coercion;
-- returns structured success, validation, policy-denial, overload, rate-limit, timeout,
-  missing-tool, and failure results;
+- returns structured success, validation, policy-denial, overload, rate-limit,
+  open-circuit, timeout, missing-tool, and failure results;
 - bounds pending invocations, registry growth, batches, value size/complexity,
-  global and per-tool concurrent work, sustained per-tool request rate, and
-  thread-pool use;
+  global and per-tool concurrent work, sustained per-tool request rate, repeated
+  calls to failing dependencies, and thread-pool use;
 - supports ordered batch invocation and cooperative async cancellation;
 - optionally requires a bounded host-owned policy decision after validation and
   before any tool code executes;
@@ -111,6 +111,8 @@ snapshot and returns `ToolPolicyDecision.ALLOW` or `DENY`; it is not an authenti
 service or a durable human-approval workflow.
 For a quota-constrained dependency with safe retry handling, run
 `python examples/rate_limited_api.py`.
+For a failing dependency with fail-fast recovery probing, run
+`python examples/circuit_breaker_api.py`.
 
 ## Connect an MCP client
 
@@ -192,6 +194,7 @@ exception. It returns a `ToolResult` with one of these states:
 - `denied`
 - `busy`
 - `rate_limited`
+- `circuit_open`
 - `timed_out`
 - `failed`
 - `runtime_closed`
@@ -217,13 +220,17 @@ them to the host's actual workload.
 Hosts can isolate a slow or quota-constrained dependency when registering its tool:
 
 ```python
-from samsarix_core import ToolRateLimit
+from samsarix_core import ToolCircuitBreaker, ToolRateLimit
 
 runtime = ToolRuntime(max_concurrency=8)
 runtime.register(
     query_warehouse,
     max_concurrency=2,
     rate_limit=ToolRateLimit(calls=60, period_seconds=60, burst=5),
+    circuit_breaker=ToolCircuitBreaker(
+        failure_threshold=3,
+        recovery_timeout_seconds=30,
+    ),
 )
 runtime.register(health_check)
 ```
@@ -240,6 +247,14 @@ warehouse calls to execute concurrently. An empty bucket returns retryable statu
 process-local deployment policy, not tenant fairness or a distributed quota service. See
 [per-tool rate limits](docs/RATE_LIMITS.md).
 
+The circuit breaker counts consecutive tool/output failures and caller-visible
+timeouts. Once open, it rejects before capacity or tokens are consumed, then admits one
+half-open recovery probe after 30 seconds. Invalid input, host policy outcomes,
+admission/rate rejections, progress-handler failure, and caller cancellation do not
+trip it. Core does not automatically retry calls. Breaker state is process-local
+deployment policy and can be inspected or manually reset by the host. See
+[per-tool circuit breakers](docs/CIRCUIT_BREAKERS.md).
+
 Hosts can attach a synchronous `lifecycle_handler` to receive paired, immutable start
 and terminal events across direct, batch, MCP, and task calls. Events contain only
 invocation ID, requested tool name, status, UTC time, and terminal duration; arguments,
@@ -272,7 +287,7 @@ for delivery semantics, privacy/cardinality cautions, and a content-free OpenTel
 See [Getting started](docs/GETTING_STARTED.md), the [API reference](docs/API_REFERENCE.md),
 [architecture](docs/ARCHITECTURE.md), [MCP bridge](docs/MCP.md),
 [lifecycle observability](docs/OBSERVABILITY.md), [best practices](docs/BEST_PRACTICES.md),
-[per-tool rate limits](docs/RATE_LIMITS.md),
+[per-tool rate limits](docs/RATE_LIMITS.md), [per-tool circuit breakers](docs/CIRCUIT_BREAKERS.md),
 [benchmark guide](docs/BENCHMARKS.md),
 the [adoption record](docs/ADOPTION.md), and the [productization
 record](docs/PRODUCTIZATION.md).

ROADMAP.md

@@ -36,6 +36,10 @@ remain separate decisions.
 - Tool registrations can also apply process-local token buckets immediately before
   execution, protecting sustained downstream request quotas with safe retry hints
   without treating the runtime as a distributed or per-tenant quota service.
+- Tool registrations can opt into process-local consecutive-failure circuit breakers
+  that fail fast before capacity and rate tokens, invalidate queued stale permits,
+  allow one half-open recovery probe, and expose safe results plus host inspection/reset
+  without automatic retry or distributed-health claims.
 - Strict `TypedDict` input and output contracts now preserve named nested fields,
   descriptions, and required/optional key semantics in JSON Schema and runtime
   validation.
@@ -84,6 +88,8 @@ remain separate decisions.
   evidence.
 - [x] Publish immutable GitHub prerelease `v2.0.0a6` with per-tool rate limiting,
   independent consumer, clean-install, checksum, and SLSA provenance evidence.
+- [ ] Prove independent consumer adoption and publish an immutable prerelease for the
+  per-tool circuit-breaker contract after exact-head review and clean-wheel evidence.
 
 ## Samsarix adoption
 

docs/API_REFERENCE.md

@@ -72,10 +72,12 @@ ToolRuntime(
 )
 ```
 
-- `register(function, *, replace=False, max_concurrency=None, rate_limit=None) -> ToolSpec`
+- `register(function, *, replace=False, max_concurrency=None, rate_limit=None, circuit_breaker=None) -> ToolSpec`
 - `await invoke(name, arguments=None, *, timeout=None, progress_handler=None) -> ToolResult`
 - `await invoke_many(calls) -> list[ToolResult]`
 - `metrics() -> RuntimeMetrics`
+- `circuit_state(name) -> ToolCircuitState | None`
+- `reset_circuit(name) -> bool`
 - `pending_sync_calls -> int`
 - `await wait_for_sync(*, timeout=None) -> bool`
 - `await aclose(*, wait_for_sync=False, timeout=None) -> bool`
@@ -122,6 +124,17 @@ per-tenant quota. Token counts are positive integers no larger than `2**53 - 1`;
 period and derived refill/retry magnitudes must be positive and finite. See
 [Per-tool rate limits](RATE_LIMITS.md).
 
+`register(..., circuit_breaker=ToolCircuitBreaker(failure_threshold=N,
+recovery_timeout_seconds=S))` adds a process-local consecutive-execution-failure
+breaker to the exact registration. It fails fast with retryable status `circuit_open`
+and safe code `tool_circuit_open`, admits one half-open probe after the recovery
+interval, and closes after a successful validated output. Invalid input, policy and
+admission outcomes, rate rejection, progress-handler failure, and caller cancellation
+do not count; tool/output failures and caller-visible timeouts do. The breaker is
+checked before capacity and again before execution so a queued stale permit cannot run
+after another call opens it. `circuit_state()` inspects configured state and
+`reset_circuit()` forces it closed. See [Per-tool circuit breakers](CIRCUIT_BREAKERS.md).
+
 Argument and output sizes are the UTF-8 byte length of compact JSON. The root is
 depth zero; every child increments depth. Each container and scalar is one node,
 while object keys are covered by the byte limit but do not count as nodes. Cyclic,
@@ -255,6 +268,9 @@ All public models are frozen, slotted dataclasses.
 - `ToolRateLimit`: positive token allocation, finite refill period, and optional burst
   capacity. `burst_capacity` resolves the default and `to_dict()` returns normalized
   deployment-local configuration.
+- `ToolCircuitBreaker`: positive consecutive-failure threshold and finite positive
+  recovery interval for one process-local exact registration.
+- `ToolCircuitState`: `closed`, `open`, or `half_open`.
 - `ToolPolicyContext`: invocation ID plus detached validated arguments and tool spec;
   unlike result models, it intentionally has no serialization helper.
 - `ToolPolicyDecision`: explicit `allow` or `deny` policy outcome.
@@ -269,13 +285,13 @@ All public models are frozen, slotted dataclasses.
 - `ToolError`: code, safe message, optional exception type/details, and retryable flag.
 - `ToolProgress`: numeric progress, optional total, and optional human-readable message.
 - `RuntimeMetrics`: content-free counters only, including policy denials, aggregate
-  process-wide rate-limit rejections, and runtime saturation.
+  process-wide rate/circuit rejections and breaker trips, and runtime saturation.
 - `ToolStatus`: `success`, `not_found`, `invalid_arguments`, `denied`, `busy`,
-  `rate_limited`, `timed_out`, `failed`, and `runtime_closed`.
+  `rate_limited`, `circuit_open`, `timed_out`, `failed`, and `runtime_closed`.
 - `TaskSupport`: the `"forbidden" | "optional" | "required"` public type alias.
 
-`ToolSpec`, `ToolRateLimit`, `ToolResult`, `ToolError`, `ToolLifecycleEvent`, and
-`RuntimeMetrics` provide `to_dict()`.
+`ToolSpec`, `ToolRateLimit`, `ToolCircuitBreaker`, `ToolResult`, `ToolError`,
+`ToolLifecycleEvent`, and `RuntimeMetrics` provide `to_dict()`.
 
 ## Supported annotations
 

docs/ARCHITECTURE.md

@@ -16,7 +16,9 @@ invocation: incoming call
     -> iterative argument resource preflight
     -> ToolRuntime argument validation
     -> optional bounded host policy decision
+    -> optional per-tool circuit permit
     -> optional per-tool execution bulkhead
+    -> queued circuit-permit revalidation
     -> optional per-tool token-bucket start check
     -> bounded async or thread-pool execution
     -> optional invocation-scoped progress handler
@@ -33,9 +35,9 @@ invocation: incoming call
   iterative resource checks, strict input validation, and JSON normalization.
 - `registry.py` stores a capped set of compiled callable contracts behind a small
   thread-safe map.
-- `runtime.py` owns concurrency, process-local per-tool rate controls, the sync thread
-  pool, timeouts, cancellation, exception redaction, batch workers, lifecycle, and
-  content-free counters.
+- `runtime.py` owns concurrency, process-local per-tool circuit/rate controls, the sync
+  thread pool, timeouts, cancellation, exception redaction, batch workers, lifecycle,
+  and content-free counters.
 - `progress.py` owns invocation-scoped async progress validation, ordering,
   resource caps, and lifecycle closure.
 - `_mcp_tasks.py` owns finite in-memory task retention, secure identifiers, TTL
@@ -97,6 +99,14 @@ global and per-tool semaphores still bound execution. This prevents workers queu
 one constrained tool from head-of-line blocking unrelated calls that fit within the
 batch's available pending capacity.
 
+An optional circuit breaker is also bound to the exact registration and checked before
+capacity is acquired. A generation-bound permit is revalidated at the execution
+boundary; opening or manually resetting a circuit invalidates queued permits and late
+outcomes. Breaker outcome is committed before async or thread-backed capacity is
+released. Open state admits no work until its monotonic recovery interval elapses, then
+reserves exactly one half-open probe. The runtime performs no automatic retries and no
+background health checks.
+
 An exact registration may also own an event-loop-local token bucket. After validation,
 policy, and concurrency acquisition, the runtime checks one token immediately before
 starting tool code. It never waits for a token. Rejection releases the acquired slots,

docs/BEST_PRACTICES.md

@@ -56,8 +56,8 @@ this tool-specific slot before a global execution slot, preserving unrelated too
 availability while the constrained tool queues.
 
 The per-tool concurrency limit is an in-process execution bulkhead, not a request-rate
-limit, tenant quota, circuit breaker, or process sandbox. When the dependency also has a
-sustained request quota, add `rate_limit=ToolRateLimit(...)`. The token bucket is checked
+limit, tenant quota, or process sandbox. When the dependency also has a sustained
+request quota, add `rate_limit=ToolRateLimit(...)`. The token bucket is checked
 immediately before tool start and returns a safe retry delay instead of waiting. Combine
 both controls: concurrency protects simultaneous downstream work while rate protects
 starts over time. Keep total admission finite and set downstream I/O deadlines.
@@ -66,6 +66,14 @@ for the reliability trade-offs and complementary controls.
 See [per-tool rate limits](RATE_LIMITS.md) for token accounting, safe retry behavior,
 primary references, and process-local limitations.
 
+When repeatedly calling a known-failing dependency would waste capacity or amplify an
+incident, add `circuit_breaker=ToolCircuitBreaker(...)`. Choose the consecutive-failure
+threshold and recovery interval from observed failure modes and recovery time. Monitor
+both trips and open-circuit rejections. Keep dependency connect/read/write deadlines:
+the breaker reacts to observed outcomes and does not interrupt a hung operation by
+itself. It also does not retry. See [per-tool circuit breakers](CIRCUIT_BREAKERS.md) for
+counting, half-open probes, manual reset, primary references, and process-local limits.
+
 Choose the runtime-wide `max_concurrency` from aggregate downstream capacity, not CPU count alone. Set
 `max_pending_invocations` to the total policy/execution work one process can safely
 hold; monitor `busy` and `peak_pending_invocations` to find sustained saturation.
@@ -105,8 +113,11 @@ Branch on `ToolStatus`; do not infer success from a truthy output. A `busy` resu
 retryable because no tool or policy code ran, but use capped exponential backoff with
 jitter rather than retrying immediately. A `rate_limited` result also means tool code
 did not run; wait at least `error.details["retry_after_ms"]`, add bounded jitter for
-competing callers, and remember that the hint does not reserve the next token. Other
-failures remain non-retryable because a timed-out sync function may still finish and
+competing callers, and remember that the hint does not reserve the next token. A
+`circuit_open` result also means this call did not run; its retry delay may be absent
+while another recovery probe is active. Retry only after bounded backoff and only when
+the tool's side-effect semantics allow it. Other failures remain non-retryable because
+a timed-out sync function may still finish and
 cause its side effect. Apply any broader retry policy only when the tool's semantics
 make that safe.