Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,10 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ['3.11', '3.12', '3.13', '3.14']
# 3.14t is the free-threaded build. The breaker's thread-safety claim
# rests on one threading.Lock; without a GIL-free interpreter nothing in
# CI can falsify it (#102).
python-version: ['3.11', '3.12', '3.13', '3.14', '3.14t']
env:
UV_PYTHON: ${{ matrix.python-version }}
INTERLOCK_TEST_REDIS_URL: redis://localhost:6379/0
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ wheels/

# Virtual environments
.venv
.venv-*

.idea/
.vscode/
Expand Down
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,18 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### Added

- The test suite now runs on **free-threaded CPython (3.14t)** in CI, alongside
3.11–3.14. interlock has always documented the breaker as thread-safe, but
every interpreter in the matrix had a GIL, so nothing could falsify that
claim. A new `tests/test_concurrency.py` drives a single breaker from many
threads at once and asserts the four properties the lock exists to provide:
window counts add up under concurrent recording, `snapshot()` never returns a
torn view, the HALF_OPEN caps (`max_concurrent_probes`,
`permitted_calls_in_half_open`) are never exceeded, and a `Registry` hands out
exactly one breaker per name. No races were found; free-threaded builds are
not a distribution target (no wheels, no classifiers) — this verifies an
existing claim rather than making a new one.

- `CircuitBreaker.close()` / `aclose()` and `Registry.close_all()` /
`aclose_all()` — a deterministic way to release a breaker's background work.
Until now a coordinated breaker's lane (a daemon thread for a sync storage,
Expand Down
26 changes: 25 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ uv sync # creates the venv and installs dev + all extras

## Running the checks

CI runs exactly these on Python 3.11–3.14. Run them locally before opening a PR:
CI runs exactly these on Python 3.11–3.14 and on the free-threaded 3.14t. Run
them locally before opening a PR:

```bash
uv run ruff format --check # formatting
Expand Down Expand Up @@ -46,6 +47,29 @@ INTERLOCK_TEST_REDIS_URL=redis://localhost:6379/0 uv run pytest

CI runs both.

### Free-threaded CPython

The breaker's thread-safety rests on a single `threading.Lock`, and a GIL-enabled
interpreter cannot falsify that claim — so CI runs the whole suite on the
free-threaded build (`3.14t`) as well. `tests/test_concurrency.py` is the part
that matters there: it drives one breaker from many threads and asserts what the
lock exists to provide (window counts add up, no torn snapshot, HALF_OPEN probe
caps hold, one name means one breaker). It runs on every interpreter, but only
on `3.14t` does failing it become likely.

Use a second environment so your main `.venv` is left alone:

```bash
UV_PROJECT_ENVIRONMENT=.venv-ft uv sync --python 3.14t
UV_PROJECT_ENVIRONMENT=.venv-ft uv run pytest
```

One caveat: `tests/test_redis_storage.py` needs a **real** server on `3.14t`.
Its `fakeredis` fallback pulls in `lupa`, whose Lua engine has not declared
free-threading support, so importing it re-enables the GIL — and
`filterwarnings = "error"` turns that warning into a collection error. Set
`INTERLOCK_TEST_REDIS_URL` (CI does) and `fakeredis` is never imported.

### Benchmarks

`benchmarks/` holds the performance suite, measured by
Expand Down
4 changes: 4 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ integrations at the transport level.
ships `py.typed` and passes mypy in strict mode.
- **Zero-dependency core.** Standard library only; everything external lives in
optional extras.
- **Thread-safe, and checked.** One `threading.Lock` guards the critical
sections; your callable runs outside it. CI drives a single breaker from many
threads at once — on free-threaded CPython (3.14t) as well as the GIL builds —
so the guarantee is verified, not merely asserted.
- **Distributed state (optional).** Coordinate tripping and recovery probing
across instances through Redis/Valkey, with graceful degradation to local
state — see the [Redis integration](integrations/redis.md).
Expand Down
4 changes: 4 additions & 0 deletions docs/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ Key facts for answering questions about interlock:
- Failure classification is a `FailureClassifier` protocol; the default treats
any raised exception as a failure. Custom classifiers add result-based rules.
- Rejection raises `CircuitOpenError(breaker_name, retry_after, last_failure)`.
- Thread-safe: one `threading.Lock` guards admission and recording, and the
protected callable runs outside it. CI runs the suite — including tests that
drive one breaker from many threads at once — on free-threaded CPython
(3.14t) as well as 3.11–3.14.
- `timeout(seconds)` is an async context manager raising `CallTimeoutError`;
`sync_timeout(seconds)` is the synchronous decorator equivalent (worker thread).
- v2.0 pipeline: `Pipeline` / `Pipeline.builder()` compose `TimeoutStrategy`,
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,9 @@ ban-relative-imports = "all"
"ARG002", # test doubles accept protocol args they intentionally ignore
"SLF001", # coordination tests drive internal lanes/coordinators deterministically
]
"tests/test_redis_storage.py" = [
"E402", # fakeredis is imported only when it is the backend (see the module)
]

[tool.mypy]
python_version = "3.11"
Expand Down
203 changes: 203 additions & 0 deletions tests/test_concurrency.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
"""Invariants that must survive real thread contention.

The rest of the suite drives one breaker from one thread, so it can only prove
the logic, never the locking. These tests run many threads through a *single*
breaker at once and assert the four properties the `threading.Lock` in
`_engine.py` exists to provide: window counts add up, no snapshot is torn,
HALF_OPEN probe caps hold, and one name means one breaker.

They pass under the GIL too, but they are weak there — bytecode-level atomicity
hides most of what they look for. Their real venue is free-threaded CPython
(3.14t), where CI runs the suite as well (#102). Nothing here sleeps: threads
rendezvous on a `Barrier` and time still comes from `FakeClock`.
"""

import contextlib
import threading
from concurrent.futures import ThreadPoolExecutor
from typing import TYPE_CHECKING

from interlock import CircuitBreaker, CircuitOpenError, Config, Registry, State

if TYPE_CHECKING:
from collections.abc import Callable

from tests.conftest import FakeClock, RecordingListener

THREADS = 8
CALLS_PER_THREAD = 24
TIMEOUT = 10.0


def _run(worker: 'Callable[[], None]', *, threads: int = THREADS) -> None:
"""Run ``worker`` on ``threads`` real threads; re-raise whatever they raise."""
with ThreadPoolExecutor(max_workers=threads) as pool:
futures = [pool.submit(worker) for _ in range(threads)]
for future in futures:
future.result(timeout=TIMEOUT)


def _boom() -> None:
raise RuntimeError('boom')


def _ok() -> None:
pass


def _alternating(breaker: CircuitBreaker, *, calls: int) -> None:
"""Run ``calls`` calls, every other one failing; swallow what comes back."""
for index in range(calls):
with contextlib.suppress(RuntimeError, CircuitOpenError):
breaker.call(_boom if index % 2 else _ok)


def test__window__concurrent_records__counts_add_up(fake_clock: 'FakeClock') -> None:
breaker = CircuitBreaker(
name='counts',
config=Config(window_size=THREADS * CALLS_PER_THREAD),
clock=fake_clock,
)
# METRICS_ONLY records every outcome but never trips, so the expected
# totals are arithmetic rather than a race with the threshold.
breaker.metrics_only()
start = threading.Barrier(THREADS)

def worker() -> None:
start.wait(timeout=TIMEOUT)
_alternating(breaker, calls=CALLS_PER_THREAD)

_run(worker)

snapshot = breaker.snapshot()
assert snapshot.total_calls == THREADS * CALLS_PER_THREAD
assert snapshot.failed_calls == THREADS * CALLS_PER_THREAD // 2


def test__snapshot__taken_during_concurrent_records__is_never_torn(
fake_clock: 'FakeClock',
) -> None:
window_size = THREADS * CALLS_PER_THREAD
breaker = CircuitBreaker(
name='torn',
config=Config(window_size=window_size),
clock=fake_clock,
)
breaker.metrics_only()
recording_done = threading.Event()
torn: list[object] = []

def read() -> None:
while not recording_done.is_set():
snapshot = breaker.snapshot()
if not (
0 <= snapshot.failed_calls <= snapshot.total_calls <= window_size
and 0 <= snapshot.slow_calls <= snapshot.total_calls
and 0.0 <= snapshot.failure_rate <= 1.0
):
torn.append(snapshot)

def write() -> None:
_alternating(breaker, calls=CALLS_PER_THREAD)

reader = threading.Thread(target=read)
reader.start()
try:
_run(write)
finally:
recording_done.set()
reader.join(timeout=TIMEOUT)

assert not reader.is_alive()
assert torn == []


def test__half_open__concurrent_probes__never_exceeds_the_caps(fake_clock: 'FakeClock') -> None:
config = Config(
window_size=2,
minimum_number_of_calls=2,
wait_duration_in_open=30.0,
permitted_calls_in_half_open=3,
max_concurrent_probes=2,
)
breaker = CircuitBreaker(name='probes', config=config, clock=fake_clock)
for _ in range(2):
with contextlib.suppress(RuntimeError):
breaker.call(_boom)
assert breaker.state is State.OPEN
fake_clock.advance(31.0)

# Every thread reaches the barrier: admitted ones while their probe is still
# in flight, rejected ones right after CircuitOpenError. So the peak is
# measured while the maximum possible number of probes is running at once.
rendezvous = threading.Barrier(THREADS)
counters = threading.Lock()
in_flight = 0
peak = 0
admitted = 0

def probe() -> None:
nonlocal in_flight, peak, admitted
with counters:
in_flight += 1
admitted += 1
peak = max(peak, in_flight)
rendezvous.wait(timeout=TIMEOUT)
with counters:
in_flight -= 1

def worker() -> None:
try:
breaker.call(probe)
except CircuitOpenError:
rendezvous.wait(timeout=TIMEOUT)

_run(worker)

assert admitted > 0, 'no probe was admitted — the test proved nothing'
assert peak <= config.max_concurrent_probes
assert admitted <= config.permitted_calls_in_half_open


def test__registry__concurrent_get__hands_out_one_breaker_per_name(
fake_clock: 'FakeClock',
) -> None:
registry = Registry(clock=fake_clock)
start = threading.Barrier(THREADS)
seen: list[CircuitBreaker] = []
collected = threading.Lock()

def worker() -> None:
start.wait(timeout=TIMEOUT)
breaker = registry.get('shared')
with collected:
seen.append(breaker)

_run(worker)

assert len(seen) == THREADS
assert len({id(breaker) for breaker in seen}) == 1


def test__trip__concurrent_failures__emits_one_state_change(
fake_clock: 'FakeClock',
listener: 'RecordingListener',
) -> None:
breaker = CircuitBreaker(
name='once',
config=Config(window_size=THREADS * CALLS_PER_THREAD, minimum_number_of_calls=2),
clock=fake_clock,
listener=listener,
)
start = threading.Barrier(THREADS)

def worker() -> None:
start.wait(timeout=TIMEOUT)
for _ in range(CALLS_PER_THREAD):
with contextlib.suppress(RuntimeError, CircuitOpenError):
breaker.call(_boom)

_run(worker)

assert breaker.state is State.OPEN
assert listener.state_changes == [(State.CLOSED, State.OPEN)]
8 changes: 7 additions & 1 deletion tests/test_redis_storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
import uuid
from collections.abc import Iterator

import fakeredis
import pytest
import redis as redis_mod
import redis.asyncio as aredis
Expand All @@ -30,6 +29,13 @@
USE_REAL_REDIS = REDIS_URL is not None
TTL = 60.0

if not USE_REAL_REDIS:
# Deferred, not top-level: fakeredis' Lua engine (lupa) does not declare
# free-threading support, so importing it on 3.14t re-enables the GIL with a
# RuntimeWarning — and filterwarnings=error turns that into a collection
# error for the whole file. A real server needs no Lua engine in-process.
import fakeredis

requires_real_redis = pytest.mark.skipif(
not USE_REAL_REDIS,
reason='atomicity is verified only against a real server (set INTERLOCK_TEST_REDIS_URL)',
Expand Down
Loading