CrashQueueLab is a deterministic fault-injection and invariant-testing laboratory for durable job queues. It makes worker crash windows observable, replays append-only transition history, and checks whether persisted queue state still satisfies its correctness model.
It is intentionally not another Celery, RQ, Dramatiq, or Huey clone. The SQLite queue is a small, inspectable system under test; the differentiator is the evidence produced around failures.
A worker commits an external effect and then crashes before acknowledging queue success. The lease expires, recovery schedules another attempt, and a second worker executes the job again. CrashQueueLab runs that exact schedule twice:
- A non-idempotent destination records two durable effects.
- A destination protected by a unique effect key records one durable effect.
Both runs contain two executions. The result demonstrates an effectively-once side effect without claiming exactly-once execution.
flowchart LR
P["Producer"] -->|"submit"| Q["SQLite durable queue"]
Q -->|"leased claim"| W1["Worker A"]
W1 -->|"commit effect"| E["Separate effect ledger"]
E -. "crash before acknowledgement" .-> X["Process exits"]
Q -->|"lease expiry and recovery"| W2["Worker B"]
W2 -->|"repeat effect"| E
W2 -->|"acknowledge"| Q
Q --> H["Append-only history"]
H --> R["Replay and invariant checker"]
CrashQueueLab requires Python 3.11 or newer and has no runtime dependencies.
git clone https://github.com/b2ty9t7yhz-source/crashqueuelab.git
cd crashqueuelab
python3 -m venv venv
source venv/bin/activate
python -m pip install -e ".[dev]"
python -m pytestRun the complete comparison in one command. The output directory must be new or empty so previous artifacts cannot be mistaken for the current run.
crashqueuelab demo --output-directory reports/demoThe deterministic summary is written to reports/demo/comparison.json:
{
"fault_point": "AFTER_EFFECT_COMMIT",
"idempotent": {
"attempt_count": 2,
"effect_count": 1,
"final_state": "SUCCEEDED",
"invariants_ok": true
},
"non_idempotent": {
"attempt_count": 2,
"effect_count": 2,
"final_state": "SUCCEEDED",
"invariants_ok": true
},
"queue_delivery_semantics": "at-least-once execution"
}The full output includes two queue databases, two effect databases, both event histories, per-scenario JSON reports, and the compact comparison. Generated databases and reports are ignored by Git.
- Durable SQLite job, attempt, and event storage in WAL mode
- Idempotent submission using canonical JSON and a payload SHA-256 digest
- Deterministic claim ordering and
BEGIN IMMEDIATEwrite transactions - Worker ID, opaque lease-generation token, heartbeat, expiry, and stale-worker fencing
- Capped exponential retry, maximum attempts, and dead-letter state
- Append-only transition history protected by SQLite triggers
- Independent history replay and database-wide invariant checking
- Thirteen named fault hooks across handler and transaction boundaries
- Abrupt child-process exit and concurrent multi-process claim tests
- Separate idempotent and non-idempotent synthetic effect ledgers
- Typed Python API, structured JSON CLI errors, and deterministic reports
The five durable states and every legal guard are documented in the state machine.
CrashQueueLab provides at-least-once execution behavior. An expired, unacknowledged lease can be executed again; therefore duplicate execution is an expected outcome, not a hidden edge case.
The project deliberately distinguishes:
| Term | V1 position |
|---|---|
| At-most-once execution | Not used; it can lose work after an early acknowledgement |
| At-least-once execution | Implemented through leases, recovery, and retry |
| Exactly-once execution | Not claimed |
| Idempotent processing | Demonstrated with a stable unique effect key |
| Effectively-once side effect | Demonstrated even though execution occurs twice |
See Delivery semantics for the two-commit crash window and the same-database exception.
The JSON CLI exposes submission, claim, heartbeat, success, failure, retry promotion, expired-lease recovery, history, replay, and invariant checking.
crashqueuelab submit \
--database demo/queue.sqlite \
--idempotency-key request-1 \
--job-type synthetic.effect \
--payload '{"effect_key":"effect-1","value":7}'
crashqueuelab history --database demo/queue.sqlite --job-id JOB_ID
crashqueuelab replay --database demo/queue.sqlite --job-id JOB_ID
crashqueuelab check --database demo/queue.sqliteThe package also exposes immutable typed models and a direct API:
from crashqueuelab import DurableQueue, ManualClock
clock = ManualClock(1_000_000)
queue = DurableQueue("demo/queue.sqlite", clock=clock)
submitted = queue.submit(
idempotency_key="request-1",
job_type="synthetic.effect",
payload={"effect_key": "effect-1", "value": 7},
)
claimed = queue.claim(worker_id="worker-a", lease_duration_us=100_000)
assert claimed is not None
queue.succeed(
job_id=submitted.job.job_id,
worker_id="worker-a",
lease_token=claimed.lease_token or "",
)Handlers are trusted in-process callables. The project never serializes or executes arbitrary Python functions from the database.
CI rejects changes unless all of these gates pass:
| Gate | Evidence enforced by CI |
|---|---|
| Compatibility | Full suite on Python 3.11, 3.12, and 3.13 |
| Fault correctness | Deterministic exception hooks and abrupt os._exit recovery |
| Concurrency | Spawned workers race through a process barrier, not sleep |
| Coverage | Branch coverage must remain at or above 95% |
| Resource safety | Unclosed SQLite ResourceWarning values fail the suite |
| Static analysis | Ruff lint/format and strict mypy |
| Packaging | Wheel and source distribution build in isolation |
| Installation | A clean virtual environment installs and runs the built wheel |
Run the same checks locally:
python -m pytest --cov --cov-report=term-missing
python -m ruff check .
python -m ruff format --check .
python -m mypy
python -m build- Architecture
- Correctness model and invariants
- State machine
- Fault model and crash points
- Delivery semantics
- Design decisions and tradeoffs
- Testing strategy
- Contributing
V1 uses Python, SQLite, synthetic payloads, and a local filesystem. It does not use Redis, PostgreSQL, Kafka, Docker, Kubernetes, a web dashboard, untrusted jobs, or network filesystems.
The tests do not prove behavior under physical disk corruption, power loss, filesystem durability violations, network partitions, clock skew, or production-scale load. These limits are intentional and documented rather than hidden behind reliability claims.
CrashQueueLab is available under the MIT License.