fix(workers): cancellable shutdown wait and per-tick watchdog (#75)

[JumpTechCode]

Closes #75.

Problem

run_forever did await tick() then await sleep(interval), checking stop only at the top of the loop:

1. Unresponsive shutdown. Setting stop does not interrupt an in-flight await asyncio.sleep(interval). The idempotency reaper's interval defaults to 3600s, so a SIGTERM arriving just after a tick waits up to an hour before run_workers returns and engine.dispose() runs — past any orchestrator grace period, so the process is SIGKILLed and clean disposal is skipped.
2. No per-tick watchdog. A tick that hangs on a slow query or lock wait pins the worker indefinitely with no cancellation and no chance to observe stop, silently wedging background reconciliation.

Both slipped through because the unit tests injected deterministic sleep/stop seams that sidestepped real asyncio.sleep blocking.

Fix

• Cancellable interval wait: when stop is present, wait with asyncio.wait_for(stop.wait(), timeout=interval) — stop firing mid-wait exits promptly; the interval elapsing (TimeoutError) is a normal boundary. Shutdown latency drops from up to a full interval to ~0.
• Per-tick watchdog: run each tick under asyncio.wait_for(tick(), timeout=tick_timeout); an overrunning tick is cancelled, logged at WARNING, and the loop recovers next interval. Timeout is Settings.worker_tick_timeout_s (default 120s), a generous backstop. Cancelling mid-tick is safe — every per-item unit of work is its own bounded transaction (ADR-0017), so no partial state is left behind.

The driver's seams change from sleep to run_tick/wait, keeping orchestration deterministic; the real _run_tick and _wait_interval helpers are unit-tested directly for the watchdog-cancel and stop-interrupt behaviors.

Scope

Implements the two core fixes (interruptible sleep + watchdog). The optional "fixed wall-clock cadence to avoid drift" and explicit gather cancellation from #75 are deliberately omitted — the interruptible wait already makes the stop event prompt, making gather cancellation redundant.

Documented in ADR-0027 (refines ADR-0017).

Tests

• test_worker_loop.py rewritten: _wait_interval returns early on stop / interrupts mid-wait; _run_tick watchdog cancels a hung tick and swallows the timeout; run_forever orchestration via fast seam fakes.
• test_run_workers.py asserts all three loops are armed with the 120s watchdog.
• test_settings.py covers the new default.

make verify green locally: 511 passed, 99.10% coverage.

🤖 Generated with Claude Code