Skip to content

Phase 0: arq task queue, durable webhooks, single-active scheduler - #3

Merged
vorflux[bot] merged 4 commits into
mainfrom
vorflux/phase0-foundations-queue-scheduler
Jul 22, 2026
Merged

Phase 0: arq task queue, durable webhooks, single-active scheduler#3
vorflux[bot] merged 4 commits into
mainfrom
vorflux/phase0-foundations-queue-scheduler

Conversation

@crewcricle

@crewcricle crewcricle commented Jul 22, 2026 •
edited by vorflux Bot
Loading

Copy link
Copy Markdown
Contributor

Phased foundations for the LocalMate Feature-Parity Initiative: arq/Redis task queue, durable webhook processing, single-active scheduler, and dead-letter reliability.

Changes

  • arq task queue (task_queue.py, worker.py): durable inbound processing tasks (Stripe/GBP/menu), outbound wrappers for all 5 integrations (Twilio, Resend, GBP, Square, DataForSEO) with retry+backoff+dead-letter, 6 cron entrypoints, and mandatory 5-min reconciliation job.
  • Durable webhooks (routers/webhooks.py): persist inbound events to webhook_events with idempotency (unique on provider, idempotency_key), dedup before provider work, return 200 fast, enqueue processing via arq. GBP inbound decodes the Pub/Sub push envelope (base64 message.data) using messageId as the idempotency key, fetches the review resource with token decrypt+refresh before enqueue.
  • Single-active scheduler (scheduler.py): all 6 cron jobs are now enqueue-only (APScheduler triggers arq enqueue, no inline business logic). main.py role-branches via WORKER_ROLE: web starts only the arq pool; scheduler starts APScheduler enqueue-only + arq pool; worker runs via arq worker.WorkerSettings.
  • Reconciliation (utils/reconcile.py): mandatory reconcile_pending_webhooks recovers stale pending and stale processing leases (using processing_started_at, not created_at). Runs every 5 minutes via arq cron.
  • Migrations 010–012: webhook_events (with processing_started_at lease + RLS), dead_letter, 5 booking-credential columns on clients. Pre-existing 003_reviews.sql RLS policy fix (FOR INSERT USING → WITH CHECK).
  • Deploy: localmate-redis (AOF, noeviction) + localmate-worker + localmate-scheduler services in deploy/docker-compose.yml; deploy script + DEPLOY.md updated; Doppler vars documented.

Testing

Test Command Result
Unit suite (88 tests) cd backend && uv run pytest tests/ -q 88 passed, 2 pre-existing warnings
Migration apply (fresh Postgres) Apply 001–012 in order on isolated postgres:16 container All 13 applied cleanly; RLS policies + constraints verified
Worker load check arq worker WorkerSettings --burst / --check against real Redis Imports cleanly; --burst drains (exit 0); --check correct
Lifespan role branching test_lifespan.py (2 tests) 2/2 passed
Compose sanity docker compose -f deploy/docker-compose.yml config All 4 services resolve with correct WORKER_ROLE
C10 staging gate bash scripts/phase0_staging_gate.sh (real Redis + Postgres) Exit 0; all 6 acceptance checks passed

C10 staging gate checks (automated, real services)

  1. Migrations 001–012 applied, tables + RLS present
  2. Deterministic _job_id dedupes duplicate enqueue
  3. Real process_stripe_event drains to done + records lease
  4. Real post_gbp_reply_task setup-failure retries → dead-letter (attempts=5)
  5. Real reconcile_pending_webhooks recovers stale processing + re-enqueues stale pending
  6. Scheduler registers 6 unique cron jobs, enqueues exactly one each, restart resumes same set

Deferred to post-deploy: real Stripe/GBP live calls, multi-container scheduler behavior, actual droplet deployment. Per agreed workflow the pre-PR gate is green unit tests + static verification — all passed.

Attached Images

[unit_test_run_045234a.log]

[migration_apply_045234a.log]

[worker_load_check_045234a.log]

[test_lifespan_045234a.log]

[compose_config_045234a.log]

[phase0_staging_gate_045234a.log]

View in Vorflux Session

Session Details

  • Session: View Session
  • Requested by: Unknown
  • Address comments on this PR. Add (aside) to your comment to have me ignore it.

@crewcricle
Phase 0: arq task queue, durable webhooks, single-active scheduler, m…
15461bf 
…igrations

- Add arq>=0.26.0 + config settings (redis_url, worker_role, dashboard_url,
  stripe_portal_config_id)
- New backend/task_queue.py (arq WorkerSettings, get_arq_pool, record_dead_letter,
  inbound processing tasks, durable outbound wrappers for all 5 integrations
  Twilio/Resend/GBP/Square/DataForSEO, cron entrypoints) + backend/worker.py
  (module named task_queue, not queue, to avoid shadowing stdlib queue that
  redis imports)
- Migrations 010_webhook_events, 011_dead_letter (both +RLS service_role_all),
  012_booking_credentials (moved to Phase 0 per C1)
- Refactor routers/webhooks.py: persist inbound events to webhook_events,
  dedupe by idempotency key, return 200 fast, enqueue processing. GBP inbound
  decodes Pub/Sub push envelope, uses messageId as key, fetches review resource
  before drafting (C3)
- Refactor scheduler.py to enqueue-only; main.py lifespan branches on worker_role
  (web = arq pool only; scheduler = APScheduler enqueue-only + arq pool)
- Mandatory utils/reconcile.py reconcile_pending_webhooks (every 5 min, C4)
- Deploy: localmate-redis (AOF, noeviction, 128MB), localmate-worker,
  localmate-scheduler on deploy_default; WORKER_ROLE=web on backend; update
  deploy_backend.sh, DEPLOY.md, Dockerfile, .env.example (REDIS_URL, WORKER_ROLE,
  STRIPE_PORTAL_CONFIG_ID, DASHBOARD_URL)
- Tests: webhook_events, dead_letter, queue_tasks, scheduler_enqueue,
  outbound_durable, reconcile, gbp_envelope_decode; new stub env in conftest
- pytest pythonpath config so tests collect without PYTHONPATH
@crewcricle
refactor(phase0): remove dead code, dedupe imports, fix stale docstring
5deb346 
- task_queue: drop unused _mark_event bump param; hoist datetime import
- reconcile: fix stale queue.WorkerSettings docstring ref -> task_queue
- main: dedupe get_arq_pool import across lifespan branches; cache logger
- tests: remove unused imports/params (json, insert_id, redundant webhooks imports)
@crewcricle
fix(phase0): address code-review blocking issues (C4 reliability, C10…
3933b1f 
… gate)

Item 1 (CRITICAL): arq retries + dead-lettering — RETRY_BACKOFF, _retry_defer,
  raise arq.Retry on non-final failures, dead-letter on MAX_TRIES; drop
  reliance on ctx["max_tries"] (not present in arq ctx).
Item 2: GBP fetch decrypts access+refresh tokens, refreshes empty, retries once
  on 401; raises on transport/auth failure instead of silently returning {}.
Item 3: dedup webhook_events SELECT before provider fetch in inbound_review.
Item 4: keep secrets out of Redis/AOF — tasks take client_id, decrypt in-worker
  (post_gbp_reply_task, square_sync_task, _load_client, _resolve_gbp_access_token).
Item 5: single reconcile trigger (arq cron only); deterministic _job_id dedup,
  stale-processing recovery, atomic pending claim.
Item 6: only ack duplicate on unique-violation (23505); re-raise other DB errors.
Item 7: propagate outbound failures — strict email/dataforseo variants, menu
  dispatch inspects sync result, holiday-skip treated as success.
Item 8 (C10): real Postgres+Redis staging gate (scripts/phase0_staging_gate.*),
  asyncpg dev dep, DEPLOY.md acceptance checks.
Add test_lifespan.py for main.py worker_role branching; update affected tests.
@crewcricle
fix(phase0): durable outbound setup failures, lease-based stale recov…
045234a 
…ery, real staging gate

Fix 1 (REVIEW BLOCKER): move client loading + credential resolution INSIDE the
coroutine passed to _run_outbound for post_gbp_reply_task and square_sync_task,
so DB outages / missing clients / decrypt / token-refresh failures get arq
retry + dead-letter instead of failing bare. Add retry-then-dead-letter tests for
client-load and decrypt failures on both tasks.

Fix 2 (REVIEW BLOCKER): add processing_started_at lease column to webhook_events
(migration 010, amended in place). Record it on the atomic pending->processing
claim, clear it on retry reset, and base stale-processing recovery on it instead
of created_at so an actively-running event is never reclaimed. Add test that a
row with old created_at but fresh lease stays processing.

Fix 3 (REVIEW BLOCKER): rewrite phase0_staging_gate to import and run the real
task_queue.FUNCTIONS (process_stripe_event, post_gbp_reply_task) and the real
reconcile_pending_webhooks against real Redis/Postgres via a psycopg Supabase
shim, instead of substitute tasks / replicated SQL. Add the single-scheduler
enqueue-exactly-once + restart/resume check using the real create_scheduler.

Fix 4 (TESTING FIND): change FOR INSERT USING to WITH CHECK in 003_reviews.sql
so the INSERT policy is valid Postgres and a clean-DB migration apply works.

Tests: 88 passed.
@vercel

vercel Bot commented Jul 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
localmate Error Error Jul 22, 2026 9:48am

@vorflux
vorflux Bot merged commit f3258c9 into main Jul 22, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@crewcricle