Thiendev 14/4

apps/tracking-api/src/services/delivery-job-dispatcher.ts

@@ -121,17 +121,23 @@ class BullMqDeliveryJobDispatcher implements DeliveryJobDispatcher {
     private readonly destinations: readonly DeliveryDestination[]
   ) {}
 
+  private buildJobId(eventId: string, destination: DeliveryDestination, replayTag?: string): string {
+    const eventPart = encodeURIComponent(eventId);
+    const destinationPart = encodeURIComponent(destination);
+    const replayPart = replayTag ? `__replay__${encodeURIComponent(replayTag)}` : '';
+    return `${eventPart}__${destinationPart}${replayPart}`;
+  }
+
   async enqueueDeliveryJobs(input: {
     eventId: string;
     destinations: readonly DeliveryDestination[];
     replayTag?: string;
   }): Promise<void> {
     const requestedAt = new Date().toISOString();
-    const jobIdSuffix = input.replayTag ? `:replay:${input.replayTag}` : '';
 
     await Promise.all(
       input.destinations.map(async (destination) => {
-        const jobId = `${input.eventId}:${destination}${jobIdSuffix}`;
+        const jobId = this.buildJobId(input.eventId, destination, input.replayTag);
         await this.queue.add(
           ROUTER_JOB_NAME,
           {

apps/tracking-api/test/delivery-job-dispatcher.test.ts

@@ -96,7 +96,7 @@ describe('delivery-job-dispatcher', () => {
       });
       expect(call[2]).toMatchObject({
         ...getDeliveryJobOptions(destination),
-        jobId: `evt-123:${destination}`
+        jobId: `evt-123__${destination}`
       });
     }
   });

apps/tracking-console/src/widgets/events/ui/events-workbench.tsx

@@ -177,7 +177,8 @@ export function EventsWorkbench() {
   const detailQuery = useQuery({
     queryKey: ['admin-event-detail', selectedEventId],
     queryFn: () => trackingApi.getEventDetail(selectedEventId ?? ''),
-    enabled: Boolean(selectedEventId)
+    enabled: Boolean(selectedEventId),
+    refetchInterval: selectedEventId ? 5_000 : false
   });
 
   const detail = detailQuery.data?.item;

docs/12-roadmap/daily-progress.md

@@ -425,3 +425,35 @@ This file tracks day-by-day execution progress for roadmap phases.
 - Delivery rows remain mostly `queued` until queue/worker path is active with destination dispatch; canary execution evidence still pending
 - Next step (next working day):
 - run canary window with worker path enabled, capture delivery status movement and parity evidence, then proceed rollback rehearsal and final go/no-go capture
+
+## 2026-04-14 (update 02)
+
+- Owner: team
+- Phase: Phase 7 - UAT and Cutover
+- Status: in_progress
+- Completed today:
+- Executed `docs/07-ops/queue-worker-go-live-checklist.md` end-to-end in local run order: Redis -> API -> Worker -> Console
+- Fixed queue enqueue blocker with minimal patch: BullMQ custom `jobId` no longer contains `:` (root cause of events stuck at `queued`)
+- Completed smoke verification for `/health` and `/ready` with `{"status":"ok"}` and `{"status":"ready"}`
+- Sent test event and verified state progression from `queued` to `failed` via worker retries when Google endpoint was intentionally unavailable
+- Executed replay for failed destination and verified final state moved to `success` after bringing up a local mock Google endpoint
+- Evidence:
+- `apps/tracking-api/src/services/delivery-job-dispatcher.ts` (jobId sanitizer for BullMQ compatibility)
+- `apps/tracking-api/.env` (`REDIS_URL=redis://127.0.0.1:6379/0`)
+- `apps/router-worker/.env` (worker env completion for `DATABASE_URL`, `REDIS_URL`, `ROUTER_QUEUE_NAME`, `ROUTER_WORKER_CONCURRENCY`)
+- `redis-cli -h 127.0.0.1 -p 6379 ping` -> `PONG`
+- `curl http://127.0.0.1:3000/health` -> `{"status":"ok"}`
+- `curl http://127.0.0.1:3000/ready` -> `{"status":"ready"}`
+- `POST /track` for `evt-e2e-fail-1776156544` -> accepted (`status: ok`)
+- `GET /admin/events?limit=5&offset=0` -> event appears in list (Events tab data source)
+- `GET /admin/events/evt-e2e-fail-1776156544` timeline:
+- initial: `deliveryOverallStatus="queued"` with `google=retrying`
+- after retries: `deliveryOverallStatus="failed"` with `google=failed`
+- `POST /admin/events/evt-e2e-fail-1776156544/replay` -> `replayedDestinations=["google"]`
+- `GET /admin/events/evt-e2e-fail-1776156544` after replay -> all destinations `delivered`, `deliveryOverallStatus="success"`
+- Worker log evidence: retry/fail then replay success for job ids `evt-e2e-fail-1776156544__google` and replay-tagged `...__replay__...`
+- Mock endpoint evidence: received replay payload at `POST /google` with same canonical `event_id`
+- Blockers:
+- none for local queue/worker smoke path; end-to-end status transitions and replay were validated
+- Next step (next working day):
+- investigate why one legacy event remained `queued` (created before jobId fix), then add a short operator note in ops docs for replaying historical stuck events after queue config bugs are fixed

docs/13-prompts/00-master-system-execution.md

@@ -0,0 +1,29 @@
+# 00 - Master System Execution (Senior)
+
+You are a Senior Engineer for `Tracking_Base_System`.
+
+Goal:
+Deliver the system to production-ready quality without unnecessary refactors.
+
+Execution rules:
+
+1. Read current docs first (`docs/README.md`, `docs/07-ops/*`, `docs/12-roadmap/*`).
+2. Work in small logical commits.
+3. Keep behavior backward-compatible unless explicitly changing contract.
+4. Prefer minimal-risk edits over broad rewrites.
+5. Always provide evidence after implementation.
+
+Required output format:
+
+1. What changed (files)
+2. Why changed (risk/impact)
+3. Evidence (commands + key results)
+4. Remaining risks
+5. Next step
+
+Done criteria:
+
+- Build passes
+- Relevant tests pass
+- Runtime flow verified for changed scope
+- Docs updated where behavior changed

docs/13-prompts/01-queue-worker-go-live.md

@@ -0,0 +1,26 @@
+# 01 - Queue Worker Go-Live
+
+You are a Senior Engineer.
+
+Goal:
+Make queue/worker flow run end-to-end so events move out of `queued`.
+
+Tasks:
+
+1. Follow `docs/07-ops/queue-worker-go-live-checklist.md`
+2. Validate env for API and worker
+3. Start Redis, API, worker, console in correct order
+4. Send test events and verify status transition `queued -> success/failed`
+5. If failed, replay from console/admin endpoint and verify update
+6. Record evidence and blockers
+
+Constraints:
+
+- No large refactor
+- Fix only blockers to go-live path
+
+Done criteria:
+
+- /health and /ready are healthy
+- Delivery states update correctly
+- Replay works for failed deliveries

docs/13-prompts/02-api-feature-delivery.md

@@ -0,0 +1,26 @@
+# 02 - API Feature Delivery
+
+You are a Senior Backend Engineer.
+
+Goal:
+Implement one API feature safely with tests and docs.
+
+Tasks:
+
+1. Define/confirm request and response contract
+2. Implement route + service/repository changes
+3. Add unit/integration tests for success + failure + validation
+4. Keep error messages actionable
+5. Update docs in `docs/04-api/*`
+
+Non-negotiables:
+
+- Idempotency where required
+- Backward compatibility
+- No secret leakage in logs
+
+Done criteria:
+
+- `npm run -w @tracking-base/tracking-api build` passes
+- Target tests pass
+- API docs updated

docs/13-prompts/03-worker-reliability-and-replay.md

@@ -0,0 +1,20 @@
+# 03 - Worker Reliability and Replay
+
+You are a Senior Reliability Engineer.
+
+Goal:
+Improve worker reliability with safe retries and replay behavior.
+
+Tasks:
+
+1. Classify errors into retryable vs terminal
+2. Tune attempts/backoff per destination
+3. Ensure delivery state writes are consistent
+4. Ensure replay does not create duplicate dangerous side effects
+5. Add tests for retry/fail/replay paths
+
+Done criteria:
+
+- Worker tests pass
+- Failed paths are observable
+- Replay path is operational and auditable

docs/13-prompts/04-frontend-console-ops.md

@@ -0,0 +1,25 @@
+# 04 - Frontend Console Operations
+
+You are a Senior Frontend Engineer.
+
+Goal:
+Make the console usable for daily operator workflows.
+
+Tasks:
+
+1. Improve Events UX (filters, sort, pagination, detail)
+2. Keep API calls typed and resilient
+3. Show clear loading/error/empty states
+4. Support replay actions for failed deliveries
+5. Avoid breaking existing routes
+
+Constraints:
+
+- Keep architecture layers (`app/pages/widgets/features/entities/shared`)
+- No UI-only fake behavior for critical ops actions
+
+Done criteria:
+
+- Typecheck passes
+- Build passes
+- Event investigation flow works end-to-end

docs/13-prompts/05-security-hardening.md

@@ -0,0 +1,20 @@
+# 05 - Security Hardening
+
+You are a Senior Security-minded Engineer.
+
+Goal:
+Harden auth and operational safety without blocking development flow.
+
+Tasks:
+
+1. Enforce admin endpoint protection (`/admin/*`)
+2. Keep `/track` auth mode explicit and documented
+3. Ensure no privileged token is sent from browser to non-admin endpoints
+4. Add tests for auth allow/deny paths
+5. Update security docs
+
+Done criteria:
+
+- Auth tests pass
+- Security docs updated
+- Dev mode behavior is explicit and visible

docs/13-prompts/06-test-and-quality-gate.md

@@ -0,0 +1,18 @@
+# 06 - Test and Quality Gate
+
+You are a Senior Engineer responsible for quality.
+
+Goal:
+Define and execute a practical quality gate for this repo.
+
+Tasks:
+
+1. Run build + relevant tests for changed scopes
+2. Add missing tests for regressions found
+3. Keep CI workflow aligned with actual project scripts
+4. Produce short pass/fail summary with actionable failures
+
+Done criteria:
+
+- CI-ready command list is clear
+- Reproducible pass status for current changes

docs/13-prompts/07-release-and-handover.md

@@ -0,0 +1,18 @@
+# 07 - Release and Handover
+
+You are a Senior Engineer preparing handover.
+
+Goal:
+Ship safely and leave clear handover notes.
+
+Tasks:
+
+1. Summarize shipped scope and impact
+2. List migration or env changes required
+3. Provide rollback notes
+4. Update daily progress and runbooks
+5. Suggest next 3 concrete tasks
+
+Done criteria:
+
+- Another engineer can run and operate the system without guessing

docs/13-prompts/README.md

@@ -0,0 +1,21 @@
+# Prompt Templates (Senior)
+
+Use these templates to drive implementation of this repository end-to-end.
+
+How to use:
+
+1. Start with `00-master-system-execution.md`
+2. Pick one focused prompt for the current sprint/task
+3. Keep constraints and done criteria unchanged for predictable quality
+4. Ask for concrete evidence output (commands, logs, screenshots, file diffs)
+
+Templates:
+
+- `00-master-system-execution.md`
+- `01-queue-worker-go-live.md`
+- `02-api-feature-delivery.md`
+- `03-worker-reliability-and-replay.md`
+- `04-frontend-console-ops.md`
+- `05-security-hardening.md`
+- `06-test-and-quality-gate.md`
+- `07-release-and-handover.md`