task: Production Test Launch: Phase 1 Validation & Smoke Testing

[clduab11]

Production Test Launch: Phase 1 Validation & Smoke Testing

Epic Context

This issue implements the critical path toward PriceHawk's first production test launch, focusing on validating core detection accuracy, worker reliability, and notification delivery across the multi-channel architecture. [github](https://github.com/clduab11/pricehawk)

Objectives

• Validate end-to-end anomaly detection pipeline with real retailer data
• Confirm BullMQ worker stability under production-like load
• Verify notification delivery across all 8 channels (Discord, Email, SMS, Telegram, WhatsApp, Twitter/X, Facebook, Webhooks)
• Establish baseline KPIs for detection latency, false positive rate, and affiliate conversion tracking
• Deploy containerized stack to Railway staging environment

Technical Scope

1. Pre-Flight Infrastructure Validation

Acceptance Criteria:

• PostgreSQL connection pooling configured for concurrent workers (min 10, max 30 connections)
• Redis persistence enabled with AOF for BullMQ job recovery
• Docker Compose health checks passing for all 6 workers (anomaly-validator, notification-sender, deal-verifier, social-poster, newsletter-digest, telegram-bot)
• Railway deployment smoke test: verify app + worker containers start successfully
• Environment variable validation: all required API keys present in .env (OpenRouter, Clerk, Stripe, Twilio, Resend, Telegram)

2. Detection Algorithm Calibration

Acceptance Criteria:

• Test all 5 anomaly detection methods (Decimal Error, Z-Score, Double MAD, Adjusted IQR, Category Thresholds) against historical pricing dataset
• Validate category-specific sensitivity tuning:
  • Grocery: 30% threshold
  • Electronics/Computers: 40%
  • Fashion: 60%
  • Home/Toys: 50-55%
• Measure false positive rate < 15% on test corpus of 100 "normal sale" prices
• Confirm DeepSeek V3 AI validation reduces false positives by ≥50% in secondary filtering
• Document detection latency: target <30 seconds from price scrape to anomaly flag

3. Scraping Reliability & Rate Limiting

Acceptance Criteria:

• Playwright scraper handles JavaScript-heavy sites (Amazon, Best Buy, Walmart) without timeouts
• Jina Reader API fallback triggers correctly when Playwright fails (timeout >15s or 403 response)
• Implement exponential backoff for rate-limited endpoints (max 3 retries, base delay 2s)
• Verify scraper respects robots.txt and avoids triggering bot detection (User-Agent rotation enabled)
• Test concurrent scraping: 20 products across 5 retailers simultaneously without failures

4. Worker Queue Stability

Acceptance Criteria:

• BullMQ workers process jobs in FIFO order with <5% queue backlog during peak load
• Dead letter queue (DLQ) configured for failed jobs after 3 retry attempts
• Monitor job completion times:
  • Anomaly Validator: <10s median
  • Deal Verifier: <15s median
  • Notification Sender: <5s median
• Confirm workers auto-scale: test with 50 simultaneous anomalies, verify no job starvation
• Implement graceful shutdown: workers finish in-flight jobs before terminating (max 30s timeout)

5. Notification Channel Integration Tests

Acceptance Criteria:

• Discord webhook delivers rich embeds with price charts, affiliate links, and expiration countdown
• Email via Resend renders HTML template correctly (test in Gmail, Outlook, Apple Mail)
• SMS via Twilio includes shortened link and retailer name (test with ≤160 characters)
• Telegram bot /deals command returns last 5 active anomalies with inline buttons
• WhatsApp template message approved and delivered without delays
• Twitter/X OAuth posts deals with #PricingError hashtag and affiliate link
• Facebook Graph API posts to designated page with image attachment
• Webhook endpoint accepts JSON payload and returns 200 status within 3s

6. Affiliate Monetization Validation

Acceptance Criteria:

• Amazon affiliate tags correctly injected into product URLs (format: tag=pricehawk-20)
• Verify click tracking via Stripe Payment Links for subscription tiers (Free, Pro, Elite)
• Test affiliate link expiration: confirm 24-hour validity window enforced
• Implement revenue attribution: associate affiliate clicks with user IDs for commission tracking
• Document conversion funnel: anomaly detected → notification sent → link clicked → purchase attributed

7. Deal Lifecycle & Expiration Logic

Acceptance Criteria:

• Deal Verifier worker re-scrapes active deals every 30 minutes
• Auto-expire deals when price returns to within 10% of historical average
• Send "Deal Expired" notifications to users who subscribed to specific anomaly alerts
• Test temporal context: anomalies detected during 2-5 AM maintenance window flagged as "low confidence"
• Verify chart generation: price history visualization includes anomaly marker and confidence score

8. Monitoring & Observability

Acceptance Criteria:

• Integrate Sentry for error tracking (production environment configured)
• Set up Grafana dashboard for real-time metrics:
  • Anomalies detected per hour
  • Worker job completion rates
  • Notification delivery success rate per channel
  • API response times (OpenRouter, Jina Reader, Twilio)
• Configure alerts for critical failures:
  • Worker offline >5 minutes
  • Database connection pool exhausted
  • Redis out of memory
  • False positive rate >25%
• Implement audit logging: all anomaly detections, notifications, and affiliate clicks persisted to Prisma DB

Testing Strategy

1. Unit Tests: Validate anomaly detection algorithms in isolation (Vitest framework)
2. Integration Tests: Test worker queue processing with mocked scraper responses
3. End-to-End Tests: Simulate live retailer scrape → anomaly detection → notification delivery
4. Load Tests: Generate 100 concurrent anomalies to stress-test worker scaling
5. User Acceptance Testing (UAT): Deploy to 5 beta testers, collect feedback on notification UX

Rollout Plan

1. Day 1-3: Infrastructure setup + worker configuration
2. Day 4-7: Algorithm calibration + scraping reliability tests
3. Day 8-10: Notification channel integration + affiliate link validation
4. Day 11-12: Monitoring dashboards + error alerting
5. Day 13-14: UAT with beta cohort
6. Day 15: Production deployment to Railway with 10-retailer watchlist

Success Metrics (KPIs)

• Detection Accuracy: ≥85% precision (true positives / total flagged anomalies)
• Notification Latency: <60 seconds from anomaly detection to user notification
• Worker Uptime: ≥99.5% availability during 7-day test period
• Affiliate Conversion: ≥3% click-through rate on shared links
• False Positive Rate: <15% (validated by AI + manual review)

Dependencies

• OpenRouter API quota sufficient for DeepSeek V3 inference (estimate: 500 requests/day)
• Railway deployment credits allocated for staging + production environments
• Twilio SMS credits loaded (minimum $20 for test phase)
• Stripe test mode configured for subscription tier validation

Documentation Requirements

• Update docs/core-flow-prod.md with production deployment checklist
• Document anomaly detection tuning parameters in docs/architecture/detection-algorithms.md
• Create runbook for worker failures: docs/guides/troubleshooting-workers.md
• Add monitoring dashboard screenshots to docs/guides/observability.md

Exit Criteria

• All acceptance criteria met with documented evidence (logs, screenshots, test reports)
• Zero critical bugs in 48-hour soak test with live retailer data
• Beta tester feedback average rating ≥4/5 for notification relevance and timeliness
• Production deployment approved by stakeholders with go/no-go checklist completed

---

Estimated Effort: 60-80 hours over 15 days
Priority: P0 - Blocks production launch
Labels: production-readiness, testing, infrastructure, monitoring
Assignee: @clduab11
Milestone: Test Launch - Phase 1

This issue provides clear acceptance criteria, measurable KPIs, technical validation steps, and a phased rollout plan to systematically prepare PriceHawk for production deployment while maintaining engineering rigor.

[linear]

FIN-26

[traycerai]

Plan

Production Test Launch: Phase 1 Implementation Plan

Observations

The PriceHawk codebase has a solid foundation with all core detection algorithms implemented (Decimal Error, Z-Score, Double MAD, Adjusted IQR, Category Thresholds), category-specific tuning configured, and 8 notification channels integrated. The architecture uses Redis Streams for worker communication and BullMQ for delayed notifications. However, critical production infrastructure components are missing: Docker health checks, Redis AOF persistence, comprehensive testing, monitoring dashboards, and Railway deployment configuration. The deal verifier worker has mocked logic requiring actual re-scraping implementation.

Approach

The plan follows a phased validation approach starting with infrastructure hardening, then algorithm calibration and testing, followed by integration validation, and finally monitoring setup. Each phase builds upon the previous, ensuring systematic validation of all acceptance criteria. The approach prioritizes critical path items (infrastructure, detection accuracy, notification delivery) before secondary concerns (monitoring dashboards, documentation). Testing will be conducted in layers: unit tests for algorithms, integration tests for workers, and end-to-end tests for the complete pipeline.

Implementation Steps

Phase 1: Pre-Flight Infrastructure Validation (Days 1-3)

1.1 PostgreSQL Connection Pooling Configuration

• Verify file:src/db/index.ts connection pool settings (currently min 10, max configurable via DATABASE_POOL_SIZE)
• Update file:.env.example to document recommended pool sizes: development (10), staging (20), production (30)
• Add connection pool monitoring to file:src/lib/monitoring/metrics.ts to track active/idle connections
• Test concurrent worker load by running all 6 workers simultaneously and monitoring pool exhaustion

1.2 Redis Persistence & Health Checks

• Modify file:docker-compose.yml to add Redis AOF persistence:

  redis:
    command: redis-server --appendonly yes --appendfsync everysec
    volumes:
      - redis_data:/data

• Add health checks to all services in file:docker-compose.yml:
  • PostgreSQL: pg_isready -U pricehawk
  • Redis: redis-cli ping
  • App: HTTP check on /api/health
  • Workers: Custom health check scripts checking Redis stream connectivity
• Implement health check endpoint in file:src/app/api/health/route.ts that validates database, Redis, and critical API keys

1.3 Worker Container Configuration

• Add missing workers to file:docker-compose.yml:
  • worker-newsletter: Run tsx src/workers/newsletter-digest.ts
  • worker-telegram: Run tsx src/workers/telegram-bot.ts
• Configure graceful shutdown handlers in all worker files to finish in-flight jobs within 30s timeout
• Add restart policies (restart: unless-stopped) to all worker services
• Verify BullMQ queue configuration in file:src/workers/notification-sender.ts and file:src/workers/social-poster.ts uses correct Redis connection settings

1.4 Environment Variable Validation

• Create validation script file:scripts/validate-env.ts that checks all required API keys from file:.env.example:
  • OpenRouter, Clerk, Stripe, Twilio, Resend, Telegram, Discord, Facebook, WhatsApp, Twitter
• Add startup validation to main app and each worker to fail fast if critical keys are missing
• Document tier-specific requirements (e.g., Stripe keys only needed for paid features)

1.5 Railway Deployment Configuration

• Create file:railway.json or file:railway.toml with service definitions for app + 6 workers
• Configure environment variable groups in Railway dashboard for staging/production
• Set up PostgreSQL and Redis add-ons in Railway
• Create deployment smoke test script that verifies all containers start and health checks pass

sequenceDiagram
    participant Env as Environment Validator
    participant DB as PostgreSQL Pool
    participant Redis as Redis (AOF)
    participant Workers as 6 Workers
    participant Health as Health Checks
    
    Env->>Env: Validate API Keys
    Env->>DB: Test Connection Pool (10-30)
    Env->>Redis: Verify AOF Persistence
    DB-->>Health: pg_isready
    Redis-->>Health: redis-cli ping
    Workers->>Redis: Connect to Streams
    Workers-->>Health: Stream Connectivity OK
    Health->>Health: All Systems Ready

Loading

---

Phase 2: Detection Algorithm Calibration (Days 4-7)

2.1 Historical Dataset Preparation

• Create test dataset in file:prisma/seed-test-data.ts with 100 products across 5 categories
• Include known pricing patterns:
  • Normal sales (20-40% off)
  • Seasonal clearance (50-70% off)
  • Pricing errors (>80% off, decimal errors)
  • Price fluctuations within normal range
• Seed historical price data spanning 30+ days for Z-score reliability

2.2 Algorithm Testing & Tuning

• Create comprehensive test suite in file:src/lib/analysis/detection.test.ts:
  • Test each detection method independently
  • Validate category-specific thresholds from file:src/lib/analysis/thresholds.ts
  • Measure false positive rate on "normal sale" corpus
• Run detection against test dataset and measure:
  • Precision: True positives / (True positives + False positives)
  • Recall: True positives / (True positives + False negatives)
  • Target: Precision ≥85%, False positive rate <15%

2.3 AI Validation Effectiveness

• Test DeepSeek V3 validation in file:src/lib/ai/validator.ts against 50 edge cases
• Measure false positive reduction: Compare initial detection confidence vs. AI-validated confidence
• Target: ≥50% reduction in false positives after AI filtering
• Document unicorn detection routing (discount >85%, Z-score >4.5, high-value items $500+)

2.4 Detection Latency Benchmarking

• Add timing instrumentation to file:src/workers/anomaly-validator.ts
• Measure end-to-end latency: price scrape → anomaly detection → AI validation → confirmed glitch
• Target: <30 seconds median latency
• Identify bottlenecks (API calls, database queries, AI inference)

2.5 Temporal Context Validation

• Test maintenance window detection in file:src/lib/analysis/thresholds.ts (2-5 AM, Sunday nights)
• Verify confidence modifiers are applied correctly
• Create test cases for anomalies detected during suspicious timing windows

---

Phase 3: Scraping Reliability & Rate Limiting (Days 4-7)

3.1 Playwright Scraper Testing

• Test file:src/scrapers/playwright-worker.ts against Amazon, Best Buy, Walmart product pages
• Verify JavaScript rendering, anti-bot detection bypass, User-Agent rotation
• Measure timeout handling (target: <15s per page, fallback to Jina if exceeded)
• Test concurrent scraping: 20 products across 5 retailers simultaneously

3.2 Jina Reader Fallback

• Verify file:src/scrapers/jina-worker.ts triggers on Playwright failures (timeout >15s, 403 responses)
• Test Jina API integration with rate limiting
• Measure fallback success rate and latency

3.3 Rate Limiting & Exponential Backoff

• Review file:src/scrapers/rate-limiter.ts implementation
• Configure retailer-specific rate limits (e.g., Amazon: 1 req/2s, Best Buy: 1 req/1s)
• Test exponential backoff: base delay 2s, max 3 retries
• Verify 429 (rate limited) responses trigger backoff correctly

3.4 Robots.txt Compliance

• Add robots.txt parser to scraper initialization
• Verify scraper respects crawl-delay and disallowed paths
• Document retailer-specific scraping policies

---

Phase 4: Worker Queue Stability (Days 8-10)

4.1 BullMQ Queue Configuration

• Review queue setup in file:src/workers/notification-sender.ts, file:src/workers/social-poster.ts, file:src/workers/deal-verifier.ts
• Configure Dead Letter Queue (DLQ) in file:src/lib/streams/dlq.ts for failed jobs after 3 retries
• Set job timeout limits: anomaly-validator (30s), deal-verifier (45s), notification-sender (15s)

4.2 Worker Performance Testing

• Measure job completion times under normal load:
  • Anomaly Validator: <10s median (target met via file:src/workers/anomaly-validator.ts)
  • Deal Verifier: <15s median (needs implementation)
  • Notification Sender: <5s median (target met via file:src/workers/notification-sender.ts)
• Monitor queue backlog during peak load (50 simultaneous anomalies)
• Verify FIFO processing order maintained

4.3 Auto-Scaling & Concurrency

• Configure BullMQ concurrency settings (currently 5 in notification worker)
• Test worker scaling: add multiple worker instances and verify job distribution
• Ensure no job starvation under high load

4.4 Graceful Shutdown Implementation

• Add SIGTERM/SIGINT handlers to all workers
• Implement 30s timeout for in-flight job completion
• Test shutdown behavior: send SIGTERM, verify jobs complete before exit

---

Phase 5: Notification Channel Integration Tests (Days 8-10)

5.1 Discord Webhook Testing

• Test file:src/lib/notifications/providers/discord.ts rich embed delivery
• Verify price charts, affiliate links, expiration countdown render correctly
• Test webhook rate limits and error handling

5.2 Email Rendering Validation

• Test file:src/lib/notifications/providers/email.ts HTML template in Gmail, Outlook, Apple Mail
• Verify Resend API integration and delivery tracking
• Test email personalization and unsubscribe links

5.3 SMS Character Limit Testing

• Test file:src/lib/notifications/providers/sms.ts message formatting
• Verify Twilio integration with shortened links (≤160 characters)
• Test international number support

5.4 Telegram Bot Commands

• Test file:src/workers/telegram-bot.ts /deals command
• Verify inline buttons and chat ID registration
• Test notification delivery to subscribed users

5.5 WhatsApp Template Approval

• Submit WhatsApp template message for approval via file:src/lib/notifications/providers/whatsapp.ts
• Test delivery after approval
• Verify 24-hour messaging window compliance

5.6 Twitter/X OAuth & Posting

• Test file:src/workers/social-poster.ts Twitter integration
• Verify OAuth token refresh in file:src/lib/notifications/twitter.ts
• Test posting with #PricingError hashtag and affiliate links
• Verify media attachment (price charts)

5.7 Facebook Graph API

• Test file:src/lib/notifications/providers/facebook.ts page posting
• Verify image attachments and link previews
• Test page access token validity

5.8 Webhook Endpoint Testing

• Test file:src/lib/notifications/providers/webhook.ts JSON payload delivery
• Verify 200 status response within 3s timeout
• Test retry logic on failures

---

Phase 6: Affiliate Monetization Validation (Days 8-10)

6.1 Amazon Affiliate Tag Injection

• Review file:src/lib/services/affiliate.ts URL transformation
• Verify Amazon affiliate tag format: tag=pricehawk-20
• Test tag injection across different Amazon URL formats (product pages, search results)

6.2 Click Tracking Implementation

• Test file:src/lib/affiliate/tracker.ts tracking link generation
• Verify redirect endpoint file:src/app/api/go/[trackingId]/route.ts logs clicks
• Test click attribution to user IDs via file:prisma/schema.prisma AuditLog

6.3 Stripe Payment Links Integration

• Test subscription tier tracking in file:src/lib/stripe.ts
• Verify Stripe webhook handling for subscription events
• Test conversion funnel: anomaly → notification → click → purchase attribution

6.4 Affiliate Link Expiration

• Implement 24-hour TTL for tracking links in file:src/lib/affiliate/tracker.ts
• Test expired link handling (redirect to product page without tracking)

6.5 Revenue Attribution Reporting

• Create analytics query to associate affiliate clicks with user subscriptions
• Test commission tracking via AuditLog entries

---

Phase 7: Deal Lifecycle & Expiration Logic (Days 11-12)

7.1 Deal Verifier Re-Scraping

• Implement actual re-scraping logic in file:src/workers/deal-verifier.ts (currently mocked)
• Schedule re-scrape every 30 minutes for active deals
• Integrate with file:src/scrapers/playwright-worker.ts for price verification

7.2 Auto-Expiration Logic

• Implement price comparison in deal verifier
• Auto-expire deals when current price returns to within 10% of historical average
• Update deal status in file:prisma/schema.prisma ValidatedGlitch table

7.3 Expiration Notifications

• Implement "Deal Expired" notification in file:src/lib/notifications/manager.ts
• Send to users who subscribed to specific anomaly alerts
• Test notification delivery across all channels

7.4 Temporal Context Filtering

• Test anomalies detected during 2-5 AM maintenance window
• Verify "low confidence" flag applied correctly in file:src/lib/analysis/thresholds.ts

7.5 Chart Generation Validation

• Test file:src/lib/charts/deal-chart-generator.ts price history visualization
• Verify anomaly marker and confidence score display
• Test chart embedding in Discord, Email, Twitter notifications

---

Phase 8: Monitoring & Observability (Days 11-12)

8.1 Sentry Integration

• Install Sentry SDK and configure in file:src/app/layout.tsx and worker files
• Set up production environment in Sentry dashboard
• Test error capture and stack trace reporting
• Configure error sampling rate (100% for critical errors, 10% for warnings)

8.2 Metrics Collection

• Enhance file:src/lib/monitoring/metrics.ts to track:
  • Anomalies detected per hour (by category, retailer)
  • Worker job completion rates (success/failure/retry)
  • Notification delivery success rate per channel
  • API response times (OpenRouter, Jina Reader, Twilio, Resend)
• Implement metrics export endpoint file:src/app/api/internal/metrics/route.ts

8.3 Grafana Dashboard Setup

• Deploy Grafana instance (Docker or Railway add-on)
• Create dashboards for:
  • Real-time anomaly detection pipeline
  • Worker queue health (backlog, processing time)
  • Notification delivery funnel
  • API latency percentiles (p50, p95, p99)
• Configure auto-refresh (30s interval)

8.4 Alert Configuration

• Configure alerts in file:src/lib/monitoring/alerts.ts:
  • Worker offline >5 minutes → Discord webhook
  • Database connection pool exhausted → Email + Discord
  • Redis out of memory → Critical alert
  • False positive rate >25% → Warning alert
• Test alert delivery and escalation

8.5 Audit Logging

• Verify file:prisma/schema.prisma AuditLog captures:
  • All anomaly detections (action: glitch.detected)
  • Notification deliveries (action: notification.sent)
  • Affiliate clicks (action: affiliate.click)
• Create audit log query endpoints for analytics

---

Phase 9: Testing Strategy (Days 13-14)

9.1 Unit Tests (Vitest)

• Create tests in file:src/lib/analysis/detection.test.ts for all 5 detection methods
• Test category threshold resolution in file:src/lib/analysis/thresholds.test.ts
• Test AI validation fallback logic in file:src/lib/ai/validator.ts
• Target: >80% code coverage for core detection logic

9.2 Integration Tests

• Test worker queue processing with mocked scraper responses
• Verify Redis Stream message flow: anomaly-detected → anomaly-confirmed
• Test BullMQ delayed notification jobs
• Test database transaction handling

9.3 End-to-End Tests

• Simulate live retailer scrape → anomaly detection → notification delivery
• Use test product URLs with known pricing patterns
• Verify complete pipeline execution within 60s
• Test error recovery (API failures, database timeouts)

9.4 Load Tests

• Generate 100 concurrent anomalies using test script
• Monitor worker scaling and queue backlog
• Verify no job starvation or deadlocks
• Measure system throughput (anomalies processed per minute)

9.5 User Acceptance Testing (UAT)

• Deploy to Railway staging environment
• Invite 5 beta testers with different subscription tiers
• Collect feedback on notification UX, timing, relevance
• Track metrics: notification open rate, click-through rate, false positive reports

---

Phase 10: Production Deployment (Day 15)

10.1 Railway Production Deployment

• Create production environment in Railway
• Configure environment variables from file:.env.example
• Deploy app + 6 workers with health checks enabled
• Verify PostgreSQL and Redis add-ons provisioned

10.2 10-Retailer Watchlist Setup

• Seed production database with 10 high-priority retailers (Amazon, Best Buy, Walmart, Target, etc.)
• Configure scraping schedules via file:prisma/schema.prisma ScheduledJob
• Start scraper orchestrator file:src/scrapers/orchestrator.ts

10.3 Smoke Testing

• Verify all 6 workers start successfully
• Test health check endpoints return 200 OK
• Trigger manual anomaly detection on test product
• Verify notification delivery across all 8 channels

10.4 48-Hour Soak Test

• Monitor system stability for 48 hours
• Track critical metrics: uptime, error rate, latency
• Verify no memory leaks or resource exhaustion
• Document any critical bugs for immediate fix

10.5 Go/No-Go Checklist

• All acceptance criteria met with documented evidence
• Zero critical bugs in soak test
• Beta tester feedback average ≥4/5
• Monitoring dashboards operational
• Runbooks documented for common failures

---

Phase 11: Documentation Requirements (Days 13-15)

11.1 Production Deployment Checklist

• Update file:docs/core-flow-prod.md with:
  • Environment variable setup
  • Database migration steps
  • Worker deployment order
  • Health check verification
  • Rollback procedures

11.2 Detection Algorithm Documentation

• Create file:docs/architecture/detection-algorithms.md with:
  • Mathematical formulas for each detection method
  • Category-specific threshold tables
  • Temporal context rules
  • AI validation prompt engineering

11.3 Troubleshooting Runbook

• Create file:docs/guides/troubleshooting-workers.md with:
  • Worker offline recovery steps
  • Redis Stream cursor reset procedures
  • Dead Letter Queue inspection
  • Database connection pool debugging

11.4 Observability Guide

• Create file:docs/guides/observability.md with:
  • Grafana dashboard screenshots
  • Metrics interpretation guide
  • Alert response procedures
  • Sentry error triage workflow

---

Success Metrics & Exit Criteria

Metric	Target	Validation Method
Detection Accuracy	≥85% precision	Test against 100-product corpus
Notification Latency	<60s end-to-end	Instrumentation in workers
Worker Uptime	≥99.5% (7 days)	Health check logs
Affiliate CTR	≥3%	Click tracking analytics
False Positive Rate	<15%	AI validation + manual review
OpenRouter Quota	500 requests/day	API usage monitoring
Soak Test	48 hours, 0 critical bugs	Production logs
Beta Feedback	≥4/5 rating	User survey

Exit Criteria:

• All acceptance criteria documented with evidence (logs, screenshots, test reports)
• Zero critical bugs in 48-hour soak test
• Beta tester feedback average ≥4/5
• Production deployment approved via go/no-go checklist

Import In IDE

🤖 Prompt for AI Agents

# Production Test Launch: Phase 1 Implementation Plan

## Observations

The PriceHawk codebase has a solid foundation with all core detection algorithms implemented (Decimal Error, Z-Score, Double MAD, Adjusted IQR, Category Thresholds), category-specific tuning configured, and 8 notification channels integrated. The architecture uses Redis Streams for worker communication and BullMQ for delayed notifications. However, critical production infrastructure components are missing: Docker health checks, Redis AOF persistence, comprehensive testing, monitoring dashboards, and Railway deployment configuration. The deal verifier worker has mocked logic requiring actual re-scraping implementation.

## Approach

The plan follows a phased validation approach starting with infrastructure hardening, then algorithm calibration and testing, followed by integration validation, and finally monitoring setup. Each phase builds upon the previous, ensuring systematic validation of all acceptance criteria. The approach prioritizes critical path items (infrastructure, detection accuracy, notification delivery) before secondary concerns (monitoring dashboards, documentation). Testing will be conducted in layers: unit tests for algorithms, integration tests for workers, and end-to-end tests for the complete pipeline.

## Implementation Steps

### Phase 1: Pre-Flight Infrastructure Validation (Days 1-3)

#### 1.1 PostgreSQL Connection Pooling Configuration
- Verify `file:src/db/index.ts` connection pool settings (currently min 10, max configurable via `DATABASE_POOL_SIZE`)
- Update `file:.env.example` to document recommended pool sizes: development (10), staging (20), production (30)
- Add connection pool monitoring to `file:src/lib/monitoring/metrics.ts` to track active/idle connections
- Test concurrent worker load by running all 6 workers simultaneously and monitoring pool exhaustion

#### 1.2 Redis Persistence & Health Checks
- Modify `file:docker-compose.yml` to add Redis AOF persistence:
  ```yaml
  redis:
    command: redis-server --appendonly yes --appendfsync everysec
    volumes:
      - redis_data:/data
  ```
- Add health checks to all services in `file:docker-compose.yml`:
  - PostgreSQL: `pg_isready -U pricehawk`
  - Redis: `redis-cli ping`
  - App: HTTP check on `/api/health`
  - Workers: Custom health check scripts checking Redis stream connectivity
- Implement health check endpoint in `file:src/app/api/health/route.ts` that validates database, Redis, and critical API keys

#### 1.3 Worker Container Configuration
- Add missing workers to `file:docker-compose.yml`:
  - `worker-newsletter`: Run `tsx src/workers/newsletter-digest.ts`
  - `worker-telegram`: Run `tsx src/workers/telegram-bot.ts`
- Configure graceful shutdown handlers in all worker files to finish in-flight jobs within 30s timeout
- Add restart policies (`restart: unless-stopped`) to all worker services
- Verify BullMQ queue configuration in `file:src/workers/notification-sender.ts` and `file:src/workers/social-poster.ts` uses correct Redis connection settings

#### 1.4 Environment Variable Validation
- Create validation script `file:scripts/validate-env.ts` that checks all required API keys from `file:.env.example`:
  - OpenRouter, Clerk, Stripe, Twilio, Resend, Telegram, Discord, Facebook, WhatsApp, Twitter
- Add startup validation to main app and each worker to fail fast if critical keys are missing
- Document tier-specific requirements (e.g., Stripe keys only needed for paid features)

#### 1.5 Railway Deployment Configuration
- Create `file:railway.json` or `file:railway.toml` with service definitions for app + 6 workers
- Configure environment variable groups in Railway dashboard for staging/production
- Set up PostgreSQL and Redis add-ons in Railway
- Create deployment smoke test script that verifies all containers start and health checks pass

```mermaid
sequenceDiagram
    participant Env as Environment Validator
    participant DB as PostgreSQL Pool
    participant Redis as Redis (AOF)
    participant Workers as 6 Workers
    participant Health as Health Checks
    
    Env->>Env: Validate API Keys
    Env->>DB: Test Connection Pool (10-30)
    Env->>Redis: Verify AOF Persistence
    DB-->>Health: pg_isready
    Redis-->>Health: redis-cli ping
    Workers->>Redis: Connect to Streams
    Workers-->>Health: Stream Connectivity OK
    Health->>Health: All Systems Ready
```

---

### Phase 2: Detection Algorithm Calibration (Days 4-7)

#### 2.1 Historical Dataset Preparation
- Create test dataset in `file:prisma/seed-test-data.ts` with 100 products across 5 categories
- Include known pricing patterns:
  - Normal sales (20-40% off)
  - Seasonal clearance (50-70% off)
  - Pricing errors (>80% off, decimal errors)
  - Price fluctuations within normal range
- Seed historical price data spanning 30+ days for Z-score reliability

#### 2.2 Algorithm Testing & Tuning
- Create comprehensive test suite in `file:src/lib/analysis/detection.test.ts`:
  - Test each detection method independently
  - Validate category-specific thresholds from `file:src/lib/analysis/thresholds.ts`
  - Measure false positive rate on "normal sale" corpus
- Run detection against test dataset and measure:
  - Precision: True positives / (True positives + False positives)
  - Recall: True positives / (True positives + False negatives)
  - Target: Precision ≥85%, False positive rate <15%

#### 2.3 AI Validation Effectiveness
- Test DeepSeek V3 validation in `file:src/lib/ai/validator.ts` against 50 edge cases
- Measure false positive reduction: Compare initial detection confidence vs. AI-validated confidence
- Target: ≥50% reduction in false positives after AI filtering
- Document unicorn detection routing (discount >85%, Z-score >4.5, high-value items $500+)

#### 2.4 Detection Latency Benchmarking
- Add timing instrumentation to `file:src/workers/anomaly-validator.ts`
- Measure end-to-end latency: price scrape → anomaly detection → AI validation → confirmed glitch
- Target: <30 seconds median latency
- Identify bottlenecks (API calls, database queries, AI inference)

#### 2.5 Temporal Context Validation
- Test maintenance window detection in `file:src/lib/analysis/thresholds.ts` (2-5 AM, Sunday nights)
- Verify confidence modifiers are applied correctly
- Create test cases for anomalies detected during suspicious timing windows

---

### Phase 3: Scraping Reliability & Rate Limiting (Days 4-7)

#### 3.1 Playwright Scraper Testing
- Test `file:src/scrapers/playwright-worker.ts` against Amazon, Best Buy, Walmart product pages
- Verify JavaScript rendering, anti-bot detection bypass, User-Agent rotation
- Measure timeout handling (target: <15s per page, fallback to Jina if exceeded)
- Test concurrent scraping: 20 products across 5 retailers simultaneously

#### 3.2 Jina Reader Fallback
- Verify `file:src/scrapers/jina-worker.ts` triggers on Playwright failures (timeout >15s, 403 responses)
- Test Jina API integration with rate limiting
- Measure fallback success rate and latency

#### 3.3 Rate Limiting & Exponential Backoff
- Review `file:src/scrapers/rate-limiter.ts` implementation
- Configure retailer-specific rate limits (e.g., Amazon: 1 req/2s, Best Buy: 1 req/1s)
- Test exponential backoff: base delay 2s, max 3 retries
- Verify 429 (rate limited) responses trigger backoff correctly

#### 3.4 Robots.txt Compliance
- Add robots.txt parser to scraper initialization
- Verify scraper respects crawl-delay and disallowed paths
- Document retailer-specific scraping policies

---

### Phase 4: Worker Queue Stability (Days 8-10)

#### 4.1 BullMQ Queue Configuration
- Review queue setup in `file:src/workers/notification-sender.ts`, `file:src/workers/social-poster.ts`, `file:src/workers/deal-verifier.ts`
- Configure Dead Letter Queue (DLQ) in `file:src/lib/streams/dlq.ts` for failed jobs after 3 retries
- Set job timeout limits: anomaly-validator (30s), deal-verifier (45s), notification-sender (15s)

#### 4.2 Worker Performance Testing
- Measure job completion times under normal load:
  - Anomaly Validator: <10s median (target met via `file:src/workers/anomaly-validator.ts`)
  - Deal Verifier: <15s median (needs implementation)
  - Notification Sender: <5s median (target met via `file:src/workers/notification-sender.ts`)
- Monitor queue backlog during peak load (50 simultaneous anomalies)
- Verify FIFO processing order maintained

#### 4.3 Auto-Scaling & Concurrency
- Configure BullMQ concurrency settings (currently 5 in notification worker)
- Test worker scaling: add multiple worker instances and verify job distribution
- Ensure no job starvation under high load

#### 4.4 Graceful Shutdown Implementation
- Add SIGTERM/SIGINT handlers to all workers
- Implement 30s timeout for in-flight job completion
- Test shutdown behavior: send SIGTERM, verify jobs complete before exit

---

### Phase 5: Notification Channel Integration Tests (Days 8-10)

#### 5.1 Discord Webhook Testing
- Test `file:src/lib/notifications/providers/discord.ts` rich embed delivery
- Verify price charts, affiliate links, expiration countdown render correctly
- Test webhook rate limits and error handling

#### 5.2 Email Rendering Validation
- Test `file:src/lib/notifications/providers/email.ts` HTML template in Gmail, Outlook, Apple Mail
- Verify Resend API integration and delivery tracking
- Test email personalization and unsubscribe links

#### 5.3 SMS Character Limit Testing
- Test `file:src/lib/notifications/providers/sms.ts` message formatting
- Verify Twilio integration with shortened links (≤160 characters)
- Test international number support

#### 5.4 Telegram Bot Commands
- Test `file:src/workers/telegram-bot.ts` `/deals` command
- Verify inline buttons and chat ID registration
- Test notification delivery to subscribed users

#### 5.5 WhatsApp Template Approval
- Submit WhatsApp template message for approval via `file:src/lib/notifications/providers/whatsapp.ts`
- Test delivery after approval
- Verify 24-hour messaging window compliance

#### 5.6 Twitter/X OAuth & Posting
- Test `file:src/workers/social-poster.ts` Twitter integration
- Verify OAuth token refresh in `file:src/lib/notifications/twitter.ts`
- Test posting with #PricingError hashtag and affiliate links
- Verify media attachment (price charts)

#### 5.7 Facebook Graph API
- Test `file:src/lib/notifications/providers/facebook.ts` page posting
- Verify image attachments and link previews
- Test page access token validity

#### 5.8 Webhook Endpoint Testing
- Test `file:src/lib/notifications/providers/webhook.ts` JSON payload delivery
- Verify 200 status response within 3s timeout
- Test retry logic on failures

---

### Phase 6: Affiliate Monetization Validation (Days 8-10)

#### 6.1 Amazon Affiliate Tag Injection
- Review `file:src/lib/services/affiliate.ts` URL transformation
- Verify Amazon affiliate tag format: `tag=pricehawk-20`
- Test tag injection across different Amazon URL formats (product pages, search results)

#### 6.2 Click Tracking Implementation
- Test `file:src/lib/affiliate/tracker.ts` tracking link generation
- Verify redirect endpoint `file:src/app/api/go/[trackingId]/route.ts` logs clicks
- Test click attribution to user IDs via `file:prisma/schema.prisma` AuditLog

#### 6.3 Stripe Payment Links Integration
- Test subscription tier tracking in `file:src/lib/stripe.ts`
- Verify Stripe webhook handling for subscription events
- Test conversion funnel: anomaly → notification → click → purchase attribution

#### 6.4 Affiliate Link Expiration
- Implement 24-hour TTL for tracking links in `file:src/lib/affiliate/tracker.ts`
- Test expired link handling (redirect to product page without tracking)

#### 6.5 Revenue Attribution Reporting
- Create analytics query to associate affiliate clicks with user subscriptions
- Test commission tracking via AuditLog entries

---

### Phase 7: Deal Lifecycle & Expiration Logic (Days 11-12)

#### 7.1 Deal Verifier Re-Scraping
- Implement actual re-scraping logic in `file:src/workers/deal-verifier.ts` (currently mocked)
- Schedule re-scrape every 30 minutes for active deals
- Integrate with `file:src/scrapers/playwright-worker.ts` for price verification

#### 7.2 Auto-Expiration Logic
- Implement price comparison in deal verifier
- Auto-expire deals when current price returns to within 10% of historical average
- Update deal status in `file:prisma/schema.prisma` ValidatedGlitch table

#### 7.3 Expiration Notifications
- Implement "Deal Expired" notification in `file:src/lib/notifications/manager.ts`
- Send to users who subscribed to specific anomaly alerts
- Test notification delivery across all channels

#### 7.4 Temporal Context Filtering
- Test anomalies detected during 2-5 AM maintenance window
- Verify "low confidence" flag applied correctly in `file:src/lib/analysis/thresholds.ts`

#### 7.5 Chart Generation Validation
- Test `file:src/lib/charts/deal-chart-generator.ts` price history visualization
- Verify anomaly marker and confidence score display
- Test chart embedding in Discord, Email, Twitter notifications

---

### Phase 8: Monitoring & Observability (Days 11-12)

#### 8.1 Sentry Integration
- Install Sentry SDK and configure in `file:src/app/layout.tsx` and worker files
- Set up production environment in Sentry dashboard
- Test error capture and stack trace reporting
- Configure error sampling rate (100% for critical errors, 10% for warnings)

#### 8.2 Metrics Collection
- Enhance `file:src/lib/monitoring/metrics.ts` to track:
  - Anomalies detected per hour (by category, retailer)
  - Worker job completion rates (success/failure/retry)
  - Notification delivery success rate per channel
  - API response times (OpenRouter, Jina Reader, Twilio, Resend)
- Implement metrics export endpoint `file:src/app/api/internal/metrics/route.ts`

#### 8.3 Grafana Dashboard Setup
- Deploy Grafana instance (Docker or Railway add-on)
- Create dashboards for:
  - Real-time anomaly detection pipeline
  - Worker queue health (backlog, processing time)
  - Notification delivery funnel
  - API latency percentiles (p50, p95, p99)
- Configure auto-refresh (30s interval)

#### 8.4 Alert Configuration
- Configure alerts in `file:src/lib/monitoring/alerts.ts`:
  - Worker offline >5 minutes → Discord webhook
  - Database connection pool exhausted → Email + Discord
  - Redis out of memory → Critical alert
  - False positive rate >25% → Warning alert
- Test alert delivery and escalation

#### 8.5 Audit Logging
- Verify `file:prisma/schema.prisma` AuditLog captures:
  - All anomaly detections (action: `glitch.detected`)
  - Notification deliveries (action: `notification.sent`)
  - Affiliate clicks (action: `affiliate.click`)
- Create audit log query endpoints for analytics

---

### Phase 9: Testing Strategy (Days 13-14)

#### 9.1 Unit Tests (Vitest)
- Create tests in `file:src/lib/analysis/detection.test.ts` for all 5 detection methods
- Test category threshold resolution in `file:src/lib/analysis/thresholds.test.ts`
- Test AI validation fallback logic in `file:src/lib/ai/validator.ts`
- Target: >80% code coverage for core detection logic

#### 9.2 Integration Tests
- Test worker queue processing with mocked scraper responses
- Verify Redis Stream message flow: anomaly-detected → anomaly-confirmed
- Test BullMQ delayed notification jobs
- Test database transaction handling

#### 9.3 End-to-End Tests
- Simulate live retailer scrape → anomaly detection → notification delivery
- Use test product URLs with known pricing patterns
- Verify complete pipeline execution within 60s
- Test error recovery (API failures, database timeouts)

#### 9.4 Load Tests
- Generate 100 concurrent anomalies using test script
- Monitor worker scaling and queue backlog
- Verify no job starvation or deadlocks
- Measure system throughput (anomalies processed per minute)

#### 9.5 User Acceptance Testing (UAT)
- Deploy to Railway staging environment
- Invite 5 beta testers with different subscription tiers
- Collect feedback on notification UX, timing, relevance
- Track metrics: notification open rate, click-through rate, false positive reports

---

### Phase 10: Production Deployment (Day 15)

#### 10.1 Railway Production Deployment
- Create production environment in Railway
- Configure environment variables from `file:.env.example`
- Deploy app + 6 workers with health checks enabled
- Verify PostgreSQL and Redis add-ons provisioned

#### 10.2 10-Retailer Watchlist Setup
- Seed production database with 10 high-priority retailers (Amazon, Best Buy, Walmart, Target, etc.)
- Configure scraping schedules via `file:prisma/schema.prisma` ScheduledJob
- Start scraper orchestrator `file:src/scrapers/orchestrator.ts`

#### 10.3 Smoke Testing
- Verify all 6 workers start successfully
- Test health check endpoints return 200 OK
- Trigger manual anomaly detection on test product
- Verify notification delivery across all 8 channels

#### 10.4 48-Hour Soak Test
- Monitor system stability for 48 hours
- Track critical metrics: uptime, error rate, latency
- Verify no memory leaks or resource exhaustion
- Document any critical bugs for immediate fix

#### 10.5 Go/No-Go Checklist
- All acceptance criteria met with documented evidence
- Zero critical bugs in soak test
- Beta tester feedback average ≥4/5
- Monitoring dashboards operational
- Runbooks documented for common failures

---

### Phase 11: Documentation Requirements (Days 13-15)

#### 11.1 Production Deployment Checklist
- Update `file:docs/core-flow-prod.md` with:
  - Environment variable setup
  - Database migration steps
  - Worker deployment order
  - Health check verification
  - Rollback procedures

#### 11.2 Detection Algorithm Documentation
- Create `file:docs/architecture/detection-algorithms.md` with:
  - Mathematical formulas for each detection method
  - Category-specific threshold tables
  - Temporal context rules
  - AI validation prompt engineering

#### 11.3 Troubleshooting Runbook
- Create `file:docs/guides/troubleshooting-workers.md` with:
  - Worker offline recovery steps
  - Redis Stream cursor reset procedures
  - Dead Letter Queue inspection
  - Database connection pool debugging

#### 11.4 Observability Guide
- Create `file:docs/guides/observability.md` with:
  - Grafana dashboard screenshots
  - Metrics interpretation guide
  - Alert response procedures
  - Sentry error triage workflow

---

## Success Metrics & Exit Criteria

| Metric | Target | Validation Method |
|--------|--------|-------------------|
| Detection Accuracy | ≥85% precision | Test against 100-product corpus |
| Notification Latency | <60s end-to-end | Instrumentation in workers |
| Worker Uptime | ≥99.5% (7 days) | Health check logs |
| Affiliate CTR | ≥3% | Click tracking analytics |
| False Positive Rate | <15% | AI validation + manual review |
| OpenRouter Quota | 500 requests/day | API usage monitoring |
| Soak Test | 48 hours, 0 critical bugs | Production logs |
| Beta Feedback | ≥4/5 rating | User survey |

**Exit Criteria:**
- All acceptance criteria documented with evidence (logs, screenshots, test reports)
- Zero critical bugs in 48-hour soak test
- Beta tester feedback average ≥4/5
- Production deployment approved via go/no-go checklist

---

Execution Information

Branch: main
Commit: d9b6714

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#28 - docs: comprehensive README update with 2026 price glitch market analy… [merged] #34 - claude/pricehawk-infrastructure-planning-ZG2hG [merged] #38 - Implement category-specific thresholds and temporal window weighting [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!