[Feature] Implement Background Job Queue for Scheduled Tasks (Billing, Reports, Sync)

[KarenZita01]

Description

The EquipChain platform requires reliable execution of background tasks that should not block API responses or require user-facing synchronization. These tasks include: scheduled billing cycles (aggregating meter readings and computing charges), periodic report generation (daily/monthly usage reports), data synchronization with the Soroban blockchain (verifying transaction statuses, syncing contract state), webhook delivery retries (from Issue #14), and cache warming for frequently accessed data.

This issue implements a background job queue system that can schedule, execute, and monitor deferred or recurring tasks. The system should support: immediate job execution (fire-and-forget), delayed execution (schedule a job for a future time), recurring/cron-based jobs (e.g., every hour, daily at midnight), job prioritization (high, normal, low), concurrency control (max N jobs running simultaneously), job retry with backoff on failure, and job result/status tracking for monitoring.

For the initial implementation, the queue can use an in-memory store (with Bull/BullMQ for Redis-backed production use planned as a future upgrade). The queue should expose a clean API: queue.add(name, data, options), queue.schedule(name, data, cronExpression), queue.getStatus(jobId), queue.cancel(jobId). Each job type should be defined as a handler function in a dedicated src/jobs/ directory.

Technical Context & Impact

• Dependencies: For MVP, no external dependencies — use Node.js EventEmitter and setTimeout. For production, plan migration to bullmq (Redis-backed). Environment variables: JOB_CONCURRENCY (default 5), JOB_RETRY_ATTEMPTS (default 3).
• Architecture: New src/services/queue.js implementing the queue. New src/jobs/ directory with handler files: billing.job.js, reports.job.js, sync.job.js, webhookRetry.job.js, cacheWarm.job.js. An src/services/scheduler.js for cron-based scheduling.
• Impact: Enables asynchronous processing that is essential for production features. Without this, billing calculations would block API responses, webhook retries would be unreliable, and scheduled reports would require manual triggering.

Step-by-Step Implementation Guide

1. Create Queue Service: Write src/services/queue.js implementing an in-memory job queue. Features: add(type, data, options) returns job ID, process() pulls from queue with concurrency control, onComplete/onFailed callbacks, retry logic with exponential backoff, and job status tracking (queued, running, completed, failed).
2. Create Job Handlers: Create src/jobs/billing.job.js — aggregates readings and computes charges. src/jobs/reports.job.js — generates daily/monthly summary reports. src/jobs/sync.job.js — syncs on-chain data with local state. Keep handlers as placeholder functions initially, with detailed implementation in linked issues.
3. Create Scheduler: Write src/services/scheduler.js that manages cron-like recurring jobs. Use setInterval for MVP (e.g., run billing every hour). Store scheduled job configurations and provide schedule(name, cronExpression, handler) and cancelSchedule(name).
4. Integrate with Application: In src/index.js, initialize the queue and scheduler on startup. Register job handlers. Start the queue processor. Add health check endpoint exposing queue stats (waiting, active, completed, failed counts).
5. Wire with Existing Features: Modify the webhook service (Issue [Documentation] Generate OpenAPI/Swagger Documentation for All API Endpoints #14) to use the queue for delivery retries instead of inline setTimeout. Add billing job registration in app bootstrap.
6. Write Tests: Create tests/unit/queue.test.js testing job lifecycle: add, process, complete, fail, retry. Test concurrency limits. Create tests/unit/scheduler.test.js testing recurring job execution timing.

Verification & Testing Steps

1. Start the server and verify the queue initializes without errors. Check health endpoint for queue stats — expect waiting: 0, active: 0.
2. Manually trigger a job via an internal mechanism (or test script) — verify the job appears in "running" status, then transitions to "completed".
3. Create a job that always fails (e.g., invalid handler) — verify it retries 3 times with exponential backoff before moving to "failed" status.
4. Schedule a recurring job with a 10-second interval — verify it runs multiple times with correct timing.
5. Add 20 jobs with concurrency limited to 5 — verify no more than 5 jobs run simultaneously. Check logs for parallel execution patterns.