Feat/worker scheduler endpoints

[Muizzyranking]

Summary

This pull request implements the core worker and scheduler architecture for the Flint job processing system. It introduces the FlintWorker for job execution and the FlintScheduler for job dispatching and system maintenance. Additionally, a comprehensive v1 API has been added to provide full control and visibility into the system, including job management, worker monitoring, and real-time event streaming.

Type of Change

• New Feature
• Bug Fix
• Refactor
• Chore (dependencies, documentation, etc.)
• Test Addition/Update

Key Changes

• Worker Component: Implemented FlintWorker which handles:
  • Worker registration and heartbeat monitoring via Redis.
  • Concurrent job processing using a pluggable JobProcessor.
  • Support for multiple queue types (HeapQueue, TimingWheel).
• Scheduler Component: Implemented FlintScheduler which handles:
  • Dispatching due jobs to the processing queue.
  • A background Aging Process that periodically increases the priority of waiting jobs to prevent starvation.
  • Automatic cleanup of stale/dead worker registrations.
• Comprehensive v1 API:
  • jobs/: Full CRUD operations, job log retrieval, and status tracking.
  • workers/: Management and monitoring of active worker nodes.
  • dlq/: Endpoints to inspect and re-queue failed jobs from the Dead Letter Queue.
  • sse/: Server-Sent Events endpoint for real-time system monitoring.
  • settings/: Dynamic configuration updates.
  • benchmark/: Performance testing suite for the job engine.
• Infrastructure & Tooling:
  • Updated docker-compose.yml to support the full distributed architecture.
  • Added a benchmark runner for performance evaluation.
  • Integrated AgingProcess to ensure fair job scheduling.

Database Migrations

• This PR includes database migrations (alembic/versions/).
• Migrations have been tested locally.

Testing Strategy

• Integration tests added for Dead Letter Queue (DLQ) operations in tests/integration/test_dlq.py.
• Manual verification of worker-scheduler coordination and Redis-based signaling.
• Verified API endpoint responses and error handling.

Checklist

• Code follows the project's style guidelines (Ruff formatted).
• uv.lock and pyproject.toml are updated.
• Documentation (in docs/) has been updated if necessary.
• I have verified the changes in a local environment.

Screenshots / Console Logs

(Manual testing confirmed worker registration and job processing loops are functioning as expected.)