Feature/Project Proposal
Implement an asynchronous message queue architecture using BullMQ and Redis to handle Gemini AI generations in the background.
Problem Statement
The aiController.js processes Google Gemini AI requests synchronously within the Express request-response cycle. This architecture ties up main thread event loops, leads to frequent API timeouts (504 Gateway Timeouts) on complex prompt generation, and severely limits horizontal scaling.
Proposed Solution
- Refactor
aiController.js to immediately enqueue an AI generation job into Redis and return a jobId to the client (202 Accepted).
- Build a dedicated background worker process (
workers/aiWorker.js) that consumes jobs, interacts with the Gemini API using exponential backoff retry logic, and saves the result to MongoDB.
- Implement a polling endpoint or WebSocket connection for the frontend to retrieve the completed AI payload.
Alternatives Considered
Instead of BullMQ, we could use Node.js standard worker_threads, but a robust queueing system like BullMQ guarantees job persistence, retries, and cluster scalability across multiple Node instances.
Benefits
Prevents server crashes under heavy AI load, eliminates request timeouts for users, and allows independent scaling of web servers and background workers.
Priority
High
Additional Context
N/A
Checklist
Feature/Project Proposal
Implement an asynchronous message queue architecture using BullMQ and Redis to handle Gemini AI generations in the background.
Problem Statement
The
aiController.jsprocesses Google Gemini AI requests synchronously within the Express request-response cycle. This architecture ties up main thread event loops, leads to frequent API timeouts (504 Gateway Timeouts) on complex prompt generation, and severely limits horizontal scaling.Proposed Solution
aiController.jsto immediately enqueue an AI generation job into Redis and return ajobIdto the client (202 Accepted).workers/aiWorker.js) that consumes jobs, interacts with the Gemini API using exponential backoff retry logic, and saves the result to MongoDB.Alternatives Considered
Instead of BullMQ, we could use Node.js standard
worker_threads, but a robust queueing system like BullMQ guarantees job persistence, retries, and cluster scalability across multiple Node instances.Benefits
Prevents server crashes under heavy AI load, eliminates request timeouts for users, and allows independent scaling of web servers and background workers.
Priority
High
Additional Context
N/A
Checklist