AI Pulse ⚡

Automated Multi-Source Tech News Aggregation & AI Executive Digest Platform
Powered by a dynamic multi-provider LLM rotation engine (Groq, Gemini, Mistral, OpenRouter, OpenAI, Anthropic).

---

📌 Overview

AI Pulse is an end-to-end tech news intelligence platform designed for developers, engineering managers, and AI researchers. It automatically fetches trending developer news, open-source AI tools, research papers, and discussions across multiple platforms (GitHub, Hacker News, Reddit, Dev.to, Twitter/X), processes them using an intelligent 2-Phase Ingestion Engine, and generates concise, structured daily digests with actionable developer takeaways.

To guarantee high availability and bypass API rate limits, AI Pulse features a Multi-Provider AI Rotation Engine that seamlessly rotates across Groq, Google Gemini, Mistral, OpenRouter, OpenAI, and Anthropic with automatic fallback.

---

✨ Key Features

• 🔄 Multi-Provider AI Rotation Pool: Dynamically balances load across multiple free and paid LLM providers. If one provider hits rate limits or fails, the engine instantly fails over to the next available provider.
• 📰 Multi-Source News Aggregation: Ingests high-signal developer content from:
  • GitHub: Trending AI repositories, releases, and topics.
  • Hacker News: Top story discussions and AI announcements.
  • Reddit: Key subreddits (r/LocalLLaMA, r/MachineLearning, r/artificial).
  • Dev.to: Community tutorials and tech posts.
  • Twitter / X: Curated AI tech tweets.
• ⚡ 2-Phase Ingestion Pipeline:
  • Phase 1 (Fetch): Ingests raw articles/repos into PostgreSQL without waiting for AI processing.
  • Phase 2 (Process): Batches unprocessed items and streams them to the AI rotation pool for TL;DR summaries, key takeaways, relevance scoring, and automatic tagging.
• 📊 Executive Dashboard UI: Modern, responsive React dashboard built with Ant Design & Recharts featuring:
  • Digest timeline & calendar browser.
  • Category, relevance score, and source filtering.
  • Interactive source management & cron status drawer.
  • One-click manual ingestion triggers ("Full Sync" vs "Retry AI Only").
• 🔒 Enterprise Clean Architecture: Decoupled NestJS architecture using the Repository Pattern and interface abstractions for database, news sources, and AI providers.

---

🏗️ System Architecture

                               ┌─────────────────────────────────────────┐
                               │             News Sources                │
                               │ GitHub | HackerNews | Reddit | Dev.to   │
                               └────────────────────┬────────────────────┘
                                                    │
                                                    ▼
                               ┌─────────────────────────────────────────┐
                               │           Phase 1: Fetching             │
                               │ Ingestion Orchestrator / Cron Service   │
                               └────────────────────┬────────────────────┘
                                                    │
                                                    ▼
                               ┌─────────────────────────────────────────┐
                               │          PostgreSQL Database            │
                               │      (DailyDigest & NewsItem models)    │
                               └────────────────────┬────────────────────┘
                                                    │
                                                    ▼
                               ┌─────────────────────────────────────────┐
                               │        Phase 2: AI Summarization        │
                               │       AI Provider Rotation Pool         │
                               │ ┌───────┬────────┬─────────┬──────────┐ │
                               │ │ Groq  │ Gemini │ Mistral │OpenRouter│ │
                               │ └───────┴────────┴─────────┴──────────┘ │
                               └────────────────────┬────────────────────┘
                                                    │
                                                    ▼
                               ┌─────────────────────────────────────────┐
                               │        React Executive Dashboard        │
                               │  Vite | Redux Toolkit | Ant Design UI   │
                               └─────────────────────────────────────────┘

---

📂 Repository Structure

ai-pulse/
├── ai-pulse-backend/       # NestJS REST API & Ingestion Engine
│   ├── src/
│   │   ├── ai/            # Multi-provider LLM adapters & pool rotation logic
│   │   ├── config/        # Environment configuration service
│   │   ├── database/      # Prisma ORM & Repository implementations
│   │   ├── digest/        # Daily Digest & Search REST API controllers
│   │   ├── ingestion/     # 2-Phase Ingestion pipeline & cron scheduler
│   │   └── news-sources/  # Adapters for GitHub, Reddit, HN, Dev.to
│   ├── prisma/            # Database schema & migrations
│   └── Dockerfile
├── ai-pulse-frontend/      # React + Vite Single Page Application
│   ├── src/
│   │   ├── components/    # Reusable UI components & drawers
│   │   ├── pages/         # HomePage, DigestDetail, SearchPage, StatsPage
│   │   ├── store/         # RTK Query API slices
│   │   └── styles/        # CSS & AntD custom styles
│   └── Dockerfile
├── docker-compose.yml     # Multi-container orchestration
├── .env.example           # Master environment variables structure
└── README.md

---

🚀 Quick Start (Docker Compose)

The fastest way to get AI Pulse running locally is using Docker Compose:

1. Clone the Repository

git clone https://github.com/your-username/ai-pulse.git
cd ai-pulse

2. Configure Environment Variables

Copy .env.example to .env in the root directory and add your API keys:

cp .env.example .env

Edit .env and add at least one AI provider key (e.g. GROQ_API_KEY or GEMINI_API_KEY):

GROQ_API_KEY=gsk_your_groq_api_key_here
GEMINI_API_KEY=your_gemini_api_key_here

3. Launch the Stack

docker-compose up --build

Access the application components:

• 🌐 Frontend Dashboard: http://localhost:8080
• 🔌 Backend REST API: http://localhost:3000
• 📚 Swagger API Docs: http://localhost:3000/api/docs

---

🛠️ Manual Development Setup

If you prefer to run services individually without Docker Compose:

Prerequisites

• Node.js: v20+
• PostgreSQL: v15+
• npm: v9+

1. Backend Setup

cd ai-pulse-backend

# Install dependencies
npm install

# Setup environment variables
cp .env.example .env

# Run Database Migrations
npx prisma migrate dev

# Seed initial source configuration
npx ts-node seed.ts

# Start backend in development mode
npm run start:dev

2. Frontend Setup

cd ai-pulse-frontend

# Install dependencies
npm install

# Start Vite development server
npm run dev

The frontend will start at http://localhost:5173 with automated API proxying to http://localhost:3000.

---

⚙️ Environment Variables Reference

Variable	Description	Default / Example	Required
NODE_ENV	Application environment	development	Yes
PORT	Backend port	3000	No
DATABASE_URL	PostgreSQL connection string	postgresql://user:pass@localhost:5432/ai_pulse_db	Yes
AI_PROVIDER_MODE	Provider mode (POOL or SINGLE)	POOL	Yes
AI_BATCH_SIZE	Items processed per AI request	5	No
AI_INTER_BATCH_DELAY_MS	Delay between AI batches (ms)	3000	No
GROQ_API_KEY	Groq API Key	gsk_...	Provider dependent
GEMINI_API_KEY	Google Gemini API Key	AQ...	Provider dependent
MISTRAL_API_KEY	Mistral AI Key	Cup9...	Provider dependent
OPENROUTER_API_KEY	OpenRouter API Key	sk-or-...	Provider dependent
GITHUB_TOKEN	GitHub Personal Access Token	github_pat_...	Recommended
INGESTION_CRON_SCHEDULE	Cron syntax for daily automated fetch	0 0 * * *	No

---

🔌 API Endpoints Summary

Method	Endpoint	Description
GET	/api/digests/latest	Fetch the latest processed daily digest
GET	/api/digests/:id	Fetch digest by UUID
GET	/api/digests	Paginated list of historical digests
POST	/api/ingestion/trigger	Trigger immediate manual news ingestion
GET	/api/sources	Get active source configurations & status
PATCH	/api/sources/:source	Enable/disable source or update fetch intervals
GET	/api/stats/overview	Ingestion metrics, source breakdown & AI stats
GET	/api/search	Search news items by query, source, or tags

Full API documentation with response schemas is available via Swagger UI at /api/docs.

---

🧪 Testing & Code Quality

# Run backend unit tests
cd ai-pulse-backend
npm run test

# Run ESLint validation
npm run lint

# Format code with Prettier
npm run format

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.