📰 News Intelligence System

An automated news intelligence pipeline that aggregates, processes, and analyzes news articles using NLP and AI. Built with FastAPI, PostgreSQL + pgvector, Celery, and Generative AI for real-time news monitoring and intelligence reporting.

---

🏗️ Architecture

┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│  News APIs   │────▶│  Celery Beat  │────▶│ Celery Worker│
│  (5 sources) │     │  (Scheduler)  │     │  (Pipeline)  │
└─────────────┘     └──────────────┘     └──────┬───────┘
                                                 │
                    ┌──────────────┐     ┌───────▼───────┐
                    │   Redis      │◀───▶│   FastAPI     │
                    │   (Queue)    │     │   (REST API)  │
                    └──────────────┘     └───────┬───────┘
                                                 │
                    ┌──────────────┐     ┌───────▼───────┐
                    │  pgvector    │◀───▶│  PostgreSQL   │
                    │  (Embeddings)│     │  (Storage)    │
                    └──────────────┘     └───────────────┘

✨ Features

• Multi-Source Aggregation: Pulls from NewsAPI, GNews, MediaStack, NewsData, and The Guardian
• NLP Processing: Entity extraction, sentiment analysis, and topic classification using spaCy
• Semantic Search: Vector-based article similarity search via pgvector embeddings
• AI Summarization: Article summarization powered by Google Generative AI and OpenAI
• Background Processing: Asynchronous pipeline via Celery + Redis
• REST API: Full CRUD API with JWT authentication, rate limiting, and pagination
• Real-time Monitoring: Scheduled news ingestion with Celery Beat

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Python 3.10+

Run with Docker

# Clone the repository
git clone https://github.com/Adejare-ml/News-Intelligence-System.git
cd News-Intelligence-System

# Copy environment variables
cp .env.example .env
# Edit .env with your API keys

# Start all services
docker-compose up -d

# API is available at http://localhost:8000
# Docs at http://localhost:8000/docs

Environment Variables

Variable	Description	Required
DATABASE_URL	PostgreSQL connection string	Yes
REDIS_URL	Redis connection string	Yes
JWT_SECRET	Secret key for JWT tokens	Yes
NEWSAPI_KEY	NewsAPI.org API key	No
GNEWS_KEY	GNews API key	No
GUARDIAN_API_KEY	The Guardian API key	No

🧪 Testing

# Run tests
pytest tests/ -v

📦 Tech Stack

Layer	Technology
Backend	FastAPI, Celery, SQLAlchemy
Database	PostgreSQL 16 + pgvector
Queue	Redis 7
NLP	spaCy, sentence-transformers
AI	Google Generative AI, OpenAI
Auth	JWT (PyJWT + passlib)
Deploy	Docker Compose

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

Built by Adelugba Adejare