Veille IA

A self-hosted AI/ML news monitoring pipeline. Triggered on demand by an @mention of a Discord bot, the pipeline collects RSS feeds, deduplicates against PostgreSQL, scores and summarises articles with Gemini, and posts grouped embeds to up to four Discord webhooks.

Architecture

Discord @mention
    -> n8n workflow (single execution, Redis-locked)
        -> Load rss_sources.json
        -> Read 25+ RSS feeds (parallel)
        -> Date filter (2 days), categorize, dedup against PostgreSQL
        -> Gemini call (relevance + summary in one round-trip)
        -> Group by source_type, top-N per group
        -> Format Discord embeds, POST to per-group webhooks
        -> UPDATE rss_articles.sent_at = NOW()
        -> Release lock

The lock uses Redis (via Webdis) with the n8n $execution.id as a fencing token; release is a Lua compare-and-delete so a stale execution cannot wipe a newer lock. Lock TTL is 15 minutes.

Articles are inserted only after passing the Gemini relevance gate (confidence >= 85). sent_at is set after the Discord POST succeeds, so the database can distinguish queued, evaluated, and actually-sent rows.

Components

Service	Image	Purpose	Host port
postgres	postgres:16-alpine	Article store, dedup, stats, retention	127.0.0.1:5433
n8n	n8nio/n8n:latest	Workflow runtime	127.0.0.1:5678
rsshub	diygod/rsshub:latest	Custom RSS feed generator (optional)	127.0.0.1:1200
redis	redis:7-alpine	Lock backend (no auth, internal network only)	internal
webdis	nicolas/webdis:latest	HTTP wrapper around Redis (used by n8n HTTP nodes)	internal

All published ports bind to 127.0.0.1. To expose any service to the network, put it behind a reverse proxy with TLS and authentication.

Repository layout

.
├── docker-compose.yml          # Service orchestration
├── env.template                # Configuration template
├── rss_sources.json            # RSS sources (single source of truth)
├── workflow-simplified.json    # Current n8n workflow
├── scripts/
│   ├── setup.sh                # First-run setup (.env + dirs)
│   ├── start.sh, stop.sh, restart.sh, status.sh, logs.sh
│   ├── backup.sh, restore.sh
│   └── init.sql                # PostgreSQL schema, functions, indexes
├── data/                       # Persistent volumes (gitignored)
└── docs/
    ├── INSTALL.md
    ├── MAINTENANCE.md
    └── TROUBLESHOOTING.md

Requirements

• Linux host (tested on Ubuntu 22.04 / Debian 12, WSL2)
• Docker 24+ and Docker Compose v2
• 2 GB RAM minimum, 4 GB recommended
• A Discord bot token with the MESSAGE CONTENT intent
• Up to four Discord webhook URLs (one per category)
• A Gemini API key (https://aistudio.google.com/apikey)

Setup

git clone <repo-url> veille-ia
cd veille-ia
chmod +x scripts/*.sh
./scripts/setup.sh

setup.sh generates .env from env.template and writes random passwords for PostgreSQL and the n8n encryption key. Edit .env and fill in the Discord and Gemini values, then start the stack:

./scripts/start.sh
./scripts/status.sh

Open n8n at http://localhost:5678, create the owner account, then import workflow-simplified.json. Wire the Postgres credential and the Discord Bot Trigger credential. See QUICKSTART.md for the full first-run procedure.

Configuration

All runtime settings live in .env. Key variables:

Variable	Required	Notes
POSTGRES_PASSWORD	yes	Auto-generated by setup.sh.
N8N_ENCRYPTION_KEY	yes	Auto-generated. Do not rotate without re-encrypting credentials.
DISCORD_BOT_TOKEN	yes	From the Discord developer portal.
DISCORD_WEBHOOK_TECH_NEWS	yes	Per-category webhook.
DISCORD_WEBHOOK_COMPANIES	yes
DISCORD_WEBHOOK_REDDIT	yes
DISCORD_WEBHOOK_OTHERS	yes
DISCORD_ALLOWED_AUTHORS	no	Comma-separated user IDs allowed to trigger. Empty = anyone in the configured channels.
GEMINI_API_KEY	yes
GEMINI_MODEL	no	Defaults to gemini-pro. gemini-2.0-flash is a good cost/latency trade-off.
POSTGRES_PORT	no	Host-side port. Defaults to 5433 to avoid colliding with a local Postgres.
TZ	no	Defaults to Europe/Paris.

The list of n8n-visible env vars is whitelisted via N8N_ENV_VARS in docker-compose.yml. Add new variables there as well as in the env section of the n8n service.

RSS sources

rss_sources.json is mounted read-only into the n8n container at /home/node/rss_sources.json. The workflow reloads it on every execution, so changes take effect immediately without restarting n8n. Each entry has url, name, category, priority, and active. Set "active": false to disable a source.

Database

Schema is in scripts/init.sql and is applied automatically on the first PostgreSQL start. Notable details:

• rss_articles.url is unique. INSERTs use ON CONFLICT (url) DO NOTHING.
• sent_at is NULL at insert and set to NOW() only after a successful Discord post. purge_old_articles() deletes by COALESCE(sent_at, created_at) < NOW() - INTERVAL '90 days'.
• article_stats view exposes total / sent / 24h / 7d counts.
• get_source_stats() returns per-source counts.
• filter_new_urls(text[]) and check_url_duplicates_json(jsonb) are helpers for n8n dedup queries.

Connect from the host:

docker compose exec postgres psql -U n8n veille_ia

Operations

Task	Command
Start	./scripts/start.sh
Stop	./scripts/stop.sh
Restart	./scripts/restart.sh
Status + stats	./scripts/status.sh
Tail logs	./scripts/logs.sh {n8n,postgres,rsshub,all}
Backup DB+workflows	./scripts/backup.sh
Restore	./scripts/restore.sh
Manual purge	SELECT purge_old_articles(); in psql

A daily backup can be scheduled with cron:

0 4 * * * cd /path/to/veille-ia && ./scripts/backup.sh >> logs/backup.log 2>&1

Security notes

• All host port bindings are restricted to 127.0.0.1. Webdis is internal-only and unauthenticated; do not publish it.
• n8n is owner-on-first-setup. Create the owner account before exposing the host.
• N8N_BLOCK_ENV_ACCESS_IN_NODE=false is required because Code nodes need $env. The exposure is bounded by the N8N_ENV_VARS whitelist; review it before adding new env vars.
• Set DISCORD_ALLOWED_AUTHORS to the user IDs of operators if the trigger channels are not fully trusted.
• The .env file is git-ignored. Keep it that way.

Troubleshooting

The most common failure modes and their fixes are documented in docs/TROUBLESHOOTING.md. A quick checklist:

• Bot does not react to mention: verify the MESSAGE CONTENT intent and the channel IDs in the Discord Bot mention1 node.
• "Already running" loop: a lock is stuck. docker compose exec redis redis-cli DEL veille_lock.
• Gemini MAX_TOKENS or 429: switch GEMINI_MODEL to gemini-2.0-flash or gemini-2.5-flash.
• relation "rss_articles" does not exist: init.sql was not applied. docker compose down -v && ./scripts/start.sh (this destroys data).

License

MIT.