VortexLog: Time-Series Log Ingestion Engine

A high-performance log ingestion engine built in Go and PostgreSQL, designed to handle high-volume log writes concurrently using a worker pool and batch processing. Logs are ingested via a REST API, buffered in RabbitMQ, and flushed to a time-partitioned Postgres table — then visualized in real time through a pre-configured Grafana dashboard.

For in-depth architectural explanations and design decisions, see Notes.md.

Architecture

HTTP Client ──▶ POST /api/logs ──▶ RabbitMQ Queue ──▶ Worker Pool (3 goroutines)
                                                              │
                                              Batch Insert (100 items or 10s timer)
                                                              │
                                                              ▼
                                                   PostgreSQL (partitioned)
                                                              │
                                                              ▼
                                                     Grafana Dashboard

• Decoupled ingestion: HTTP handlers publish logs into RabbitMQ; background workers consume and process them in batches.
• Batch writes: Each worker accumulates up to 100 logs or flushes every 10 seconds — whichever comes first.
• Weekly partitioning: pg_partman auto-creates weekly range partitions on ingested_at, keeping indexes small and fast.
• Advanced indexing: Composite B-Tree (service + level + time), GIN with jsonb_path_ops (metadata), and Trigram (message text search).
• Graceful shutdown: On SIGINT/SIGTERM, the app stops accepting requests, flushes all in-memory buffers to Postgres, then exits cleanly.

Tech Stack

Layer	Technology
Application	Go 1.25, net/http, pgx/v5 connection pool
Message Broker	RabbitMQ 3.13
Database	PostgreSQL 18, pg_partman, pg_cron
Code Generation	sqlc (type-safe SQL → Go)
Migrations	goose
Observability	Grafana
Dev Tooling	pgcli — interactive Postgres CLI with auto-completion and syntax highlighting
Containerization	Docker & Docker Compose

Project Structure

.
├── main.go                  # Entrypoint — server, signal handling, worker pool init
├── internal/
│   ├── api/                 # HTTP handlers and JSON payload types
│   ├── database/            # sqlc-generated query code
│   ├── queue/               # RabbitMQ connection and publisher logic
│   └── worker/              # Batch worker pool (RabbitMQ consumer)
├── sql/
│   ├── schema/              # Goose migration files
│   └── queries/             # sqlc query definitions
├── Dockerfile               # Custom Postgres 18 image with pg_partman + pg_cron
├── docker-compose.yml       # Postgres + Adminer + Grafana
├── sqlc.yaml                # sqlc configuration
└── .env.example             # Environment variable template

🚀 Quick Start

1. Prerequisites

Ensure you have the following installed:

• Docker & Docker Compose
• Go (v1.22+)
• goose — database migration tool
• Git

Optional (recommended for development):

• pgcli — a better Postgres CLI with auto-completion, syntax highlighting, and multi-line query editing (used throughout this guide instead of psql)
• sqlc — SQL code generator (only needed if modifying queries)

2. Clone and Configure

git clone https://github.com/sudovishal/vortexlog.git
cd vortexlog

Copy the example environment file and fill in your values:

cp .env.example .env

Edit .env with your credentials:

DB_URL="postgres://<username>:<password>@localhost:5432/<db_name>?search_path=logsingest"
DB_USER="<username>"
DB_PASSWORD="<password>"
DB_NAME="logsdb"

GRAFANA_ADMIN_PASSWORD="<your_grafana_password>"

3. Launch the Infrastructure

This starts Postgres (custom image with pg_partman + pg_cron), RabbitMQ, Adminer, and Grafana:

docker compose up --build -d

Verify the containers are running:

docker compose ps

4. Run Database Migrations

cd sql/schema/
goose postgres "postgres://<username>:<password>@localhost:5432/<db_name>" up

Check migration status:

goose postgres "postgres://<username>:<password>@localhost:5432/<db_name>" status

5. Configure pg_cron (One-Time Setup)

Connect to your database using psql or pgcli:

pgcli -h localhost -p 5432 -U <db_user> -d <db_name>

Then run these SQL commands:

-- Activate pg_cron in your logs database
CREATE EXTENSION pg_cron;

-- Schedule pg_partman maintenance to run every night at midnight
SELECT cron.schedule(
    'partman-weekly-maintenance',
    '0 0 * * *',
    $$SELECT partman.run_maintenance();$$
);

-- Set the schema search path
SET search_path TO logsingest, public;

Why is this manual? pg_cron stores schedules in the system-level postgres database as cluster-wide background workers, so they can't be managed through application-level goose migrations. See Notes.md for details.

6. Start the Go Application

The Go app runs outside Docker (the compose file only manages the infrastructure services):

go run .

The API server will start on port 3001.

📡 API Reference

POST /api/logs

Ingest an array of log entries. The logs are published to RabbitMQ and batch-inserted asynchronously.

Request:

curl -X POST http://localhost:3001/api/logs \
  -H "Content-Type: application/json" \
  -d '[{
    "service_name": "auth-service",
    "level": "ERROR",
    "msg": "Failed to validate JWT token",
    "time": "2026-06-02T10:30:00Z",
    "trace_id": "abc-123-def-456",
    "metadata": {
      "user_id": "u_9281",
      "endpoint": "/api/login",
      "status_code": 401
    }
  }]'

Response: 201 Created

{ "status": "queued" }

Payload Fields:

Field	Type	Required	Description
service_name	string	✅	Name of the originating service
level	string	✅	Log severity (INFO, WARN, ERROR, etc.)
msg	string	✅	Log message text
time	string (ISO 8601)	✅	Timestamp when the event occurred
trace_id	string	❌	Distributed tracing correlation ID
metadata	object (JSONB)	❌	Arbitrary key-value metadata

📊 Dashboards & Tools

Tool	URL	Description
Grafana	http://localhost:3000	Log dashboard with live streaming, expression filters, and JSON unpacking
RabbitMQ	http://localhost:15672	RabbitMQ Management UI to monitor queues and workers
Adminer	http://localhost:8080	Lightweight database management UI

Grafana credentials: Username: admin / Password: <your GRAFANA_ADMIN_PASSWORD from .env> RabbitMQ credentials: Username: guest / Password: guest

🛑 Stopping the App

# Stop the Go app with Ctrl+C (sends SIGINT)
# It will flush all remaining memory buffers to Postgres before exiting

# Tear down the infrastructure
docker compose down -v

The app utilizes a graceful shutdown pattern to flush all remaining memory buffers to Postgres before closing down, preventing data loss.