Break-Free

A reliable Go microservice implementing the outbox pattern for guaranteed message delivery to Kafka.

Overview

Break-Free monitors a PostgreSQL outbox table and reliably delivers messages to Kafka topics. It ensures at-least-once delivery by polling the database for unprocessed events and marking them as processed after successful delivery.

Features

• Reliable Message Delivery: Implements the outbox pattern for guaranteed delivery
• At-Least-Once Delivery: Messages are guaranteed to be delivered to Kafka
• Multiple Topic Support: Route messages to different Kafka topics
• Retry Logic: Handles temporary failures with retry mechanism
• Header Support: Forward custom headers to Kafka messages
• Automatic Recovery: Gracefully handles Kafka downtime
• Health Monitoring: Built-in logging and status tracking

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│                 │    │                 │    │                 │
│   Application   │───▶│   PostgreSQL    │    │      Kafka      │
│                 │    │   Outbox Table  │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                │                       ▲
                                │                       │
                                ▼                       │
                       ┌─────────────────┐              │
                       │                 │              │
                       │  Break-Free     │──────────────┘
                       │   Service       │
                       │                 │
                       └─────────────────┘

Quick Start

Prerequisites

• Docker and Docker Compose
• Go 1.21+

Running the Demo

git clone <repository>
cd break-free
./scripts/test.sh

This will:

1. Start PostgreSQL and Kafka infrastructure
2. Build the application
3. Open 3 colored terminals for interactive testing
4. Provide step-by-step instructions

Usage

Command Line Flags

./break-free \
  -kafka-brokers "localhost:9092" \
  -kafka-user "your-user" \
  -kafka-pass "your-password" \
  -db-name "your-database" \
  -db-user "your-db-user" \
  -db-pass "your-db-password"

Required Flags

• -kafka-brokers: Comma-separated list of Kafka broker addresses
• -db-name: PostgreSQL database name
• -db-user: Database username

Optional Flags

• -kafka-user: Kafka SASL username (for authenticated clusters)
• -kafka-pass: Kafka SASL password
• -db-host: Database host (default: localhost)
• -db-port: Database port (default: 5433)
• -db-pass: Database password
• -db-sslmode: Database SSL mode (default: disable)

Database Schema

The service automatically creates the break_free_outbox table:

CREATE TABLE break_free_outbox (
    id BIGSERIAL PRIMARY KEY,
    payload TEXT NOT NULL,
    topic VARCHAR(255) NOT NULL,
    key VARCHAR(255),
    headers JSONB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    processed BOOLEAN DEFAULT FALSE,
    processed_at TIMESTAMP,
    retry_count INTEGER DEFAULT 0
);

Column Descriptions

Column	Type	Description
id	BIGSERIAL	Unique identifier for each message
payload	TEXT	JSON message payload to send to Kafka
topic	VARCHAR(255)	Kafka topic destination
key	VARCHAR(255)	Optional Kafka message key
headers	JSONB	Optional Kafka headers as JSON object
created_at	TIMESTAMP	When the record was created
processed	BOOLEAN	Whether message was successfully sent
processed_at	TIMESTAMP	When the message was processed
retry_count	INTEGER	Number of retry attempts

Message Format

Basic Message

INSERT INTO break_free_outbox (payload, topic) 
VALUES ('{"user_id": 123, "action": "created"}', 'user-events');

Message with Key

INSERT INTO break_free_outbox (payload, topic, key) 
VALUES ('{"order_id": 456}', 'order-events', 'order_456');

Message with Headers

INSERT INTO break_free_outbox (payload, topic, key, headers) 
VALUES (
    '{"notification": "welcome"}', 
    'notifications', 
    'user_123',
    '{"source": "user-service", "version": "1.0"}'
);

Configuration

Environment Variables

You can also use environment variables instead of command line flags:

export KAFKA_BROKERS="localhost:9092"
export KAFKA_USER="username"
export KAFKA_PASS="password"
export DB_HOST="localhost"
export DB_PORT="5433"
export DB_NAME="myapp"
export DB_USER="postgres"
export DB_PASS="password"
export DB_SSLMODE="disable"

Service Configuration

The service polls the outbox table every 5 seconds by default. Messages are processed in batches of up to 100 records per poll cycle.

Building

Local Build

go build -o break-free ./cmd/break-free

Docker Build

docker build -t break-free .

Testing

See TESTING.md for comprehensive testing instructions.

Project Structure

break-free/
├── cmd/
│   └── break-free/
│       └── main.go          # Application entry point
├── internal/
│   └── outbox/
│       └── outbox.go        # Core outbox service logic
├── scripts/
│   ├── test.sh              # Automated testing script
│   └── kafka-monitor.sh     # Kafka consumer helper
├── examples/
│   └── example.sql          # Sample data for testing
├── go.mod                   # Go module dependencies
├── go.sum                   # Dependency checksums
├── docker-compose.yml       # Local infrastructure setup
├── README.md                # This file
└── TESTING.md               # Detailed testing guide

Monitoring

Service Logs

The service provides detailed logging:

Starting outbox service...
Creating outbox table...
Outbox table created successfully
Service ready - monitoring outbox table every 5 seconds
Successfully published event 1 to topic user-events
Successfully published event 2 to topic order-events

Health Checks

Monitor the processed column in the outbox table:

-- Check processing status
SELECT topic, COUNT(*) as total, 
       COUNT(CASE WHEN processed THEN 1 END) as processed,
       COUNT(CASE WHEN NOT processed THEN 1 END) as pending
FROM break_free_outbox 
GROUP BY topic;

Error Handling

The service handles various error scenarios:

• Database Connection Loss: Automatic reconnection
• Kafka Downtime: Messages queue in database until Kafka recovers
• Message Serialization Errors: Logged and marked for retry
• Network Failures: Exponential backoff retry logic

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For questions or issues:

1. Check the TESTING.md guide
2. Review the troubleshooting section
3. Open an issue on GitHub
4. Contact: lazarbogosavljevic98@gmail.com