Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
655 lines (478 loc) · 27.2 KB

ARCHITECTURE.md

File metadata and controls

655 lines (478 loc) · 27.2 KB
Raw

RaidHub Services Architecture

Overview

RaidHub Services is a microservices architecture built in Go, managing data collection, processing, and analysis for Destiny 2 raid completion tracking. The system is designed to handle high-throughput PGCR (Post-Game Carnage Report) crawling, intelligent queue-based processing, and comprehensive cheat detection.

Folder Structure

RaidHub-Services/
├── apps/                        # Long-running application services
│   ├── atlas/                   # Intelligent PGCR crawler with adaptive scaling
│   ├── zeus/                    # Bungie API reverse proxy with IPv6 load balancing
│   └── hermes/                  # Queue worker manager with self-scaling topics
├── lib/                         # Shared libraries and business logic
│   ├── database/                # Database connection singletons
│   │   ├── postgres/            # PostgreSQL connection management
│   │   └── clickhouse/          # ClickHouse connection management
│   ├── messaging/               # RabbitMQ messaging infrastructure
│   │   ├── processing/          # Topic managers and workers
│   │   ├── queue-workers/        # Queue worker topic definitions
│   │   │   ├── activity_history.go      # Player activity history processing
│   │   │   ├── character_fill.go        # Character data completion
│   │   │   ├── clan_crawl.go            # Clan information crawler
│   │   │   ├── pgcr_blocked_retry.go    # Retry mechanism for blocked PGCRs
│   │   │   ├── instance_cheat_check.go  # Post-storage cheat detection
│   │   │   ├── instance_store.go        # Primary PGCR data storage
│   │   │   ├── pgcr_crawl.go            # General PGCR processing (legacy)
│   │   │   └── player_crawl.go          # Player profile data crawler
│   │   ├── routing/             # Queue routing constants
│   │   ├── rabbit/              # RabbitMQ connection singleton
│   │   └── messages/            # Message type definitions
│   ├── services/                # Domain-specific business logic
│   │   ├── pgcr_processing/     # PGCR fetch and processing logic
│   │   ├── instance_storage/    # Multi-database storage orchestration
│   │   ├── cheat_detection/     # Comprehensive cheat detection system
│   │   ├── player/              # Player data management
│   │   ├── character/           # Character data operations
│   │   ├── clan/                # Clan data operations
│   │   ├── instance/            # Instance data queries
│   │   └── stats/               # Statistical calculations
│   ├── web/                     # External API clients
│   │   ├── bungie/              # Bungie.net API client
│   │   ├── discord/             # Discord webhook client
│   │   └── gm_report/           # gm report webhooks
│   ├── monitoring/              # Prometheus metrics
│   ├── utils/                   # Common utilities
│   └── env/                     # Environment configuration
├── tools/                       # Utilities and maintenance tools
│   ├── activity-history-update/ # Batch activity history updates
│   ├── cheat-detection/        # Cheat detection and account maintenance (used by cron)
│   ├── fix-sherpa-clears/      # Data correction utilities
│   ├── flag-restricted-pgcrs/  # Batch PGCR flagging
│   ├── leaderboard-clan-crawl/ # Clan crawler for leaderboard players (used by cron)
│   ├── manifest-downloader/    # Destiny 2 manifest downloader (used by cron)
│   ├── process-missed-pgcrs/   # Processes missed PGCRs (used by cron)
│   ├── process-single-pgcr/    # Individual PGCR processing
│   ├── refresh-view/           # Materialized view refresher (used by cron)
│   ├── seed/                   # Database seeding utility
│   └── update-skull-hashes/    # Manifest hash updates
├── infrastructure/              # Infrastructure configuration (NO application code)
│   ├── postgres/                # PostgreSQL infrastructure
│   │   ├── migrations/          # Schema migrations
│   │   ├── init/                # Database initialization scripts
│   │   └── tools/               # Migration and seeding utilities
│   ├── clickhouse/              # ClickHouse analytics database
│   │   ├── migrations/          # ClickHouse schema migrations
│   │   ├── views/               # Materialized views for analytics
│   │   └── tools/               # ClickHouse utilities
│   ├── cron/                    # Scheduled task configuration
│   ├── prometheus/              # Monitoring configuration
│   └── rabbitmq/                # Message queue configuration
├── docs/                        # Architecture and API documentation
├── bin/                         # Built application binaries
├── volumes/                     # Docker persistent volumes
└── logs/                        # Application log files

Key Architectural Principles

1. Microservices with Clear Boundaries

  • apps/ - Long-running application services (hermes, atlas, zeus)
  • lib/messaging/queue-workers/ - Topic-based asynchronous processing definitions
  • lib/ - Shared business logic and infrastructure libraries
  • tools/ - Maintenance utilities and scheduled tasks
  • infrastructure/ - Pure infrastructure configuration (NO application code)

2. Event-Driven Architecture

  • Message Queues: RabbitMQ with topic-based routing for async processing
  • Self-Scaling Workers: Automatic scaling based on queue depth and processing metrics
  • Side Effects: Storage operations trigger downstream processing via message queues
  • Failure Recovery: Dedicated retry mechanisms and missed item recovery

3. Infrastructure vs Application Code

  • infrastructure/ contains ONLY configuration and tooling:
    • Database schemas, migrations, and views
    • Service configuration files
    • Cron job definitions
    • Migration and deployment tools
  • lib/ contains ONLY application logic:
    • Database connection management
    • Business domain logic
    • External API clients
    • Message queue processing framework
    • Monitoring and utilities

4. Domain-Driven Design

  • Services organized by business domain (player, instance, cheat_detection)
  • Clear boundaries between domains with well-defined interfaces
  • Dependency injection through singleton pattern for shared resources
  • Separation of concerns between data access, business logic, and external integrations

Application Services

Long-Running Services

These services run continuously and are started with make up:

Atlas - Intelligent PGCR Crawler

Purpose: Crawls PostGame Carnage Reports from the Bungie API with intelligent scaling and recovery mechanisms.

Key Features:

  • Adaptive Worker Scaling: Dynamically adjusts worker count based on 404 rates and lag metrics
  • Offload Workers: Handles problematic PGCRs with exponential backoff retry logic
  • Gap Detection: Automatically identifies and handles missing PGCR sequences
  • Rate Limiting: Respects Bungie API limits with intelligent throttling
  • Monitoring: Comprehensive Prometheus metrics and Discord alerting

Configuration:

  • Default workers: 25 (configurable via flags)
  • Buffer distance: 10,000 IDs behind latest
  • Monitoring port: 8080

Zeus - Bungie API Reverse Proxy

Purpose: Provides rate limiting and optional load balancing for Bungie API requests.

Key Features:

  • Optional IPv6 Load Balancing: When ZEUS_IPV6 environment variable is set, distributes requests across sequential IPv6 addresses (round robin)
  • Differentiated Rate Limiting: Separate limits for stats.bungie.net vs www.bungie.net (always enabled)
  • Health Monitoring: BetterUptime probe support
  • Development Mode: Use --dev flag to disable round robin (single transport) while keeping rate limiting enabled

Configuration:

  • Default port: 7777
  • IPv6 configuration: Optional via ZEUS_IPV6 environment variable (base address)
  • Configurable IPv6 interface and address count via flags (--interface, --v6_n)
  • Stats API: 40 requests/second per IP, 90 burst
  • WWW API: 12 requests/second per IP, 25 burst
  • Development mode: --dev flag disables round robin but keeps rate limiting enabled (used by Tilt)

Hermes - Queue Worker Manager

Purpose: Manages all queue workers with self-scaling topic managers for different processing types.

Key Features:

  • Topic Management: Coordinates multiple queue types with independent scaling
  • Contest Mode Support: Higher worker counts during contest weekends (disabled autoscaling)
  • Dynamic Scaling: Scales workers up/down based on queue depth and processing metrics
  • Bungie API Availability Monitoring: Polls Bungie Settings API and blocks workers when API is disabled
  • Graceful Shutdown: Proper cleanup and resource management

Managed Topics:

  • player_crawl: Player profile data processing
  • activity_history_crawl: Player activity history updates
  • character_fill: Missing character data completion
  • clan_crawl: Clan information processing
  • pgcr_blocked_retry: Retry mechanism for failed PGCRs
  • pgcr_crawl: General PGCR processing (existence checking)
  • instance_store: Primary PGCR data storage
  • instance_cheat_check: Post-storage cheat detection

Scheduled Services (Cron Jobs)

These tools run on a schedule via system crontab:

Process Missed PGCRs

Purpose: Processes PGCRs that were missed during normal crawling operations.

Key Features:

  • Gap Processing: Handles missing PGCR sequences with configurable range limits
  • Retry Logic: Intelligent retry with exponential backoff
  • Progress Tracking: Detailed reporting of recovery success/failure rates
  • Safety Limits: Prevents processing of overly large gaps

Usage:

  • ./bin/process-missed-pgcrs: Process missed PGCRs
  • ./bin/process-missed-pgcrs --gap: Process gaps in sequences

Leaderboard Clan Crawl

Purpose: Keeps player data fresh by crawling top players from various leaderboards.

Key Features:

  • Leaderboard Analysis: Processes top N players from individual and raid leaderboards
  • Clan Discovery: Discovers and processes clans from top players
  • Bulk Operations: Efficient batch processing with configurable concurrency
  • Member Validation: Ensures discovered players exist in the system

Usage: ./bin/leaderboard-clan-crawl -top 1500 -reqs 14

Cheat Detection

Purpose: Runs comprehensive cheat detection analysis and player account maintenance.

Key Features:

  • Player Cheat Level Analysis: Calculates and updates player cheat levels
  • Instance Re-checking: Re-processes instances for high-risk players
  • Blacklist Management: Automatically blacklists flagged instances and player instances
  • Statistical Reporting: Provides detailed cheat detection statistics

Usage: ./bin/cheat-detection

Manifest Downloader

Purpose: Downloads and processes Destiny 2 manifest data for weapon and feature definitions.

Key Features:

  • Manifest Fetching: Downloads latest manifest from Bungie API
  • Definition Processing: Extracts weapon and feature definitions
  • Database Updates: Updates PostgreSQL with latest definitions
  • Version Management: Only processes new manifests

Processing:

  • Weapon definitions (hash, name, icon, element, ammo type, slot, type, rarity)
  • Activity feat definitions (skulls/modifiers for raids)

Usage: ./bin/manifest-downloader --out=<directory> [--force] [--disk]

Queue Worker System

Message Queue Architecture

The system uses RabbitMQ with a topic-based architecture where each queue type is managed as a "topic" with:

  • Self-Scaling Workers: Automatically scales based on queue depth and processing metrics
  • Bungie API Availability: Workers are blocked when Bungie API is disabled (monitored via Settings API)
  • Contest Mode Support: Higher worker counts during contest periods (autoscaling disabled)
  • Configurable Parameters: Min/max workers, scale thresholds, prefetch counts, check intervals
  • Failure Handling: Built-in retry mechanisms and error handling

Queue Types

Primary Data Flow Queues

  1. instance_store - Primary PGCR storage pipeline

    • Purpose: Stores processed PGCRs to PostgreSQL and ClickHouse
    • Triggers: Character fill, player crawl, cheat check side effects
    • Workers: 1-50 (10 desired, 20 contest)

  2. instance_cheat_check - Post-storage cheat detection

    • Purpose: Runs cheat detection algorithms on stored instances
    • Workers: 1-10 (2 desired, 5 contest)

Support Queues

  1. player_crawl - Player data updates

    • Purpose: Fetches and updates player profile data
    • Workers: 5-70 (20 desired, 40 contest)

  2. activity_history_crawl - Activity history processing

    • Purpose: Updates player activity history from Bungie API
    • Workers: 1-20 (3 desired, 10 contest)

  3. character_fill - Character data completion

    • Purpose: Fills missing character information
    • Workers: 1-15 (3 desired, 8 contest)

  4. clan_crawl - Clan information updates

    • Purpose: Processes clan data and membership
    • Workers: 1-5 (1 desired, 2 contest)

  5. pgcr_blocked_retry - Failed PGCR retry mechanism

    • Purpose: Retries PGCRs that failed due to permissions or rate limiting with floodgate detection
    • Workers: 10-500 (50 desired, 200 contest)

  6. pgcr_crawl - General PGCR processing

    • Purpose: Checks PGCR existence in database
    • Workers: 1-20 (2 desired, 5 contest)

Scaling Parameters

Each topic has configurable scaling parameters:

  • MinWorkers/MaxWorkers: Hard limits on worker count
  • DesiredWorkers: Target worker count under normal conditions
  • ContestWeekendWorkers: Higher target during contest periods
  • ScaleUpThreshold: Queue depth that triggers scaling up
  • ScaleDownThreshold: Queue depth that triggers scaling down
  • ScaleUpPercent/ScaleDownPercent: Rate of scaling changes
  • ScaleCheckInterval: How often to check queue depth and Bungie API availability (default: 5 min)
  • API Availability Monitoring: Workers are blocked (not scaled down) when Bungie API is disabled
  • MaxRetryCount: Maximum number of retries before sending message to dead letter queue (0 = unlimited)

Error Handling and Retry Mechanisms

The queue worker system implements sophisticated error handling with configurable retry limits:

Retryable vs Unretryable Errors

  • Retryable Errors (Default): Most errors are treated as transient and will be retried by requeuing the message

    • Network failures, temporary API errors, rate limiting
    • Messages are NACKed with requeue=true to retry later
    • Retry count is tracked via RabbitMQ's x-death header

  • Unretryable Errors: Permanent failures that won't succeed on retry

    • Use processing.NewUnretryableError(err) to wrap permanent failures
    • Messages are NACKed with requeue=false and sent to dead letter queue (if configured)
    • Examples: invalid data format, authentication failures, not found errors for deleted resources

MaxRetryCount Configuration

Each topic can configure a maximum retry count:

  • MaxRetryCount = 0: Unlimited retries (default for most queues)
  • MaxRetryCount > 0: After this many retries, message is sent to DLQ regardless of error type
  • Per-Queue Configuration: Each queue sets its own limit based on message importance:
    • instance_store: 3 retries (critical data, but has DLQ fallback)
    • player_crawl: 12 retries (important for data collection)
    • pgcr_crawl: 20 retries (critical main functionality)
    • pgcr_blocked_retry: 25 retries (designed for retries, but still needs limit)
    • character_fill: 4 retries (useful but not critical)
    • clan_crawl: 5 retries
    • activity_history: 3 retries
    • instance_cheat_check: 5 retries

Retry Count Tracking

  • Retry count is extracted from RabbitMQ's x-death header
  • Each NACK with requeue=true increments the retry count
  • Retry count is logged in worker logs for debugging
  • When MaxRetryCount is exceeded, the message is automatically sent to DLQ with an error log

Error Handling Best Practices

  • Transient Errors: Return the error directly - it will be retried automatically
  • Permanent Errors: Wrap with processing.NewUnretryableError(err) to skip retries
  • Bungie API Errors: Use bungie.IsTransientError() to determine if an error should be retried
  • Logging: Include retry count in logs when available for better debugging

Retry Configuration

The system uses a configurable retry mechanism with exponential backoff and jitter. Retry configurations are defined in lib/utils/network/retry.go and lib/utils/retry/retry.go.

Available Retry Configurations

TransientNetworkErrorRetryConfig(): Retries transient network errors (timeouts, connection errors, server errors 5xx)

  • MaxAttempts: 3
  • InitialDelay: 50ms
  • MaxDelay: 5s
  • Multiplier: 2.0
  • Jitter: 10%
  • Retries: Timeout, connection, and server errors (502, 504)

CloudflareRetryConfig(logger, loggingFields): Optimized for Cloudflare blocking errors

  • MaxAttempts: 6
  • InitialDelay: 3s
  • MaxDelay: 120s
  • Multiplier: 3.0
  • Jitter: 20%
  • Retries: Only Cloudflare errors (detected by error message content)
  • Parameters:
    • logger: Logger instance for retry attempt logging
    • loggingFields: Map of fields to include in retry logs (fields prefixed with $ become Sentry tags)

PublishingRetryConfig: Used for RabbitMQ message publishing

  • MaxAttempts: 5
  • InitialDelay: 500ms
  • MaxDelay: 5s
  • Multiplier: 1.25
  • Jitter: 5%
  • Retries: Timeout and connection errors

Retry Behavior

  • Context Cancellation: All retry operations respect context cancellation. When a context is cancelled (e.g., worker shutdown), retries are immediately stopped and a ContextCancelledError is returned (this error is not sent to Sentry).
  • Exponential Backoff: Delays increase exponentially: delay = initialDelay * multiplier^attempt
  • Jitter: Random variation (±jitter percentage) prevents thundering herd problems
  • Error Filtering: Only errors that match the ShouldRetry function are retried

Usage Example

import (
    "context"
    "raidhub/lib/utils/network"
    "raidhub/lib/utils/retry"
)

// Simple retry with transient network errors
err := retry.WithRetry(ctx, network.TransientNetworkErrorRetryConfig(), func() error {
    return someNetworkOperation()
})

// Retry with Cloudflare-specific configuration
fields := map[string]any{
    "$queue": queueName,
    "operation": "get_profile",
}
err := retry.WithRetry(ctx, network.CloudflareRetryConfig(logger, fields), func() error {
    return bungieAPI.GetProfile(...)
})

// Retry with result
result, err := retry.WithRetryForResult(ctx, config, func() (ResultType, error) {
    return someOperation()
})

LoggingFields Parameter

The loggingFields parameter in CloudflareRetryConfig is used to add context to retry logs:

  • Fields are copied to retry attempt logs
  • Fields prefixed with $ are converted to Sentry tags (see Sentry Integration)
  • The attempt number is automatically added to retry logs
  • Example: {"$queue": "player_crawl", "operation": "get_profile"} creates tags and extra data in Sentry

Data Flow Architecture

Primary PGCR Processing Flow

  1. Atlas crawls PGCR IDs sequentially from Bungie API
  2. Zeus proxies API requests with load balancing and rate limiting
  3. PGCR Processing validates and transforms raw API responses
  4. Instance Store Queue receives successful PGCRs for storage
  5. Orchestrated Storage saves to both PostgreSQL and ClickHouse atomically
  6. Side Effects trigger downstream processing:
    • Character fill for missing character data
    • Player crawl for new or stale players
    • Cheat check for completed instances

Recovery and Retry Mechanisms

  1. Offload Workers (Atlas): Handle slow or problematic PGCRs
  2. Missed Log Processing (process-missed-pgcrs tool): Recovers PGCRs that failed completely
  3. Blocked Retry Queue: Handles permission-based failures with floodgate detection
  4. Gap Detection: Identifies and fills missing PGCR sequences

Cheat Detection Pipeline

  1. Instance Analysis: Examines completed instances for suspicious patterns
  2. Heuristic Application: Applies multiple cheat detection algorithms
  3. Player Flag Management: Updates player cheat levels and flags
  4. Blacklist Management: Automatically promotes high-confidence flags
  5. Webhook Notifications: Sends Discord alerts for flagged content

Domain Services Architecture

Service Organization

Services in lib/services/ are organized by business domain with clear boundaries:

pgcr_processing/ - PGCR Domain

  • FetchAndProcessPGCR(ctx, instanceID): Coordinates API fetch and data transformation (requires context for cancellation)
  • FetchPGCR(ctx, instanceID): Fetches PGCR from Bungie API (requires context for cancellation)
  • ParsePGCRToInstance(): Converts Bungie API format to internal structure
  • CalculateDateCompleted(): Determines instance completion timestamp
  • Result Types: Success, NotFound, NonRaid, SystemDisabled, etc.

instance_storage/ - Storage Orchestration

  • StorePGCR(): Orchestrates multi-database storage with atomicity
  • StoreRawJSON(): Compressed JSON storage in PostgreSQL
  • Store(): Structured instance data storage
  • StoreToClickHouse(): Analytics database storage
  • Side Effect Management: Triggers downstream queue processing

cheat_detection/ - Anti-Cheat System

  • CheckForCheats(): Main cheat detection entry point
  • Heuristic Algorithms: Lowman, speedrun, kill analysis, time dilation
  • Player Management: Cheat level calculation and blacklist management
  • Webhook Integration: Discord notifications for flagged content

player/ - Player Domain

  • Crawl(ctx, membershipId): Fetches player data from Bungie API (requires context for cancellation)
  • UpdateActivityHistory(ctx, membershipId): Processes player activity timeline (requires context for cancellation)
  • GetPlayerCharacters(ctx, membershipId): Retrieves character IDs from player profile (requires context for cancellation)
  • Data Management: Player profiles, characters, statistics

character/ - Character Domain

  • Fill(ctx, membershipId, characterId, instanceId): Fetches and fills missing character data (requires context for cancellation)

clan/ - Clan Domain

  • Crawl(ctx, groupId): Fetches and processes clan data (requires context for cancellation)

  • Fill(): Completes missing character information

clan/ - Clan Domain

  • Crawl(): Fetches clan data and membership
  • ParseClanDetails(): Processes clan banner and metadata

Database Architecture

PostgreSQL - Primary Data Store

  • Multi-schema structure: core, definitions, clan, extended, raw, flagging, leaderboard
  • ACID Compliance: Ensures data consistency for critical operations
  • Relationship Management: Complex queries across normalized tables
  • JSON Storage: Compressed raw PGCR data for replay capability

ClickHouse - Analytics Database

  • Time-series Optimization: Optimized for analytical queries
  • Materialized Views: Pre-computed aggregations
  • Column Storage: Efficient compression and query performance
  • Real-time Ingestion: Receives data from PostgreSQL storage operations

Infrastructure Components

Docker Services

  • PostgreSQL: Primary relational database with persistent volumes
  • RabbitMQ: Message queue with management interface
  • ClickHouse: Analytics database with custom configuration
  • Prometheus: Metrics collection and storage

Monitoring & Alerting

  • Prometheus Metrics: Comprehensive application and infrastructure metrics
  • Discord Webhooks: Real-time alerts for critical events and cheat detection
  • Health Checks: BetterUptime integration for service monitoring
  • Performance Tracking: Request latency, queue depths, processing rates

Development Workflow

Setup

# Clone and setup environment
./bootstrap.sh
# Copy and configure environment
cp example.env .env
# Start infrastructure services
make up

Building & Running

make bin        # Build all binaries
make <service>  # Build specific service
make dev        # Build and start all services with hot reload (Tilt)

Database Management

make migrate    # Run database migrations
make seed       # Populate seed data

Service Management

make up         # Start core services (hermes, atlas, zeus)
make services   # Start all services
make logs       # View aggregated logs
make down       # Stop all services

Environment Configuration

See example.env for all configuration options.

Critical Variables

  • BUNGIE_API_KEY: Primary Bungie API authentication
  • ZEUS_API_KEYS: Comma-separated list of API keys for Zeus rotation
  • ZEUS_IPV6: Base IPv6 address for Zeus load balancing
  • Database credentials: POSTGRES_*, CLICKHOUSE_*, RABBITMQ_*
  • Webhook URLs: ATLAS_WEBHOOK_URL

Deployment Architecture

Service Types

Long-Running Services (Docker Compose)

  • Atlas: PGCR crawler
  • Zeus: API proxy
  • Hermes: Queue manager

Scheduled Tasks (Cron)

  • Process Missed PGCRs: Recovery processing every 15 minutes
  • Leaderboard Clan Crawl: Weekly player updates
  • Cheat Detection: Cheat detection maintenance (4 times daily)
  • Manifest Downloader: Manifest updates (multiple times daily)
  • Refresh View: Materialized view refreshes (daily)

On-Demand Tools

  • tools/: Various maintenance and data correction utilities

Performance Characteristics

Throughput

  • Atlas: Processes ~1000-5000 PGCRs per minute
  • Queue Workers: Self-scaling based on load
  • API Rate Limits: Managed through Zeus proxy with multiple IPs/keys

Scalability

  • Horizontal: Multiple Zeus instances for higher API throughput
  • Vertical: Worker count scaling based on queue depth
  • Database: PostgreSQL primary with ClickHouse for analytics

Contributing

  1. Follow the architectural principles and service boundaries
  2. Keep infrastructure configuration separate from application code
  3. Use domain-driven design for new services
  4. Implement proper error handling and monitoring
  5. Update documentation for significant architectural changes

Future Improvements

  1. Enhanced Monitoring: Real-time dashboards and advanced alerting
  2. API Rate Optimization: Intelligent request batching and caching
  3. Horizontal Scaling: Kubernetes deployment for better resource utilization
  4. Data Pipeline Optimization: Streaming analytics and real-time processing
  5. Advanced Cheat Detection: Machine learning integration for pattern recognition