ATLAS.md

Atlas Documentation

Overview

Atlas is a PGCR (Post Game Carnage Report) crawler service that continuously processes instance IDs from the Destiny 2 API. It uses an adaptive autoscaling system to dynamically adjust the number of worker goroutines and the period length (number of instances processed per cycle) based on real-time metrics.

Purpose and Context

What Atlas Does

Atlas is responsible for discovering and fetching PGCRs (Post Game Carnage Reports) from the Bungie API. PGCRs contain detailed information about completed activities in Destiny 2, including raid completions, player performance, and activity metadata. These instance IDs are pseudo-sequential integers that generally increment as activities are completed.

Role in RaidHub Services

Atlas serves as the entry point for the RaidHub data pipeline:

1. Discovery: Crawls sequential instance IDs, maintaining position relative to the latest available instance
2. Fetching: Retrieves PGCR data from the Bungie API via Zeus (Bungie API reverse proxy)
3. Processing: Validates and processes PGCR data to extract raid completion information
4. Publishing: Publishes successfully processed PGCRs to the InstanceStore queue for asynchronous storage by queue workers
5. Recovery: Handles gaps, errors, and missed instances through specialized recovery mechanisms

Key Design Goals

• Adaptive Scaling: Automatically adjusts worker count based on system health and lag metrics
• Resilience: Handles API errors, rate limits, and missing instances gracefully
• Self-Healing: Detects and recovers from gaps in instance sequences automatically
• Efficiency: Maintains optimal throughput while respecting API rate limits
• Observability: Comprehensive metrics and Discord alerting for operational visibility

Data Flow

Destiny 2 API → Zeus (Proxy) → Atlas → InstanceStore Queue → Queue Workers → Databases

Atlas fetches PGCRs sequentially, processes them, and publishes successful results to RabbitMQ. Queue workers (managed by Hermes) then handle the actual storage operations, allowing Atlas to focus on crawling without blocking on database operations.

What Are PGCRs?

PGCR Overview

PGCR stands for Post Game Carnage Report - detailed reports generated by Bungie's Destiny 2 API after any activity completes. Each PGCR represents a single completed activity instance and contains comprehensive data about that activity.

Instance IDs

• Pseudo-Sequential Identifiers: Each PGCR has a unique instance_id - a 64-bit integer that is generally sequential but not strictly so
• Global Sequence: Instance IDs increment globally across all Destiny 2 activities (raids, strikes, crucible, etc.), but gaps and non-sequential values exist
• Chronological Order: Higher instance IDs generally correspond to activities completed more recently, but the relationship is not perfect
• Not Truly Sequential: Instance IDs have gaps, missing values, and may not always increment strictly by 1

PGCR Processing Flow

When Atlas processes a PGCR:

1. Fetch: Retrieves raw PGCR JSON from Bungie API
2. Validate: Checks if it's a valid PGCR response
3. Parse: Extracts activity metadata and player information
4. Filter: Identifies if it's a raid activity (non-raids are discarded after recording lag metrics)
5. Process: Converts to internal activity format
6. Publish: Sends to queue for storage
7. Store: Queue workers store processed data in PostgreSQL and raw JSON in ClickHouse

Non-Raid Activities

Atlas processes all PGCRs but only stores raid-related data:

• Raid Activities: Stored in database, used for leaderboards and statistics
• Non-Raid Activities: Discarded after recording lag metrics (used for tracking crawl position relative to live instances)
• This filtering happens during processing to avoid storing irrelevant data

Crawling Strategy

Sequential Instance ID Processing

Atlas uses a sequential crawling strategy to process instance IDs from the Destiny 2 API. Instance IDs are pseudo-sequential integers that generally increment as activities are completed in Destiny 2, though gaps and non-sequential values exist. Atlas maintains a cursor (LatestId) that tracks the current position in the sequence.

How Atlas Processes Instance IDs

1. Starting Position

• On startup, Atlas determines the starting instance ID:
  • If targetInstanceId is specified: Starts at targetInstanceId - buffer
  • Otherwise: Queries database for the latest stored instance ID, then starts buffer IDs behind (default: 10,000 IDs)
• This buffer ensures Atlas doesn't miss any small gaps by starting slightly behind the latest known instance

2. Sequential Incrementing

• Each period, Atlas spawns workers that process periodLength instance IDs
• Instance IDs are incremented atomically: LatestId += (Skip + 1) for each ID
• In normal operation, Skip = 0, so IDs are processed sequentially: 1000000, 1000001, 1000002...
• In dev mode only, Skip can be set to skip IDs (e.g., Skip = 4 processes every 5th ID: 1000000, 1000005, 1000010...)
• Workers pull IDs from an unbuffered channel, ensuring immediate processing

API Polling Behavior

Per Instance Processing:

1. Worker receives instance ID from channel
2. Retries with exponential backoff:
   • Base delay: 5.5 seconds (retryDelayTime)
   • Random variation: ±1/3 of base delay
   • Increases with attempt number: timeout = baseDelay - variation + random(baseDelay * attempt)
3. Fetches PGCR from Bungie API via Zeus proxy
4. Processes result:
   • Success: Publishes to InstanceStore queue, breaks retry loop
   • NotFound (404): Increments notFoundCount, retries up to 3 times
   • NonRaid: Records lag metric and discards (non-raid activities don't need storage)
   • SystemDisabled: Observes 0 lag, waits 45 seconds, retries
   • RateLimited/InsufficientPrivileges: Publishes to retry queue, breaks
   • BadFormat/ExternalError: Logs as missed, offloads if multiple errors

Retry Logic:

• Workers retry failed requests up to 3 times for 404s or 2 times for errors
• After max retries, instance is logged as "missed" and sent to offload channel
• Offload worker handles problematic instances separately with longer retry delays

Buffer and Skip Configuration

Buffer (default: 10,000):

• Distance behind latest instance ID to start crawling
• Prevents any small gaps from being missed
• Can be configured via --buffer flag

Dev Skip (dev mode only, default: 4):

• Number of instance IDs to skip between processed IDs
• Allows faster testing by processing every Nth ID (e.g., skip=4 processes every 5th ID)
• Only active when --dev flag is set
• Example: Skip=4 means process IDs: N, N+5, N+10, N+15...

Worker Distribution

Unbuffered Channel Design:

• Instance IDs are distributed via an unbuffered channel
• Ensures IDs don't sit in buffer - workers receive IDs immediately
• Provides backpressure: if workers are slow, ID distribution slows down

Parallel Processing:

• Multiple workers process different instance IDs concurrently
• Each worker independently handles retries and error cases
• Workers don't block each other - slow instances don't delay fast ones

Architecture

Core Components

1. Scaling Loop (scaling.go): Main control loop that spawns workers, collects metrics, and makes scaling decisions
2. Metrics Service (metrics_service.go): Fetches metrics from Prometheus for scaling decisions
3. Workers (worker.go): Individual goroutines that process PGCR instances
4. Gap Checker (gap_checker.go): Detects and handles gaps in instance sequences independently
5. Alerting (alerting.go): Sends status updates and alerts to Discord

Key Metrics

The autoscaling system uses the following metrics collected from Prometheus:

• P20Lag: 20th percentile lag (how far behind the latest instance we are, in seconds)

  • Calculated using histogram_quantile(0.20, sum(rate(pgcr_crawl_summary_lag_seconds_bucket[2m])) by (le))
  • Lower values indicate we're closer to catching up to live instances
  • Default fallback: 900 seconds if no data available

• Fraction404: Fraction of requests that result in 404 (Not Found) responses

  • Indicates gaps in instance sequences or instances that don't exist
  • High values (>0.50) trigger gap detection mode

• ErrorFraction: Fraction of requests that result in errors (status codes 6-10)

  • Includes various error types like rate limiting, system disabled, etc.

• PGCRRate: Rate of successful PGCR processing (PGCRs per second)

  • Used to calculate appropriate period lengths

• Count404: Absolute count of 404 responses in the time window

  • Used for gap detection thresholds

Architecture

Core Components

1. Scaling Loop (scaling.go): Main control loop that spawns workers, collects metrics, and makes scaling decisions
2. Metrics Service (metrics_service.go): Fetches metrics from Prometheus for scaling decisions
3. Workers (worker.go): Individual goroutines that process PGCR instances
4. Gap Checker (gap_checker.go): Detects and handles gaps in instance sequences independently
5. Alerting (alerting.go): Sends status updates and alerts to Discord

Key Metrics

The autoscaling system uses the following metrics collected from Prometheus:

• P20Lag: 20th percentile lag (how far behind the latest instance we are, in seconds)

  • Calculated using histogram_quantile(0.20, sum(rate(pgcr_crawl_summary_lag_seconds_bucket[2m])) by (le))
  • Lower values indicate we're closer to catching up to live instances
  • Default fallback: 900 seconds if no data available

• Fraction404: Fraction of requests that result in 404 (Not Found) responses

  • Indicates gaps in instance sequences or instances that don't exist
  • High values (>0.50) trigger gap detection mode

• ErrorFraction: Fraction of requests that result in errors (status codes 6-10)

  • Includes various error types like rate limiting, system disabled, etc.

• PGCRRate: Rate of successful PGCR processing (PGCRs per second)

  • Used to calculate appropriate period lengths

• Count404: Absolute count of 404 responses in the time window

  • Used for gap detection thresholds

Autoscaling Strategy

Overall Approach

Atlas uses an adaptive, metrics-driven autoscaling strategy that dynamically adjusts worker count and processing periods based on real-time system health. The strategy is designed to balance multiple competing goals:

1. Stay Caught Up: Minimize lag behind the latest available instance IDs
2. Efficiency: Avoid wasting resources when processing gaps or invalid instances
3. Resilience: Handle API errors, rate limits, and missing instances gracefully
4. Predictability: Maintain consistent 5-minute periods for stable metrics collection

Strategic Principles

1. Metrics-Driven Decisions

• Every scaling decision is based on Prometheus metrics collected after each period
• Uses weighted averages over time windows to smooth out noise
• Tracks multiple dimensions: lag, 404 rate, error rate, and throughput

2. State-Aware Scaling

• Recognizes three distinct operational states and applies appropriate strategies
• Transitions between states based on clear thresholds
• Each state has optimized formulas for its specific conditions

3. Adaptive Period Length

• Period length adapts to current throughput to maintain ~5 minutes of work
• Shorter periods during gaps for faster detection
• Longer periods during catch-up to maximize throughput

4. Conservative Scaling

• Scales up aggressively when clearly behind
• Scales down gradually when ahead to avoid oscillation
• Maintains minimum worker count to ensure continuous operation

Strategy Selection

Atlas evaluates three key metrics after each period:

1. Fraction404 (404 rate): Indicates whether we're in a gap or processing valid instances
2. P20Lag: How far behind the latest instance we are (20th percentile)
3. PGCRRate: Current processing throughput

Based on these metrics, Atlas selects one of three strategies:

High 404 Rate (>50%)    → Strategy 1: Gap Detection Mode

Low 404 (<0.1%) OR      → Strategy 2: Catch-Up Mode
High Lag (≥600s)

Normal Operations       → Strategy 3: Fine-Tuning Mode
(0.1% ≤ 404 ≤ 50%)

The strategy selection happens every ~5 minutes (or after each period completes), allowing Atlas to quickly adapt to changing conditions.

Strategy 1: High 404 Rate (>50%) - Gap Detection Mode

Trigger: Fraction404 > 0.50

When It Activates: Atlas detects it's encountering many missing instances, indicating we're likely in a gap where instance IDs don't exist or are missing.

Behavior:

• Sets newPeriodLength = 2500 (smaller period for faster detection)
• Sets newWorkers = 25 (moderate worker count)
• Rapid probe to detect gap boundaries

Rationale: When encountering high 404 rates, we're likely in a gap. A smaller period with moderate workers helps quickly identify gap boundaries without wasting resources trying to process many non-existent instances. Once gap boundaries are identified, the gap checker can handle skipping the gap.

Transition: Once 404 rate drops below 50%, Atlas transitions to Normal Operation or Catch-Up Mode depending on lag.

Strategy 2: Catch-Up Mode - Aggressive Scaling

Trigger: Fraction404 < 0.001 || P20Lag >= 600

When It Activates:

• Very low 404 rate (<0.1%) indicates continuous valid instances ahead
• High lag (≥600 seconds) indicates we're significantly behind

Behavior:

• Aggressive scaling to catch up to live instances
• Period length: max(pow(effectiveWorkers * (ceil(effectiveLag) - 20), 0.824), 10_000)
  • effectiveWorkers = max(workers, minWorkers) (ensures at least 5 workers)
  • effectiveLag = max(P20Lag, 20) (clamps lag to minimum of 20 seconds)
• Worker count: ceil(workers * (1 + (effectiveLag - 20) / 100))
  • Uses consistent ceil operation and effectiveLag clamping
  • Scales up proportionally to lag (1% increase per second of lag above 20s)

Rationale: When we're far behind (600+ seconds) or have very few 404s (indicating continuous valid instances), we aggressively scale up workers to catch up quickly. The period length formula scales with lag and worker count to maximize throughput.

Edge Case Handling:

• Low Lag: P20Lag is clamped to a minimum of 20 seconds to prevent negative scaling when close to live instances
• Zero Workers: Uses effectiveWorkers (minimum of minWorkers) to ensure period length calculation always has valid worker count

Transition: Once lag drops below 600 seconds and 404 rate increases above 0.1%, Atlas transitions to Normal Operation.

Strategy 3: Normal Operation - Fine-Tuning Mode

Trigger: Default case (404 rate between 0.1% and 50%)

When It Activates: Normal operational conditions where we're processing a mix of valid and invalid instances, or when we're reasonably caught up.

Behavior:

• Gradual worker adjustment based on 404 rate
• Adjustment formula: workers - sign(adjf) * decreaseFraction * workers
  • adjf = Fraction404 - 0.025 (ensures workers don't go below 2.5% baseline)
  • decreaseFraction = min(pow(retryDelayTime/8 * abs(adjf), 0.88) / 100, 0.65)
  • Scales down when 404 rate is high (we're ahead of valid instances)
  • Scales up when 404 rate is low (we're behind valid instances)
• Period length calculation (targets ~5 minutes of work):
  • If PGCRRate == 0: 600 * newWorkers (fallback estimate)
  • If Fraction404 >= 0.075: 100 * PGCRRate (reduced period when encountering gaps)
  • Otherwise: 300 * PGCRRate (5 minutes = 300 seconds at current rate)

Rationale: Fine-tune worker count based on 404 rate. Higher 404 rates indicate we're ahead of valid instances, so we scale down to avoid wasting resources. Lower 404 rates indicate we're behind, so we scale up. The adjustment is gradual to prevent oscillation. Period length adapts to maintain ~5 minutes of work, with shorter periods when encountering gaps.

Transition:

• If 404 rate exceeds 50% → Strategy 1 (Gap Detection)
• If 404 rate drops below 0.1% or lag exceeds 600s → Strategy 2 (Catch-Up)

Period-Based Execution

Atlas operates in discrete periods:

1. Ensure Minimum Workers: Workers is ensured to be at least minWorkers before spawning
2. Spawn Workers: Create workers goroutines that process instances from a channel
3. Distribute IDs: Feed periodLength instance IDs to workers (incrementing by Skip + 1 each time)
4. Wait for Completion: Wait for all workers to finish processing their assigned instances
5. Collect Metrics: Fetch metrics from Prometheus for the elapsed time
   • On metrics failure: Log error and continue with previous worker count/period length
6. Make Scaling Decision: Calculate new worker count and period length based on metrics
7. Repeat: Start next period with new configuration

Period Length Calculation

Period length determines how many instances each worker processes before the next scaling decision. The period length is carefully designed to target approximately 5 minutes of work when healthy, ensuring consistent scaling decision intervals while adapting to current throughput.

Period length is calculated differently based on the scaling strategy:

• High 404: Fixed at 2500 (quick probe for gap detection)
• Catch-Up: Dynamic based on lag and worker count (may exceed 5 minutes to accelerate catch-up)
• Normal Operation (designed for ~5 minutes):
  • If PGCRRate == 0: 600 * newWorkers (fallback estimate)
  • If Fraction404 >= 0.075: 100 * PGCRRate (reduced period when encountering gaps)
  • Otherwise: 300 * PGCRRate (targets 5 minutes = 300 seconds at current processing rate)

The design ensures that under normal healthy conditions, each period takes approximately 5 minutes, providing regular scaling decision intervals, adequate time for metrics collection, and stable throughput measurement windows.

Gap Detection and Handling

The gap checker (gap_checker.go) runs independently every 5 minutes:

1. Detection: Checks if Fraction404 > 0.8 && Count404 > 50
2. Supercharge: Spawns 500 additional workers to process 10,000 instances
3. Re-evaluation: Collects metrics after supercharge
4. Binary Search: If Fraction404 > 0.99, performs binary search to find the start of the next valid block
5. Skip: Updates LatestId to skip the gap

Gap Block Skip

When a gap is detected and skipped:

• All instance IDs in the gap are logged as "missed"
• The crawler jumps forward to the next valid instance ID
• Workers continue processing from the new position

Worker Processing

Each worker follows the API polling behavior described in the Crawling Strategy section:

1. Receives instance ID from channel
2. Fetches PGCR for assigned instance ID with retry logic
3. Processes the PGCR (validates, extracts data)
4. Publishes to storage queue if successful
5. Offloads problematic instances to a separate channel after max retries

See "API Polling Behavior" in the Crawling Strategy section for detailed retry logic and error handling.

Metrics Collection

Metrics are collected from Prometheus using query ranges:

• Time Window: Uses min(4, elapsedTime.Minutes) for most metrics
• Lag Metric: Uses fixed [2m] window
• Weighted Average: Calculates weighted average over time range (recent points weighted more)

Metrics are collected from Prometheus after each period completes, providing a view of system performance over the measurement window.

Error Handling: If metrics collection fails, the system logs an error and continues with the previous worker count and period length, ensuring continuous operation even when Prometheus is temporarily unavailable.

Configuration

Worker Limits

• Minimum Workers: 5 (minWorkers)
• Maximum Workers: 250 (maxWorkers) or configured MaxWorkers

Startup Configuration

• Initial Workers: Set via --workers flag (default: 25)
• Target Instance ID: Can specify starting instance ID via --target flag or use latest from database
• Buffer: Offset from target/latest instance ID (see "Buffer and Skip Configuration" in Crawling Strategy)
• Dev Skip: Number of instances to skip between processed instances (see "Buffer and Skip Configuration" in Crawling Strategy)

Scaling Decision Flow

┌─────────────────┐
│ Ensure Min      │
│ Workers         │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Spawn Workers   │
│ Period Length   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Process Instances│
│ (Wait for all)  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Collect Metrics │
│ (Prometheus)    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Metrics Success?│
│ Yes: Continue   │
│ No: Log & Loop  │
└────────┬────────┘
         │ (Yes)
         ▼
┌─────────────────┐
│ Evaluate State  │
│ - Fraction404   │
│ - P20Lag        │
│ - ErrorFraction │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Select Strategy │
│ 1. High 404     │
│ 2. Catch-Up     │
│ 3. Normal       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Calculate New   │
│ - Workers       │
│ - Period Length │
│ (with edge case │
│  safeguards)    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Apply Bounds    │
│ (min/max)       │
└────────┬────────┘
         │
         └──────────────┐
                        │
                        ▼
                    (Repeat)

Known Limitations and Issues

Critical Issues

1. No Dead Zone: Small metric changes trigger scaling, causing oscillation
2. Gap Checker Interference: Gap checker spawns workers independently, affecting metrics

Edge Cases

1. Time Windows: Lag metric uses [2m] while others use [intervalMins]m

Monitoring and Alerting

Atlas sends status updates to Discord including:

• Lag Behind Head (P20)
• 404 Percentage
• Error Percentage
• Workers Used
• Worker Start/Stop events
• Gap detection alerts
• Runaway error alerts

Best Practices

1. Monitor Worker Oscillation: Watch for rapid worker count changes
2. Track Metrics Freshness: Ensure Prometheus is collecting data regularly
3. Monitor Gap Detection: High 404 rates should trigger gap detection
4. Verify Scaling Effectiveness: Compare metrics before/after scaling decisions

Future Improvements

1. Add dead zone to prevent oscillation
2. Track actual throughput per worker
3. Coordinate gap checker with main scaling loop
4. Implement feedback loop to verify scaling effectiveness
5. Use consistent time windows for all metrics