Threshold: A Distributed API Gateway Demonstrating Redis Lua-Based Rate Limiting

Threshold is a highly available, load-balanced API Gateway cluster built with Express, TypeScript, and Docker. It implements a Redis-backed Token Bucket rate limiter using atomic Lua scripting and includes additional Fixed Window and Sliding Window implementations for comparison and experimentation. The mock backend service is intentionally lightweight so the project focuses entirely on gateway infrastructure rather than business logic. The system is fully instrumented with Prometheus, Grafana, k6, and structured JSON logging.

---

🎯 Why Threshold Exists

Threshold explores how production API gateways enforce distributed rate limiting while remaining observable, fault-tolerant, and horizontally scalable.

The project focuses on infrastructure and system engineering (concurrency, race condition prevention, atomic scripting, load balancing, and failure policies) rather than boilerplate CRUD application development.

---

✨ Key Highlights

• Distributed Gateway Cluster: 3 load-balanced gateway instances operating behind Nginx.
• Atomic Rate Limiting: Redis Lua-based Token Bucket engine resolving race conditions.
• Outage Resilience: Fail-open and fail-closed policies safeguarding availability/security.
• Observability Stack: Custom Prometheus scrape metrics integrated into live Grafana dashboards.
• Stress Benchmarking: Dynamic k6 tests validating request rates under high concurrency.
• Request Tracing: Structured JSON logs correlated using unique request IDs (requestId).

---

🏗️ Architecture

graph TD
    Client["Client Requests"] --> Nginx["Nginx Load Balancer (Port 80)"]
    
    subgraph Gateway Cluster
        Nginx --> Gateway1["Gateway Instance 1 (Port 3000)"]
        Nginx --> Gateway2["Gateway Instance 2 (Port 3000)"]
        Nginx --> Gateway3["Gateway Instance 3 (Port 3000)"]
    end
    
    Gateway1 --> Redis["Redis (Rate Limiter)"]
    Gateway2 --> Redis
    Gateway3 --> Redis
    
    Gateway1 --> Backend["Backend API Service (Port 8000)"]
    Gateway2 --> Backend
    Gateway3 --> Backend
    
    Prometheus["Prometheus Scraper"] --> Gateway1
    Prometheus --> Gateway2
    Prometheus --> Gateway3
    Prometheus --> Grafana["Grafana Dashboard (Port 3001)"]

Loading

---

⏱️ Request Flow

sequenceDiagram
    autonumber
    actor Client
    participant Nginx as Nginx Load Balancer
    participant Gateway as Gateway Instance
    participant Redis as Redis (Lua Script)
    participant Backend as Backend Service

    Client->>Nginx: HTTP GET /users
    Nginx->>Gateway: Forward Request (Round Robin)
    
    rect rgb(35, 38, 46)
        Note over Gateway,Redis: Rate Limiter Middleware
        Gateway->>Redis: EVAL tokenBucket.lua
        Note over Redis: 1. Accumulate Refilled Tokens<br/>2. Deduct Token if available<br/>3. Return Success and Retry-After
        Redis-->>Gateway: Return success and retryAfter status
    end

    alt success is true
        Gateway->>Backend: Forward Request
        Backend-->>Gateway: HTTP 200 OK
        Gateway-->>Client: HTTP 200 OK
    else success is false
        Gateway-->>Client: HTTP 429 Too Many Requests
    end

Loading

---

⚙️ How It Works

1. Client sends a request to the Nginx reverse proxy load balancer on port 80.
2. Nginx forwards the request to one of three API gateway instances using a round-robin routing policy.
3. The gateway instance catches the request and executes an atomic Token Bucket Lua script inside Redis.
4. If a token is available, the request is allowed and forwarded downstream to the mock backend service.
5. If the bucket is empty, the gateway blocks the request and returns an HTTP 429 Too Many Requests response.
6. Throughout the lifecycle, Prometheus scrapes gateway metrics in real-time while Grafana visualizes the charts.

---

🛠️ Tech Stack

• Language: TypeScript, JavaScript, Lua
• Runtime: Node.js (v24)
• Framework: Express
• Infrastructure: Docker, Docker Compose, Nginx
• Data Store: Redis
• Observability: Prometheus, Grafana
• Load Testing: k6
• Logging: Pino, Pino HTTP

---

🐳 Docker Compose Stack

The system launches as a single, containerized stack composed of the following services:

• nginx: Port 80 reverse proxy round-robing requests.
• gateway1, gateway2, gateway3: Load balanced API gateway nodes.
• backend: Service endpoint containing mock resource APIs.
• redis: Atomic rate limiter state database.
• prometheus: Scraper fetching gateways operational metrics.
• grafana: Web console visualizing scraping reports (Port 3001).
• k6: Load testing utility container.

---

💡 Engineering Challenges

• Race Condition Prevention: Resolving read-modify-write data hazards across separate gateway instances by packaging algorithms into single atomic Redis Lua transactions.
• Fractional Token Refill: Implementing continuous token replenishment based on elapsed time in milliseconds, avoiding scheduled refill cron jobs while maintaining precision.
• Resilience Configuration: Designing a dual fail-open/fail-closed protection layer wrapped in a 250ms Redis query timeout boundary to keep gateways responsive.
• Telemetry Instrumentation: Capturing and exposing real-time HTTP metrics (latencies, counts, blocks) without introducing overhead.
• Request Correlation: Generating and propagating unique correlation IDs (X-Request-Id) across proxies, gateway layers, and JSON logger blocks to isolate distributed traces.

---

📂 Project Structure

threshold/
├── gateway/          # API Gateway service source code (TypeScript)
│   ├── src/
│   │   ├── lua/      # Atomic Lua scripts (tokenBucket.lua, slidingWindow.lua, etc.)
│   │   └── ...
├── backend/          # Mock Backend API service (TypeScript)
├── nginx/            # Nginx Load Balancer templates
├── prometheus/       # Prometheus scraper configurations
├── grafana/          # Grafana dashboard & datasource provisioning configs
├── k6/               # load-test.js script definitions
└── docker-compose.yml

---

🚦 API Endpoints

Gateway Cluster Endpoints

• GET /: Home endpoint returning greeting.
• GET /live: Liveness check returning process state.
• GET /ready: Readiness check evaluating active database sockets.
• GET /health: Evaluates Gateway connection status to dependencies.
• GET /metrics: Exposes Prometheus metrics output. Logs custom metrics:
  • http_requests_total: Total count of processed requests.
  • http_request_duration_seconds: Histogram measuring latency durations.
  • rate_limit_blocks_total: Total count of rate-limited (HTTP 429) events.

Backend Endpoints (Load Balanced)

• GET /users: Returns mock user data.
• GET /products: Returns mock product data.

---

⚙️ Configuration

Controlled via the root .env environment configuration:

• PUBLIC_PORT: Port exposed by Nginx to public clients (default: 80).
• RATE_LIMIT_FAILURE_MODE: Define limiter outage policies (fail-open or fail-closed).
• GATEWAY_PORT: Port mapping internal gateway instances (default: 3000).
• TOKEN_BUCKET_CAPACITY: Maximum burst tokens stored (default: 3).
• TOKEN_REFILL_RATE: Time duration in seconds to refill the bucket (default: 60).
• REDIS_TIMEOUT_MS: Milliseconds boundary before failing rate-limiting requests (default: 250).

---

📊 Rate Limiting Algorithm Comparison

Threshold implements multiple algorithms. Here is a comparison of their trade-offs:

Algorithm	Memory Usage	Burst Support	Accuracy	Redis Data Structure
Fixed Window	Low (1 key per window)	Poor (Double limits at boundaries)	Medium	String (with INCR)
Sliding Window	High (Keeps timestamps of all reqs)	Good (Smooth enforcement)	High	Sorted Set (ZSET)
Token Bucket	Low (Stores capacity, last refill epoch)	Excellent (Handles bursts cleanly)	High	Hash (HMSET)

---

📈 Benchmark Results

Under a 100 Virtual Users (VUs) concurrent stress test for 1 minute:

Metric	Result
Throughput	793.7 Requests/Sec (55,127 total requests)
Average Latency	25.3 ms (p95: 4.22 ms)
Blocked Rate	99.98% (55,121 requests rejected with 429)
Error Rate	0.03% (TCP timeouts under maximum concurrency)

Note

Understanding Latency Metrics (Why Average > p95): Under high-concurrency rate limiting, 99.98% of requests are rejected instantly with an HTTP 429 status code (average round-trip duration: 1.5 ms). Since these fast 429 responses make up the overwhelming majority of the request volume, they dominate the percentiles, keeping the 95th percentile (p95) extremely low at 4.22 ms.

In contrast, the remaining 0.02% of requests that are allowed to pass through must be proxied downstream to the backend container, experiencing TCP queuing delays and processing overhead (average duration: 2.14 s, max: 12.81 s). These slow tail-latency outliers pull the arithmetic average up to 25.3 ms, making the average duration significantly larger than the p95 percentile.

The load test intentionally flooded the system, exceeding the configured Token Bucket rate-limit capacity. The benchmark results prove the gateway correctly and safely enforced rate limits by rejecting excess requests with HTTP 429 while keeping latencies low and minimizing internal errors.

---

🖼️ Visuals & Screenshots

---

🚀 Setup & Execution

Prerequisites

• Docker & Docker Compose installed.

Commands

1. Spin up the entire stack:

docker compose up -d --build

2. Verify container statuses:

docker compose ps

3. Run a k6 load test:

docker compose run --rm k6

4. Shut down the stack:

docker compose down

---

📖 Reference & Documentation Links

• Prometheus Setup: Prometheus Scraper Config Reference
• Grafana Provisioning: Grafana Provisioning Guide
• Grafana k6: k6 JavaScript API Reference
• Redis Scripting: Redis Lua Programmability (EVAL)
• Pino Logging: Pino API Documentation
• Docker Compose: Docker Compose Overview & CLI