Skip to content

SIMULATOR.md

Latest commit

 

History

History
147 lines (116 loc) · 5.92 KB

SIMULATOR.md

File metadata and controls

147 lines (116 loc) · 5.92 KB
Raw

ReadIssue Simulator

A safe, internet-deployable version of the ReadIssue overload lab. It shows the same dashboard and the same story as the real lab — overload the database, watch p95 latency and CPU climb, then apply fixes (index, cache, connection pooling, read replicas, autoscaling, load shedding) and watch the system recover — but it runs no real infrastructure.

Why this exists

The real lab (docker-compose.yml) spins up real Postgres, Redis, Traefik, and worker containers, and the control plane mounts the Docker socket to spawn and kill them on demand. That is great on your own machine and terrible on a public server: the Docker socket is effectively root on the host, and none of the API is authenticated. It is explicitly documented as "localhost only."

The simulator replaces all of that with a queueing model. Instances, load, latency, CPU, connections, cache behaviour, errors, and 429s are all computed. There is nothing to exploit: no Docker socket, no database, no cache, no shell. That makes it safe to put on the internet behind a login, so it can be a live portfolio demo.

Real lab vs. simulator

Real lab (docker-compose.yml) Simulator (docker-compose.sim.yml)
Workers Real FastAPI containers, spawned via Docker socket Numbers in engine.py
Database Real PostgreSQL (+ replica) Modelled as the shared bottleneck
Cache Real Redis Modelled hit-ratio warm-up
Load generator Real multi-process HTTP client Modelled offered vs. served rate
Latency / CPU Measured from docker stats + workers Derived (M/M/1 sojourn, Little's law)
Docker socket Mounted (host-root risk) Not used
Auth None HTTP basic auth at the reverse proxy
Safe on public internet No Yes
Dashboard & API — Identical (frontend is unchanged)

The dashboard cannot tell the difference: the simulator serves the exact same REST + WebSocket contract (/api/*, /ws/metrics), so frontend/ is reused byte-for-byte.

How the model works

The database is the scarce resource (that's the point of the lab). Each tick (~1s) the engine:

  1. Splits offered load across running instances.
  2. Subtracts cache hits (a warmed cache absorbs a hot-key share before the DB).
  3. Computes DB utilisation ρ = arrival / capacity, where capacity depends on the query type and the active optimizations.
  4. Derives latency from an M/M/1 sojourn time service_time / (1 - ρ), then in-flight and CPU from Little's law.

Each control changes the model the way it changes the real system:

  • DB index — flips the slow read from a sequential scan (low capacity, high per-call cost) to an index scan → p95 collapses.
  • Redis cache — a hot-key share of reads skips the DB → DB active-query count drops, hit ratio climbs.
  • Read replicas — reads move onto replica lanes → primary active queries drop to ~0 while the replica lane takes the load.
  • PgBouncer — bounds DB connections (n·pool_size → a small pooled count).
  • Load shedding — rejects excess load (429) instead of queueing it, so accepted-request latency stays bounded under overload.
  • Autoscaling — the same threshold logic as the real lab, reacting to the modelled CPU/p95.

Constants live in sim/app/config.py if you want to retune the numbers.

Run it locally

No Docker needed for development:

python -m venv .venv && . .venv/bin/activate
pip install -r sim/requirements.txt

# API + simulator on :8000
uvicorn sim.app.main:app --port 8000

# dashboard (dev server, proxies /api and /ws to :8000)
cd frontend && npm install && npm run dev   # http://localhost:5173

Or run the whole thing (UI + API) from one process like production does:

cd frontend && npm run build && cd ..
STATIC_DIR=frontend/dist uvicorn sim.app.main:app --port 8000
# open http://localhost:8000

Tests

# engine model (fast, no server needed)
python -m pytest sim/tests/ -q

# live end-to-end: boots a real server, drives it over HTTP + WebSocket
python sim/verify_live.py

Deploy it safely (Option B)

The stack is two containers: sim (the app, not exposed to the host) and caddy (the only public service — it does TLS, HTTP basic auth, and reverse proxies everything to sim, WebSocket included).

docker compose -f docker-compose.sim.yml up -d --build

Steps only you can do

  1. Provision a host. Any small Linux VM with Docker works (AWS EC2 t3.micro/t4g.small, or a DigitalOcean/Hetzner droplet — roughly $5–15/month). Install Docker + the compose plugin.
  2. Open the firewall for inbound TCP 80 and 443 (on EC2 this is the security group).
  3. Point DNS. Create an A record for your chosen subdomain (e.g. readissue.yourdomain.com) pointing at the VM's public IP. Automatic HTTPS only works once DNS resolves to the box.
  4. Set the credentials. On the VM: 
    cp .env.sim.example .env
# generate a password hash:
docker run --rm caddy:2.8 caddy hash-password --plaintext 'your-strong-password'
# edit .env: set SITE_ADDRESS (your domain), BASIC_AUTH_USER, BASIC_AUTH_HASH
  5. Launch: 
    docker compose -f docker-compose.sim.yml up -d --build
    Visit https://readissue.yourdomain.com, log in, and the dashboard is live.

To test without a domain first, set SITE_ADDRESS=:80 (plain HTTP) and hit the server's IP directly.

Security notes

  • The sim container runs as a non-root user and has no host mounts, no Docker socket, and no outbound dependencies.
  • sim has no published ports — it is only reachable through Caddy.
  • All access (dashboard, API, WebSocket) is behind HTTP basic auth. This is a demo-grade control: it keeps the public and bots out. It is not a substitute for real user auth, and the simulator intentionally holds no data worth protecting.
  • Nothing user-supplied is executed or stored; the API only flips model toggles.