tests: add k6 stress and resilience suite

.github/workflows/docker-integration.yml

@@ -10,6 +10,7 @@ on:
       - 'gunicorn/asgi/**'
       - 'gunicorn/workers/**'
       - 'tests/docker/h2spec/**'
+      - 'tests/docker/stress/**'
       - '.github/workflows/docker-integration.yml'
   pull_request:
     paths:
@@ -19,6 +20,7 @@ on:
       - 'gunicorn/asgi/**'
       - 'gunicorn/workers/**'
       - 'tests/docker/h2spec/**'
+      - 'tests/docker/stress/**'
       - '.github/workflows/docker-integration.yml'
 
 permissions:
@@ -69,3 +71,21 @@ jobs:
       - name: Run h2spec against each HTTP/2 worker
         run: |
           pytest tests/docker/h2spec/ -v --tb=short
+
+  stress-smoke:
+    name: Stress smoke (k6)
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    steps:
+      - uses: actions/checkout@v7
+      - name: Set up Python
+        uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+      - name: Install test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install pytest pytest-cov requests
+      - name: Run the k6 stress smoke matrix
+        run: |
+          pytest tests/docker/stress/test_stress.py -v --tb=short

.github/workflows/stress-nightly.yml

@@ -0,0 +1,41 @@
+name: Stress Nightly
+
+on:
+  schedule:
+    - cron: '0 3 * * *'   # 03:00 UTC daily
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+env:
+  FORCE_COLOR: 1
+
+jobs:
+  stress-heavy:
+    name: Stress and resilience (k6 + toxiproxy)
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v7
+      - name: Set up Python
+        uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+      - name: Install test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install pytest pytest-cov requests
+      - name: Run the full stress suite (smoke + resilience + faults)
+        env:
+          GUNICORN_STRESS_HEAVY: "1"
+        run: |
+          pytest tests/docker/stress -v --tb=short
+      - name: Upload k6 results and container logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: stress-results
+          path: |
+            tests/docker/stress/_results/**
+          if-no-files-found: ignore

tests/docker/README.md

@@ -47,6 +47,7 @@ to 8000 and collides with `asgi_compliance`.
 | `asgi_framework_compat` | 8001 to 8006 |
 | `http2` | 8443, 8444 |
 | `h2spec` | 8451, 8452, 8453 |
+| `stress` | 8460 to 8466, 8474, 8475 |
 | `uwsgi` | 8080 |
 
 If a suite fails everywhere at once, check for a port collision before looking
@@ -66,16 +67,18 @@ lsof -nP -iTCP:8000 -sTCP:LISTEN
 | `dirty_ttin_ttou` | scaling dirty workers with TTIN/TTOU |
 | `http2` | HTTP/2 over TLS, direct and behind nginx |
 | `per_app_allocation` | per-app worker allocation end to end |
+| `stress` | k6 load and resilience across workers, direct and behind nginx |
 | `uwsgi` | uWSGI binary protocol behind nginx |
 
 `asgi/` and `test_asgi_uwsgi/` are shell-driven demos (`test_asgi.sh`,
 `test_uwsgi.sh`), not pytest suites, and the runner skips them.
 
 ## CI
 
-`.github/workflows/docker-integration.yml` runs `tests/docker/uwsgi/` only. The
-other suites are not gated, so run them locally before changing the areas they
-cover.
+`.github/workflows/docker-integration.yml` runs `uwsgi`, `h2spec`, and the
+`stress` smoke matrix. The heavier `stress` resilience and fault scenarios run
+in `.github/workflows/stress-nightly.yml`. The other suites are not gated, so
+run them locally before changing the areas they cover.
 
 ## Certificates
 

tests/docker/stress/.gitignore

@@ -0,0 +1,3 @@
+certs/
+_results/
+*.log

tests/docker/stress/Dockerfile.gunicorn

@@ -0,0 +1,17 @@
+# syntax=docker/dockerfile:1
+FROM python:3.14-slim
+
+RUN apt-get update && apt-get install -y --no-install-recommends gcc curl procps \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY . /gunicorn-src
+RUN pip install --no-cache-dir "/gunicorn-src/[http2,gevent,testing]" \
+    && pip install --no-cache-dir "starlette>=0.35.0" "websockets>=12.0"
+
+WORKDIR /app
+COPY tests/docker/stress/apps /app
+COPY tests/docker/stress/entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+
+EXPOSE 8000 8443
+ENTRYPOINT ["/entrypoint.sh"]

tests/docker/stress/Dockerfile.nginx

@@ -0,0 +1,6 @@
+FROM nginx:1.29-alpine
+RUN apk add --no-cache curl
+RUN rm -f /etc/nginx/conf.d/default.conf
+COPY tests/docker/stress/nginx.conf /etc/nginx/nginx.conf
+COPY tests/docker/stress/uwsgi_params /etc/nginx/uwsgi_params
+EXPOSE 8461 8463 8464 8466

tests/docker/stress/README.md

@@ -0,0 +1,97 @@
+# Stress and resilience suite
+
+Drives real load at gunicorn with [k6](https://k6.io) and asserts the server
+stays correct under it: no failed requests, intact bodies and checksums, the
+negotiated protocol, and no worker tracebacks. Use it when a change touches the
+workers, the ASGI/HTTP-2 paths, or process management and you need to know it
+holds up under concurrency, not just on a single request.
+
+Load runs as a pinned k6 container inside the compose network; network faults
+run through a pinned [Toxiproxy](https://github.com/Shopify/toxiproxy). Nothing
+is installed on the host and Grafana Cloud is not used. HTTP/2 and HPACK
+conformance stay with the `h2spec` suite; k6 is a load driver, not a fuzzer.
+
+## Run it
+
+```sh
+PYTEST=".venv/bin/python -m pytest" scripts/run_docker_tests.sh stress
+```
+
+This runs the smoke matrix only. The heavier resilience and Toxiproxy scenarios
+are opt-in:
+
+```sh
+GUNICORN_STRESS_HEAVY=1 .venv/bin/python -m pytest tests/docker/stress -v
+```
+
+Requires a running Docker daemon. The first run builds the gunicorn and nginx
+images and pulls `grafana/k6` and `ghcr.io/shopify/toxiproxy`; later runs reuse
+them.
+
+## What it covers
+
+Smoke matrix (wired and verified):
+
+| Config | Worker | Path | Protocol |
+| --- | --- | --- | --- |
+| `sync-direct-h1` | sync | direct | HTTP/1.1 |
+| `gthread-nginx-h1` | gthread | behind nginx | HTTP/1.1 |
+| `asgi-direct-h1` | asgi | direct | HTTP/1.1 |
+| `asgi-nginx-h1` | asgi | behind nginx | HTTP/1.1 |
+| `asgi-ws-nginx` | asgi | behind nginx | WebSocket |
+| `h2-direct-tls` | asgi | direct | HTTP/2 (TLS) |
+
+nginx terminates HTTP/2 and the uWSGI protocol downstream only; the gunicorn
+upstream always sees HTTP/1.1 (proxy) or the uWSGI protocol (`uwsgi_pass`). The
+`uwsgi_pass` topology targets the sync worker, gunicorn's supported uWSGI path;
+the asgi worker's uWSGI parser is not exercised here.
+
+Heavy scenarios (`GUNICORN_STRESS_HEAVY=1`): kill a worker, HUP reload, TTIN/TTOU
+scaling, and Toxiproxy latency, bandwidth, and connection-reset faults, each
+applied while load runs.
+
+## Selecting worker, topology, protocol, and profile
+
+Every combination is reachable by environment. The k6 scenario is chosen with
+`SCENARIO` (`smoke`, `constant`, `ramping`, `spike`, `churn`, `soak`) and tuned
+with `RATE`, `DURATION`, `VUS`, `MAXVUS`, `FAIL_BUDGET`, `MAX_P95`, `MAX_P99`.
+Gunicorn services are shaped by the compose environment: `MAX_REQUESTS`,
+`ASGI_LOOP` (`auto`/`asyncio`/`uvloop`), and `H2_WORKER` (which worker backs the
+HTTP/2 service). For example, to run the asgi worker on uvloop with request
+recycling:
+
+```sh
+ASGI_LOOP=uvloop MAX_REQUESTS=1000 docker compose -p gunicorn_stress \
+  -f tests/docker/stress/docker-compose.yml up -d --build
+```
+
+## Ports
+
+Fixed host ports (run one docker suite at a time; a collision answers from the
+wrong stack):
+
+| Port | Target |
+| --- | --- |
+| 8460 | sync, direct |
+| 8461 | gthread, behind nginx |
+| 8462 | asgi, direct |
+| 8463 | asgi, behind nginx (HTTP + WebSocket) |
+| 8464 | HTTP/2 (TLS) behind nginx |
+| 8465 | asgi HTTP/2 (TLS), direct |
+| 8466 | sync behind nginx `uwsgi_pass` |
+| 8474 | Toxiproxy admin API |
+| 8475 | Toxiproxy proxy |
+
+## Resource expectations
+
+The smoke matrix runs in a few minutes on a laptop. The nightly profiles
+(`ramping`, `spike`, `soak`) and the heavy resilience/fault tests run longer and
+push more concurrency; give them a host with a few spare cores and ~2 GB free.
+The `soak` scenario defaults to 30 minutes (`DURATION` overrides it).
+
+## Evidence
+
+k6 writes a machine-readable summary per run to `_results/<config>-<scenario>.json`
+(git-ignored) and prints error rate, check rate, p95/p99/max latency, and dropped
+iterations. Container logs are available with
+`docker compose -p gunicorn_stress logs <service>`.