nntp-client-bench

A client-neutral, reproducible benchmark harness for NNTP download clients.

The project measures a defined workload against a deterministic local NNTP server. It is built to make comparisons auditable—not to promote a particular client. A published result is valid only when it includes its JSON artifact, image digests or versions, command line, host details, and all failed legs.

What it measures

The bundled raw and queue scenarios exercise the download path:

Scenario	Workload	Completion condition
raw	One synthetic binary NZB	Client terminal success and expected payload bytes on disk
queue	Multiple separately submitted synthetic NZBs	Every job terminally succeeds and expected total payload bytes exist

The following post-processing scenarios are also available. Every one is validated by recursively locating expected output files and matching their SHA-256 and size, not merely by observing a client success state.

Scenario	Workload
par2-basic	Deterministic basic-loss PAR2 recovery
par2-high-loss	Deterministic high-loss PAR2 recovery
direct-unpack	RAR fixture with direct unpack enabled
archive-zip, archive-7z, archive-rar	Nested archive extraction
api-validation	Authenticated client status, queue, and history API contract

Post-processing fixtures are built before the comparison clock. The mock serves those fixture articles from its shared volume and intentionally returns 430 for a stable spread of primary-data articles in PAR2 legs. RAR creation uses a licensed writer supplied through FIXTURE_RAR_CMD; without it, the attempted RAR leg is retained as an error rather than masquerading as a ZIP fixture.

Timing boundary

The comparison clock starts immediately before the first NZB is submitted and ends only when the client reports terminal success and the expected decoded payload is visible in its download directory. The reported elapsed_s and throughput therefore measure NZB submission to validated output.

Container startup, client boot, configuration installation, API readiness, fixture cleanup, log collection, and teardown are deliberately excluded from that metric. They are retained as diagnostic timeline events so they can be investigated without being mistaken for download performance.

The mock server deterministically generates each yEnc article from its message ID. It contains no downloaded content and has no provider, retention, routing, or article-availability variable. This means the benchmark is useful for comparing client overhead, decoding, write behavior, queue handling, CPU, RSS, and disk footprint under an explicitly bounded workload.

It is deliberately not a claim about real-world Usenet performance. TLS, WAN latency, provider limits, and provider-specific availability behavior are outside the defined local fixture.

Included adapters

The initial adapter set is RustNZB, NZBFast, SABnzbd, and NZBGet. They are peers: no client is treated as a baseline, omitted from reports when it fails, or given a different completion criterion. Additional clients are welcome; see clients/README.md.

Images are version-pinned in docker-compose.yml where a public immutable tag is available. The RustNZB image is configurable because release-image availability varies by environment:

RUSTNZB_IMAGE=registry.example/rustnzb:1.3.7 ./run.sh

For a formal report, pin every image to an immutable digest and retain that exact compose file with the results.

A configurable image default can silently lag behind the source tree you're testing. docker-compose.yml has no build: section for the adapter services, so unless you override an image variable (for example RUSTNZB_IMAGE) explicitly, the harness pulls whatever the default tag currently resolves to — which may predate the commit you're actually trying to benchmark. Before drawing any conclusion from a client source change, build a local image from the exact checkout under test and pass it explicitly, for example:

RUSTNZB_IMAGE=rustnzb:local ./run.sh

Requirements

• Docker Engine with the Compose plugin
• Python 3.10+; the controller uses only the standard library
• Enough free local disk for the selected payload size and client working data

The fixture exposes client APIs only on 127.0.0.1, while it is running:

Client	Port
RustNZB	19090
SABnzbd	18080
NZBGet	16789
NZBFast	16790

Run a benchmark

# Validate the environment with a small, single-round workload.
./run.sh --size 32MiB --rounds 1

# Default: raw and queue scenarios, three rotated rounds, 512 MiB per job.
./run.sh

# Download-path comparison at equal, explicit concurrency.
./run.sh --scenarios raw --size 5GiB --connections 16 --rounds 5

# Report a separate leg for each requested connection limit.
./run.sh --scenarios raw,par2-basic,par2-high-loss,direct-unpack,archive-zip,archive-7z,api-validation \
  --size 1GiB --connection-limits 8,20,50,100 --rounds 1

# Or run the standard 8/20/50/100 matrix wrapper.
./scripts/run-connection-matrix.sh

# Run a subset while developing an adapter.
./run.sh --clients sabnzbd,nzbget --scenarios raw --size 128MiB --rounds 1

# Apply the same hard Docker memory ceiling to whichever client is active.
./run.sh --scenarios raw --size 5GiB --rounds 3 --memory-limit 512MiB

# Use the Compose-network API route when the controller cannot reach
# Docker's loopback-published ports (for example, from a dev container).
./run.sh --scenarios raw --size 1GiB --rounds 1 --api-transport network

Each leg gets fresh client state, output directories, and a fresh Docker fixture. Client order rotates every round to reduce systematic first/last effects. The harness records client-reported speed, Docker CPU/RSS, and disk usage, working download-space footprint, and block I/O once per profiling interval. Profiling runs separately from completion polling, so it does not add filesystem-scan time to the comparison clock. CPU, memory, and block I/O are available when the controller can read the client container's /proc tree; a remote Docker daemon leaves those values unavailable rather than distorting the elapsed result. Memory is process RSS in this local-proc mode; block I/O is a cumulative cgroup/process counter and may be zero with some storage drivers.

Results are written to results/comparison_<timestamp>.json. A terminal success is not enough: the harness checks expected output recursively by SHA-256 and size in the client download directory. Failures, timeouts, API errors, and validation failures remain in the artifact. Summary medians use successful, validated legs only and always include the attempted and passed counts.

Each leg also retains a structured trace with phase start/finish events: prepare_state, compose_up, install_config, wait_ready, configure_server, submit, download, validate_output, and teardown. The benchmark_clock trace event is the precise start of the reported comparison interval. Per-poll samples record the client-reported transfer rate, CPU, RSS, decoded bytes, and observed queue state. The last 200 client log lines are included by default; change this with --client-log-lines (or use 0 to disable retention). Trace messages are printed by default and can be suppressed with --no-trace.

Render a portable HTML view of a retained artifact with:

python3 render_report.py results/comparison_<timestamp>.json

The renderer produces comparison_<timestamp>.html next to the JSON and includes all attempted legs, including failures. Its detailed phase timeline and expandable recent client diagnostics make it possible to distinguish a slow download from slow startup, configuration, final file movement, or teardown. It intentionally does not name a baseline or declare a winner.

For a longer sequential matrix (raw 1 GiB, 5 GiB, 20 GiB, then 5 GiB with a 512 MiB client-container cap), run:

./scripts/run-extended-matrix.sh

Set LOW_MEMORY_LIMIT to use a different cap. The script creates one JSON, HTML report, and log per matrix point; clients never overlap.

For faster iteration, use the short matrix, which deliberately runs only the uncapped 1 GiB and 5 GiB raw legs:

./scripts/run-quick-matrix.sh

It defaults to the Compose-network API route; set API_TRANSPORT=auto or API_TRANSPORT=host only when that route is appropriate for the controller.

Fairness contract

The default scenarios use the same conditions for every included client:

• one local mock host and identical generated articles;
• the same NNTP credentials, plaintext transport, connection count, and no speed cap;
• fresh client state and output directories for every leg;
• scenario-specific repair and extraction settings, with unsupported client capabilities retained as unsupported legs;
• completion requires a successful terminal state plus SHA-256 verified output;
• comparison elapsed time starts at NZB submission and ends at validated output; setup and teardown are diagnostic-only;
• rotated execution order and retained failures.

Client tuning must be declared in its config template. A parity change must be made for all applicable clients, described in the README, and represented in the result command/configuration. Do not compare results from different host conditions or versions as if they came from a single run.

--memory-limit applies the same hard Docker memory ceiling to the active client container for every leg; the mock server is deliberately not capped. The requested limit is stored in the JSON artifact arguments.

Running on a schedule

For recurring, unattended runs (for example, re-running the benchmark whenever any of the bundled clients cuts a new release), pin every adapter's image to an explicit tag or digest rather than relying on docker-compose.yml's :latest-style defaults — a floating tag can silently point at a stale image relative to the release that triggered the run. A scheduled setup should:

• track each client's release feed (container registry tags or GitHub releases) for all bundled adapters, not only one;
• invoke ./run.sh with an explicit *_IMAGE override per client, pinned to the digest of the release that triggered the run;
• retain the resulting JSON/HTML artifacts and the exact image digests alongside them so a later reader can reproduce the comparison; and
• treat every adapter identically — a new release from any bundled client is an equally valid trigger, per the peer contract above.

This repository does not currently ship the scheduler itself (e.g. a GitHub Actions cron workflow or release-webhook listener); the above is the contract such automation should satisfy when it's added.

Publishing results responsibly

Publish the complete JSON, command, git revision, image digests, client versions, CPU/RAM/storage description, OS/kernel, and any deviations from the fairness contract. Include every attempted client and all non-success legs. Do not turn a mock-NNTP result into a claim about provider throughput or end-to-end media completion.

License

MIT. See LICENSE.