Inference Bottleneck Lab

ibl synthesize --scenario healthy --samples 5 --seed 7 --output baseline.json
ibl synthesize --scenario queue --samples 5 --seed 7 --output candidate.json
ibl analyze --input candidate.json --format markdown
ibl compare --baseline baseline.json --candidate candidate.json --format markdown

Inference Bottleneck Lab is a dependency-free, offline-first Python tool for turning a declared inference trace into an evidence-backed bottleneck hypothesis and a next experiment. It produces hypotheses, not root-cause proof, and never changes a runtime.

Install

Python 3.12 or newer is required.

python -m pip install .

For a checkout used during development:

python -m pip install -e ".[dev]"

What it does

A trace is an ordered JSON record of declared observations such as waiting requests, completion rate, KV-cache usage, transfer time, and declared resource fractions. The analyzer applies six deterministic rules:

• Queue pressure: waiting work persists while arrivals exceed completions.
• Compute pressure: waiting work coincides with declared compute pressure without stronger bandwidth evidence.
• Memory-bandwidth pressure: waiting work coincides with declared bandwidth pressure and compute headroom.
• KV-capacity pressure: high key-value (KV) cache use coincides with churn or an admission signal.
• Transfer pressure: state movement consumes a material share of time to first token (TTFT), or transfer failures increase.
• Scheduler pressure: work waits despite declared resource headroom and small admitted batches.

Every rule reports observations, derived signals, thresholds, counterevidence, missing evidence, alternatives, and an experiment that could disprove the hypothesis. Missing required evidence is inconclusive; multiple classes can be supported together.

Commands

Generate synthetic traces

Synthetic scenarios are CPU-only educational fixtures. Their abstract values are not runtime measurements or benchmarks.

ibl synthesize --scenario scheduler --samples 5 --seed 11 --output scheduler.json

Scenarios are healthy, queue, compute, bandwidth, kv, transfer, and scheduler. Both --samples and --seed are required; at least three samples are needed so persistence and counter changes can be represented.

Analyze a neutral trace

ibl analyze --input scheduler.json --format json
ibl analyze --input - --format markdown < scheduler.json

The effective educational thresholds are checked in at src/inference_bottleneck_lab/defaults.json. A JSON override can replace individual documented values without changing rule logic:

ibl analyze --input scheduler.json --thresholds examples/thresholds-lower-batch-floor.json

Compare declared traces

ibl compare --baseline baseline.json --candidate candidate.json --format markdown

Each comparison row uses the arithmetic mean of available samples, then requires both absolute and relative thresholds when both deltas are defined. A zero baseline emits not_defined for relative delta and uses the absolute threshold only. A regression signal is not causal proof, and the report deliberately has no aggregate score.

Normalize public runtime metrics

adapt accepts exactly one source: a file, standard input, or one explicitly supplied HTTP(S) URL. URL fetching has a timeout, body-size limit, no retry loop, no redirect following, and no credential persistence.

ibl adapt --runtime vllm --input tests/fixtures/vllm.prom --runtime-version public-example
ibl adapt --runtime sglang --stdin < tests/fixtures/sglang.prom
ibl adapt --runtime vllm --url https://example.invalid/metrics --timeout-seconds 3 --max-bytes 1048576

The output is a neutral trace with the adapter name, source kind, deterministic aggregation policy, mapped fields, and unmapped metric names. The tool does not contact a runtime unless the URL is explicitly supplied.

Input and output boundaries

• Runtime dependencies: none. The installed package uses only Python’s standard library.
• Input: neutral JSON, saved Prometheus text, standard input, or one explicit HTTP(S) metrics URL.
• Output: deterministic JSON or Markdown.
• Runtime behavior: read-only. This project does not serve models, issue load, profile accelerators, tune settings, or make automatic changes.
• Publication boundary: tracked fixtures are synthetic and adapter aliases come from public interfaces only. See publication policy.

Documentation

• Read a diagnosis
• Neutral trace schema
• vLLM mapping
• SGLang mapping
• Write a rule
• Compare baselines
• Limitations and next experiments
• Contribution and security
• Public source register
• Implementation notes

Development checks

The runtime has no dependencies. Optional development tools are declared in pyproject.toml; they are not required by installed users.

python -m unittest discover -s tests -v
ruff check .
ruff format --check .
mypy src tests scripts build_backend.py
mkdocs build --strict
python scripts/audit_publication.py --denylist /path/to/private-denylist.txt
python scripts/build.py --outdir /tmp/ibl-dist

The denylist file is external verification input and is never bundled, copied, or named by the project.

License

Licensed under Apache-2.0. See NOTICE, THIRD_PARTY, SECURITY.md, and CONTRIBUTING.md.