DRAFT: feat(config): consolidate all durations and deadlines into one Timeouts model

.pre-commit-config.yaml

@@ -54,7 +54,7 @@ repos:
         entry: python scripts/regenerate_templates.py
         language: system
         pass_filenames: false
-        files: ^(src/inference_endpoint/config/(schema\.py|templates/.*)|src/inference_endpoint/endpoint_client/config\.py|scripts/regenerate_templates\.py)$
+        files: ^(src/inference_endpoint/config/((schema|enums|audit|model_params|datasets|settings|timeouts)\.py|templates/.*)|src/inference_endpoint/endpoint_client/config\.py|scripts/regenerate_templates\.py)$
 
       - id: add-license-header
         name: Add license headers

AGENTS.md

@@ -92,7 +92,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint
 | **Dataset Manager**                 | `src/inference_endpoint/dataset_manager/`                                                                          | Loads JSONL, HuggingFace, CSV, JSON, Parquet datasets. `Dataset` base class with `load_sample()`/`num_samples()` interface                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | **Metrics Aggregator**              | `src/inference_endpoint/async_utils/services/metrics_aggregator/`                                                  | Subprocess. Subscribes to events, aggregates per-sample metrics into a `MetricsRegistry` (counters + HDR-histogram series + raw values), publishes `MetricsSnapshot` over IPC PUB at a configurable cadence (`SessionState`: `INITIALIZE` → `LIVE` → `DRAINING` → {`COMPLETE` \| `INTERRUPTED`}). Final snapshot is atomically written to `final_snapshot.json` as the **primary** Report source; the terminal pub/sub frame is a TUI "run finished" signal only.                                                                                                                                                                                                                                                                                                              |
 | **Report**                          | `src/inference_endpoint/metrics/report.py`                                                                         | `Report.from_snapshot(dict)` — pure-function builder consuming the dict form (`snapshot_to_dict`). Reads `final_snapshot.json` directly via `json.loads` (no Struct decode). Plumbs `complete = (state == "complete" and n_pending_tasks == 0)`; renders an explicit warning for `INTERRUPTED` runs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
-| **Config**                          | `src/inference_endpoint/config/`, `endpoint_client/config.py`                                                      | Pydantic-based YAML schema (`schema.py`), `HTTPClientConfig` (single Pydantic model for CLI/YAML/runtime), `RuntimeSettings`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+| **Config**                          | `src/inference_endpoint/config/`, `endpoint_client/config.py`                                                      | Pydantic-based YAML schema split into focused modules (`schema.py` = BenchmarkConfig + re-export hub; `enums.py`, `audit.py`, `model_params.py`, `datasets.py`, `settings.py`), `Timeouts` (`config/timeouts.py` — all global durations/deadlines in one frozen model at `settings.timeouts`, incl. the whole-run `run_timeout_s` watchdog), `HTTPClientConfig` (single Pydantic model for CLI/YAML/runtime), `RuntimeSettings`                                                                                                                                                                                                                                                                                                                                                |
 | **CLI**                             | `src/inference_endpoint/main.py`, `commands/benchmark/cli.py`                                                      | cyclopts-based, auto-generated from `schema.py` and `HTTPClientConfig` Pydantic models. Flat shorthands via `cyclopts.Parameter(alias=...)`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | **Async Utils**                     | `src/inference_endpoint/async_utils/`                                                                              | `LoopManager` (uvloop + eager_task_factory), ZMQ transport layer, generic `MessageCodec[T]`-parametrized pub/sub, event publisher                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
 | **OpenAI/SGLang**                   | `src/inference_endpoint/openai/`, `sglang/`                                                                        | Protocol adapters and response accumulators for different API formats. `openai_completions` adapter (`completions_adapter.py`) sends pre-tokenized token IDs to `/v1/completions`, bypassing the server chat template — required for gpt-oss-120b on vLLM. `sglang` adapter sends to `/generate` via `input_ids`. Both apply `Harmonize()` client-side.                                                                                                                                                                                                                                                                                                                                                                                                                        |
@@ -118,7 +118,7 @@ The aggregator is a separate process (`python -m inference_endpoint.async_utils.
 
 - **Series storage**: each `SeriesSampler` keeps three parallel views: O(1) cheap rollups (count/total/min/max/sum_sq, exact), an HDR Histogram (cheap live percentiles), and an in-memory `array.array` of raw values (for exact percentiles in the `COMPLETE` snapshot). Hot path is `registry.record(name, value)` — no allocation, no I/O.
 - **Counter API**: `registry.increment(name, delta=1)` for sample-event counters. `registry.set_counter(name, value)` only for the three derived-duration counters (`total_duration_ns` max-of-elapsed, `tracked_duration_ns` sum-of-blocks, `legacy_loadgen_window_duration_ns` first-issue→last-issued-completion span for LoadGen-parity QPS/TPS).
-- **Lifecycle**: `INITIALIZE` (constructed, awaiting first `STARTED`) → `LIVE` (run in progress, ticking every `--publish-interval` seconds) → `DRAINING` (set on `ENDED`; tick continues; bounded by the `--drain-timeout` budget — schema default 0 = unlimited) → terminal: `COMPLETE` (clean end via `publish_final`, exact stats) **or** `INTERRUPTED` (signal-handler-triggered final via SIGTERM/SIGINT; best-effort partial stats). Drain timeout detected by consumers as `state == COMPLETE and n_pending_tasks > 0`; interrupted runs are detected as `state == INTERRUPTED` directly.
+- **Lifecycle**: `INITIALIZE` (constructed, awaiting first `STARTED`) → `LIVE` (run in progress, ticking every `--publish-interval` seconds) → `DRAINING` (set on `ENDED`; tick continues; bounded by the `--drain-timeout` budget — argv 0 = unlimited; schema `settings.timeouts.metrics_drain_timeout_s` uses None = unlimited, converted at the argv boundary) → terminal: `COMPLETE` (clean end via `publish_final`, exact stats) **or** `INTERRUPTED` (signal-handler-triggered final via SIGTERM/SIGINT; best-effort partial stats). Drain timeout detected by consumers as `state == COMPLETE and n_pending_tasks > 0`; interrupted runs are detected as `state == INTERRUPTED` directly.
 - **Final delivery is dual-path with separated concerns**: `publish_final` atomically writes `final_snapshot.json` (`tmp + fsync(file) + rename + fsync(parent_dir)`) — this is the **primary** Report source — AND emits the terminal-state snapshot over pub/sub as a TUI shutdown signal. Each path is wrapped in its own try/except so one failure cannot suppress the other. Main process consumer reads `final_snapshot.json` (via `json.loads` to dict, no Struct decode); falls back to the subscriber's `latest` live snapshot only if the file is missing (e.g. SIGKILL / OOM before the signal handler ran). The dict form is the canonical consumer contract (see `snapshot_to_dict`).
 - **Histogram bucket edges are dynamic per snapshot**: log-spaced over the observed `[min, max]`. Bucket count is fixed at construction; consumers MUST re-render from the snapshot's `(lo, hi, count)` triples each frame and MUST NOT track bucket-by-index across snapshots.
 
@@ -127,7 +127,7 @@ The aggregator is a separate process (`python -m inference_endpoint.async_utils.
 CLI is auto-generated from `config/schema.py` Pydantic models via cyclopts. Fields annotated with `cyclopts.Parameter(alias="--flag")` get flat shorthands; all other fields get auto-generated dotted flags (kebab-case).
 
 - **CLI mode** (`offline`/`online`): cyclopts constructs `OfflineBenchmarkConfig`/`OnlineBenchmarkConfig` (subclasses in `config/schema.py`) directly from CLI args. Type locked via `Literal`. `--dataset` is repeatable with TOML-style format `[perf|acc:]<path>[,key=value...]` (e.g. `--dataset data.csv,samples=500,parser.prompt=article`). Full accuracy support via `accuracy_config.eval_method=pass_at_1` etc.
-- **YAML mode** (`from-config`): `BenchmarkConfig.from_yaml_file()` loads YAML, resolves env vars, and auto-selects the right subclass via Pydantic discriminated union. Optional `--timeout`/`--mode` overrides via `config.with_updates()`.
+- **YAML mode** (`from-config`): `BenchmarkConfig.from_yaml_file()` loads YAML, resolves env vars, and auto-selects the right subclass via Pydantic discriminated union. Optional `--timeout` (maps to `settings.timeouts.run_timeout_s`)/`--mode` overrides via `config.with_updates()`.
 - **eval**: Not yet implemented (raises `CLIError` with a tracking issue link)
 
 ### Config Construction & Validation
@@ -239,7 +239,13 @@ src/inference_endpoint/
 │   ├── metric.py              # Metric types (Throughput, etc.)
 │   └── results_plots.py       # Standardized run-artifact plots (matplotlib-guarded); CLI: scripts/plot_results.py
 ├── config/
-│   ├── schema.py              # Single source of truth: Pydantic models + cyclopts annotations
+│   ├── schema.py              # BenchmarkConfig + EndpointConfig; re-export hub for the schema surface
+│   ├── enums.py               # Shared schema enums (TestType, LoadPatternType, StreamingMode, ...)
+│   ├── audit.py               # Audit config models (audit: YAML block)
+│   ├── model_params.py        # ModelParams, OSLDistribution, SubmissionReference
+│   ├── datasets.py            # Dataset, AccuracyConfig, AgenticInferenceConfig
+│   ├── settings.py            # Settings + Runtime/LoadPattern/Warmup/Metrics/Profiling configs
+│   ├── timeouts.py            # Timeouts — all global durations + deadlines (settings.timeouts)
 │   ├── runtime_settings.py    # RuntimeSettings + SampleOrderSpec dataclasses
 │   ├── ruleset_base.py        # BenchmarkSuiteRuleset base
 │   ├── ruleset_registry.py    # Ruleset registry

docs/CLI_DESIGN.md

@@ -204,5 +204,5 @@ class HTTPClientConfig(WithUpdatesMixin, BaseModel):
 `BenchmarkConfig` is frozen. Use `with_updates()` to produce new instances with re-validation:
 
 ```python
-config = config.with_updates(timeout=300, datasets=["new_data.jsonl"])
+config = config.with_updates(report_dir="results/run1", datasets=["new_data.jsonl"])
 ```

docs/CLI_QUICK_REFERENCE.md

@@ -96,8 +96,9 @@ Flag names shown as `--full.dotted.path --alias`. Both forms work.
 - `--model-params.max-new-tokens --max-output-tokens` - Max output tokens (default: 1024)
 - `--model-params.osl-distribution.min --min-output-tokens` - Min output tokens (default: 1)
 - `--model-params.streaming --streaming` - Streaming mode: auto/on/off (default: auto)
-- `--runtime.min-duration-ms --duration` - Min duration: ms default, or with suffix (600s, 10m) (default: 600000)
-- `--runtime.n-samples-to-issue --num-samples` - Explicit sample count override
+- `--min-duration-ms --duration` - Min perf-phase duration: ms default, or with suffix (600s, 10m); sample count = QPS x duration
+- `--runtime.n-samples-to-issue --num-samples` - Explicit sample count
+  - `--duration` and `--num-samples` are mutually exclusive; omit both to issue the dataset once (the default)
 - `--client.num-workers --workers` - HTTP workers (-1=auto, default: -1)
 - `--client.max-connections --max-connections` - Max TCP connections (-1=unlimited)
 - `--endpoint-config.api-key --api-key` - API authentication
@@ -106,7 +107,7 @@ Flag names shown as `--full.dotted.path --alias`. Both forms work.
   Note: applies to CLI-driven `benchmark offline` / `benchmark online`; `benchmark from-config`
   does not expose a CLI override for `report_dir`. Set it in the YAML only if you need to control
   the output location; otherwise a default report directory is used.
-- `--timeout` - Global timeout in seconds
+- `--timeout` - Whole-run watchdog in seconds (off by default). If it fires, the run is aborted and the report is marked INTERRUPTED (exits non-zero).
 - `--enable-cpu-affinity / --no-cpu-affinity` - NUMA-aware CPU pinning (default: true)
 
 **Online-specific:**
@@ -284,10 +285,10 @@ datasets:
 
 settings:
   runtime:
-    min_duration_ms: 600000 # 10 minutes
-    n_samples_to_issue: null # Optional: explicit sample count (null = auto-calculate)
     scheduler_random_seed: 42 # For Poisson/distribution sampling
     dataloader_random_seed: 42 # For dataset shuffling
+  timeouts:
+    min_duration_ms: 600000 # 10 minutes (or set runtime.n_samples_to_issue instead; omit both = dataset once)
   load_pattern:
     type: "max_throughput"
     target_qps: 10.0
@@ -325,8 +326,8 @@ Note: For submission configs, `model_params.name` is optional when `submission_r
 
 **Sample Count Control:**
 
-- Priority: `--num-samples` > calculated (target_qps × duration) > dataset size
-- Default duration: 600000ms (10 minutes)
+- `--num-samples` and `--duration` are mutually exclusive: explicit count, or calculated (target_qps × duration)
+- Default (neither set): issue the dataset once
 
 **Mode Requirements:**
 

docs/LOCAL_TESTING.md

@@ -74,8 +74,7 @@ Waiting for 5 responses...
 uv run inference-endpoint -v benchmark offline \
   --endpoints http://localhost:8765 \
   --model Qwen/Qwen3-8B \
-  --dataset tests/assets/datasets/dummy_1k.jsonl \
-  --duration 0
+  --dataset tests/assets/datasets/dummy_1k.jsonl
 
 # Production test with custom params and report generation
 uv run inference-endpoint -v benchmark offline \
@@ -115,7 +114,6 @@ uv run inference-endpoint -v benchmark online \
   --endpoints http://localhost:8765 \
   --model Qwen/Qwen3-8B \
   --dataset tests/assets/datasets/dummy_1k.jsonl \
-  --duration 0 \
   --load-pattern poisson \
   --target-qps 100 \
   --report-dir online_benchmark_report
@@ -311,9 +309,8 @@ uv run inference-endpoint benchmark online \
 
 **Sample Count Control:**
 
-- Use `--duration 0` when you want a local test to stop after exhausting the dataset instead of running for the default timed duration
-- Sample priority: `--num-samples` > dataset size (when `--duration 0`) > calculated (target_qps × duration)
-- Default duration: 600000ms (10 minutes)
+- By default (no `--num-samples`, no `--duration`) a run stops after issuing the dataset once
+- `--num-samples` (explicit count) and `--duration` (count = target_qps × duration) are mutually exclusive
 
 **Testing & Debugging:**
 

docs/config/DESIGN.md

@@ -61,8 +61,8 @@ Immutable snapshot of all parameters needed to execute a run.
 | -------------------- | -------------- | --------------------------------------- |
 | `load_pattern`       | `LoadPattern`  | config                                  |
 | `n_samples_to_issue` | `int`          | calculated: QPS × duration, or explicit |
-| `min_duration_ms`    | `int`          | runtime config                          |
-| `max_duration_ms`    | `int`          | runtime config                          |
+| `min_duration_ms`    | `int`          | `settings.timeouts`                     |
+| `max_duration_ms`    | `int`          | `settings.timeouts`                     |
 | `min_sample_count`   | `int`          | current default / future ruleset hook   |
 | `metric_target`      | `Metric`       | primary target driving scheduler logic  |
 | `reported_metrics`   | `list[Metric]` | metrics validated after the run         |

examples/02_ServerBenchmarking/offline_llama3_8b_cnn.yaml

@@ -19,11 +19,13 @@ datasets:
 
 settings:
   runtime:
-    min_duration_ms: 6000 # 6 seconds
-    max_duration_ms: 60000 # 1 minute
     scheduler_random_seed: 137 # For Poisson/distribution sampling
     dataloader_random_seed: 111 # For dataset shuffling
 
+  timeouts:
+    min_duration_ms: 6000 # 6 seconds
+    max_duration_ms: 60000 # 1 minute
+
   load_pattern:
     type: "max_throughput"
 

examples/02_ServerBenchmarking/online_llama2_70b_cnn.yaml

@@ -19,11 +19,13 @@ datasets:
 
 settings:
   runtime:
-    min_duration_ms: 60000 # 1 minute
-    max_duration_ms: 180000 # 3 minutes
     scheduler_random_seed: 42 # For Poisson/distribution sampling
     dataloader_random_seed: 42 # For dataset shuffling
 
+  timeouts:
+    min_duration_ms: 60000 # 1 minute
+    max_duration_ms: 180000 # 3 minutes
+
   load_pattern:
     type: "poisson"
     target_qps: 10

examples/03_BenchmarkComparison/compare_with_vllm.py

@@ -177,16 +177,13 @@ def generate_ie_config(
         ],
         "settings": {
             "runtime": {
-                "min_duration_ms": 0,
-                "max_duration_ms": timeout * 1000,
                 "n_samples_to_issue": num_requests,
             },
             "load_pattern": {"type": "max_throughput"},
             "client": {"num_workers": workers},
         },
         "endpoint_config": {"endpoints": [endpoint_url]},
         "report_dir": str(report_dir),
-        "timeout": timeout,
     }
 
     with open(config_path, "w") as f: