DRAFT: feat(config): consolidate all durations and deadlines into one Timeouts model

[viraatc]

What does this PR do?

Consolidates every global time knob into one frozen pydantic model — Timeouts in the new src/inference_endpoint/config/timeouts.py, mounted at settings.timeouts — gives the previously dead --timeout flag real semantics, splits the 1400-line config/schema.py into focused modules, and makes the sample-count knobs mutually exclusive.

The headline bug this fixes

BenchmarkConfig.timeout (--timeout, top-level timeout: YAML key) was consumed nowhere — a silent no-op. Several example YAMLs set it believing it bounded the run. It is now settings.timeouts.run_timeout_s: a real whole-run watchdog. When it fires, it SIGTERMs the metrics aggregator (whose handler writes an INTERRUPTED final_snapshot.json), stops the session, and run_benchmark raises ExecutionError (non-zero exit) after finalization — a fired watchdog can never yield a COMPLETE report and always leaves result_summary.json with complete: false. Locked by tests/integration/commands/test_run_timeout.py.

The one model

settings:
  timeouts:
    # workload durations (benchmark definition — reaching them is a normal end)
    min_duration_ms: null          # --duration (suffixes: 600s, 10m); sample count = QPS × duration
    max_duration_ms: null          # whole-perf-phase-only cap; never bounds warmup/accuracy
    # give-up deadlines (failure handling — None = wait indefinitely)
    run_timeout_s: null            # --timeout, whole-run watchdog (covers drains too), fires → INTERRUPTED
    service_ready_timeout_s: 30.0  # --service-ready-timeout
    warmup_drain_timeout_s: 240.0  # --warmup-drain-timeout
    performance_drain_timeout_s: null  # ⚠️ default changed: was 240 — now unbounded by request
    accuracy_drain_timeout_s: null # deliberately unbounded: every sample must complete
    metrics_drain_timeout_s: null  # None = unlimited (was 0 = unlimited)
  metrics:
    tokenizer_workers: 2           # --metrics-tokenizer-workers (not a timeout → own block)

All CLI flag aliases are unchanged. Only YAML key locations and auto-generated dotted flags moved.

Sample count: explicit XOR duration-derived (new)

runtime.n_samples_to_issue (--num-samples) and timeouts.min_duration_ms (--duration) are now mutually exclusive (Settings validator). Previously an explicit count silently won and the configured duration was dead weight — several examples carried both. Omitting both issues the dataset once, which also deletes the old min_duration_ms: 0 = "all samples" and max_duration_ms: 0 = "no limit" sentinels (0 is now rejected loudly).

⚠️ Reviewer callouts (deliberate behavior changes):

• Bare-config default changed: neither knob set previously derived qps × 600s worth of samples; it now runs the dataset once.
• performance_drain_timeout_s default is now None (unbounded). A default run has no time backstop on a wedged endpoint — set run_timeout_s for unattended runs.
• metrics_drain_timeout_s: 0 / min_duration_ms: 0 / max_duration_ms: 0 now error loudly.
• Example run_timeout_s values were dropped, not carried over (old top-level timeout: was inert; enforcing stale values would hard-kill valid runs). Also fixed compare_with_vllm.py, which capped the same budget twice via max_duration_ms and --timeout.

schema.py split

config/schema.py now owns only BenchmarkConfig/EndpointConfig and re-exports the full schema surface (import sites unchanged). Models moved to config/enums.py, audit.py, model_params.py, datasets.py, settings.py (+ existing timeouts.py). Dead code deleted: SystemDefaults (DEFAULT_TIMEOUT had zero consumers), TEMPLATE_TYPE_MAP.

Design invariants (multi-model design review: Codex/Opus/Grok/Gemini)

• run_timeout_s never derives per-stage deadlines (no remaining-budget propagation).
• min/max_duration_ms bound the whole performance phase (single phase; not per-dataset); max_duration_ms remains the sole perf ceiling.
• turn_timeout_s stays dataset-scoped; worker lifecycle timeouts stay on settings.client.*.
• Hot path untouched: everything resolves to plain values at setup.
• No back-compat shims: old keys hard-error; templates/examples/docs/tests migrated in this PR.

Multi-AI review hardening

A Codex review + adversary-council pass found and fixed: watchdog now covers the unbounded metrics drain; watchdog SIGTERMs only the aggregator (event logger kept its buffered events.jsonl); timed-out runs skip accuracy scoring (artifacts still salvaged); teardown race can no longer skip result_summary.json; run_audit maps a watchdog fire to ExecutionError instead of the Ctrl-C exit-130 path.

Type of change

• Bug fix
• New feature
• Refactor/cleanup

Testing

• Tests added/updated — watchdog locking test, audit run-timeout test, sample-count mutual-exclusion + dataset-once tests; all suites migrated
• All tests pass locally — tests/unit: 1522 passed, 5 skipped; integration command suites green (incl. full test_cli.py e2e)
• Manual testing completed — every example YAML load-validated; CLI --help verified aliases unchanged

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated — AGENTS.md, CLI docs, LOCAL_TESTING, config DESIGN docs, example READMEs

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request refactors the configuration schema by centralizing all global durations, deadlines, and timeouts into a new frozen Pydantic model Timeouts (accessible via settings.timeouts). This separates workload durations from failure-handling deadlines. Additionally, a whole-run watchdog (run_timeout_s) has been introduced to gracefully abort stuck runs, signaling managed subprocesses via SIGTERM to write an interrupted final snapshot before exiting non-zero. All configuration templates, examples, and tests have been updated to align with this new schema, and new integration tests have been added to verify the watchdog behavior. No review comments were provided, so there is no feedback to address.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.