diff --git a/docs/benchmark-modes/timing-modes-reference.md b/docs/benchmark-modes/timing-modes-reference.md index 507c3e6e9e..6161525cc8 100644 --- a/docs/benchmark-modes/timing-modes-reference.md +++ b/docs/benchmark-modes/timing-modes-reference.md @@ -17,7 +17,6 @@ AIPerf determines how to schedule requests based on which CLI options you specif | `--concurrency` (alone) | Saturation/throughput testing | Send requests as fast as possible within concurrency limits | | `--fixed-schedule` | Trace replay | Replay requests at exact timestamps from dataset | | `--user-centric-rate` | KV cache benchmarking | Per-user rate limiting with consistent turn gaps | -| `--adaptive-scale` | SLA boundary discovery | Adapt one load-control variable during a duration-based phase, then sustain near the last passing boundary | ### Option Priority @@ -28,7 +27,7 @@ When multiple options are specified, AIPerf uses this priority: 3. `--request-rate` → Rate-based scheduling with arrival patterns 4. `--concurrency` only → Burst mode (as fast as possible within limits) -`--adaptive-scale` wraps the selected duration-based profiling phase after normal scheduling is chosen. It adapts one control variable (`concurrency`, `prefill_concurrency`, `request_rate`, or `users`) and rejects fixed ramps on that same variable. See [Adaptive Scale](../tutorials/adaptive-scale.md) for examples and artifact semantics. +`adaptive_scale` is not a scheduling option. It is a YAML-only controller overlay that wraps a duration-based profiling phase after normal scheduling is chosen. It adapts one control variable (`concurrency`, `prefill_concurrency`, `request_rate`, or `users`) and rejects fixed ramps on that same variable. See [Adaptive Scale](../tutorials/adaptive-scale.md) for examples and artifact semantics. --- diff --git a/docs/cli-options.md b/docs/cli-options.md index 6b57d2d488..8edfc4efd8 100644 --- a/docs/cli-options.md +++ b/docs/cli-options.md @@ -954,43 +954,6 @@ Duration in seconds to ramp prefill concurrency from 1 to target. Duration in seconds to ramp request rate from a proportional minimum to target. Start rate is calculated as target * (update_interval / duration), ensuring correct behavior for target rates below 1 QPS. Useful for gradual warm-up of the target system.
_Constraints: > 0_ -#### `--adaptive-scale` - -Enable stable single-run adaptive scale control. Use --adaptive-scale-control variable:min,max:type to choose the controlled variable and bounds, and --adaptive-scale-sla metric:stat:op:threshold to define pass/fail criteria. Also requires --benchmark-duration and --adaptive-sustain-duration. -
_Flag (no value required)_ - -#### `--adaptive-sustain-duration` `` - -Duration in seconds to sustain load near the discovered adaptive scale boundary. -
_Constraints: > 0_ - -#### `--adaptive-assessment-period`, `--adaptive-scale-assessment-period` `` - -Duration in seconds for each adaptive scale SLA assessment window. -
_Constraints: ≥ 1.0_ - -#### `--adaptive-scale-control` `` - -Compact adaptive scale control spec in variable:min,max:type form. The variable is one of concurrency, prefill_concurrency, request_rate, or users; min and max are required explicit bounds; type is int for discrete controls and float for rate controls. Examples: concurrency:1,1000:int, prefill_concurrency:1,8:int, request_rate:1,200:float, users:10,500:int. Do not combine this with expanded --adaptive-control-* flags. - -#### `--adaptive-control-variable` `` - -Adaptive scale control variable: concurrency, prefill_concurrency, request_rate, or users. - -#### `--adaptive-control-min` `` - -Minimum adaptive scale control value. -
_Constraints: > 0_ - -#### `--adaptive-control-max` `` - -Maximum adaptive scale control value. Inferred from the phase target when omitted. -
_Constraints: > 0_ - -#### `--adaptive-scale-sla` `` - -SLA filter for adaptive scale. Format: 'metric_tag:stat:op:threshold'. Latency-family metrics (request_latency, time_to_first_token/ttft, inter_token_latency/itl/tpot) support percentile stats; window scalar/rate metrics (request_throughput, output_token_throughput, goodput, goodput_ratio, success_rate, error_rate, cancellation_rate) support {avg, min, max}. Full metric/stat table: [Adaptive SLA metric support](tutorials/yaml-config.md#adaptive-sla-metric-support). op in {lt, le, gt, ge}; threshold is a float. Repeatable. Example: --adaptive-scale-sla 'request_latency:p95:le:30000'. - ### Warmup #### `--warmup-request-count`, `--num-warmup-requests` `` @@ -2412,43 +2375,6 @@ Duration in seconds to ramp prefill concurrency from 1 to target. Duration in seconds to ramp request rate from a proportional minimum to target. Start rate is calculated as target * (update_interval / duration), ensuring correct behavior for target rates below 1 QPS. Useful for gradual warm-up of the target system.
_Constraints: > 0_ -#### `--adaptive-scale` - -Enable stable single-run adaptive scale control. Use --adaptive-scale-control variable:min,max:type to choose the controlled variable and bounds, and --adaptive-scale-sla metric:stat:op:threshold to define pass/fail criteria. Also requires --benchmark-duration and --adaptive-sustain-duration. -
_Flag (no value required)_ - -#### `--adaptive-sustain-duration` `` - -Duration in seconds to sustain load near the discovered adaptive scale boundary. -
_Constraints: > 0_ - -#### `--adaptive-assessment-period`, `--adaptive-scale-assessment-period` `` - -Duration in seconds for each adaptive scale SLA assessment window. -
_Constraints: ≥ 1.0_ - -#### `--adaptive-scale-control` `` - -Compact adaptive scale control spec in variable:min,max:type form. The variable is one of concurrency, prefill_concurrency, request_rate, or users; min and max are required explicit bounds; type is int for discrete controls and float for rate controls. Examples: concurrency:1,1000:int, prefill_concurrency:1,8:int, request_rate:1,200:float, users:10,500:int. Do not combine this with expanded --adaptive-control-* flags. - -#### `--adaptive-control-variable` `` - -Adaptive scale control variable: concurrency, prefill_concurrency, request_rate, or users. - -#### `--adaptive-control-min` `` - -Minimum adaptive scale control value. -
_Constraints: > 0_ - -#### `--adaptive-control-max` `` - -Maximum adaptive scale control value. Inferred from the phase target when omitted. -
_Constraints: > 0_ - -#### `--adaptive-scale-sla` `` - -SLA filter for adaptive scale. Format: 'metric_tag:stat:op:threshold'. Latency-family metrics (request_latency, time_to_first_token/ttft, inter_token_latency/itl/tpot) support percentile stats; window scalar/rate metrics (request_throughput, output_token_throughput, goodput, goodput_ratio, success_rate, error_rate, cancellation_rate) support {avg, min, max}. Full metric/stat table: [Adaptive SLA metric support](tutorials/yaml-config.md#adaptive-sla-metric-support). op in {lt, le, gt, ge}; threshold is a float. Repeatable. Example: --adaptive-scale-sla 'request_latency:p95:le:30000'. - ### Warmup #### `--warmup-request-count`, `--num-warmup-requests` `` diff --git a/docs/tutorials/adaptive-scale.md b/docs/tutorials/adaptive-scale.md index dc5c7cf5f5..5758f989a7 100644 --- a/docs/tutorials/adaptive-scale.md +++ b/docs/tutorials/adaptive-scale.md @@ -11,7 +11,7 @@ Use adaptive scale when you want one benchmark invocation to push a service unti ## YAML example -YAML is the preferred way to configure adaptive scale because it keeps the control variable, assessment windows, sustain period, and SLA filters together. The canonical shape uses a nested `adaptive_scale.control` block: +Adaptive scale is configured in YAML because it keeps the control variable, assessment windows, sustain period, and SLA filters together with the phase they control. The examples below set `kind: profiling` explicitly to show the current phase model, but existing configs that use canonical `warmup`/`profiling` names without `kind` remain valid because the loader infers it. The canonical shape uses a nested `adaptive_scale.control` block: ```yaml schemaVersion: "2.0" @@ -28,6 +28,7 @@ benchmark: prompts: {isl: 512, osl: 128} phases: - name: profiling + kind: profiling type: concurrency concurrency: 200 prefill_concurrency: 64 @@ -82,6 +83,7 @@ benchmark: osl: 16 phases: - name: profiling + kind: profiling type: concurrency concurrency: 4 duration: 45 @@ -155,39 +157,25 @@ Exact numbers depend on your server and hardware. If the run passes at `max`, lo For `users`, adaptive scale changes population pressure rather than acting as another spelling of request rate. -## CLI quick start +## YAML-only configuration -The CLI is useful for scripts and simple one-phase runs. Prefer YAML for reusable benchmark definitions. - -```bash -aiperf profile \ - --url http://localhost:8000/v1/chat/completions \ - --model meta-llama/Llama-3.1-8B-Instruct \ - --endpoint-type chat \ - --streaming \ - --concurrency 400 \ - --benchmark-duration 3600 \ - --adaptive-scale \ - --adaptive-scale-control concurrency:1,400:int \ - --adaptive-scale-assessment-period 60 \ - --adaptive-sustain-duration 1800 \ - --adaptive-scale-sla request_latency:p95:le:30000 -``` - -The compact control form is `--adaptive-scale-control variable:min,max:type`. Use `int` for `concurrency`, `prefill_concurrency`, and `users`; use `float` for `request_rate`. Do not mix compact control with expanded `--adaptive-control-*` flags. +Adaptive scale does not expose standalone CLI flags. Put the adaptive settings in the target phase's `adaptive_scale` block and run the benchmark with `aiperf profile --config `. General CLI flags such as `--tokenizer`, `--ui`, and `--output-artifact-dir` can still be used around the YAML benchmark definition. ## Artifacts -Adaptive scale writes these timing-owned artifacts into the run artifact directory: +Adaptive scale writes phase-scoped timing artifacts plus a manifest into the run artifact directory: ```text -adaptive_scale_events.jsonl -adaptive_scale_summary.json +phases//adaptive_scale_events.jsonl +phases//adaptive_scale_summary.json +adaptive_scale_manifest.json ``` +A run with multiple adaptive profiling phases gets one event/summary pair per phase and one manifest entry per adaptive phase. + `adaptive_scale_events.jsonl` is an event stream for orchestration and post-processing. Each line includes `schema_version`, timestamps, `event`, `control_variable`, `control_value_before`, `control_value_after`, `boundary_value`, `last_passing_value`, `first_failing_value`, `sla_values`, and `binding_sla` fields. Pollers should key off explicit events such as `sustain_started` rather than sleeping for a fixed amount of time. -`adaptive_scale_summary.json` is the final controller summary. It records the discovered boundary, final control value, last passing value, first failing value, sustain status, throughput, sample counts, error counts, cancellation counts, and the evaluated candidate windows. +`adaptive_scale_summary.json` is the final controller summary for one adaptive phase. It records the discovered boundary, final control value, last passing value, first failing value, sustain status, throughput, sample counts, error counts, cancellation counts, and the evaluated candidate windows. Artifact fields are intended for orchestration-facing consumers, so treat schema changes and field renames as compatibility events. diff --git a/docs/tutorials/yaml-config.md b/docs/tutorials/yaml-config.md index 2922dc029e..39fda42c2c 100644 --- a/docs/tutorials/yaml-config.md +++ b/docs/tutorials/yaml-config.md @@ -48,8 +48,8 @@ benchmark: entries: 500 prompts: {isl: 512, osl: 128} phases: - - {name: warmup, type: concurrency, concurrency: 8, requests: 50, exclude_from_results: true} - - {name: profiling, type: concurrency, requests: 500} + - {name: warmup, kind: warmup, type: concurrency, concurrency: 8, requests: 50} + - {name: profiling, kind: profiling, type: concurrency, requests: 500} artifacts: dir: ./artifacts/my-test @@ -156,11 +156,39 @@ benchmark: datasets: - {name: main, type: synthetic, prompts: {isl: 512, osl: 128}} phases: - - {name: warmup, type: concurrency, concurrency: 4, requests: 50, exclude_from_results: true} - - {name: profiling, type: poisson, rate: 30.0, duration: 120} + - {name: warmup, kind: warmup, type: concurrency, concurrency: 4, requests: 50} + - {name: profiling, kind: profiling, type: poisson, rate: 30.0, duration: 120} ``` -You can mix and match — the loader auto-expands `model:` into a one-element `models:` list, `dataset:` into a one-entry `datasets:` list named `default`, and a flat `phases:` block into a one-element list named `profiling`. The normalized `datasets:` form is future-facing but currently accepts exactly one dataset; multiple datasets are a roadmap item. +You can mix and match — the loader auto-expands `model:` into a one-element `models:` list, `dataset:` into a one-entry `datasets:` list named `default`, and a flat `phases:` block into a one-element profiling phase named `profiling`. Named phases are additive: existing configs that use canonical phase names such as `warmup` and `profiling` without an explicit `kind` remain valid, and the loader infers the matching semantic kind. Explicit phase lists may set `name` as the unique workflow identifier and `kind` as the semantic (`warmup` or `profiling`); custom names such as `storm_1` must set `kind: profiling` or `kind: warmup`. The normalized `datasets:` form is future-facing but currently accepts exactly one dataset; multiple datasets are a roadmap item. + +Named phases are useful for long soaks with multiple profiling windows: + +```yaml +benchmark: + phases: + - name: warmup + kind: warmup + type: concurrency + duration: 5m + - name: low_cancel_1 + kind: profiling + type: concurrency + duration: 30m + cancellation: {rate: 5, delay: 0} + - name: storm_1 + kind: profiling + type: concurrency + duration: 5m + cancellation: {rate: 50, delay: 0} + - name: recovery_1 + kind: profiling + type: concurrency + duration: 30m + cancellation: {rate: 0, delay: 0} +``` + +Phase names must be strict identifiers (`^[A-Za-z_][A-Za-z0-9_-]*$`) and are unique case-insensitively because they are used in sweep paths and artifact directories. ### Inline datasets @@ -336,6 +364,7 @@ benchmark: prompts: {isl: 512, osl: 128} phases: - name: profiling + kind: profiling type: concurrency concurrency: 200 prefill_concurrency: 64 @@ -365,7 +394,7 @@ benchmark: Adaptive scale rejects fixed ramps on the same variable it controls. For example, do not combine `control.variable: prefill_concurrency` with `prefill_ramp`. Fixed ramps for other variables are allowed. -The CLI exposes a compact sweep-like control flag for the common single-phase case: `--adaptive-scale-control variable:min,max:type`, plus repeated `--adaptive-scale-sla metric:stat:op:threshold` flags. For example: `--adaptive-scale-control "concurrency:1,1000:int" --adaptive-scale-sla "request_latency:p95:le:30000"`. Expanded `--adaptive-control-variable`, `--adaptive-control-min`, and `--adaptive-control-max` flags remain supported for advanced scripting; if expanded `--adaptive-control-max` is omitted, AIPerf infers it from the matching phase target such as `--concurrency`, `--prefill-concurrency`, `--request-rate`, or `--num-users`. Do not mix compact and expanded control forms. +Adaptive scale is a YAML-only phase feature. Configure the control variable, min/max bounds, assessment windows, sustain duration, strategy, and SLA filters in the phase `adaptive_scale` and `sla` blocks, then run the benchmark with `aiperf profile --config `. Existing command-line workflows for other settings and canonical YAML phase shapes remain valid. Adaptive scale combines SLA filters with simple AND semantics. A window passes only when every configured filter passes. Step sizing uses the smallest normalized passing margin, so the closest SLA boundary controls the next increase. There are no weights, formulas, or multi-objective scoring in single-run adaptive scale. @@ -466,13 +495,16 @@ benchmark: le: 0.10 ``` -Adaptive scale writes two timing-owned artifacts into the run directory: +Adaptive scale writes phase-scoped timing artifacts plus a manifest into the run directory: ```text -adaptive_scale_events.jsonl -adaptive_scale_summary.json +phases//adaptive_scale_events.jsonl +phases//adaptive_scale_summary.json +adaptive_scale_manifest.json ``` +A run with multiple adaptive profiling phases gets one event/summary pair per phase and one manifest entry per adaptive phase. + These artifacts use schema version 2 and generic control fields such as `control_variable`, `control_value_before`, `control_value_after`, `boundary_value`, `last_passing_value`, and `first_failing_value`. Every `adaptive_window` event includes all evaluated SLA values and the binding constraint. Dynamo-style pollers should gate fault injection on explicit events such as `sustain_started` rather than fixed sleeps. Use adaptive scale when you want continuous pressure inside one benchmark invocation. Use `sweep` or `adaptive_search` when you want offline multi-run exploration across many independent trials. @@ -507,12 +539,14 @@ benchmark: duration: 120 ``` -The `parameters:` keys are dot-paths into the `benchmark:` body. For lists, the second segment is the entry's `name`: +The `parameters:` keys are dot-paths into the `benchmark:` body. For phase lists, the second segment resolves in this order: numeric index, exact unique phase `name`, then legacy `phases.profiling.*` shorthand for the unique profiling-kind phase when no phase is named `profiling`: -- `phases.profiling.rate` → the phase named `profiling`, field `rate` +- `phases.storm_1.cancellation.rate` → the phase named `storm_1`, field `cancellation.rate` +- `phases.1.concurrency` → the second phase, field `concurrency` +- `phases.profiling.rate` → the phase named `profiling`, or the unique profiling-kind phase when unambiguous - `datasets.default.prompts.isl` → the dataset named `default` (the singular `dataset:` shorthand auto-names it `default`) -The 12 most-swept phase fields also have bare-name sugar: `concurrency`, `prefill_concurrency`, `rate`, `requests`, `duration`, `sessions`, `users`, `smoothness`, `grace_period`, `concurrency_ramp`, `prefill_ramp`, `rate_ramp`. Each expands to `phases.profiling.` (resolves to the unique non-warmup phase). The two forms are equivalent — see [Bare-Name Aliases](sweeps.md#bare-name-aliases-for-common-phase-fields). +The 12 most-swept phase fields also have bare-name sugar: `concurrency`, `prefill_concurrency`, `rate`, `requests`, `duration`, `sessions`, `users`, `smoothness`, `grace_period`, `concurrency_ramp`, `prefill_ramp`, `rate_ramp`. Each expands to `phases.profiling.` and must still resolve unambiguously. The two forms are equivalent — see [Bare-Name Aliases](sweeps.md#bare-name-aliases-for-common-phase-fields). Other sweep modes available in YAML: @@ -578,7 +612,7 @@ aiperf profile --config benchmark.yaml \ --artifact-dir ./run-2026-05-09 ``` -This loads `benchmark.yaml` as the base, then overrides the *profiling* phase's `concurrency` with `32` and the artifact directory with the new path. (CLI loadgen flags overlay onto the phase named `profiling` — they don't broadcast to every named phase, so multi-phase configs need YAML edits to tweak warmup or other phases.) Useful when most of your config is stable but you want to tweak one knob from a script or CI job. +This loads `benchmark.yaml` as the base, then overrides the unique profiling phase's `concurrency` with `32` and the artifact directory with the new path. CLI loadgen flags target `kind: profiling` and work only when that target is unambiguous. If a config has multiple profiling phases, set per-phase values in YAML. Useful when most of your config is stable but you want to tweak one knob from a script or CI job. The precedence order, lowest to highest: diff --git a/src/aiperf/common/accumulator_protocols.py b/src/aiperf/common/accumulator_protocols.py index e4a7c421ed..06d17c7cb8 100644 --- a/src/aiperf/common/accumulator_protocols.py +++ b/src/aiperf/common/accumulator_protocols.py @@ -73,6 +73,15 @@ class ExportContext: phase: CreditPhase | None = None """Credit phase represented by this export, or None when phase-agnostic.""" + phase_index: int | None = None + """Concrete runtime phase index for phase-local exports, when available.""" + + phase_name: str | None = None + """User-provided phase name for diagnostics/export metadata.""" + + phase_kind: str | None = None + """Semantic phase kind for diagnostics/export metadata.""" + error_summary: list[ErrorDetailsCount] | None = None """De-duplicated profile-run error counts to surface in the export, if any.""" diff --git a/src/aiperf/common/models/__init__.py b/src/aiperf/common/models/__init__.py index 92163ef0f8..64401ed9b3 100644 --- a/src/aiperf/common/models/__init__.py +++ b/src/aiperf/common/models/__init__.py @@ -77,6 +77,7 @@ MetricValue, ParsedResponse, ParsedResponseRecord, + PhaseProfileResults, ProcessRecordsResult, ProfileResults, RAGSources, @@ -223,6 +224,7 @@ "ParsedResponse", "ParsedResponseRecord", "PhaseRecordsStats", + "PhaseProfileResults", "ProcessHealth", "ProcessRecordsResult", "ProcessServerMetricsResult", diff --git a/src/aiperf/common/models/credit_models.py b/src/aiperf/common/models/credit_models.py index 6e33b41c9a..b870b3e72e 100644 --- a/src/aiperf/common/models/credit_models.py +++ b/src/aiperf/common/models/credit_models.py @@ -8,6 +8,7 @@ from aiperf.common.constants import NANOS_PER_SECOND from aiperf.common.enums import CreditPhase from aiperf.common.models.base_models import AIPerfBaseModel +from aiperf.common.types import PhaseKind class BasePhaseStats(AIPerfBaseModel): @@ -18,6 +19,20 @@ class BasePhaseStats(AIPerfBaseModel): phase: CreditPhase = Field( ..., description="The type of credit phase, such as warmup or profiling." ) + phase_index: int | None = Field( + default=None, ge=0, description="Absolute index in the ordered phases list." + ) + profiling_index: int | None = Field( + default=None, + ge=0, + description="Index among profiling-kind phases; None for warmup.", + ) + phase_name: str | None = Field( + default=None, description="User-provided unique phase name." + ) + phase_kind: PhaseKind | None = Field( + default=None, description="Phase semantic kind: warmup or profiling." + ) # Timestamp fields start_ns: int | None = Field( diff --git a/src/aiperf/common/models/record_models.py b/src/aiperf/common/models/record_models.py index bb6f958235..8663065fb7 100644 --- a/src/aiperf/common/models/record_models.py +++ b/src/aiperf/common/models/record_models.py @@ -39,7 +39,7 @@ from aiperf.common.models.model_endpoint_info import ModelEndpointInfo from aiperf.common.models.trace_models import BaseTraceData, TraceDataExport from aiperf.common.models.usage_models import Usage -from aiperf.common.types import JsonObject, MetricTagT +from aiperf.common.types import JsonObject, MetricTagT, PhaseKind from aiperf.common.utils import load_json_str _logger = AIPerfLogger(__name__) @@ -208,6 +208,20 @@ class MetricRecordMetadata(AIPerfBaseModel): ..., description="The benchmark phase of the record, either warmup or profiling.", ) + phase_index: int | None = Field( + default=None, ge=0, description="Absolute index in the ordered phases list." + ) + profiling_index: int | None = Field( + default=None, + ge=0, + description="Index among profiling-kind phases; None for warmup.", + ) + phase_name: str | None = Field( + default=None, description="User-provided unique phase name." + ) + phase_kind: PhaseKind | None = Field( + default=None, description="Phase semantic kind: warmup or profiling." + ) was_cancelled: bool = Field( default=False, description="Whether the request was cancelled during execution.", @@ -276,6 +290,49 @@ def _coerce_metric_results(cls, value: Any) -> Any: return value +class PhaseProfileResults(AIPerfBaseModel): + """Metric summary for one concrete named phase.""" + + phase_index: int | None = Field( + default=None, ge=0, description="Absolute index in the ordered phases list." + ) + profiling_index: int | None = Field( + default=None, + ge=0, + description="Index among profiling-kind phases; None for warmup.", + ) + phase_name: str = Field(description="User-provided unique phase name.") + phase_kind: PhaseKind = Field( + description="Phase semantic kind: warmup or profiling." + ) + records: list[MetricResult] = Field( + default_factory=list, description="Metric results scoped to this phase." + ) + start_ns: int | None = Field( + default=None, + ge=0, + description="Phase start time in nanoseconds, when available.", + ) + end_ns: int | None = Field( + default=None, + ge=0, + description="Phase request completion time in nanoseconds, when available.", + ) + was_cancelled: bool = Field( + default=False, description="Whether this phase was cancelled early." + ) + successful_request_count: int = Field( + default=0, ge=0, description="Successful records for this phase." + ) + error_request_count: int = Field( + default=0, ge=0, description="Errored records for this phase." + ) + error_summary: list[ErrorDetailsCount] = Field( + default_factory=list, + description="A list of the unique phase error details and their counts", + ) + + class ProfileResults(AIPerfBaseModel): """The results of a profile run.""" @@ -332,6 +389,10 @@ class ProfileResults(AIPerfBaseModel): "runs. Forwarded to profile_export_aiperf.json under the " "``branch_stats`` key when present.", ) + phase_records: list[PhaseProfileResults] | None = Field( + default=None, + description="Internal per-phase metric summaries used for phase artifacts.", + ) def get(self, tag: MetricTagT) -> MetricResult | None: """Get a metric result by tag, if it exists.""" @@ -645,6 +706,20 @@ class RecordContext(AIPerfBaseModel): ..., description="The type of credit phase (either warmup or profiling)", ) + phase_index: int | None = Field( + default=None, ge=0, description="Absolute index in the ordered phases list." + ) + profiling_index: int | None = Field( + default=None, + ge=0, + description="Index among profiling-kind phases; None for warmup.", + ) + phase_name: str | None = Field( + default=None, description="User-provided unique phase name." + ) + phase_kind: PhaseKind | None = Field( + default=None, description="Phase semantic kind: warmup or profiling." + ) conversation_id: str = Field( ..., description="The ID of the conversation (if applicable).", @@ -1320,7 +1395,7 @@ class MetricRecordInfo(AIPerfBaseModel): metadata: MetricRecordMetadata = Field( ..., - description="The metadata of the record. Should match the metadata in the RecordsMessage.", + description="The metadata of the record. Should match the metadata in the MetricRecordsMessage.", ) metrics: dict[str, MetricValue] = Field( ..., @@ -1343,7 +1418,7 @@ class RawRecordInfo(AIPerfBaseModel): metadata: MetricRecordMetadata = Field( ..., - description="The metadata of the record. Should match the metadata in the RecordsMessage.", + description="The metadata of the record. Should match the metadata in the MetricRecordsMessage.", ) start_perf_ns: int = Field( default_factory=time.perf_counter_ns, diff --git a/src/aiperf/common/phase.py b/src/aiperf/common/phase.py new file mode 100644 index 0000000000..fd6ae68749 --- /dev/null +++ b/src/aiperf/common/phase.py @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Shared helpers for named benchmark phase identity.""" + +from __future__ import annotations + +from aiperf.common.enums import CreditPhase +from aiperf.common.types import PhaseKind + +PhaseRuntimeKey = int | CreditPhase + + +def infer_legacy_phase_kind( + name: object, kind: PhaseKind | None = None +) -> PhaseKind | None: + """Infer phase kind from reserved legacy canonical names when omitted.""" + if kind is not None: + return kind + if name in {"warmup", "profiling"}: + return name # type: ignore[return-value] + return None + + +def phase_runtime_key( + phase: CreditPhase, phase_index: int | None = None +) -> PhaseRuntimeKey: + """Return the runtime key used for per-phase maps and concurrency slots.""" + return phase_index if phase_index is not None else phase diff --git a/src/aiperf/common/types.py b/src/aiperf/common/types.py index 66ec8cf4b2..4fa6b7716f 100644 --- a/src/aiperf/common/types.py +++ b/src/aiperf/common/types.py @@ -7,7 +7,7 @@ from collections.abc import Awaitable, Callable from types import UnionType -from typing import TYPE_CHECKING, Any, TypeAlias, TypeVar, Union +from typing import TYPE_CHECKING, Any, Literal, TypeAlias, TypeVar, Union from aiperf.common.enums import ( CaseInsensitiveStrEnum, @@ -56,6 +56,7 @@ MetricTagT: TypeAlias = str ModelEndpointInfoT = TypeVar("ModelEndpointInfoT", bound="ModelEndpointInfo") OutputT = TypeVar("OutputT", bound=Any) +PhaseKind: TypeAlias = Literal["warmup", "profiling"] PluginClassT = TypeVar("PluginClassT", bound=Any) ProtocolT = TypeVar("ProtocolT", bound=Any) RawRequestT = TypeVar("RawRequestT", bound=Any, contravariant=True) diff --git a/src/aiperf/config/config.py b/src/aiperf/config/config.py index ddc532f78b..72802254bc 100644 --- a/src/aiperf/config/config.py +++ b/src/aiperf/config/config.py @@ -437,24 +437,26 @@ def parse_datasets(cls, v: Any) -> list[Any]: @model_validator(mode="after") def validate_phase_names_unique(self) -> Self: - """Reject duplicate phase names — they must be unique within the list.""" - seen: set[str] = set() + """Reject duplicate phase names, case-insensitively, within the list.""" + seen: dict[str, str] = {} for phase in self.phases: - if phase.name in seen: + key = phase.name.lower() + if key in seen: raise ValueError( - f"duplicate phase name '{phase.name}' — names must be unique. " + f"duplicate phase name '{phase.name}' conflicts with " + f"'{seen[key]}' — names must be unique case-insensitively. " f"Found names: {[p.name for p in self.phases]}" ) - seen.add(phase.name) + seen[key] = phase.name return self @model_validator(mode="after") def validate_profiling_phase_required(self) -> Self: - """Require at least one 'profiling' phase — warmup alone is not a benchmark.""" - if not any(p.name == "profiling" for p in self.phases): + """Require at least one profiling-kind phase — warmup alone is not a benchmark.""" + if not any(p.kind == "profiling" for p in self.phases): raise ValueError( "a 'profiling' phase is required; " - f"got phases: {[p.name for p in self.phases]}" + f"got phases: {[(p.name, p.kind) for p in self.phases]}" ) return self diff --git a/src/aiperf/config/flags/_adaptive_control_cli.py b/src/aiperf/config/flags/_adaptive_control_cli.py deleted file mode 100644 index 45d8a9d7e5..0000000000 --- a/src/aiperf/config/flags/_adaptive_control_cli.py +++ /dev/null @@ -1,82 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -"""Adaptive-scale compact CLI parsing helpers.""" - -from __future__ import annotations - -from collections.abc import Set -from typing import Any - -EXPANDED_ADAPTIVE_CONTROL_FIELDS = frozenset( - { - "adaptive_control_variable", - "adaptive_control_min", - "adaptive_control_max", - } -) - - -def reject_mixed_adaptive_control_cli(fields_set: Set[str]) -> None: - if "adaptive_scale_control" not in fields_set: - return - if EXPANDED_ADAPTIVE_CONTROL_FIELDS & fields_set: - raise ValueError( - "Use either --adaptive-scale-control or " - "--adaptive-control-variable/--adaptive-control-min/" - "--adaptive-control-max, not both." - ) - - -def parse_adaptive_scale_control(value: str) -> dict[str, Any]: - """Parse ``variable:min,max:type`` into canonical adaptive control fields.""" - try: - variable, bounds, value_type = value.split(":", 2) - minimum_text, maximum_text = bounds.split(",", 1) - except ValueError as exc: - raise ValueError( - "--adaptive-scale-control must use variable:min,max:type, " - "for example concurrency:1,1000:int" - ) from exc - - variable = variable.strip() - value_type = value_type.strip() - if not variable: - raise ValueError("--adaptive-scale-control requires a control variable") - - if value_type == "int": - minimum = _parse_int_bound(minimum_text, "min") - maximum = _parse_int_bound(maximum_text, "max") - elif value_type == "float": - minimum = _parse_float_bound(minimum_text, "min") - maximum = _parse_float_bound(maximum_text, "max") - else: - raise ValueError( - "--adaptive-scale-control type must be 'int' or 'float', " - f"got {value_type!r}" - ) - - return { - "adaptive_control_variable": variable, - "adaptive_control_min": minimum, - "adaptive_control_max": maximum, - } - - -def _parse_int_bound(value: str, name: str) -> int: - value = value.strip() - try: - return int(value) - except ValueError as exc: - raise ValueError( - f"--adaptive-scale-control {name} bound must be an integer" - ) from exc - - -def _parse_float_bound(value: str, name: str) -> float: - value = value.strip() - try: - return float(value) - except ValueError as exc: - raise ValueError( - f"--adaptive-scale-control {name} bound must be a number" - ) from exc diff --git a/src/aiperf/config/flags/_converter_profiling.py b/src/aiperf/config/flags/_converter_profiling.py index da57905542..48a4400960 100644 --- a/src/aiperf/config/flags/_converter_profiling.py +++ b/src/aiperf/config/flags/_converter_profiling.py @@ -7,12 +7,6 @@ from typing import TYPE_CHECKING, Any -from aiperf.config.flags._adaptive_control_cli import ( - parse_adaptive_scale_control, - reject_mixed_adaptive_control_cli, -) -from aiperf.orchestrator.search_planner.parsing import parse_sla_filter - if TYPE_CHECKING: from pathlib import Path @@ -29,13 +23,6 @@ ("users", "num_users"), ("rate", "request_rate"), ("rate", "user_centric_rate"), - ("adaptive_scale", "adaptive_scale"), - ("adaptive_sustain_duration", "adaptive_sustain_duration"), - ("adaptive_assessment_period", "adaptive_assessment_period"), - ("adaptive_scale_control", "adaptive_scale_control"), - ("adaptive_control_variable", "adaptive_control_variable"), - ("adaptive_control_min", "adaptive_control_min"), - ("adaptive_control_max", "adaptive_control_max"), ) @@ -84,41 +71,6 @@ def _apply_profiling_ramps(prof: dict[str, Any], cli: CLIConfig) -> None: prof[key] = {"duration": getattr(cli, field)} -def _parse_adaptive_scale_sla_filter(value: str) -> dict[str, Any]: - try: - return parse_sla_filter(value).model_dump(mode="json") - except TypeError as exc: - message = str(exc).replace("--search-sla", "--adaptive-scale-sla") - raise TypeError(message) from exc - - -def _apply_adaptive_scale_sla(prof: dict[str, Any], cli: CLIConfig) -> None: - if "adaptive_scale_sla" not in cli.model_fields_set or not cli.adaptive_scale_sla: - return - - parsed_sla = [ - _parse_adaptive_scale_sla_filter(value) for value in cli.adaptive_scale_sla - ] - if not prof.get("adaptive_scale"): - raise ValueError("--adaptive-scale-sla requires --adaptive-scale") - - prof["sla"] = parsed_sla - - -def _apply_adaptive_scale_control(prof: dict[str, Any], cli: CLIConfig) -> None: - reject_mixed_adaptive_control_cli(cli.model_fields_set) - if "adaptive_scale_control" not in cli.model_fields_set: - if "adaptive_control_variable" in cli.model_fields_set: - prof["_adaptive_control_variable_source"] = "--adaptive-control-variable" - return - if not prof.get("adaptive_scale"): - raise ValueError("--adaptive-scale-control requires --adaptive-scale") - control = parse_adaptive_scale_control(cli.adaptive_scale_control or "") - prof.update(control) - prof["_adaptive_control_variable_source"] = "--adaptive-scale-control" - prof.pop("adaptive_scale_control", None) - - def _reject_orphan_load_generator_flags(prof: dict[str, Any], cli: CLIConfig) -> None: """Reject CLI flags whose load-generator partner wasn't supplied. @@ -132,40 +84,6 @@ def _reject_orphan_load_generator_flags(prof: dict[str, Any], cli: CLIConfig) -> fields_set = cli.model_fields_set phase_type = prof["type"] - if prof.get("adaptive_scale"): - variable = prof.get("adaptive_control_variable", "concurrency") - required = { - "concurrency": ("concurrency", "--concurrency"), - "prefill_concurrency": ("prefill_concurrency", "--prefill-concurrency"), - "request_rate": ("rate", "--request-rate"), - "users": ("users", "--num-users"), - }.get(variable) - if required is None: - source = prof.get( - "_adaptive_control_variable_source", - "--adaptive-control-variable", - ) - raise ValueError(f"unsupported {source} variable {variable!r}") - required_key, required_flag = required - if required_key not in prof and "adaptive_control_max" not in prof: - if variable == "concurrency": - raise ValueError( - "--adaptive-scale requires --concurrency or --adaptive-control-max" - ) - raise ValueError( - f"--adaptive-scale control variable {variable!r} requires " - f"{required_flag} or --adaptive-control-max" - ) - if ( - prof.get("adaptive_scale") - and "search_sla" in fields_set - and "adaptive_scale_sla" not in fields_set - ): - raise ValueError( - "--adaptive-scale uses --adaptive-scale-sla; --search-sla is reserved " - "for adaptive-search/grid runs" - ) - if "num_users" in fields_set and phase_type != PhaseType.USER_CENTRIC: raise ValueError( "--num-users requires --user-centric-rate. Pass --user-centric-rate " @@ -416,6 +334,8 @@ def _first_record_has_timestamp(file_path: object) -> bool: data = load_json_str(stripped) except (ValueError, TypeError): return False + if not isinstance(data, dict): + return False return data.get("timestamp") is not None except OSError: return False @@ -466,12 +386,7 @@ def build_profiling(cli: CLIConfig) -> dict[str, Any]: _apply_profiling_ramps(prof, cli) prof["type"] = _profiling_phase_type(cli) - _apply_adaptive_scale_control(prof, cli) - _apply_adaptive_scale_sla(prof, cli) - _reject_orphan_load_generator_flags(prof, cli) - prof.pop("_adaptive_control_variable_source", None) - _apply_phase_specific_routes(prof, cli) if prof["type"] == PhaseType.FIXED_SCHEDULE and "start_offset" in prof: diff --git a/src/aiperf/config/flags/_resolver_adaptive.py b/src/aiperf/config/flags/_resolver_adaptive.py deleted file mode 100644 index 6911bb2235..0000000000 --- a/src/aiperf/config/flags/_resolver_adaptive.py +++ /dev/null @@ -1,164 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -"""Adaptive-scale CLI overlay helpers for YAML resolver.""" - -from __future__ import annotations - -from typing import TYPE_CHECKING, Any - -from aiperf.config.flags._adaptive_control_cli import ( - parse_adaptive_scale_control, - reject_mixed_adaptive_control_cli, -) - -if TYPE_CHECKING: - from aiperf.config.flags import CLIConfig - - -BASIC_ADAPTIVE_CLI_FIELDS = frozenset( - { - "adaptive_scale", - "adaptive_sustain_duration", - "adaptive_assessment_period", - "adaptive_scale_control", - "adaptive_control_variable", - "adaptive_control_min", - "adaptive_control_max", - "adaptive_scale_sla", - } -) - - -def apply_basic_adaptive_scale_overrides( - target: dict[str, Any], cli: CLIConfig -) -> None: - """Overlay the small adaptive-scale CLI surface onto a YAML phase.""" - fields_set = cli.model_fields_set & BASIC_ADAPTIVE_CLI_FIELDS - if not fields_set: - return - _reject_search_sla_for_adaptive(cli) - - adaptive_block = _current_adaptive_block(target) - _apply_adaptive_enabled(target, cli, fields_set, adaptive_block) - _apply_adaptive_durations(target, cli, fields_set) - reject_mixed_adaptive_control_cli(cli.model_fields_set) - _apply_compact_control(target, cli, fields_set) - _apply_explicit_control(target, cli, fields_set) - _apply_adaptive_sla(target, cli, fields_set) - - -def _reject_search_sla_for_adaptive(cli: CLIConfig) -> None: - if ( - "search_sla" in cli.model_fields_set - and "adaptive_scale_sla" not in cli.model_fields_set - ): - raise ValueError( - "--adaptive-scale uses --adaptive-scale-sla; --search-sla is reserved " - "for adaptive-search/grid runs" - ) - - -def _current_adaptive_block(target: dict[str, Any]) -> dict[str, Any] | None: - existing = target.get("adaptive_scale") - return existing if isinstance(existing, dict) else None - - -def _adaptive_scale_enabled(target: dict[str, Any], cli: CLIConfig) -> bool: - if "adaptive_scale" in cli.model_fields_set: - return bool(cli.adaptive_scale) - existing = target.get("adaptive_scale") - if isinstance(existing, dict): - return bool(existing.get("enabled", True)) - return bool(existing) - - -def _ensure_adaptive_block(target: dict[str, Any]) -> dict[str, Any]: - adaptive_block = _current_adaptive_block(target) - if adaptive_block is None: - adaptive_block = {} - target["adaptive_scale"] = adaptive_block - return adaptive_block - - -def _apply_adaptive_enabled( - target: dict[str, Any], - cli: CLIConfig, - fields_set: set[str], - adaptive_block: dict[str, Any] | None, -) -> None: - if "adaptive_scale" not in fields_set: - return - if adaptive_block is not None: - adaptive_block["enabled"] = bool(cli.adaptive_scale) - else: - target["adaptive_scale"] = bool(cli.adaptive_scale) - - -def _apply_adaptive_durations( - target: dict[str, Any], cli: CLIConfig, fields_set: set[str] -) -> None: - duration_fields = { - "adaptive_sustain_duration", - "adaptive_assessment_period", - } - if not fields_set.intersection(duration_fields): - return - adaptive_block = _ensure_adaptive_block(target) - if cli.adaptive_sustain_duration is not None: - adaptive_block["sustain_duration"] = cli.adaptive_sustain_duration - if cli.adaptive_assessment_period is not None: - adaptive_block["assessment_period"] = cli.adaptive_assessment_period - - -def _apply_compact_control( - target: dict[str, Any], cli: CLIConfig, fields_set: set[str] -) -> None: - if "adaptive_scale_control" not in fields_set or not cli.adaptive_scale_control: - return - adaptive_block = _ensure_adaptive_block(target) - control = adaptive_block.setdefault("control", {}) - parsed_control = parse_adaptive_scale_control(cli.adaptive_scale_control) - control["variable"] = parsed_control["adaptive_control_variable"] - control["min"] = parsed_control["adaptive_control_min"] - control["max"] = parsed_control["adaptive_control_max"] - - -def _apply_explicit_control( - target: dict[str, Any], cli: CLIConfig, fields_set: set[str] -) -> None: - explicit_fields = { - "adaptive_control_variable", - "adaptive_control_min", - "adaptive_control_max", - } - if not fields_set.intersection(explicit_fields): - return - control = _ensure_adaptive_block(target).setdefault("control", {}) - if "adaptive_control_variable" in fields_set and cli.adaptive_control_variable: - control["variable"] = cli.adaptive_control_variable - if "adaptive_control_min" in fields_set and cli.adaptive_control_min is not None: - control["min"] = cli.adaptive_control_min - if "adaptive_control_max" in fields_set and cli.adaptive_control_max is not None: - control["max"] = cli.adaptive_control_max - - -def _apply_adaptive_sla( - target: dict[str, Any], cli: CLIConfig, fields_set: set[str] -) -> None: - if "adaptive_scale_sla" not in fields_set or not cli.adaptive_scale_sla: - return - from aiperf.orchestrator.search_planner.parsing import parse_sla_filter - - parsed_sla: list[dict[str, Any]] = [] - for value in cli.adaptive_scale_sla: - try: - parsed_sla.append(parse_sla_filter(value).model_dump(mode="json")) - except TypeError as exc: - message = str(exc).replace("--search-sla", "--adaptive-scale-sla") - raise TypeError(message) from exc - if not _adaptive_scale_enabled(target, cli): - raise ValueError("--adaptive-scale-sla requires --adaptive-scale") - target["sla"] = parsed_sla - adaptive_block = _current_adaptive_block(target) - if adaptive_block is not None: - adaptive_block["sla"] = parsed_sla diff --git a/src/aiperf/config/flags/_section_fields.py b/src/aiperf/config/flags/_section_fields.py index deae2cce04..f4b542faad 100644 --- a/src/aiperf/config/flags/_section_fields.py +++ b/src/aiperf/config/flags/_section_fields.py @@ -147,14 +147,6 @@ LOADGEN_FIELDS: frozenset[str] = frozenset( { - "adaptive_assessment_period", - "adaptive_control_max", - "adaptive_control_min", - "adaptive_control_variable", - "adaptive_scale", - "adaptive_scale_control", - "adaptive_scale_sla", - "adaptive_sustain_duration", "arrival_pattern", "arrival_smoothness", "benchmark_duration", diff --git a/src/aiperf/config/flags/cli_config.py b/src/aiperf/config/flags/cli_config.py index bd469b6998..9bb9d55dc0 100644 --- a/src/aiperf/config/flags/cli_config.py +++ b/src/aiperf/config/flags/cli_config.py @@ -1946,118 +1946,6 @@ def url(self) -> str: ), ] = None - adaptive_scale: Annotated[ - bool, - Field( - description=( - "Enable stable single-run adaptive scale control. Use " - "--adaptive-scale-control variable:min,max:type to choose the " - "controlled variable and bounds, and --adaptive-scale-sla " - "metric:stat:op:threshold to define pass/fail criteria. Also " - "requires --benchmark-duration and --adaptive-sustain-duration." - ), - ), - CLIParameter(name=("--adaptive-scale",), group=Groups.LOAD_GENERATOR), - ] = False - - adaptive_sustain_duration: Annotated[ - float | None, - Field( - gt=0, - description="Duration in seconds to sustain load near the discovered adaptive scale boundary.", - ), - CLIParameter( - name=("--adaptive-sustain-duration",), group=Groups.LOAD_GENERATOR - ), - ] = None - - adaptive_assessment_period: Annotated[ - float | None, - Field( - ge=1.0, - description="Duration in seconds for each adaptive scale SLA assessment window.", - ), - CLIParameter( - name=("--adaptive-assessment-period", "--adaptive-scale-assessment-period"), - group=Groups.LOAD_GENERATOR, - ), - ] = None - - adaptive_scale_control: Annotated[ - str | None, - Field( - default=None, - description=( - "Compact adaptive scale control spec in variable:min,max:type " - "form. The variable is one of concurrency, prefill_concurrency, " - "request_rate, or users; min and max are required explicit " - "bounds; type is int for discrete controls and float for rate " - "controls. Examples: concurrency:1,1000:int, " - "prefill_concurrency:1,8:int, request_rate:1,200:float, " - "users:10,500:int. Do not combine this with expanded " - "--adaptive-control-* flags." - ), - ), - CLIParameter(name=("--adaptive-scale-control",), group=Groups.LOAD_GENERATOR), - ] = None - - adaptive_control_variable: Annotated[ - str | None, - Field( - default=None, - description="Adaptive scale control variable: concurrency, prefill_concurrency, request_rate, or users.", - ), - CLIParameter( - name=("--adaptive-control-variable",), group=Groups.LOAD_GENERATOR - ), - ] = None - - adaptive_control_min: Annotated[ - float | None, - Field( - gt=0, - default=None, - description="Minimum adaptive scale control value.", - ), - CLIParameter(name=("--adaptive-control-min",), group=Groups.LOAD_GENERATOR), - ] = None - - adaptive_control_max: Annotated[ - float | None, - Field( - gt=0, - default=None, - description="Maximum adaptive scale control value. Inferred from the phase target when omitted.", - ), - CLIParameter(name=("--adaptive-control-max",), group=Groups.LOAD_GENERATOR), - ] = None - - adaptive_scale_sla: Annotated[ - list[str] | None, - Field( - default=None, - description=( - "SLA filter for adaptive scale. Format: " - "'metric_tag:stat:op:threshold'. Latency-family metrics " - "(request_latency, time_to_first_token/ttft, " - "inter_token_latency/itl/tpot) support percentile stats; " - "window scalar/rate metrics (request_throughput, " - "output_token_throughput, goodput, " - "goodput_ratio, success_rate, " - "error_rate, cancellation_rate) support {avg, min, max}. " - "Full metric/stat table: " - "[Adaptive SLA metric support]" - "(tutorials/yaml-config.md#adaptive-sla-metric-support). " - "op in {lt, le, gt, ge}; threshold is a float. Repeatable. " - "Example: --adaptive-scale-sla 'request_latency:p95:le:30000'." - ), - ), - CLIParameter( - name=("--adaptive-scale-sla",), - group=Groups.LOAD_GENERATOR, - ), - ] = None - ############################################################################## # Warmup ############################################################################## diff --git a/src/aiperf/config/flags/converter.py b/src/aiperf/config/flags/converter.py index 28152ada02..9730ed6c0b 100644 --- a/src/aiperf/config/flags/converter.py +++ b/src/aiperf/config/flags/converter.py @@ -512,8 +512,8 @@ def _assemble_envelope_dict(cli: CLIConfig) -> dict[str, Any]: phases: list[dict[str, Any]] = [] if (warmup := build_warmup(cli)) is not None: - phases.append({"name": "warmup", **warmup}) - phases.append({"name": "profiling", **prof}) + phases.append({"name": "warmup", "kind": "warmup", **warmup}) + phases.append({"name": "profiling", "kind": "profiling", **prof}) ds = build_dataset(cli) diff --git a/src/aiperf/config/flags/resolver.py b/src/aiperf/config/flags/resolver.py index e976e14267..55ab2e69d0 100644 --- a/src/aiperf/config/flags/resolver.py +++ b/src/aiperf/config/flags/resolver.py @@ -19,9 +19,7 @@ from typing import TYPE_CHECKING, Any from aiperf.common.enums import DatasetType -from aiperf.config.flags._resolver_adaptive import ( - apply_basic_adaptive_scale_overrides, -) +from aiperf.common.phase import infer_legacy_phase_kind from aiperf.config.flags._resolver_helpers import promote_benchmark_magic_lists from aiperf.config.flags._resolver_server_metrics import ( build_server_metrics_override, @@ -81,6 +79,7 @@ def resolve_config( from aiperf.config.loader import load_config_dict yaml_dict = load_config_dict(config_file) + _normalize_loaded_benchmark_shorthands(yaml_dict) # Build the recipe's view of BenchmarkConfig from YAML + the # endpoint/input CLI overrides ONLY: the recipe inspects fields like # ``endpoint.streaming`` (via ``require_streaming``) before emitting @@ -116,6 +115,22 @@ def resolve_config( return AIPerfConfig.model_validate(merged) +def _normalize_loaded_benchmark_shorthands(yaml_dict: dict[str, Any]) -> None: + """Normalize YAML benchmark shorthands before raw-dict CLI overlays. + + ``AIPerfConfig.model_validate`` already accepts conveniences such as + ``model:``, ``dataset:``, and single-dict ``phases: {type: ...}``, but + resolver overlay helpers inspect the loaded YAML dict before final + validation. Normalizing once here gives those helpers the same canonical + shape while preserving CLI-over-YAML precedence. + """ + from aiperf.config.loader.normalizers import normalize_benchmark_input + + benchmark = yaml_dict.get("benchmark") + if isinstance(benchmark, dict): + yaml_dict["benchmark"] = normalize_benchmark_input(benchmark) + + def deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]: """Recursively merge ``override`` onto ``base``; non-dict values replace. @@ -391,11 +406,6 @@ def _apply_dataset_filter_overrides(merged: dict[str, Any], cli: CLIConfig) -> N ("request_rate", "rate"), ("user_centric_rate", "rate"), ("num_users", "users"), - ("adaptive_sustain_duration", "adaptive_sustain_duration"), - ("adaptive_assessment_period", "adaptive_assessment_period"), - ("adaptive_control_variable", "adaptive_control_variable"), - ("adaptive_control_min", "adaptive_control_min"), - ("adaptive_control_max", "adaptive_control_max"), ) @@ -406,9 +416,9 @@ def _apply_phase_loadgen_overrides(merged: dict[str, Any], cli: CLIConfig) -> No YAML configs land ``phases`` as a list under ``benchmark.phases``; ``deep_merge`` replaces lists wholesale, so the CLI flags otherwise silently no-op when the YAML already sets ``phases[*].requests``. This - walks the merged envelope, finds the phase named ``profiling`` (or the - sole phase entry if there's only one), and writes each user-set - loadgen field onto it. Other phases (warmup) are left untouched so a + walks the merged envelope, finds the unique profiling-kind phase, and + writes each user-set loadgen field onto it. Other phases (warmup) are + left untouched so a user passing ``--request-count 10`` with ``warmup_profiling.yaml`` doesn't clobber the warmup ramp. """ @@ -428,7 +438,6 @@ def _apply_phase_loadgen_overrides(merged: dict[str, Any], cli: CLIConfig) -> No return _reject_loadgen_target_collisions(fields_set) - apply_basic_adaptive_scale_overrides(target, cli) for attr, key in _LOADGEN_PHASE_FIELD_MAP: if attr not in fields_set: @@ -466,26 +475,30 @@ def _reject_loadgen_target_collisions(fields_set: set[str]) -> None: def _find_profiling_phase(phases: list[Any]) -> dict[str, Any] | None: - """Return the phase entry whose CLI loadgen flags should overlay onto. + """Return the unique profiling-kind phase for CLI loadgen overlays. - Prefers the phase named ``profiling``. If no phase has that name (e.g. - a single-entry list authored without ``name``), returns the sole entry - when there's exactly one. Otherwise returns ``None`` and the override - silently no-ops -- matching the convention that ambiguous YAML wins. + Legacy YAML may omit ``kind`` on canonical names, so infer it for this + pre-validation merge pass. Ambiguous multi-profiling configs must express + values directly in YAML for v1. """ - profiling: dict[str, Any] | None = None - non_warmup: list[dict[str, Any]] = [] + candidates: list[dict[str, Any]] = [] for entry in phases: if not isinstance(entry, dict): continue - name = entry.get("name") - if name == "profiling": - profiling = entry - break - if name != "warmup": - non_warmup.append(entry) - if profiling is not None: - return profiling - if len(non_warmup) == 1: - return non_warmup[0] + kind = infer_legacy_phase_kind(entry.get("name"), entry.get("kind")) + if kind is not None: + entry["kind"] = kind + if kind == "profiling": + candidates.append(entry) + if len(candidates) == 1: + return candidates[0] + if len(candidates) > 1: + from aiperf.config.loader.errors import ConfigurationError + + names = [str(entry.get("name")) for entry in candidates] + raise ConfigurationError( + "CLI loadgen flags target the profiling phase, but this config has " + f"{len(candidates)} profiling phases: {', '.join(names)}. Set the " + "value in YAML or use an explicit phase path." + ) return None diff --git a/src/aiperf/config/loader/helpers.py b/src/aiperf/config/loader/helpers.py index 79b0c2fa63..d9bb70dce9 100644 --- a/src/aiperf/config/loader/helpers.py +++ b/src/aiperf/config/loader/helpers.py @@ -59,19 +59,19 @@ def get_default_dataset(self) -> DatasetConfig: return self.datasets[0] # type: ignore[attr-defined] def get_profiling_phases(self) -> list[PhaseConfig]: - """Get phase configs with exclude_from_results=False.""" + """Get profiling-kind phase configs.""" return [ phase for phase in self.phases # type: ignore[attr-defined] - if not phase.exclude_from_results + if phase.kind == "profiling" ] def get_warmup_phases(self) -> list[PhaseConfig]: - """Get warmup phase configs (excluded from results).""" + """Get warmup-kind phase configs.""" return [ phase for phase in self.phases # type: ignore[attr-defined] - if phase.exclude_from_results + if phase.kind == "warmup" ] # ========================================================================== diff --git a/src/aiperf/config/loader/normalizers.py b/src/aiperf/config/loader/normalizers.py index e5f8bf57cf..c4a7ccbd3c 100644 --- a/src/aiperf/config/loader/normalizers.py +++ b/src/aiperf/config/loader/normalizers.py @@ -38,6 +38,23 @@ ) +def _infer_phase_kind(phase: Any) -> Any: + if not isinstance(phase, dict): + return phase + name = phase.get("name") + if phase.get("kind") is None and name in {"warmup", "profiling"}: + phase["kind"] = name + return phase + + +def _infer_phase_kinds(data: dict[str, Any]) -> None: + phases = data.get("phases") + if isinstance(phases, list): + data["phases"] = [_infer_phase_kind(phase) for phase in phases] + elif isinstance(phases, dict): + data["phases"] = _infer_phase_kind(phases) + + def _check_mutual_exclusivity(data: dict[str, Any]) -> None: has_warmup = "warmup" in data has_profiling = "profiling" in data @@ -65,12 +82,12 @@ def _normalize_warmup_profiling_to_phases(data: dict[str, Any]) -> None: if has_warmup: warmup = data.pop("warmup") if isinstance(warmup, dict): - warmup = {"name": "warmup", **warmup} + warmup = {"name": "warmup", "kind": "warmup", **warmup} phases.append(warmup) if has_profiling: prof = data.pop("profiling") if isinstance(prof, dict): - prof = {"name": "profiling", **prof} + prof = {"name": "profiling", "kind": "profiling", **prof} phases.append(prof) data["phases"] = phases @@ -100,7 +117,7 @@ def _normalize_dataset_and_phases(data: dict[str, Any]) -> None: phases = data["phases"] # Single flat-dict shorthand: phases: {type: concurrency, ...} if isinstance(phases, dict) and "type" in phases: - data["phases"] = [{"name": "profiling", **phases}] + data["phases"] = [{"name": "profiling", "kind": "profiling", **phases}] def normalize_benchmark_input(data: Any) -> Any: @@ -120,6 +137,7 @@ def normalize_benchmark_input(data: Any) -> Any: _normalize_warmup_profiling_to_phases(data) _normalize_models(data) _normalize_dataset_and_phases(data) + _infer_phase_kinds(data) return data diff --git a/src/aiperf/config/phases.py b/src/aiperf/config/phases.py index c8b7551e4f..4146c6fe5e 100644 --- a/src/aiperf/config/phases.py +++ b/src/aiperf/config/phases.py @@ -19,6 +19,8 @@ model_validator, ) +from aiperf.common.phase import infer_legacy_phase_kind +from aiperf.common.types import PhaseKind from aiperf.config.adaptive_scale_phase import AdaptiveScalePhaseMixin from aiperf.config.base import BaseConfig from aiperf.config.cancellation import CancellationConfig @@ -42,6 +44,7 @@ "PhaseConfig", "PhaseType", "PhaseTypeStr", + "PhaseKind", "PoissonPhase", "RampConfig", "RampSpec", @@ -70,11 +73,29 @@ class BasePhaseConfig(AdaptiveScalePhaseMixin, BaseConfig): model_config = ConfigDict(extra="forbid") name: Annotated[ - Literal["warmup", "profiling"], + str, Field( - description="Phase identifier — must be 'warmup' or 'profiling'. " - "The credit pipeline only distinguishes these two phase kinds. " - "Used in logs, status, sweep targeting, and result file naming.", + pattern=r"^[A-Za-z_][A-Za-z0-9_-]*$", + description="Unique workflow label for this phase, such as " + "'low', 'storm', or 'recover'. This is distinct from phase kind: " + "multiple phases may share kind='profiling' while each keeps a " + "different name. Used in logs, status, sweep targeting, artifact " + "paths, and result file naming. Must be a strict identifier: " + "letters, numbers, underscores, and hyphens; must start with a " + "letter or underscore.", + ), + ] + + kind: Annotated[ + PhaseKind | None, + Field( + default=None, + description="Semantic runtime role for the phase. Only 'warmup' " + "and 'profiling' are valid kinds because the credit/results " + "pipeline distinguishes those two roles. This field is nullable " + "only as an input compatibility bridge: legacy canonical names " + "('warmup' and 'profiling') infer kind during normalization, " + "while validated phases always carry a concrete kind.", ), ] @@ -100,9 +121,9 @@ class BasePhaseConfig(AdaptiveScalePhaseMixin, BaseConfig): Field( default=False, description="Exclude this phase's metrics from final results. " - "Forced by phase name: 'warmup' is always excluded, " - "'profiling' is always included. Explicitly setting this " - "field to a value inconsistent with the phase name is rejected.", + "Forced by phase kind: kind='warmup' is always excluded, " + "kind='profiling' is always included. Explicitly setting this " + "field to a value inconsistent with the phase kind is rejected.", ), ] @@ -226,6 +247,11 @@ class BasePhaseConfig(AdaptiveScalePhaseMixin, BaseConfig): # ``aiperf.config.flags._converter_profiling``); YAML users must be # explicit. _stop_condition_required: ClassVar[bool] = True + _windows_reserved_phase_names: ClassVar[frozenset[str]] = frozenset( + {"CON", "PRN", "AUX", "NUL"} + | {f"COM{idx}" for idx in range(1, 10)} + | {f"LPT{idx}" for idx in range(1, 10)} + ) # ========================================================================= # VALIDATORS @@ -234,19 +260,37 @@ class BasePhaseConfig(AdaptiveScalePhaseMixin, BaseConfig): @model_validator(mode="after") def _validate_phase_constraints(self) -> Self: """Validate stop condition and cross-field constraints.""" - required = {"warmup": True, "profiling": False}.get(self.name) - if required is not None: - if ( - "exclude_from_results" in self.model_fields_set - and self.exclude_from_results != required - ): - raise ValueError( - f"Phase '{self.name}': exclude_from_results must be " - f"{required} (warmup is always excluded; profiling is " - f"always included)" - ) - if self.exclude_from_results != required: - self.exclude_from_results = required + windows_basename = self.name.split(".", 1)[0].upper() + if windows_basename in self._windows_reserved_phase_names: + raise ValueError( + f"Phase name '{self.name}' is reserved by Windows and cannot " + "be used as an artifact directory name." + ) + + self.kind = infer_legacy_phase_kind(self.name, self.kind) + if self.kind is None: + raise ValueError( + f"Phase '{self.name}': kind is required for non-canonical phase " + "names. Set kind to 'warmup' or 'profiling'." + ) + if self.name in {"warmup", "profiling"} and self.kind != self.name: + raise ValueError( + f"Phase name '{self.name}' is reserved for kind '{self.name}'; " + f"got kind '{self.kind}'." + ) + + required = self.kind == "warmup" + if ( + "exclude_from_results" in self.model_fields_set + and self.exclude_from_results != required + ): + raise ValueError( + f"Phase '{self.name}': exclude_from_results must be " + f"{required} for kind '{self.kind}' (warmup is always " + "excluded; profiling is always included)" + ) + if self.exclude_from_results != required: + self.exclude_from_results = required if ( self._stop_condition_required and self.requests is None diff --git a/src/aiperf/config/schema/aiperf-config.schema.json b/src/aiperf/config/schema/aiperf-config.schema.json index e0d1be58e1..2173a4238f 100644 --- a/src/aiperf/config/schema/aiperf-config.schema.json +++ b/src/aiperf/config/schema/aiperf-config.schema.json @@ -1042,14 +1042,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "concurrency", "description": "Concurrency-controlled immediate dispatch.", @@ -1058,7 +1072,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -1340,14 +1354,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "poisson", "description": "Poisson-distributed rate-controlled arrivals.", @@ -1356,7 +1384,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -1664,14 +1692,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "gamma", "description": "Gamma-distributed rate-controlled arrivals.", @@ -1680,7 +1722,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -2002,14 +2044,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "constant", "description": "Constant rate-controlled arrivals.", @@ -2018,7 +2074,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -2326,14 +2382,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "user_centric", "description": "N users sharing a global request rate.", @@ -2342,7 +2412,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -2657,14 +2727,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "fixed_schedule", "description": "Replay requests at trace timestamps.", @@ -2673,7 +2757,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -3678,14 +3762,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "concurrency", "description": "Concurrency-controlled immediate dispatch.", @@ -3694,7 +3792,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -3976,14 +4074,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "poisson", "description": "Poisson-distributed rate-controlled arrivals.", @@ -3992,7 +4104,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -4300,14 +4412,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "gamma", "description": "Gamma-distributed rate-controlled arrivals.", @@ -4316,7 +4442,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -4638,14 +4764,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "constant", "description": "Constant rate-controlled arrivals.", @@ -4654,7 +4794,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -4962,14 +5102,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "user_centric", "description": "N users sharing a global request rate.", @@ -4978,7 +5132,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -5293,14 +5447,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "fixed_schedule", "description": "Replay requests at trace timestamps.", @@ -5309,7 +5477,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -5648,14 +5816,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "concurrency", "description": "Concurrency-controlled immediate dispatch.", @@ -5664,7 +5846,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -5946,14 +6128,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "poisson", "description": "Poisson-distributed rate-controlled arrivals.", @@ -5962,7 +6158,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -6270,14 +6466,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "gamma", "description": "Gamma-distributed rate-controlled arrivals.", @@ -6286,7 +6496,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -6608,14 +6818,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "constant", "description": "Constant rate-controlled arrivals.", @@ -6624,7 +6848,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -6932,14 +7156,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "user_centric", "description": "N users sharing a global request rate.", @@ -6948,7 +7186,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -7263,14 +7501,28 @@ "type": "number" }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "fixed_schedule", "description": "Replay requests at trace timestamps.", @@ -7279,7 +7531,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -7850,14 +8102,28 @@ "x-jinja2-supported": true }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "concurrency", "description": "Concurrency-controlled immediate dispatch.", @@ -7866,7 +8132,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -8377,14 +8643,28 @@ "x-jinja2-supported": true }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "constant", "description": "Constant rate-controlled arrivals.", @@ -8393,7 +8673,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -10009,14 +10289,28 @@ "x-jinja2-supported": true }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "fixed_schedule", "description": "Replay requests at trace timestamps.", @@ -10025,7 +10319,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -10600,14 +10894,28 @@ "x-jinja2-supported": true }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "gamma", "description": "Gamma-distributed rate-controlled arrivals.", @@ -10616,7 +10924,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -12942,14 +13250,28 @@ "x-jinja2-supported": true }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "poisson", "description": "Poisson-distributed rate-controlled arrivals.", @@ -12958,7 +13280,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, @@ -15942,14 +16264,28 @@ "x-jinja2-supported": true }, "name": { - "description": "Phase identifier — must be 'warmup' or 'profiling'. The credit pipeline only distinguishes these two phase kinds. Used in logs, status, sweep targeting, and result file naming.", - "enum": [ - "warmup", - "profiling" - ], + "description": "Unique workflow label for this phase, such as 'low', 'storm', or 'recover'. This is distinct from phase kind: multiple phases may share kind='profiling' while each keeps a different name. Used in logs, status, sweep targeting, artifact paths, and result file naming. Must be a strict identifier: letters, numbers, underscores, and hyphens; must start with a letter or underscore.", + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$", "title": "Name", "type": "string" }, + "kind": { + "anyOf": [ + { + "enum": [ + "warmup", + "profiling" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Semantic runtime role for the phase. Only 'warmup' and 'profiling' are valid kinds because the credit/results pipeline distinguishes those two roles. This field is nullable only as an input compatibility bridge: legacy canonical names ('warmup' and 'profiling') infer kind during normalization, while validated phases always carry a concrete kind.", + "title": "Kind" + }, "type": { "const": "user_centric", "description": "N users sharing a global request rate.", @@ -15958,7 +16294,7 @@ }, "excludeFromResults": { "default": false, - "description": "Exclude this phase's metrics from final results. Forced by phase name: 'warmup' is always excluded, 'profiling' is always included. Explicitly setting this field to a value inconsistent with the phase name is rejected.", + "description": "Exclude this phase's metrics from final results. Forced by phase kind: kind='warmup' is always excluded, kind='profiling' is always included. Explicitly setting this field to a value inconsistent with the phase kind is rejected.", "title": "Excludefromresults", "type": "boolean" }, diff --git a/src/aiperf/config/sweep/expand.py b/src/aiperf/config/sweep/expand.py index 6ae863fbdb..a011106c4d 100644 --- a/src/aiperf/config/sweep/expand.py +++ b/src/aiperf/config/sweep/expand.py @@ -493,10 +493,9 @@ def _set_nested_value(data: dict, path: str, value: Any) -> None: raise ValueError rather than silently appending a phantom entry. Special case for ``phases.profiling.``: when no phase named - ``profiling`` exists, fall back to the unique non-warmup phase (if - exactly one exists). YAML's ``phases: {type: ..}`` shorthand emits - a phase named ``default``, but search recipes hard-code the - ``profiling`` segment. See ``_find_phase_or_recipe_alias``. + ``profiling`` exists, fall back to the unique profiling-kind phase. + Legacy pre-kind configs with exactly one non-warmup phase keep the + old recipe-friendly fallback. See ``_find_phase_or_recipe_alias``. """ keys = path.split(".") current: Any = data @@ -506,10 +505,8 @@ def _set_nested_value(data: dict, path: str, value: Any) -> None: current, key, parent_key=keys[i - 1] if i > 0 else "" ) if match is None: - names = [item.get("name") for item in current] - raise ValueError( - f"sweep path {path!r}: no entry named {key!r} found " - f"(existing: {names}). Add the entry first or fix the typo." + _raise_named_list_resolution_error( + path, key, current, parent_key=keys[i - 1] if i > 0 else "" ) current = match continue @@ -524,10 +521,8 @@ def _set_nested_value(data: dict, path: str, value: Any) -> None: current, last, parent_key=keys[-2] if len(keys) >= 2 else "" ) if match is None: - names = [item.get("name") for item in current] - raise ValueError( - f"sweep path {path!r}: no entry named {last!r} found " - f"(existing: {names}). Add the entry first or fix the typo." + _raise_named_list_resolution_error( + path, last, current, parent_key=keys[-2] if len(keys) >= 2 else "" ) match[last] = value elif isinstance(current, dict): @@ -551,24 +546,75 @@ def _raise_traversal_type_error( ) +def _raise_named_list_resolution_error( + path: str, name: str, items: list[dict[str, Any]], *, parent_key: str +) -> None: + names = [item.get("name") for item in items] + if parent_key == "phases" and name == "profiling": + candidates, _legacy_candidates = _profiling_alias_candidates(items) + if len(candidates) > 1: + candidate_names = [str(item.get("name")) for item in candidates] + suffix = "" + prefix = "phases.profiling." + if path.startswith(prefix): + suffix = "." + path[len(prefix) :] + name_example = f"phases.{candidate_names[0]}{suffix}" + index_example = f"phases.0{suffix}" + raise ValueError( + f"sweep path {path!r} is ambiguous: {len(candidates)} " + f"profiling phases exist ({', '.join(candidate_names)}). " + f"Use an explicit phase name like {name_example!r} or " + f"a numeric index like {index_example!r}." + ) + raise ValueError( + f"sweep path {path!r}: no entry named {name!r} found " + f"(existing: {names}). Add the entry first or fix the typo." + ) + + +def _profiling_alias_candidates( + items: list[dict[str, Any]], +) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + candidates = [] + legacy_candidates = [] + for item in items: + item_name = item.get("name") + kind = item.get("kind") + if kind is None and item_name in {"warmup", "profiling"}: + kind = item_name + if kind == "profiling": + candidates.append(item) + elif kind is None and item_name != "warmup": + legacy_candidates.append(item) + return candidates, legacy_candidates + + def _find_phase_or_recipe_alias( items: list[dict[str, Any]], name: str, *, parent_key: str ) -> dict[str, Any] | None: - """Resolve `_find_named` with a `phases.profiling` recipe-friendly fallback. + """Resolve phase path segments by index, name, then legacy profiling kind. - When ``name`` is ``profiling`` and the parent segment is ``phases`` and - no phase by that name exists, return the unique non-warmup phase if one - exists; otherwise return None (caller raises with the existing-names - list, same as a plain miss). + Resolution order follows the named-phase plan: numeric index, exact unique + ``name``, then ``phases.profiling`` as shorthand for the unique profiling + kind only when no exact name match exists. """ + if parent_key == "phases" and name.isdigit(): + index = int(name) + if 0 <= index < len(items): + return items[index] + return None + direct = _find_named(items, name) if direct is not None: return direct if name != "profiling" or parent_key != "phases": return None - candidates = [item for item in items if item.get("name") != "warmup"] + + candidates, legacy_candidates = _profiling_alias_candidates(items) if len(candidates) == 1: return candidates[0] + if not candidates and len(legacy_candidates) == 1: + return legacy_candidates[0] return None diff --git a/src/aiperf/credit/callback_handler.py b/src/aiperf/credit/callback_handler.py index 19da73c6a2..a2e053036c 100644 --- a/src/aiperf/credit/callback_handler.py +++ b/src/aiperf/credit/callback_handler.py @@ -20,6 +20,7 @@ from aiperf.common.aiperf_logger import AIPerfLogger from aiperf.common.enums import CreditPhase +from aiperf.timing.concurrency import PhaseRuntimeKey if TYPE_CHECKING: from aiperf.credit.messages import CreditReturn, FirstToken @@ -86,7 +87,7 @@ def __init__(self, concurrency_manager: ConcurrencyManager) -> None: concurrency_manager: Manages concurrency slots (shared across phases). """ self._concurrency_manager = concurrency_manager - self._phase_handlers: dict[CreditPhase, PhaseCallbackContext] = {} + self._phase_handlers: dict[PhaseRuntimeKey, PhaseCallbackContext] = {} self._branch_orchestrator: BranchOrchestrator | None = None def set_branch_orchestrator(self, orchestrator: BranchOrchestrator | None) -> None: @@ -147,10 +148,17 @@ def _on_orchestrator_abort(self) -> None: handler.lifecycle.cancel() handler.progress.all_credits_returned_event.set() + @staticmethod + def _phase_key( + phase: CreditPhase, phase_index: int | None = None + ) -> PhaseRuntimeKey: + return phase_index if phase_index is not None else phase + def register_phase( self, *, phase: CreditPhase, + phase_index: int | None = None, progress: PhaseProgressTracker, lifecycle: PhaseLifecycle, stop_checker: StopConditionChecker, @@ -170,7 +178,8 @@ def register_phase( """ handle_credit_result = getattr(strategy, "handle_credit_result", None) handle_first_token = getattr(strategy, "handle_first_token", None) - self._phase_handlers[phase] = PhaseCallbackContext( + key = self._phase_key(phase, phase_index) + self._phase_handlers[key] = PhaseCallbackContext( progress=progress, lifecycle=lifecycle, stop_checker=stop_checker, @@ -183,9 +192,13 @@ def register_phase( if inspect.iscoroutinefunction(handle_first_token) else None, ) - _logger.debug(lambda: f"Registered callback handler for phase {phase}") + _logger.debug( + lambda: f"Registered callback handler for phase {phase} key={key}" + ) - def unregister_phase(self, phase: CreditPhase) -> None: + def unregister_phase( + self, phase: CreditPhase, phase_index: int | None = None + ) -> None: """Unregister phase when done. Called by PhaseRunner after phase completes. @@ -194,9 +207,12 @@ def unregister_phase(self, phase: CreditPhase) -> None: Args: phase: Phase to unregister. """ - if phase in self._phase_handlers: - del self._phase_handlers[phase] - _logger.debug(lambda: f"Unregistered callback handler for phase {phase}") + key = self._phase_key(phase, phase_index) + if key in self._phase_handlers: + del self._phase_handlers[key] + _logger.debug( + lambda: f"Unregistered callback handler for phase {phase} key={key}" + ) async def on_credit_return( self, worker_id: str, credit_return: CreditReturn @@ -269,11 +285,12 @@ def _lookup_active_phase_handler( a None return. """ phase = credit.phase - handler = self._phase_handlers.get(phase) + key = self._phase_key(credit.phase, credit.phase_index) + handler = self._phase_handlers.get(key) if not handler: _logger.debug( lambda: ( - f"Credit return for unregistered phase {phase}, " + f"Credit return for unregistered phase {phase} key={key}, " f"credit_id={credit.id}, worker={worker_id}" ) ) @@ -281,7 +298,7 @@ def _lookup_active_phase_handler( if handler.lifecycle.is_complete: _logger.warning( lambda: ( - f"Credit return after phase {phase} complete, " + f"Credit return after phase {phase} key={key} complete, " f"credit_id={credit.id}, worker={worker_id}" ) ) @@ -310,7 +327,11 @@ def _count_and_release( # 3. Release concurrency slots self._release_slots_for_return( - credit.phase, credit, credit_return, is_final_returned, handler + self._phase_key(credit.phase, credit.phase_index), + credit, + credit_return, + is_final_returned, + handler, ) # 4. Signal completion if this was the final return — but defer for @@ -407,7 +428,7 @@ async def _intercept_for_dag(self, credit: Credit) -> bool: def _release_slots_for_return( self, - phase: CreditPhase, + phase: PhaseRuntimeKey, credit: Credit, credit_return: CreditReturn, is_final_returned: bool, @@ -464,12 +485,13 @@ async def on_first_token(self, first_token: FirstToken) -> None: first_token: TTFT event details including credit_id and phase. """ phase = first_token.phase - handler = self._phase_handlers.get(phase) + key = self._phase_key(first_token.phase, first_token.phase_index) + handler = self._phase_handlers.get(key) if not handler: _logger.debug( lambda: ( - f"TTFT for unregistered phase {phase}, " + f"TTFT for unregistered phase {phase} key={key}, " f"credit_id={first_token.credit_id}" ) ) @@ -482,4 +504,4 @@ async def on_first_token(self, first_token: FirstToken) -> None: handler.progress.increment_prefill_released() # Release the prefill slot - handler.concurrency_manager.release_prefill_slot(phase) + handler.concurrency_manager.release_prefill_slot(key) diff --git a/src/aiperf/credit/issuer.py b/src/aiperf/credit/issuer.py index 41a14da326..9fe6903080 100644 --- a/src/aiperf/credit/issuer.py +++ b/src/aiperf/credit/issuer.py @@ -18,6 +18,7 @@ from typing import TYPE_CHECKING from aiperf.common.enums import CreditPhase +from aiperf.common.phase import phase_runtime_key from aiperf.credit.structs import Credit, TurnToSend from aiperf.timing.url_samplers import URLSelectionStrategyProtocol @@ -55,6 +56,10 @@ def __init__( self, *, phase: CreditPhase, + phase_index: int | None = None, + profiling_index: int | None = None, + phase_name: str | None = None, + phase_kind: str | None = None, stop_checker: StopConditionChecker, progress: PhaseProgressTracker, concurrency_manager: ConcurrencyManager, @@ -77,6 +82,10 @@ def __init__( balancing. If None, url_index will be None in credits. """ self._phase = phase + self._phase_index = phase_index + self._profiling_index = profiling_index + self._phase_name = phase_name + self._phase_kind = phase_kind self._stop_checker = stop_checker self._progress = progress self._concurrency_manager = concurrency_manager @@ -85,10 +94,14 @@ def __init__( self._lifecycle = lifecycle self._url_selection_strategy = url_selection_strategy + @property + def _phase_key(self) -> int | CreditPhase: + return phase_runtime_key(self._phase, self._phase_index) + def can_acquire_and_start_new_session(self) -> bool: """Check if a session slot can be acquired and a new session can be started.""" return ( - self._concurrency_manager.session_slot_available(self._phase) + self._concurrency_manager.session_slot_available(self._phase_key) and self._stop_checker.can_start_new_session() ) @@ -130,7 +143,7 @@ async def issue_credit(self, turn: TurnToSend) -> bool: # Controls how many multi-turn conversations can be active simultaneously. if is_first_turn: acquired = await self._concurrency_manager.acquire_session_slot( - self._phase, self._stop_checker.can_start_new_session + self._phase_key, self._stop_checker.can_start_new_session ) if not acquired: return False @@ -138,12 +151,12 @@ async def issue_credit(self, turn: TurnToSend) -> bool: # Prefill concurrency: one slot per request, released when TTFT arrives. # Limits concurrent prompt processing which is the GPU-intensive phase. acquired = await self._concurrency_manager.acquire_prefill_slot( - self._phase, can_proceed_fn + self._phase_key, can_proceed_fn ) if not acquired: # CRITICAL: Release session slot if we acquired it to maintain symmetry if is_first_turn: - self._concurrency_manager.release_session_slot(self._phase) + self._concurrency_manager.release_session_slot(self._phase_key) return False # Slots acquired - proceed with credit issuance @@ -178,18 +191,18 @@ async def try_issue_credit(self, turn: TurnToSend) -> bool | None: if is_first_turn: acquired = self._concurrency_manager.try_acquire_session_slot( - self._phase, can_proceed_fn + self._phase_key, can_proceed_fn ) if not acquired: return None # No slot - credit not issued acquired = self._concurrency_manager.try_acquire_prefill_slot( - self._phase, can_proceed_fn + self._phase_key, can_proceed_fn ) if not acquired: # CRITICAL: Release session slot if we acquired it to maintain symmetry if is_first_turn: - self._concurrency_manager.release_session_slot(self._phase) + self._concurrency_manager.release_session_slot(self._phase_key) return None # No slot - credit not issued return await self._issue_credit_internal(turn) @@ -222,6 +235,10 @@ async def _issue_credit_internal(self, turn: TurnToSend) -> bool: credit = Credit( id=credit_index, phase=self._phase, + phase_index=self._phase_index, + profiling_index=self._profiling_index, + phase_name=self._phase_name, + phase_kind=self._phase_kind, conversation_id=turn.conversation_id, x_correlation_id=turn.x_correlation_id, turn_index=turn.turn_index, @@ -343,7 +360,7 @@ async def _dispatch_dag_turn(self, turn: TurnToSend) -> bool: if not self._stop_checker.can_send_dag_child_turn(): return False acquired = await self._concurrency_manager.acquire_prefill_slot( - self._phase, self._stop_checker.can_send_dag_child_turn + self._phase_key, self._stop_checker.can_send_dag_child_turn ) if not acquired: return False diff --git a/src/aiperf/credit/messages.py b/src/aiperf/credit/messages.py index 00a092b5f5..2bf56821ae 100644 --- a/src/aiperf/credit/messages.py +++ b/src/aiperf/credit/messages.py @@ -148,6 +148,7 @@ class FirstToken(Struct, frozen=True, kw_only=True, tag_field="t", tag="ft"): credit_id: int phase: CreditPhase ttft_ns: int + phase_index: int | None = None # Union type for decoding worker -> router messages diff --git a/src/aiperf/credit/structs.py b/src/aiperf/credit/structs.py index 0a80926ff3..1257f6f51a 100644 --- a/src/aiperf/credit/structs.py +++ b/src/aiperf/credit/structs.py @@ -11,6 +11,7 @@ from msgspec import Struct from aiperf.common.enums import ConversationBranchMode, CreditPhase +from aiperf.common.types import PhaseKind if TYPE_CHECKING: from aiperf.common.models.dataset_models import TurnMetadata @@ -54,6 +55,10 @@ class Credit( id: int phase: CreditPhase + phase_index: int | None = None + profiling_index: int | None = None + phase_name: str | None = None + phase_kind: PhaseKind | None = None conversation_id: str x_correlation_id: str turn_index: int diff --git a/src/aiperf/dataset/composer/custom.py b/src/aiperf/dataset/composer/custom.py index 384f447cf4..318f42b013 100644 --- a/src/aiperf/dataset/composer/custom.py +++ b/src/aiperf/dataset/composer/custom.py @@ -133,15 +133,18 @@ def _infer_dataset_type(self, file_path: str) -> CustomDatasetType: # For files, read first non-empty line and use both content and path detection with open(file_path, encoding="utf-8") as f: - for line in f: - if not (line := line.strip()): - continue - try: - data = load_json_str(line) - except (orjson.JSONDecodeError, UnicodeDecodeError): - # Non-JSON file (e.g. CSV) — fall back to filename-based detection - return self._infer_type(data=None, filename=file_path) - return self._infer_type(data=data, filename=file_path) + try: + for line in f: + if not (line := line.strip()): + continue + try: + data = load_json_str(line) + except orjson.JSONDecodeError: + # Non-JSON file (e.g. CSV) — fall back to filename-based detection + return self._infer_type(data=None, filename=file_path) + return self._infer_type(data=data, filename=file_path) + except UnicodeDecodeError: + return self._infer_type(data=None, filename=file_path) except ValueError as e: self.exception( diff --git a/src/aiperf/exporters/exporter_manager.py b/src/aiperf/exporters/exporter_manager.py index 4e3b690c58..bb11ba3066 100644 --- a/src/aiperf/exporters/exporter_manager.py +++ b/src/aiperf/exporters/exporter_manager.py @@ -3,8 +3,10 @@ import asyncio import io -from typing import TYPE_CHECKING +from pathlib import Path +from typing import TYPE_CHECKING, Any +import orjson from rich.console import Console from aiperf.common.environment import Environment @@ -12,11 +14,14 @@ ConsoleExporterDisabled, DataExporterDisabled, ) +from aiperf.common.finite import scrub_non_finite from aiperf.common.mixins import AIPerfLoggerMixin from aiperf.common.models import ProfileResults from aiperf.common.models.export_models import TelemetryExportData from aiperf.common.models.server_metrics_models import ServerMetricsResults from aiperf.exporters.exporter_config import ExporterConfig, FileExportInfo +from aiperf.exporters.metrics_csv_exporter import MetricsCsvExporter +from aiperf.exporters.metrics_json_exporter import MetricsJsonExporter from aiperf.exporters.protocols import ConsoleExporterProtocol, DataExporterProtocol from aiperf.plugin import plugins from aiperf.plugin.enums import DataExporterType, PluginType @@ -97,6 +102,11 @@ async def export_data(self) -> None: await asyncio.gather(*self._tasks, return_exceptions=True) self._tasks.clear() + try: + await self._export_phase_metric_artifacts() + except (OSError, ValueError) as exc: + self.warning(f"Failed to export phase metric artifacts: {exc}") + for exporter in deferred_exporters: self.debug(f"Running deferred exporter: {exporter.__class__.__name__}") task = asyncio.create_task(exporter.export()) @@ -107,6 +117,121 @@ async def export_data(self) -> None: self._tasks.clear() self.debug("Exporting all records completed") + async def _export_phase_metric_artifacts(self) -> None: + phase_records = getattr(self._results, "phase_records", None) or [] + if not phase_records: + return + manifest_entries: list[dict[str, Any]] = [] + for phase_result in phase_records: + phase_dir = self._run.cfg.artifacts.dir / "phases" / phase_result.phase_name + await asyncio.to_thread(phase_dir.mkdir, parents=True, exist_ok=True) + completed = ( + phase_result.successful_request_count + phase_result.error_request_count + ) + phase_profile = ProfileResults( + records=phase_result.records, + completed=completed, + start_ns=phase_result.start_ns or self._results.start_ns, + end_ns=phase_result.end_ns or self._results.end_ns, + was_cancelled=phase_result.was_cancelled, + successful_request_count=phase_result.successful_request_count, + error_request_count=phase_result.error_request_count, + error_summary=phase_result.error_summary, + ) + entry: dict[str, Any] = { + "phase_index": phase_result.phase_index, + "profiling_index": phase_result.profiling_index, + "phase_name": phase_result.phase_name, + "phase_kind": phase_result.phase_kind, + "start_ns": phase_result.start_ns, + "end_ns": phase_result.end_ns, + "was_cancelled": phase_result.was_cancelled, + "successful_request_count": phase_result.successful_request_count, + "error_request_count": phase_result.error_request_count, + "total_request_count": completed, + "error_summary": [ + item.model_dump(mode="json") for item in phase_result.error_summary + ], + } + await self._write_phase_export( + exporter_cls=MetricsJsonExporter, + phase_profile=phase_profile, + file_path=phase_dir + / self._run.cfg.artifacts.profile_export_json_file.name, + manifest_entry=entry, + manifest_key="metrics_json", + ) + await self._write_phase_export( + exporter_cls=MetricsCsvExporter, + phase_profile=phase_profile, + file_path=phase_dir + / self._run.cfg.artifacts.profile_export_csv_file.name, + manifest_entry=entry, + manifest_key="metrics_csv", + ) + manifest_entries.append(entry) + try: + await asyncio.to_thread(self._write_phase_manifest, manifest_entries) + except (OSError, ValueError) as exc: + self.warning(f"Failed to write phase artifact manifest: {exc}") + + async def _write_phase_export( + self, + *, + exporter_cls: type[MetricsJsonExporter] | type[MetricsCsvExporter], + phase_profile: ProfileResults, + file_path: Path, + manifest_entry: dict[str, Any], + manifest_key: str, + ) -> None: + config = ExporterConfig( + results=phase_profile, + cfg=self._run.cfg, + telemetry_results=None, + server_metrics_results=None, + run=self._run, + ) + try: + exporter = exporter_cls(exporter_config=config) + except DataExporterDisabled: + return + except Exception as exc: + self.error( + f"Error creating phase exporter {exporter_cls.__name__} " + f"for {manifest_entry.get('phase_name')}: {exc!r}" + ) + return + try: + content = exporter._generate_content() + await asyncio.to_thread(file_path.write_text, content, encoding="utf-8") + except Exception as exc: + self.error(f"Failed to write phase export {file_path}: {exc!r}") + return + manifest_entry[manifest_key] = file_path.relative_to( + self._run.cfg.artifacts.dir + ).as_posix() + + def _write_phase_manifest(self, entries: list[dict[str, Any]]) -> None: + manifest_path = self._run.cfg.artifacts.dir / "phase_manifest.json" + payload = { + "schema_version": 1, + "phases": sorted( + entries, + key=lambda item: ( + item.get("phase_index") + if item.get("phase_index") is not None + else 10**9 + ), + ), + } + manifest_path.write_bytes( + orjson.dumps( + scrub_non_finite(payload), + option=orjson.OPT_INDENT_2 | orjson.OPT_SORT_KEYS, + ) + + b"\n" + ) + def get_exported_file_infos(self) -> list[FileExportInfo]: """Get the file infos for all exported files.""" file_infos = [] diff --git a/src/aiperf/metrics/accumulator.py b/src/aiperf/metrics/accumulator.py index 0fadf07b40..8859d0343f 100644 --- a/src/aiperf/metrics/accumulator.py +++ b/src/aiperf/metrics/accumulator.py @@ -209,6 +209,9 @@ async def process_record(self, record: MetricRecordsData) -> None: "worker_id": meta.worker_id, "record_processor_id": meta.record_processor_id, "benchmark_phase": str(meta.benchmark_phase), + "phase_index": str(meta.phase_index) + if meta.phase_index is not None + else None, "x_correlation_id": meta.x_correlation_id, "conversation_id": meta.conversation_id, }, @@ -260,6 +263,10 @@ def _mask_for_export_context(self, ctx: ExportContext | None) -> BoolArray | Non mask &= self._column_store.mask_for_categorical( "benchmark_phase", phase_value ) + if ctx.phase_index is not None: + mask &= self._column_store.mask_for_categorical( + "phase_index", str(ctx.phase_index) + ) return mask if ctx.start_ns is not None: mask &= self._column_store.start_ns[:n] >= ctx.start_ns diff --git a/src/aiperf/orchestrator/search_planner/_pooled_percentile.py b/src/aiperf/orchestrator/search_planner/_pooled_percentile.py index 5d498a839a..e1cb765057 100644 --- a/src/aiperf/orchestrator/search_planner/_pooled_percentile.py +++ b/src/aiperf/orchestrator/search_planner/_pooled_percentile.py @@ -9,7 +9,7 @@ correct statistic; mean-of-percentiles is a heuristic. This helper walks each ``RunResult.artifacts_path / "profile_export.jsonl"`` -file -- written by :mod:`aiperf.post_processors.record_export_jsonl_writer` +file -- written by :mod:`aiperf.post_processors.record_export_results_processor` when ``--export-level records`` (or ``raw``) is set -- accumulates raw metric samples into a single bag, and returns ``np.percentile(bag, pct)``. diff --git a/src/aiperf/records/error_tracker.py b/src/aiperf/records/error_tracker.py index 58bf8cda22..c396f15e15 100644 --- a/src/aiperf/records/error_tracker.py +++ b/src/aiperf/records/error_tracker.py @@ -2,10 +2,13 @@ # SPDX-License-Identifier: Apache-2.0 from collections import defaultdict +from typing import TypeAlias from aiperf.common.enums import CreditPhase from aiperf.common.models import ErrorDetails, ErrorDetailsCount +PhaseErrorKey: TypeAlias = CreditPhase | tuple[CreditPhase, int] + class PhaseErrorTracker: """Phase Error Tracker. This is used to track the errors encountered during a credit phase. @@ -41,24 +44,34 @@ class ErrorTracker: """ def __init__(self) -> None: - self._phase_error_trackers: dict[CreditPhase, PhaseErrorTracker] = {} + self._phase_error_trackers: dict[PhaseErrorKey, PhaseErrorTracker] = {} - def _get_phase_error_tracker(self, phase: CreditPhase) -> PhaseErrorTracker: + def _get_phase_error_tracker( + self, phase: CreditPhase, phase_index: int | None = None + ) -> PhaseErrorTracker: """Get the phase error tracker.""" - if phase not in self._phase_error_trackers: - self._phase_error_trackers.setdefault(phase, PhaseErrorTracker(phase)) - return self._phase_error_trackers[phase] + key: PhaseErrorKey = phase if phase_index is None else (phase, phase_index) + if key not in self._phase_error_trackers: + self._phase_error_trackers.setdefault(key, PhaseErrorTracker(phase)) + return self._phase_error_trackers[key] def increment_error_count_for_phase( - self, phase: CreditPhase, error: ErrorDetails + self, phase: CreditPhase, error: ErrorDetails, phase_index: int | None = None ) -> None: """Increment the error count for an error in a phase.""" phase_error_tracker = self._get_phase_error_tracker(phase) phase_error_tracker.increment_error_count(error) + if phase_index is not None: + indexed_phase_error_tracker = self._get_phase_error_tracker( + phase, phase_index=phase_index + ) + indexed_phase_error_tracker.increment_error_count(error) def get_error_summary_for_phase( - self, phase: CreditPhase + self, phase: CreditPhase, phase_index: int | None = None ) -> list[ErrorDetailsCount]: """Get the error summary for a phase.""" - phase_error_tracker = self._get_phase_error_tracker(phase) + phase_error_tracker = self._get_phase_error_tracker( + phase, phase_index=phase_index + ) return phase_error_tracker.get_error_summary() diff --git a/src/aiperf/records/record_processor_service.py b/src/aiperf/records/record_processor_service.py index 56644e57fd..04f2b6259a 100644 --- a/src/aiperf/records/record_processor_service.py +++ b/src/aiperf/records/record_processor_service.py @@ -233,6 +233,10 @@ def _create_metric_record_metadata( turn_index=record.request_info.turn_index, record_processor_id=self.service_id, benchmark_phase=record.request_info.credit_phase, + phase_index=record.request_info.phase_index, + profiling_index=record.request_info.profiling_index, + phase_name=record.request_info.phase_name, + phase_kind=record.request_info.phase_kind, x_request_id=record.request_info.x_request_id, x_correlation_id=record.request_info.x_correlation_id, session_num=record.request_info.credit_num, @@ -396,9 +400,17 @@ async def _forward_failed_record( if record.request_info is not None: session_num = record.request_info.credit_num benchmark_phase = record.request_info.credit_phase + phase_index = record.request_info.phase_index + profiling_index = record.request_info.profiling_index + phase_name = record.request_info.phase_name + phase_kind = record.request_info.phase_kind else: session_num = -1 benchmark_phase = CreditPhase.PROFILING + phase_index = None + profiling_index = None + phase_name = None + phase_kind = None metadata = MetricRecordMetadata( session_num=session_num, request_start_ns=record.timestamp_ns, @@ -406,6 +418,10 @@ async def _forward_failed_record( worker_id=message.service_id, record_processor_id=self.service_id, benchmark_phase=benchmark_phase, + phase_index=phase_index, + profiling_index=profiling_index, + phase_name=phase_name, + phase_kind=phase_kind, ) error = record.error or ErrorDetails.from_exception(exc) # The producers didn't run, so ship a RecordsMessage carrying a single diff --git a/src/aiperf/records/records_manager.py b/src/aiperf/records/records_manager.py index cd9fdb6f83..ce73d06e28 100644 --- a/src/aiperf/records/records_manager.py +++ b/src/aiperf/records/records_manager.py @@ -52,6 +52,7 @@ ErrorDetails, ErrorDetailsCount, MetricResult, + PhaseProfileResults, PhaseRecordsStats, ProcessRecordsResult, ProcessServerMetricsResult, @@ -419,6 +420,8 @@ def __init__( # warmup vs profiling separation. self._phase_branch_stats: dict[CreditPhase, BranchStats] = {} self._complete_credit_phases: set[CreditPhase] = set() + self._credits_complete_received = False + self._all_records_received_phases: set[CreditPhase] = set() self._telemetry_state = ErrorTrackingState() self._server_metrics_state = ErrorTrackingState() @@ -566,6 +569,24 @@ def _log_routing_table(self) -> None: handler_names = [handler.__class__.__name__ for handler in handlers] self.debug(lambda rt=record_type, hn=handler_names: f" {rt} -> {hn}") + def _has_multiple_profiling_phases(self) -> bool: + """Return whether this run has more than one profiling-kind phase.""" + try: + phases = self.run.cfg.phases + except AttributeError: + return False + return sum(1 for phase in phases if phase.kind == "profiling") > 1 + + def _check_all_records_received( + self, phase: CreditPhase, phase_index: int | None = None + ) -> bool: + """Check record completion, aggregating only for multi-profiling runs.""" + if phase == CreditPhase.PROFILING and self._has_multiple_profiling_phases(): + return self._records_tracker.check_and_set_all_records_received_for_phase( + phase + ) + return self._records_tracker.check_and_set_all_records_received_for_phase(phase) + @on_pull_message(MessageType.RECORDS) async def _on_records(self, message: RecordsMessage) -> None: """Handle a per-request records envelope generically. @@ -588,22 +609,29 @@ async def _on_records(self, message: RecordsMessage) -> None: phase = message.metadata.benchmark_phase self._records_tracker.update_from_request(message.metadata, message.error) if message.error: - self._error_tracker.increment_error_count_for_phase(phase, message.error) + self._error_tracker.increment_error_count_for_phase( + phase, message.error, phase_index=message.metadata.phase_index + ) # A metric accumulator/exporter that failed to ingest this record yields # incomplete metrics; surface it in the phase error summary rather than # marking the record cleanly processed and silently dropping the failure. for error in dispatch_errors: self._error_tracker.increment_error_count_for_phase( - phase, ErrorDetails.from_exception(error) + phase, + ErrorDetails.from_exception(error), + phase_index=message.metadata.phase_index, ) if ( phase in self._complete_credit_phases - and self._records_tracker.check_and_set_all_records_received_for_phase( - phase + and ( + phase != CreditPhase.PROFILING + or not self._has_multiple_profiling_phases() + or self._credits_complete_received ) + and self._check_all_records_received(phase, message.metadata.phase_index) ): - await self._handle_all_records_received(phase) + await self._handle_all_records_received_once(phase) @on_pull_message(MessageType.TELEMETRY_RECORDS) async def _on_telemetry_records(self, message: TelemetryRecordsMessage) -> None: @@ -645,13 +673,26 @@ async def _on_network_latency_records( elif message.error: self._network_latency_state.error_counts[message.error] += 1 + async def _handle_all_records_received_once(self, phase: CreditPhase) -> None: + """Idempotently trigger all-records finalization for one phase kind.""" + handled = getattr(self, "_all_records_received_phases", set()) + if phase in handled: + return + handled.add(phase) + self._all_records_received_phases = handled + await self._handle_all_records_received(phase) + async def _handle_all_records_received(self, phase: CreditPhase) -> None: """Handle the case where all records have been received.""" if phase != CreditPhase.PROFILING: self.debug(lambda: f"Skipping non-profiling phase: {phase}") return - phase_stats = self._records_tracker.create_stats_for_phase(phase) + phase_stats = ( + self._records_tracker.create_aggregate_stats_for_phase(phase) + if phase == CreditPhase.PROFILING + else self._records_tracker.create_stats_for_phase(phase) + ) self.info( lambda: ( f"Processed {phase_stats.success_records} valid requests and {phase_stats.error_records} errors ({phase_stats.total_records} total)." @@ -674,7 +715,11 @@ async def _finalize_and_process_results( This runs as a background task to avoid blocking the message pump. """ - phase_stats = self._records_tracker.create_stats_for_phase(phase) + phase_stats = ( + self._records_tracker.create_aggregate_stats_for_phase(phase) + if phase == CreditPhase.PROFILING + else self._records_tracker.create_stats_for_phase(phase) + ) # Send a message to the event bus to signal that we received all the records await self.publish( @@ -700,7 +745,7 @@ async def _finalize_and_process_results( # Wait for server metrics flush period to allow final metrics to be collected # This ensures metrics that are still being processed by the server are captured flush_period = Environment.SERVER_METRICS.COLLECTION_FLUSH_PERIOD - phase_stats = self._records_tracker.create_stats_for_phase( + phase_stats = self._records_tracker.create_aggregate_stats_for_phase( CreditPhase.PROFILING ) flush_end_ns = (phase_stats.requests_end_ns or time.time_ns()) + ( @@ -787,10 +832,14 @@ async def _on_credit_phase_complete( # This check is to prevent a race condition where the records manager processes # all records before the timing manager has sent the final completed count. - if self._records_tracker.check_and_set_all_records_received_for_phase( - message.stats.phase + if ( + message.stats.phase != CreditPhase.PROFILING + or not self._has_multiple_profiling_phases() + or self._credits_complete_received + ) and self._check_all_records_received( + message.stats.phase, message.stats.phase_index ): - await self._handle_all_records_received(message.stats.phase) + await self._handle_all_records_received_once(message.stats.phase) def _snapshot_branch_stats(self, phase: CreditPhase) -> BranchStats | None: """Return the orchestrator-published BranchStats for ``phase``. @@ -807,13 +856,14 @@ async def _on_credits_complete(self, message: CreditsCompleteMessage) -> None: self.info( "All credits complete, please wait for the results to be processed..." ) + self._credits_complete_received = True if ( CreditPhase.PROFILING in self._complete_credit_phases and self._records_tracker.check_and_set_all_records_received_for_phase( CreditPhase.PROFILING ) ): - await self._handle_all_records_received(CreditPhase.PROFILING) + await self._handle_all_records_received_once(CreditPhase.PROFILING) @background_task( interval=Environment.RECORD.PROGRESS_REPORT_INTERVAL, immediate=False @@ -1187,7 +1237,7 @@ async def _summarize_metric_record_accumulators( timeslices: list[TimesliceResult] = [] error_results: list[ErrorDetails] = [] - phase_stats = self._records_tracker.create_stats_for_phase(phase) + phase_stats = RecordsManager._create_result_stats_for_phase(self, phase) summary_ctx = SummaryContext( accumulators=dict(self._accumulators), start_ns=phase_stats.start_ns or 0, @@ -1229,14 +1279,105 @@ async def _summarize_metric_record_accumulators( timeslices = ts return records_results, timeslices, error_results, summary_ctx + def _create_result_stats_for_phase(self, phase: CreditPhase) -> PhaseRecordsStats: + if phase == CreditPhase.PROFILING and self._has_multiple_profiling_phases(): + return self._records_tracker.create_aggregate_stats_for_phase(phase) + return self._records_tracker.create_stats_for_phase(phase) + + def _iter_concrete_phase_stats( + self, phase: CreditPhase | None = None + ) -> list[PhaseRecordsStats]: + phase_trackers = getattr(self._records_tracker, "_phase_trackers", {}) + if not isinstance(phase_trackers, dict): + return [] + stats = [ + tracker.create_stats() + for (tracker_phase, phase_index), tracker in phase_trackers.items() + if (phase is None or tracker_phase == phase) and phase_index is not None + ] + return sorted( + stats, + key=lambda item: item.phase_index + if item.phase_index is not None + else 10**9, + ) + + async def _build_phase_profile_results( + self, phase: CreditPhase, cancelled: bool + ) -> list[PhaseProfileResults] | None: + if phase != CreditPhase.PROFILING or not self._has_multiple_profiling_phases(): + return None + + concrete_phase_stats = self._iter_concrete_phase_stats( + phase=CreditPhase.PROFILING + ) + + phase_results: list[PhaseProfileResults] = [] + acc_items = [ + (acc_type, acc) + for acc_type, acc in self._accumulators.items() + if acc in self._metric_record_accumulators + ] + if not acc_items: + return None + + for stats in concrete_phase_stats: + if stats.phase_index is None: + continue + records_results: list[MetricResult] = [] + error_results: list[ErrorDetails] = [] + ctx = ExportContext( + start_ns=stats.start_ns, + end_ns=stats.requests_end_ns, + phase=stats.phase, + phase_index=stats.phase_index, + phase_name=stats.phase_name, + phase_kind=stats.phase_kind or stats.phase.value, + error_summary=self._error_tracker.get_error_summary_for_phase( + stats.phase, phase_index=stats.phase_index + ), + cancelled=cancelled, + ) + summaries = await asyncio.gather( + *[ + self._summarize_one_accumulator(acc_type, acc, ctx) + for acc_type, acc in acc_items + ], + return_exceptions=False, + ) + for acc_type, summary in summaries: + self._bucket_accumulator_summary( + acc_type, summary, records_results, error_results + ) + phase_results.append( + PhaseProfileResults( + phase_index=stats.phase_index, + profiling_index=stats.profiling_index, + phase_name=stats.phase_name + or f"{stats.phase.value}_{stats.phase_index}", + phase_kind=stats.phase_kind or stats.phase.value, + records=records_results, + start_ns=stats.start_ns, + end_ns=stats.requests_end_ns, + was_cancelled=stats.was_cancelled or cancelled, + successful_request_count=stats.success_records, + error_request_count=stats.error_records, + error_summary=self._error_tracker.get_error_summary_for_phase( + stats.phase, phase_index=stats.phase_index + ), + ) + ) + return phase_results or None + def _has_records_for_phase(self, phase: CreditPhase) -> bool: phase_trackers = getattr(self._records_tracker, "_phase_trackers", {}) if not isinstance(phase_trackers, dict): return False - tracker = phase_trackers.get(phase) - if tracker is None: - return False - return tracker.total_records > 0 + return any( + tracker.total_records > 0 + for (tracker_phase, _), tracker in phase_trackers.items() + if tracker_phase == phase + ) async def _summarize_warmup_metric_records(self) -> list[MetricResult] | None: """Return warmup-only inference metrics, or None when no warmup records exist.""" @@ -1395,7 +1536,10 @@ async def _process_results_impl( await self._finalize_stream_exporters() - phase_stats = self._records_tracker.create_stats_for_phase(phase) + phase_stats = RecordsManager._create_result_stats_for_phase(self, phase) + phase_records = await RecordsManager._build_phase_profile_results( + self, phase, cancelled + ) # Snapshot count BEFORE extending with derived aggregates (efficiency, # analyzers) — `completed` reports request-derived records only. records_completed = len(records_results) @@ -1419,6 +1563,7 @@ async def _process_results_impl( branch_stats=self._latest_branch_stats if phase == CreditPhase.PROFILING else None, + phase_records=phase_records, ), errors=error_results, ) @@ -1496,7 +1641,7 @@ async def _process_telemetry_results(self) -> ProcessTelemetryResult: # Note: end_ns is left None to include the final telemetry scrape that # occurs after PROFILE_COMPLETE but before export_results is called. # If start_ns is None (no profiling phase), include all data. - phase_stats = self._records_tracker.create_stats_for_phase( + phase_stats = self._records_tracker.create_aggregate_stats_for_phase( CreditPhase.PROFILING ) telemetry_export_data = await self._gpu_telemetry_accumulator.export_results( @@ -1585,7 +1730,7 @@ async def _process_server_metrics_results(self) -> ProcessServerMetricsResult: # Get timing from profiling phase stats (warmup is automatically excluded) # TimeFilter will be constructed per-endpoint in accumulator with per-endpoint end times - phase_stats = self._records_tracker.create_stats_for_phase( + phase_stats = self._records_tracker.create_aggregate_stats_for_phase( CreditPhase.PROFILING ) profiling_start_ns = phase_stats.start_ns or time.time_ns() diff --git a/src/aiperf/records/records_tracker.py b/src/aiperf/records/records_tracker.py index b6f79188d6..ef997525b9 100644 --- a/src/aiperf/records/records_tracker.py +++ b/src/aiperf/records/records_tracker.py @@ -38,6 +38,10 @@ def __init__(self, phase: CreditPhase, **kwargs) -> None: super().__init__(**kwargs) # Must be set by the caller self._phase: CreditPhase = phase + self._phase_index: int | None = None + self._profiling_index: int | None = None + self._phase_name: str | None = None + self._phase_kind: str | None = None self._total_expected_requests: int | None = None # Timestamp fields @@ -88,6 +92,10 @@ def create_stats(self) -> PhaseRecordsStats: """Create a new immutable RecordsPhaseStats object for the phase (for use in messages).""" return PhaseRecordsStats( phase=self._phase, + phase_index=self._phase_index, + profiling_index=self._profiling_index, + phase_name=self._phase_name, + phase_kind=self._phase_kind, start_ns=self._start_ns, sent_end_ns=self._sent_end_ns, requests_end_ns=self._requests_end_ns, @@ -105,6 +113,10 @@ def create_stats(self) -> PhaseRecordsStats: def update_from_credit_phase_stats(self, credit_stats: CreditPhaseStats) -> None: """Update the phase info.""" + self._phase_index = credit_stats.phase_index + self._profiling_index = credit_stats.profiling_index + self._phase_name = credit_stats.phase_name + self._phase_kind = credit_stats.phase_kind self._start_ns = credit_stats.start_ns self._sent_end_ns = credit_stats.sent_end_ns self._requests_end_ns = credit_stats.requests_end_ns @@ -162,13 +174,27 @@ class RecordsTracker: """ def __init__(self) -> None: - self._phase_trackers: dict[CreditPhase, CreditPhaseRecordsTracker] = {} - - def _get_phase_tracker(self, phase: CreditPhase) -> CreditPhaseRecordsTracker: - """Get the phase tracker.""" - if phase not in self._phase_trackers: - self._phase_trackers[phase] = CreditPhaseRecordsTracker(phase) - return self._phase_trackers[phase] + self._phase_trackers: dict[ + tuple[CreditPhase, int | None], CreditPhaseRecordsTracker + ] = {} + self._latest_phase_index: dict[CreditPhase, int | None] = {} + + def _get_phase_tracker( + self, phase: CreditPhase, phase_index: int | None = None + ) -> CreditPhaseRecordsTracker: + """Get the phase tracker for one concrete phase instance.""" + key = (phase, phase_index) + if key not in self._phase_trackers: + self._phase_trackers[key] = CreditPhaseRecordsTracker(phase) + self._latest_phase_index[phase] = phase_index + return self._phase_trackers[key] + + def _latest_tracker_for_phase( + self, phase: CreditPhase + ) -> CreditPhaseRecordsTracker | None: + if phase not in self._latest_phase_index: + return None + return self._get_phase_tracker(phase, self._latest_phase_index[phase]) def create_overall_worker_stats(self) -> dict[str, WorkerProcessingStats]: """Create a new dictionary of WorkerProcessingStats objects for ALL phases.""" @@ -181,14 +207,77 @@ def create_overall_worker_stats(self) -> dict[str, WorkerProcessingStats]: all_worker_stats[worker_id].error_records += worker_stats.error_records return dict(all_worker_stats) - def create_stats_for_phase(self, phase: CreditPhase) -> PhaseRecordsStats: - """Create a new immutable RecordsPhaseStats object for the phase (for use in messages).""" - phase_tracker = self._get_phase_tracker(phase) - return phase_tracker.create_stats() + def create_stats_for_phase( + self, phase: CreditPhase, phase_index: int | None = None + ) -> PhaseRecordsStats: + """Create stats for one concrete phase instance. + + When ``phase_index`` is omitted, this preserves the legacy behavior by + returning the latest tracker for that phase kind. + """ + if phase_index is None: + phase_tracker = self._latest_tracker_for_phase(phase) + if phase_tracker is None: + return CreditPhaseRecordsTracker(phase).create_stats() + return phase_tracker.create_stats() + return self._get_phase_tracker(phase, phase_index).create_stats() + + def create_aggregate_stats_for_phase(self, phase: CreditPhase) -> PhaseRecordsStats: + """Create stats spanning all concrete instances of ``phase``.""" + concrete_stats = [ + tracker.create_stats() + for (tracker_phase, phase_index), tracker in self._phase_trackers.items() + if tracker_phase == phase and phase_index is not None + ] + stats = concrete_stats or [ + tracker.create_stats() + for (tracker_phase, _), tracker in self._phase_trackers.items() + if tracker_phase == phase + ] + if not stats: + return self.create_stats_for_phase(phase) + + starts = [s.start_ns for s in stats if s.start_ns is not None] + sent_ends = [s.sent_end_ns for s in stats if s.sent_end_ns is not None] + request_ends = [ + s.requests_end_ns for s in stats if s.requests_end_ns is not None + ] + record_ends = [s.records_end_ns for s in stats if s.records_end_ns is not None] + + def optional_sum(values: list[int | None]) -> int | None: + concrete = [v for v in values if v is not None] + return sum(concrete) if concrete else None + + return PhaseRecordsStats( + phase=phase, + start_ns=min(starts) if starts else None, + sent_end_ns=max(sent_ends) if sent_ends else None, + requests_end_ns=max(request_ends) if request_ends else None, + records_end_ns=max(record_ends) if record_ends else None, + total_expected_requests=optional_sum( + [s.total_expected_requests for s in stats] + ), + success_records=sum(s.success_records for s in stats), + error_records=sum(s.error_records for s in stats), + final_requests_completed=optional_sum( + [s.final_requests_completed for s in stats] + ), + final_requests_cancelled=optional_sum( + [s.final_requests_cancelled for s in stats] + ), + final_request_errors=optional_sum([s.final_request_errors for s in stats]), + timeout_triggered=any(s.timeout_triggered for s in stats), + grace_period_timeout_triggered=any( + s.grace_period_timeout_triggered for s in stats + ), + was_cancelled=any(s.was_cancelled for s in stats), + ) def update_phase_info(self, credit_phase_stats: CreditPhaseStats) -> None: """Update the phase tracker.""" - phase_tracker = self._get_phase_tracker(credit_phase_stats.phase) + phase_tracker = self._get_phase_tracker( + credit_phase_stats.phase, credit_phase_stats.phase_index + ) phase_tracker.update_from_credit_phase_stats(credit_phase_stats) def update_from_request( @@ -199,7 +288,9 @@ def update_from_request( Drives the per-request lockstep off the request envelope: a request is counted as a success when ``error is None``, otherwise as an error. """ - phase_tracker = self._get_phase_tracker(metadata.benchmark_phase) + phase_tracker = self._get_phase_tracker( + metadata.benchmark_phase, metadata.phase_index + ) if error is None: phase_tracker.increment_success_records() phase_tracker.increment_worker_success_records(metadata.worker_id) @@ -209,8 +300,11 @@ def update_from_request( def was_phase_cancelled(self, phase: CreditPhase) -> bool: """Check if the phase was cancelled.""" - phase_tracker = self._get_phase_tracker(phase) - return phase_tracker._was_cancelled + return any( + tracker._was_cancelled + for (tracker_phase, _), tracker in self._phase_trackers.items() + if tracker_phase == phase + ) def mark_phase_cancelled(self, phase: CreditPhase) -> None: """Mark a phase as cancelled (e.g., from ProfileCancelCommand). @@ -219,13 +313,59 @@ def mark_phase_cancelled(self, phase: CreditPhase) -> None: the cancelled state is tracked even before the CreditPhaseCompleteMessage arrives with the updated stats. """ - phase_tracker = self._get_phase_tracker(phase) - phase_tracker._was_cancelled = True + for (tracker_phase, _), tracker in self._phase_trackers.items(): + if tracker_phase == phase: + tracker._was_cancelled = True - def check_and_set_all_records_received_for_phase(self, phase: CreditPhase) -> bool: + def check_and_set_all_records_received_for_phase( + self, phase: CreditPhase, phase_index: int | None = None + ) -> bool: """Check if all records have been received and set the flag if so.""" - phase_tracker = self._get_phase_tracker(phase) - return phase_tracker.check_and_set_all_records_received() + if phase_index is not None: + phase_tracker = self._get_phase_tracker(phase, phase_index) + return phase_tracker.check_and_set_all_records_received() + + concrete_phase_trackers = [ + tracker + for (tracker_phase, phase_index), tracker in self._phase_trackers.items() + if tracker_phase == phase and phase_index is not None + ] + phase_trackers = concrete_phase_trackers or [ + tracker + for (tracker_phase, _), tracker in self._phase_trackers.items() + if tracker_phase == phase + ] + if not phase_trackers: + return False + + all_counts_received = True + for tracker in phase_trackers: + stats = tracker.create_stats() + if ( + stats.final_requests_completed is None + or stats.total_records < stats.final_requests_completed + ): + all_counts_received = False + break + if not all_counts_received: + return False + + newly_completed = False + for tracker in phase_trackers: + if tracker._sent_all_records_received: + continue + newly_completed = ( + tracker.check_and_set_all_records_received() or newly_completed + ) + return newly_completed + + def check_and_set_all_records_received_for_stats( + self, stats: CreditPhaseStats | PhaseRecordsStats + ) -> bool: + """Check completion for the concrete phase represented by stats.""" + return self.check_and_set_all_records_received_for_phase( + stats.phase, stats.phase_index + ) def create_active_phase_stats_list(self) -> list[PhaseRecordsStats]: """Get the active phase stats.""" diff --git a/src/aiperf/timing/concurrency.py b/src/aiperf/timing/concurrency.py index 4adcc2b7da..0f57b51cfc 100644 --- a/src/aiperf/timing/concurrency.py +++ b/src/aiperf/timing/concurrency.py @@ -17,6 +17,8 @@ from aiperf.common.enums import CreditPhase +PhaseRuntimeKey = int | CreditPhase + @dataclass(slots=True) class ConcurrencyStats: @@ -209,14 +211,14 @@ def __init__(self) -> None: """Initialize as disabled with empty phase limits and a global limit of 0.""" self._enabled = False self._global_limit = DynamicConcurrencyLimit() - self._phase_limits: dict[CreditPhase, DynamicConcurrencyLimit] = {} + self._phase_limits: dict[PhaseRuntimeKey, DynamicConcurrencyLimit] = {} @property def enabled(self) -> bool: """Whether concurrency limiting is enabled for this limiter.""" return self._enabled - def configure_for_phase(self, phase: CreditPhase, limit: int | None) -> None: + def configure_for_phase(self, phase: PhaseRuntimeKey, limit: int | None) -> None: """Configure limits for a new phase. Args: @@ -234,7 +236,7 @@ def configure_for_phase(self, phase: CreditPhase, limit: int | None) -> None: self._global_limit.set_limit(limit) async def acquire( - self, phase: CreditPhase, can_proceed_fn: Callable[[], bool] + self, phase: PhaseRuntimeKey, can_proceed_fn: Callable[[], bool] ) -> bool: """Acquire a concurrency slot. @@ -283,7 +285,7 @@ async def acquire( raise def try_acquire( - self, phase: CreditPhase, can_proceed_fn: Callable[[], bool] + self, phase: PhaseRuntimeKey, can_proceed_fn: Callable[[], bool] ) -> bool: """Try to acquire a concurrency slot without blocking. @@ -322,7 +324,7 @@ def try_acquire( return True - def release(self, phase: CreditPhase) -> None: + def release(self, phase: PhaseRuntimeKey) -> None: """Release a concurrency slot. Args: @@ -337,7 +339,7 @@ def release(self, phase: CreditPhase) -> None: self._global_limit.release() self._phase_limits[phase].release() - def slot_available(self, phase: CreditPhase) -> bool: + def slot_available(self, phase: PhaseRuntimeKey) -> bool: """Check if a slot is available without blocking. Args: @@ -353,7 +355,7 @@ def slot_available(self, phase: CreditPhase) -> bool: not self._global_limit.locked() and not self._phase_limits[phase].locked() ) - def get_held_slots(self, phase: CreditPhase) -> int: + def get_held_slots(self, phase: PhaseRuntimeKey) -> int: """Get the number of slots currently held for a specific phase. Args: @@ -374,7 +376,7 @@ def global_stats(self) -> ConcurrencyStats: """Global concurrency stats across all phases.""" return self._global_limit.stats - def get_phase_stats(self, phase: CreditPhase) -> ConcurrencyStats | None: + def get_phase_stats(self, phase: PhaseRuntimeKey) -> ConcurrencyStats | None: """Get stats for a specific phase. Args: @@ -411,7 +413,7 @@ def __init__(self) -> None: def configure_for_phase( self, - phase: CreditPhase, + phase: PhaseRuntimeKey, concurrency: int | None, prefill_concurrency: int | None, ) -> None: @@ -431,7 +433,7 @@ def configure_for_phase( self._session_limiter.configure_for_phase(phase, concurrency) self._prefill_limiter.configure_for_phase(phase, prefill_concurrency) - def session_slot_available(self, phase: CreditPhase) -> bool: + def session_slot_available(self, phase: PhaseRuntimeKey) -> bool: """Check if a session slot is available without blocking. Returns True when concurrency limiting is disabled (no limit configured). @@ -441,7 +443,7 @@ def session_slot_available(self, phase: CreditPhase) -> bool: return self._session_limiter.slot_available(phase) async def acquire_session_slot( - self, phase: CreditPhase, can_proceed_fn: Callable[[], bool] + self, phase: PhaseRuntimeKey, can_proceed_fn: Callable[[], bool] ) -> bool: """Acquire a session concurrency slot. @@ -461,7 +463,7 @@ async def acquire_session_slot( return await self._session_limiter.acquire(phase, can_proceed_fn) def try_acquire_session_slot( - self, phase: CreditPhase, can_proceed_fn: Callable[[], bool] + self, phase: PhaseRuntimeKey, can_proceed_fn: Callable[[], bool] ) -> bool: """Try to acquire a session concurrency slot without blocking. @@ -481,7 +483,7 @@ def try_acquire_session_slot( return can_proceed_fn() return self._session_limiter.try_acquire(phase, can_proceed_fn) - def release_session_slot(self, phase: CreditPhase) -> None: + def release_session_slot(self, phase: PhaseRuntimeKey) -> None: """Release a session concurrency slot. Called when a session ends (final turn completed or cancelled) or during @@ -494,7 +496,7 @@ def release_session_slot(self, phase: CreditPhase) -> None: self._session_limiter.release(phase) async def acquire_prefill_slot( - self, phase: CreditPhase, can_proceed_fn: Callable[[], bool] + self, phase: PhaseRuntimeKey, can_proceed_fn: Callable[[], bool] ) -> bool: """Acquire a prefill concurrency slot. @@ -518,7 +520,7 @@ async def acquire_prefill_slot( return await self._prefill_limiter.acquire(phase, can_proceed_fn) def try_acquire_prefill_slot( - self, phase: CreditPhase, can_proceed_fn: Callable[[], bool] + self, phase: PhaseRuntimeKey, can_proceed_fn: Callable[[], bool] ) -> bool: """Try to acquire a prefill concurrency slot without blocking. @@ -538,7 +540,7 @@ def try_acquire_prefill_slot( return can_proceed_fn() return self._prefill_limiter.try_acquire(phase, can_proceed_fn) - def release_prefill_slot(self, phase: CreditPhase) -> None: + def release_prefill_slot(self, phase: PhaseRuntimeKey) -> None: """Release a prefill concurrency slot. Called when TTFT is received (normal path) or when a credit is returned @@ -550,7 +552,7 @@ def release_prefill_slot(self, phase: CreditPhase) -> None: if self._prefill_limiter.enabled: self._prefill_limiter.release(phase) - def release_stuck_slots(self, phase: CreditPhase) -> tuple[int, int]: + def release_stuck_slots(self, phase: PhaseRuntimeKey) -> tuple[int, int]: """Release all stuck slots for a phase during force-completion. Called when cancel drain timeout expires and credits will never return. @@ -577,7 +579,7 @@ def release_stuck_slots(self, phase: CreditPhase) -> tuple[int, int]: return session_released, prefill_released def get_session_stats( - self, phase: CreditPhase | None = None + self, phase: PhaseRuntimeKey | None = None ) -> ConcurrencyStats | None: """Get session concurrency stats. @@ -593,7 +595,7 @@ def get_session_stats( return self._session_limiter.get_phase_stats(phase) return self._session_limiter.global_stats - def set_session_limit(self, phase: CreditPhase, limit: int) -> None: + def set_session_limit(self, phase: PhaseRuntimeKey, limit: int) -> None: """Set session concurrency limit for both global and phase. Args: @@ -606,7 +608,7 @@ def set_session_limit(self, phase: CreditPhase, limit: int) -> None: if phase in self._session_limiter._phase_limits: self._session_limiter._phase_limits[phase].set_limit(limit) - def set_prefill_limit(self, phase: CreditPhase, limit: int) -> None: + def set_prefill_limit(self, phase: PhaseRuntimeKey, limit: int) -> None: """Set prefill concurrency limit for both global and phase. Args: diff --git a/src/aiperf/timing/config.py b/src/aiperf/timing/config.py index 86d7514c45..51b750a14f 100644 --- a/src/aiperf/timing/config.py +++ b/src/aiperf/timing/config.py @@ -10,6 +10,7 @@ from aiperf.common.enums import CreditPhase from aiperf.common.models.base_models import AIPerfBaseModel +from aiperf.common.types import PhaseKind from aiperf.config.dataset.defaults import InputDefaults from aiperf.config.sweep.adaptive import SLAFilter from aiperf.plugin.enums import ( @@ -87,34 +88,44 @@ class TimingConfig(AIPerfBaseModel): def from_run(cls, run: BenchmarkRun) -> TimingConfig: """Build ordered list of credit-phase configs from a ``BenchmarkRun``. - Iterates ``run.cfg.get_warmup_phases()`` first (each becomes a WARMUP - CreditPhaseConfig) followed by ``run.cfg.get_profiling_phases()`` - (each becomes a PROFILING CreditPhaseConfig). The cancellation policy - is sourced from the first profiling phase that declares one; URLs and - url-selection strategy come from the endpoint section. + Preserves the ordered ``cfg.phases`` list. Each executable phase gets + stable identity metadata: absolute ``phase_index``, profiling-only + ``profiling_index``, ``phase_name``, and ``phase_kind``. A phase-local + cancellation config is attached to each timing phase; omitted per-phase + cancellation inherits the run default. """ cfg = run.cfg artifact_dir = cfg.artifacts.dir + profiling_default_cancellation = _default_cancellation_config(cfg.phases) + warmup_default_cancellation = RequestCancellationConfig() + configs: list[CreditPhaseConfig] = [] - for phase in cfg.get_warmup_phases(): - configs.append(_build_warmup_config(phase, artifact_dir=artifact_dir)) - for phase in cfg.get_profiling_phases(): - configs.append(_build_profiling_config(phase, artifact_dir=artifact_dir)) - - cancellation_config: RequestCancellationConfig = RequestCancellationConfig() - for phase in cfg.get_profiling_phases(): - if getattr(phase, "cancellation", None) is not None: - cancellation_config = RequestCancellationConfig( - rate=phase.cancellation.rate, - delay=phase.cancellation.delay, + profiling_index = 0 + for phase_index, phase in enumerate(cfg.phases): + current_profiling_index = None + if phase.kind == "profiling": + current_profiling_index = profiling_index + profiling_index += 1 + default_cancellation = ( + profiling_default_cancellation + if phase.kind == "profiling" + else warmup_default_cancellation + ) + configs.append( + _build_phase_config( + phase, + artifact_dir=artifact_dir, + default_cancellation=default_cancellation, + phase_index=phase_index, + profiling_index=current_profiling_index, ) - break + ) return cls( phase_configs=configs, - request_cancellation=cancellation_config, + request_cancellation=profiling_default_cancellation, urls=list(cfg.endpoint.urls), url_selection_strategy=cfg.endpoint.url_strategy, ) @@ -132,6 +143,24 @@ class CreditPhaseConfig(AIPerfBaseModel): model_config = ConfigDict(frozen=True) phase: CreditPhase = Field(..., description="The phase of the credit phase.") + phase_index: int | None = Field( + default=None, ge=0, description="Absolute index in the ordered phases list." + ) + profiling_index: int | None = Field( + default=None, + ge=0, + description="Index among profiling-kind phases; None for warmup.", + ) + phase_name: str | None = Field( + default=None, description="User-provided unique phase name." + ) + phase_kind: PhaseKind | None = Field( + default=None, description="Phase semantic kind: warmup or profiling." + ) + request_cancellation: RequestCancellationConfig = Field( + default_factory=RequestCancellationConfig, + description="Phase-local request cancellation policy.", + ) timing_mode: TimingMode = Field( ..., description="The timing mode of the credit phase. Used to determine " @@ -344,8 +373,61 @@ def _phase_arrival_pattern(phase: PhaseConfig) -> ArrivalPattern: return _PHASE_TYPE_TO_ARRIVAL_PATTERN.get(phase.type, ArrivalPattern.POISSON) +def _default_cancellation_config( + phases: list[PhaseConfig], +) -> RequestCancellationConfig: + for phase in phases: + if getattr(phase, "kind", None) != "profiling": + continue + cancellation = getattr(phase, "cancellation", None) + if cancellation is not None: + return RequestCancellationConfig( + rate=cancellation.rate, delay=cancellation.delay + ) + return RequestCancellationConfig() + + +def _phase_cancellation_config( + phase: PhaseConfig, default_cancellation: RequestCancellationConfig +) -> RequestCancellationConfig: + cancellation = getattr(phase, "cancellation", None) + if cancellation is None: + return default_cancellation + return RequestCancellationConfig(rate=cancellation.rate, delay=cancellation.delay) + + +def _build_phase_config( + phase: PhaseConfig, + *, + artifact_dir: Path | None = None, + default_cancellation: RequestCancellationConfig, + phase_index: int, + profiling_index: int | None, +) -> CreditPhaseConfig: + if phase.kind == "warmup": + return _build_warmup_config( + phase, + artifact_dir=artifact_dir, + default_cancellation=default_cancellation, + phase_index=phase_index, + profiling_index=profiling_index, + ) + return _build_profiling_config( + phase, + artifact_dir=artifact_dir, + default_cancellation=default_cancellation, + phase_index=phase_index, + profiling_index=profiling_index, + ) + + def _build_warmup_config( - phase: PhaseConfig, *, artifact_dir: Path | None = None + phase: PhaseConfig, + *, + artifact_dir: Path | None = None, + default_cancellation: RequestCancellationConfig, + phase_index: int, + profiling_index: int | None, ) -> CreditPhaseConfig: """Build a warmup CreditPhaseConfig from a warmup PhaseConfig. @@ -363,6 +445,11 @@ def _build_warmup_config( return CreditPhaseConfig( phase=CreditPhase.WARMUP, + phase_index=phase_index, + profiling_index=profiling_index, + phase_name=phase.name, + phase_kind=phase.kind, + request_cancellation=_phase_cancellation_config(phase, default_cancellation), # Warmup phase is always request rate timing mode timing_mode=TimingMode.REQUEST_RATE, total_expected_requests=phase.requests, @@ -385,7 +472,12 @@ def _build_warmup_config( def _build_profiling_config( - phase: PhaseConfig, *, artifact_dir: Path | None = None + phase: PhaseConfig, + *, + artifact_dir: Path | None = None, + default_cancellation: RequestCancellationConfig, + phase_index: int, + profiling_index: int | None, ) -> CreditPhaseConfig: """Build a profiling CreditPhaseConfig from a profiling PhaseConfig. @@ -395,6 +487,11 @@ def _build_profiling_config( """ return CreditPhaseConfig( phase=CreditPhase.PROFILING, + phase_index=phase_index, + profiling_index=profiling_index, + phase_name=phase.name, + phase_kind=phase.kind, + request_cancellation=_phase_cancellation_config(phase, default_cancellation), timing_mode=_phase_timing_mode(phase), expected_duration_sec=phase.duration, total_expected_requests=phase.requests, diff --git a/src/aiperf/timing/phase/progress_tracker.py b/src/aiperf/timing/phase/progress_tracker.py index 06b38eaf5f..a86afdb0f4 100644 --- a/src/aiperf/timing/phase/progress_tracker.py +++ b/src/aiperf/timing/phase/progress_tracker.py @@ -187,6 +187,10 @@ def create_stats(self, lifecycle: PhaseLifecycle) -> CreditPhaseStats: """ return CreditPhaseStats( phase=self._config.phase, + phase_index=self._config.phase_index, + profiling_index=self._config.profiling_index, + phase_name=self._config.phase_name, + phase_kind=self._config.phase_kind, # Timestamps from lifecycle start_ns=lifecycle.started_at_ns, sent_end_ns=lifecycle.sending_complete_at_ns, diff --git a/src/aiperf/timing/phase/runner.py b/src/aiperf/timing/phase/runner.py index 19dd63746f..f34c310d28 100644 --- a/src/aiperf/timing/phase/runner.py +++ b/src/aiperf/timing/phase/runner.py @@ -16,6 +16,7 @@ from aiperf.common.environment import Environment from aiperf.common.loop_scheduler import LoopScheduler from aiperf.common.mixins import TaskManagerMixin +from aiperf.common.phase import phase_runtime_key from aiperf.credit.issuer import CreditIssuer from aiperf.plugin import plugins from aiperf.plugin.enums import PluginType, TimingMode @@ -24,6 +25,7 @@ from aiperf.timing.phase.progress_tracker import PhaseProgressTracker from aiperf.timing.phase.stop_conditions import StopConditionChecker from aiperf.timing.ramping import Ramper, RamperConfig, RampType +from aiperf.timing.request_cancellation import RequestCancellationSimulator from aiperf.timing.strategies.core import RateSettableProtocol from aiperf.timing.url_samplers import URLSelectionStrategyProtocol @@ -77,7 +79,7 @@ def __init__( phase_publisher: PhasePublisher, credit_router: CreditRouterProtocol, concurrency_manager: ConcurrencyManager, - cancellation_policy: RequestCancellationSimulator, + cancellation_policy: RequestCancellationSimulator | None = None, callback_handler: CreditCallbackHandler, url_selection_strategy: URLSelectionStrategyProtocol | None = None, branch_orchestrator: BranchOrchestrator | None = None, @@ -119,7 +121,9 @@ def __init__( self._phase_publisher = phase_publisher self._credit_router = credit_router self._concurrency_manager = concurrency_manager - self._cancellation_policy = cancellation_policy + self._cancellation_policy = cancellation_policy or RequestCancellationSimulator( + self._config.request_cancellation + ) self._callback_handler = callback_handler self._on_phase_complete: Callable[[], None] | None = None @@ -149,6 +153,10 @@ def _build_credit_issuer( ergonomics file-size cap.""" return CreditIssuer( phase=self._config.phase, + phase_index=self._config.phase_index, + profiling_index=self._config.profiling_index, + phase_name=self._config.phase_name, + phase_kind=self._config.phase_kind, stop_checker=self._stop_checker, progress=self._progress, concurrency_manager=self._concurrency_manager, @@ -179,6 +187,10 @@ def _maybe_construct_branch_orchestrator( sticky_router=sticky_router, ) + @property + def _phase_key(self) -> int | CreditPhase: + return phase_runtime_key(self._config.phase, self._config.phase_index) + @property def phase(self) -> CreditPhase: """Phase enum (WARMUP or PROFILING).""" @@ -329,6 +341,7 @@ def _register_strategy_with_callback_handler( """ self._callback_handler.register_phase( phase=self._config.phase, + phase_index=self._config.phase_index, progress=self._progress, lifecycle=self._lifecycle, stop_checker=self._stop_checker, @@ -357,7 +370,7 @@ async def _run_strategy( lifecycle state) lives in the caller's ``except``. """ self._concurrency_manager.configure_for_phase( - self._config.phase, + self._phase_key, self._config.concurrency, self._config.prefill_concurrency, ) @@ -479,7 +492,7 @@ def _create_rampers(self, strategy: TimingStrategyProtocol) -> None: def setter(limit: float) -> None: return self._concurrency_manager.set_session_limit( - config.phase, int(limit) + self._phase_key, int(limit) ) self._rampers.append(Ramper(setter=setter, config=ramp_config)) @@ -499,7 +512,7 @@ def setter(limit: float) -> None: def setter(limit: float) -> None: return self._concurrency_manager.set_prefill_limit( - config.phase, int(limit) + self._phase_key, int(limit) ) self._rampers.append(Ramper(setter=setter, config=ramp_config)) @@ -535,7 +548,13 @@ def setter(limit: float) -> None: def _format_phase_started(self, stats: CreditPhaseStats) -> str: """Format a concise log message for phase start.""" - parts = [f"Phase {stats.phase} started"] + parts = [ + f"Phase {stats.phase_name or stats.phase} ({stats.phase_kind or stats.phase}) started" + ] + if stats.phase_index is not None: + parts.append(f"phase_index={stats.phase_index}") + if stats.profiling_index is not None: + parts.append(f"profiling_index={stats.profiling_index}") targets = [] if stats.total_expected_requests: targets.append(f"{stats.total_expected_requests:,} requests") @@ -549,7 +568,9 @@ def _format_phase_started(self, stats: CreditPhaseStats) -> str: def _format_phase_sending_complete(self, stats: CreditPhaseStats) -> str: """Format a concise log message for phase sending complete.""" - parts = [f"Phase {stats.phase} sending complete"] + parts = [ + f"Phase {stats.phase_name or stats.phase} ({stats.phase_kind or stats.phase}) sending complete" + ] parts.append( f"sent={stats.requests_sent:,}, " f"completed={stats.requests_completed:,}, " @@ -566,7 +587,9 @@ def _format_phase_sending_complete(self, stats: CreditPhaseStats) -> str: def _format_phase_complete(self, stats: CreditPhaseStats) -> str: """Format a concise log message for phase complete.""" - parts = [f"Phase {stats.phase} complete"] + parts = [ + f"Phase {stats.phase_name or stats.phase} ({stats.phase_kind or stats.phase}) complete" + ] parts.append( f"completed={stats.final_requests_completed:,}, " f"cancelled={stats.final_requests_cancelled:,}, " @@ -701,7 +724,7 @@ async def _wait_for_returning_complete(self) -> None: def _release_stuck_slots(self) -> None: """Release concurrency slots for credits that will never return.""" session_released, prefill_released = ( - self._concurrency_manager.release_stuck_slots(self._config.phase) + self._concurrency_manager.release_stuck_slots(self._phase_key) ) if session_released or prefill_released: self.warning( diff --git a/src/aiperf/timing/phase_orchestrator.py b/src/aiperf/timing/phase_orchestrator.py index d829747c52..3474403416 100644 --- a/src/aiperf/timing/phase_orchestrator.py +++ b/src/aiperf/timing/phase_orchestrator.py @@ -129,9 +129,6 @@ def __init__( self._dataset_metadata, self._dataset_sampler ) self._concurrency_manager = ConcurrencyManager() - self._cancellation_policy = RequestCancellationSimulator( - config.request_cancellation - ) # URL sampler for multi-URL load balancing (None if single URL) self._url_sampler: URLSelectionStrategyProtocol | None = None @@ -205,7 +202,9 @@ async def _execute_phases(self) -> None: phase_publisher=self._phase_publisher, credit_router=self._credit_router, concurrency_manager=self._concurrency_manager, - cancellation_policy=self._cancellation_policy, + cancellation_policy=RequestCancellationSimulator( + phase_config.request_cancellation + ), callback_handler=self._callback_handler, url_selection_strategy=self._url_sampler, ) diff --git a/src/aiperf/timing/strategies/adaptive_scale.py b/src/aiperf/timing/strategies/adaptive_scale.py index 0210b7f74e..afe7bbc017 100644 --- a/src/aiperf/timing/strategies/adaptive_scale.py +++ b/src/aiperf/timing/strategies/adaptive_scale.py @@ -205,10 +205,29 @@ def _sla_margin(sla: SLAFilter, observed: float | None) -> float | None: return (observed - sla.threshold) / threshold def _resolve_artifact_path(self, filename: str) -> Path | None: - return self._artifacts.resolve_path(self._config.artifact_dir, filename) + return self._artifacts.phase_scoped_path( + self._config.artifact_dir, self._config.phase_name, filename + ) + + def _write_adaptive_manifest_entry(self) -> None: + if self._config.artifact_dir is None or self._config.phase_name is None: + return + phase_root = f"phases/{self._config.phase_name}" + self._artifacts.write_manifest_entry( + self._config.artifact_dir, + { + "phase_index": self._config.phase_index, + "profiling_index": self._config.profiling_index, + "phase_name": self._config.phase_name, + "phase_kind": self._config.phase_kind, + "events_path": f"{phase_root}/{self.EVENT_FILE}", + "summary_path": f"{phase_root}/{self.SUMMARY_FILE}", + }, + ) async def setup_phase(self) -> None: await self._artifacts.start() + self._write_adaptive_manifest_entry() setup_complete = False try: self._set_control(self._control.minimum) diff --git a/src/aiperf/timing/strategies/adaptive_scale_artifacts.py b/src/aiperf/timing/strategies/adaptive_scale_artifacts.py index 14cdf2416a..98358205e0 100644 --- a/src/aiperf/timing/strategies/adaptive_scale_artifacts.py +++ b/src/aiperf/timing/strategies/adaptive_scale_artifacts.py @@ -86,6 +86,14 @@ def resolve_path(artifact_dir: Path | None, filename: str) -> Path | None: return None return artifact_dir / filename + @staticmethod + def phase_scoped_path( + artifact_dir: Path | None, phase_name: str | None, filename: str + ) -> Path | None: + if artifact_dir is None or phase_name is None: + return None + return artifact_dir / "phases" / phase_name / filename + @staticmethod def correlation_payload( *, @@ -98,11 +106,17 @@ def correlation_payload( phase_start_ts: str | None = None, phase_end_ts: str | None = None, fault_window_id: str | None = None, + phase_index: int | None = None, + profiling_index: int | None = None, + phase_kind: str | None = None, ) -> dict[str, Any]: return { "run_id": run_id, "phase_id": phase_id, + "phase_index": phase_index, + "profiling_index": profiling_index, "phase_name": phase_name, + "phase_kind": phase_kind, "phase_start_ts": phase_start_ts, "phase_end_ts": phase_end_ts, "adaptive_iteration": adaptive_iteration, @@ -335,3 +349,37 @@ def write() -> None: path.write_bytes(encoded + b"\n") self._schedule_write(write) + + def write_manifest_entry( + self, artifact_dir: Path | None, entry: dict[str, Any] + ) -> None: + if artifact_dir is None: + return + manifest_path = artifact_dir / "adaptive_scale_manifest.json" + + def write() -> None: + manifest_path.parent.mkdir(parents=True, exist_ok=True) + if manifest_path.exists(): + payload = orjson.loads(manifest_path.read_bytes()) + else: + payload = {"schema_version": SCHEMA_VERSION, "adaptive_phases": []} + phases = payload.setdefault("adaptive_phases", []) + phases = [ + phase + for phase in phases + if phase.get("phase_name") != entry.get("phase_name") + or phase.get("phase_index") != entry.get("phase_index") + ] + phases.append(entry) + phases.sort( + key=lambda item: item.get("phase_index") + if item.get("phase_index") is not None + else 10**9 + ) + payload["adaptive_phases"] = phases + encoded = orjson.dumps( + payload, option=orjson.OPT_INDENT_2 | orjson.OPT_SORT_KEYS + ) + manifest_path.write_bytes(encoded + b"\n") + + self._schedule_write(write) diff --git a/src/aiperf/timing/strategies/adaptive_scale_backends.py b/src/aiperf/timing/strategies/adaptive_scale_backends.py index 303ebfd1bf..24e72615c6 100644 --- a/src/aiperf/timing/strategies/adaptive_scale_backends.py +++ b/src/aiperf/timing/strategies/adaptive_scale_backends.py @@ -7,9 +7,9 @@ from dataclasses import dataclass from typing import Any, Protocol -from aiperf.common.enums import CreditPhase from aiperf.plugin.enums import ArrivalPattern from aiperf.timing.adaptive_config import AdaptiveControlVariable +from aiperf.timing.concurrency import PhaseRuntimeKey class AdaptiveControlBackend(Protocol): @@ -50,7 +50,7 @@ def snapshot(self) -> dict[str, Any]: class SessionConcurrencyControlBackend(_BaseControlBackend): def __init__( - self, *, concurrency_manager, phase: CreditPhase, minimum: int, maximum: int + self, *, concurrency_manager, phase: PhaseRuntimeKey, minimum: int, maximum: int ): super().__init__("concurrency", minimum, maximum, minimum) self._concurrency_manager = concurrency_manager @@ -63,7 +63,7 @@ def set(self, value: float) -> None: class PrefillConcurrencyControlBackend(_BaseControlBackend): def __init__( - self, *, concurrency_manager, phase: CreditPhase, minimum: int, maximum: int + self, *, concurrency_manager, phase: PhaseRuntimeKey, minimum: int, maximum: int ): super().__init__("prefill_concurrency", minimum, maximum, minimum) self._concurrency_manager = concurrency_manager @@ -154,7 +154,7 @@ def _build_concurrency_backend(*, strategy: Any, concurrency_manager, config): _validate_bounds(min_value, max_value, "concurrency") return SessionConcurrencyControlBackend( concurrency_manager=concurrency_manager, - phase=config.phase, + phase=config.phase_index if config.phase_index is not None else config.phase, minimum=min_value, maximum=max_value, ) @@ -185,7 +185,7 @@ def _build_prefill_concurrency_backend(*, strategy: Any, concurrency_manager, co _validate_bounds(min_value, max_value, "prefill_concurrency") return PrefillConcurrencyControlBackend( concurrency_manager=concurrency_manager, - phase=config.phase, + phase=config.phase_index if config.phase_index is not None else config.phase, minimum=min_value, maximum=max_value, ) diff --git a/src/aiperf/timing/strategies/adaptive_scale_runtime.py b/src/aiperf/timing/strategies/adaptive_scale_runtime.py index 19f542f904..e4b0f00a1c 100644 --- a/src/aiperf/timing/strategies/adaptive_scale_runtime.py +++ b/src/aiperf/timing/strategies/adaptive_scale_runtime.py @@ -32,7 +32,7 @@ def _emit_event( step_size: float | None = None, sla_values: dict[str, float] | None = None, ) -> None: - phase_name = getattr(self._config, "name", None) + phase_name = getattr(self._config, "phase_name", None) phase_id = phase_name or CreditPhase.PROFILING run = getattr(self, "run", None) run_id = getattr(run, "benchmark_id", None) @@ -66,6 +66,9 @@ def _emit_event( run_id=run_id, phase_id=phase_id, phase_name=phase_name, + phase_index=getattr(self._config, "phase_index", None), + profiling_index=getattr(self._config, "profiling_index", None), + phase_kind=getattr(self._config, "phase_kind", None), adaptive_iteration=self._adaptive_iteration, candidate_value=(self._control.current if before is None else before), accepted_value=self._control.current, @@ -176,4 +179,12 @@ def _write_summary( max_step_multiplier=self._config.adaptive_scale_max_step_multiplier, step_percent=self._config.adaptive_scale_step_percent, ) + summary.update( + { + "phase_index": getattr(self._config, "phase_index", None), + "profiling_index": getattr(self._config, "profiling_index", None), + "phase_name": getattr(self._config, "phase_name", None), + "phase_kind": getattr(self._config, "phase_kind", None), + } + ) self._artifacts.write_summary(self._summary_path, summary) diff --git a/src/aiperf/workers/worker.py b/src/aiperf/workers/worker.py index 7ebc82ef92..1cd523adbd 100644 --- a/src/aiperf/workers/worker.py +++ b/src/aiperf/workers/worker.py @@ -480,6 +480,7 @@ async def first_token_callback(ttft_ns: int, message: SSEMessage) -> bool: FirstToken( credit_id=credit.id, phase=credit.phase, + phase_index=credit.phase_index, ttft_ns=ttft_ns, ) ) @@ -731,6 +732,10 @@ def _create_request_info( model_endpoint=self.model_endpoint, credit_num=credit.id, credit_phase=credit.phase, + phase_index=credit.phase_index, + profiling_index=credit.profiling_index, + phase_name=credit.phase_name, + phase_kind=credit.phase_kind, cancel_after_ns=credit.cancel_after_ns, x_request_id=x_request_id, x_correlation_id=session.x_correlation_id, @@ -817,6 +822,10 @@ async def _request_conversation_from_dataset_manager( turns=[], credit_num=credit_context.credit.id, credit_phase=credit_context.credit.phase, + phase_index=credit_context.credit.phase_index, + profiling_index=credit_context.credit.profiling_index, + phase_name=credit_context.credit.phase_name, + phase_kind=credit_context.credit.phase_kind, x_request_id=str(uuid.uuid4()), x_correlation_id=credit_context.credit.x_correlation_id, drop_perf_ns=credit_context.drop_perf_ns, diff --git a/tests/component_integration/timing/test_adaptive_scale.py b/tests/component_integration/timing/test_adaptive_scale.py index cb1b37301c..f2c84ff2e2 100644 --- a/tests/component_integration/timing/test_adaptive_scale.py +++ b/tests/component_integration/timing/test_adaptive_scale.py @@ -25,20 +25,54 @@ def _load_jsonl(path: Path) -> list[dict]: @pytest.mark.component_integration -def test_adaptive_scale_profile_writes_controller_artifacts(cli: AIPerfCLI) -> None: - """Exercise CLI -> config -> TimingConfig -> AdaptiveScaleStrategy wiring.""" +def test_adaptive_scale_profile_writes_controller_artifacts( + cli: AIPerfCLI, tmp_path: Path +) -> None: + """Exercise YAML config -> TimingConfig -> AdaptiveScaleStrategy wiring.""" + config_path = tmp_path / "adaptive_scale_smoke.yaml" + config_path.write_text( + f""" +schemaVersion: "2.0" + +benchmark: + model: {defaults.model} + endpoint: + url: http://localhost:8000 + type: chat + streaming: true + dataset: + type: synthetic + entries: 1000 + prompts: + isl: 550 + osl: 8 + phases: + - name: profiling + kind: profiling + type: concurrency + concurrency: 4 + duration: 2.5 + adaptive_scale: + enabled: true + control: + variable: concurrency + min: 1 + max: 4 + assessment_period: 1.0 + min_completed_requests: 1 + sustain_duration: 1.0 + sla: + request_latency: + p95: + le: 10000 +""".lstrip(), + encoding="utf-8", + ) + result = cli.run_sync( f""" aiperf profile \ - --model {defaults.model} \ - --streaming \ - --concurrency 4 \ - --benchmark-duration 2.5 \ - --adaptive-scale \ - --adaptive-sustain-duration 1.0 \ - --adaptive-assessment-period 1.0 \ - --adaptive-scale-sla request_latency:p95:le:10000 \ - --osl 8 \ + --config "{config_path}" \ --extra-inputs ignore_eos:true \ --ui {defaults.ui} """, @@ -49,8 +83,9 @@ def test_adaptive_scale_profile_writes_controller_artifacts(cli: AIPerfCLI) -> N assert result.request_count > 0 assert result.json.was_cancelled is False - event_path = result.artifacts_dir / "adaptive_scale_events.jsonl" - summary_path = result.artifacts_dir / "adaptive_scale_summary.json" + phase_dir = result.artifacts_dir / "phases" / "profiling" + event_path = phase_dir / "adaptive_scale_events.jsonl" + summary_path = phase_dir / "adaptive_scale_summary.json" assert event_path.exists() assert summary_path.exists() @@ -162,7 +197,9 @@ def deterministic_passes_sla( assert result.request_count > 0 assert result.json.was_cancelled is False - events = _load_jsonl(result.artifacts_dir / "adaptive_scale_events.jsonl") + events = _load_jsonl( + result.artifacts_dir / "phases" / "profiling" / "adaptive_scale_events.jsonl" + ) boundary_events = [ event for event in events if event["event"] == "boundary_discovered" ] @@ -195,7 +232,12 @@ def deterministic_passes_sla( assert boundary["sla_value"] > boundary["sla_bound"] summary = orjson.loads( - (result.artifacts_dir / "adaptive_scale_summary.json").read_bytes() + ( + result.artifacts_dir + / "phases" + / "profiling" + / "adaptive_scale_summary.json" + ).read_bytes() ) assert summary["boundary_value"] == boundary["boundary_value"] assert summary["first_failing_value"] == boundary["first_failing_value"] diff --git a/tests/integration/test_adaptive_scale.py b/tests/integration/test_adaptive_scale.py index 9b6705946a..21e0b30e76 100644 --- a/tests/integration/test_adaptive_scale.py +++ b/tests/integration/test_adaptive_scale.py @@ -4,6 +4,7 @@ from __future__ import annotations +import asyncio import re from collections.abc import Callable from pathlib import Path @@ -109,12 +110,35 @@ async def test_adaptive_scale_subprocess_contract_with_deterministic_saturation( assert result.json is not None assert result.json.was_cancelled is False - event_path = result.artifacts_dir / "adaptive_scale_events.jsonl" - summary_path = result.artifacts_dir / "adaptive_scale_summary.json" - assert event_path.exists() - assert summary_path.exists() + manifest_path = result.artifacts_dir / "adaptive_scale_manifest.json" + assert await asyncio.to_thread(manifest_path.exists) + manifest_bytes = await asyncio.to_thread(manifest_path.read_bytes) + manifest = orjson.loads(manifest_bytes) + assert manifest["schema_version"] == 2 + [adaptive_phase] = manifest["adaptive_phases"] + assert adaptive_phase["phase_name"] == "profiling" + assert adaptive_phase["phase_kind"] == "profiling" + assert adaptive_phase["phase_index"] == 0 + assert adaptive_phase["profiling_index"] == 0 + + assert adaptive_phase["events_path"] == ( + "phases/profiling/adaptive_scale_events.jsonl" + ) + assert adaptive_phase["summary_path"] == ( + "phases/profiling/adaptive_scale_summary.json" + ) + event_path = result.artifacts_dir / adaptive_phase["events_path"] + summary_path = result.artifacts_dir / adaptive_phase["summary_path"] + assert await asyncio.to_thread(event_path.exists) + assert await asyncio.to_thread(summary_path.exists) + assert not await asyncio.to_thread( + (result.artifacts_dir / "adaptive_scale_events.jsonl").exists + ) + assert not await asyncio.to_thread( + (result.artifacts_dir / "adaptive_scale_summary.json").exists + ) - events = _load_jsonl(event_path) + events = await asyncio.to_thread(_load_jsonl, event_path) assert events event_names = {event["event"] for event in events} assert "adaptive_phase_started" in event_names diff --git a/tests/unit/cli_commands/test_config.py b/tests/unit/cli_commands/test_config.py index f65b79cf04..6de1ca1a66 100644 --- a/tests/unit/cli_commands/test_config.py +++ b/tests/unit/cli_commands/test_config.py @@ -60,6 +60,31 @@ def _write_adaptive_sweep(tmp_path: Path) -> Path: return path +def _write_named_phase_legacy_sweep(tmp_path: Path) -> Path: + yaml_str = textwrap.dedent("""\ + sweep: + type: grid + parameters: + phases.profiling.concurrency: [4] + benchmark: + models: [llama] + endpoint: + urls: ["http://x:8000/v1/chat/completions"] + datasets: + - name: main + type: synthetic + phases: + - name: storm + kind: profiling + type: concurrency + requests: 10 + concurrency: 1 + """) + path = tmp_path / "named_phase_legacy_sweep.yaml" + path.write_text(yaml_str, encoding="utf-8") + return path + + class TestConfigInit: """End-to-end behavior of `aiperf config init`.""" @@ -233,6 +258,19 @@ def test_adaptive_search_rejects_with_hint( assert "adaptive_search" in err assert "dynamically" in err + def test_legacy_profiling_path_targets_unique_profiling_kind( + self, tmp_path: Path, capsys: pytest.CaptureFixture[str] + ) -> None: + path = _write_named_phase_legacy_sweep(tmp_path) + + expand(path, full=True, fmt="json") + + payload = json.loads(capsys.readouterr().out) + phase = payload["variations"][0]["benchmark"]["phases"][0] + assert phase["name"] == "storm" + assert phase["kind"] == "profiling" + assert phase["concurrency"] == 4 + class TestConfigValidate: """End-to-end behavior of `aiperf config validate`.""" @@ -248,6 +286,72 @@ def test_clean_config_reports_valid( assert "warning" not in captured.out assert captured.err == "" + def test_single_dict_phase_shorthand_reports_valid( + self, tmp_path: Path, capsys: pytest.CaptureFixture[str] + ) -> None: + yaml_str = textwrap.dedent("""\ + benchmark: + models: [llama] + endpoint: + urls: ["http://x:8000/v1/chat/completions"] + datasets: + - name: main + type: synthetic + phases: + type: concurrency + requests: 4 + concurrency: 1 + """) + path = tmp_path / "single_phase_shorthand.yaml" + path.write_text(yaml_str, encoding="utf-8") + + validate(path) + + captured = capsys.readouterr() + assert "Configuration valid" in captured.out + assert captured.err == "" + + def test_ambiguous_legacy_profiling_sweep_path_reports_actionable_error( + self, tmp_path: Path, capsys: pytest.CaptureFixture[str] + ) -> None: + yaml_str = textwrap.dedent("""\ + sweep: + type: grid + parameters: + phases.profiling.concurrency: [4] + benchmark: + models: [llama] + endpoint: + urls: ["http://x:8000/v1/chat/completions"] + datasets: + - name: main + type: synthetic + phases: + - name: low + kind: profiling + type: concurrency + requests: 10 + concurrency: 1 + - name: storm + kind: profiling + type: concurrency + requests: 10 + concurrency: 1 + """) + path = tmp_path / "ambiguous_legacy_profiling_sweep.yaml" + path.write_text(yaml_str, encoding="utf-8") + + with pytest.raises(SystemExit) as exc: + validate(path) + + assert exc.value.code == 1 + err = capsys.readouterr().err + assert "phases.profiling.concurrency" in err + assert "ambiguous" in err + assert "2 profiling phases exist" in err + assert "phases.low.concurrency" in err + assert "phases.0.concurrency" in err + def test_warnings_print_to_stderr_but_exit_zero( self, tmp_path: Path, capsys: pytest.CaptureFixture[str] ) -> None: diff --git a/tests/unit/common/models/test_credit_models.py b/tests/unit/common/models/test_credit_models.py index 80a536081d..c46bbaed1b 100644 --- a/tests/unit/common/models/test_credit_models.py +++ b/tests/unit/common/models/test_credit_models.py @@ -60,6 +60,42 @@ def _create(**kwargs) -> ProcessingStats: return _create +class TestBasePhaseStatsMetadata: + """Phase identity metadata should round-trip with stats payloads.""" + + def test_phase_identity_metadata_roundtrips(self) -> None: + stats = CreditPhaseStats( + phase=CreditPhase.PROFILING, + phase_index=3, + profiling_index=2, + phase_name="storm_2", + phase_kind="profiling", + ) + + dumped = stats.model_dump() + assert dumped["phase_index"] == 3 + assert dumped["profiling_index"] == 2 + assert dumped["phase_name"] == "storm_2" + assert dumped["phase_kind"] == "profiling" + + restored = CreditPhaseStats.model_validate(dumped) + assert restored.phase_index == 3 + assert restored.profiling_index == 2 + assert restored.phase_name == "storm_2" + assert restored.phase_kind == "profiling" + + @pytest.mark.parametrize( + "field", + [ + pytest.param("phase_index", id="phase-index"), + pytest.param("profiling_index", id="profiling-index"), + ], # fmt: skip + ) + def test_phase_indexes_must_be_non_negative(self, field: str) -> None: + with pytest.raises(ValueError, match="greater than or equal to 0"): + CreditPhaseStats(phase=CreditPhase.PROFILING, **{field: -1}) + + class TestBasePhaseStatsProperties: """Test BasePhaseStats property methods.""" diff --git a/tests/unit/common/models/test_record_context.py b/tests/unit/common/models/test_record_context.py index 61f2e5c19d..31eac1cb90 100644 --- a/tests/unit/common/models/test_record_context.py +++ b/tests/unit/common/models/test_record_context.py @@ -39,6 +39,19 @@ def test_explicit_dag_fields(self): assert ctx.agent_depth == 3 assert ctx.parent_correlation_id == "root" + def test_phase_identity_fields(self): + ctx = _make_record_context( + phase_index=2, + profiling_index=1, + phase_name="recovery", + phase_kind="profiling", + ) + + assert ctx.phase_index == 2 + assert ctx.profiling_index == 1 + assert ctx.phase_name == "recovery" + assert ctx.phase_kind == "profiling" + class TestRequestInfoIsRecordContext: def test_request_info_inherits_record_context(self): @@ -56,6 +69,10 @@ def test_request_info_has_transport_extras(self): assert "turns" not in extras assert "system_message" not in extras assert "user_context_message" not in extras + assert "phase_index" not in extras + assert "profiling_index" not in extras + assert "phase_name" not in extras + assert "phase_kind" not in extras class TestRequestRecordHoldsRecordContext: diff --git a/tests/unit/config/test_baseten_replay_flags.py b/tests/unit/config/test_baseten_replay_flags.py index edca62471f..2066183eb8 100644 --- a/tests/unit/config/test_baseten_replay_flags.py +++ b/tests/unit/config/test_baseten_replay_flags.py @@ -20,9 +20,12 @@ import logging from pathlib import Path +import pytest + +pytest.importorskip("pyarrow") + import pyarrow as pa import pyarrow.parquet as pq -import pytest from pydantic import ValidationError from pytest import param diff --git a/tests/unit/config/test_converter_profiling_phase_routes.py b/tests/unit/config/test_converter_profiling_phase_routes.py index e2cd7f195c..05ff3fd3f5 100644 --- a/tests/unit/config/test_converter_profiling_phase_routes.py +++ b/tests/unit/config/test_converter_profiling_phase_routes.py @@ -20,6 +20,8 @@ from __future__ import annotations +from typing import ClassVar + import pytest from aiperf.config.flags._converter_profiling import build_profiling @@ -256,196 +258,44 @@ def test_rate_ramp_with_request_rate_succeeds(self): assert prof.get("rate_ramp") == {"duration": 30} -class TestAdaptiveScaleRoutes: - def test_adaptive_scale_cli_fields_route_to_profiling_phase( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - adaptive_scale=True, - adaptive_sustain_duration=120.0, - adaptive_assessment_period=30.0, - adaptive_scale_sla=["request_latency:p95:le:30000"], - benchmark_duration=600.0, - concurrency=200, - ) - user = _make_user(loadgen=loadgen) - prof = build_profiling(user) - - assert prof["type"] == PhaseType.CONCURRENCY - assert prof["adaptive_scale"] is True - assert prof["adaptive_sustain_duration"] == 120.0 - assert prof["adaptive_assessment_period"] == 30.0 - assert prof["sla"] == [ - { - "metric_tag": "request_latency", - "stat": "p95", - "op": "le", - "threshold": 30000.0, - } - ] - - def test_adaptive_scale_compact_control_routes_to_profiling_phase( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - adaptive_scale=True, - adaptive_scale_control="request_rate:1,200:float", - adaptive_sustain_duration=120.0, - adaptive_scale_sla=["request_latency:p95:le:30000"], - benchmark_duration=600.0, - request_rate=200.0, - ) - user = _make_user(loadgen=loadgen) - prof = build_profiling(user) - - assert prof["adaptive_control_variable"] == "request_rate" - assert prof["adaptive_control_min"] == 1.0 - assert prof["adaptive_control_max"] == 200.0 - assert "adaptive_scale_control" not in prof - - @pytest.mark.parametrize( - ("control", "match"), - [ - pytest.param("concurrency:1", "variable:min,max:type", id="missing-type"), - pytest.param( - ":1,10:int", "requires a control variable", id="blank-variable" - ), - pytest.param( - "concurrency:1.5,10:int", - "min bound must be an integer", - id="bad-int-min", - ), - pytest.param( - "request_rate:one,10:float", - "min bound must be a number", - id="bad-float-min", - ), - pytest.param( - "concurrency:1,10:string", - "type must be 'int' or 'float'", - id="bad-type", - ), - ], +class TestAdaptiveScaleCliRemoval: + REMOVED_FIELDS: ClassVar[frozenset[str]] = frozenset( + { + "adaptive_scale", + "adaptive_sustain_duration", + "adaptive_assessment_period", + "adaptive_scale_control", + "adaptive_control_variable", + "adaptive_control_min", + "adaptive_control_max", + "adaptive_scale_sla", + } ) - def test_adaptive_scale_rejects_invalid_compact_control( - self: TestAdaptiveScaleRoutes, control: str, match: str - ) -> None: - loadgen = CLIConfig( - adaptive_scale=True, - adaptive_scale_control=control, - adaptive_sustain_duration=120.0, - adaptive_scale_sla=["request_latency:p95:le:30000"], - benchmark_duration=600.0, - concurrency=100, - ) - user = _make_user(loadgen=loadgen) - with pytest.raises(ValueError, match=match): - build_profiling(user) + def test_removed_adaptive_scale_cli_fields_are_not_on_cli_config(self) -> None: + assert self.REMOVED_FIELDS.isdisjoint(CLIConfig.model_fields) - def test_adaptive_scale_rejects_mixed_compact_and_expanded_control( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - adaptive_scale=True, - adaptive_scale_control="concurrency:1,100:int", - adaptive_control_variable="concurrency", - adaptive_sustain_duration=120.0, - adaptive_scale_sla=["request_latency:p95:le:30000"], - benchmark_duration=600.0, - concurrency=100, - ) - user = _make_user(loadgen=loadgen) + def test_removed_adaptive_scale_cli_fields_are_not_loadgen_routes(self) -> None: + from aiperf.config.flags._section_fields import LOADGEN_FIELDS - with pytest.raises(ValueError, match="Use either --adaptive-scale-control"): - build_profiling(user) - - def test_adaptive_scale_requires_concurrency( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - adaptive_scale=True, - adaptive_sustain_duration=120.0, - benchmark_duration=600.0, - request_rate=10.0, - ) - user = _make_user(loadgen=loadgen) - - with pytest.raises(ValueError, match="--adaptive-scale requires --concurrency"): - build_profiling(user) - - def test_adaptive_scale_rejects_search_sla( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - adaptive_scale=True, - adaptive_sustain_duration=120.0, - benchmark_duration=600.0, - concurrency=200, - search_sla=["request_latency:p95:le:30000"], - ) - user = _make_user(loadgen=loadgen) - - with pytest.raises(ValueError, match="--adaptive-scale-sla"): - build_profiling(user) - - def test_adaptive_scale_sla_requires_adaptive_scale( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - benchmark_duration=600.0, - concurrency=200, - adaptive_scale_sla=["request_latency:p95:le:30000"], - ) - user = _make_user(loadgen=loadgen) - - with pytest.raises(ValueError, match="--adaptive-scale-sla requires"): - build_profiling(user) + assert self.REMOVED_FIELDS.isdisjoint(LOADGEN_FIELDS) - def test_adaptive_scale_sla_still_reports_malformed_filter_without_adaptive_scale( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - benchmark_duration=600.0, - concurrency=200, - adaptive_scale_sla=["bad"], + def test_build_profiling_does_not_emit_adaptive_scale_from_cli(self) -> None: + user = _make_user( + loadgen=CLIConfig( + concurrency=8, + benchmark_duration=60, + request_count=100, + ) ) - user = _make_user(loadgen=loadgen) - - with pytest.raises(TypeError, match="--adaptive-scale-sla"): - build_profiling(user) - def test_adaptive_scale_cli_accepts_itl_and_goodput_slas( - self: TestAdaptiveScaleRoutes, - ) -> None: - loadgen = CLIConfig( - adaptive_scale=True, - adaptive_sustain_duration=120.0, - benchmark_duration=600.0, - concurrency=200, - adaptive_scale_sla=[ - "itl:p95:le:100", - "goodput:avg:ge:20", - ], - ) - user = _make_user(loadgen=loadgen) prof = build_profiling(user) - assert prof["sla"] == [ - { - "metric_tag": "itl", - "stat": "p95", - "op": "le", - "threshold": 100.0, - }, - { - "metric_tag": "goodput", - "stat": "avg", - "op": "ge", - "threshold": 20.0, - }, - ] + assert self.REMOVED_FIELDS.isdisjoint(prof) + assert "adaptive_scale" not in prof + +class TestAdaptiveScaleRoutes: def test_adaptive_scale_rejects_concurrency_ramp( self: TestAdaptiveScaleRoutes, ) -> None: diff --git a/tests/unit/config/test_named_phase_kind.py b/tests/unit/config/test_named_phase_kind.py new file mode 100644 index 0000000000..90fca97dc6 --- /dev/null +++ b/tests/unit/config/test_named_phase_kind.py @@ -0,0 +1,456 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from aiperf.config.config import BenchmarkConfig +from aiperf.config.flags import CLIConfig +from aiperf.config.flags.resolver import _apply_phase_loadgen_overrides +from aiperf.config.resolution.plan import BenchmarkRun +from aiperf.config.sweep import expand_sweep +from aiperf.timing.config import TimingConfig + +_ENDPOINT = {"urls": ["http://localhost:8000/v1/chat/completions"]} +_DATASETS = [ + { + "name": "main", + "type": "synthetic", + "entries": 10, + "prompts": {"isl": 16, "osl": 8}, + } +] +_PHASE = {"type": "concurrency", "requests": 2, "concurrency": 1} + + +def _cfg(phases: list[dict]) -> BenchmarkConfig: + return BenchmarkConfig.model_validate( + { + "models": ["m"], + "endpoint": _ENDPOINT, + "datasets": _DATASETS, + "phases": phases, + } + ) + + +def _envelope(phases: list[dict]) -> dict: + return { + "benchmark": { + "models": ["m"], + "endpoint": _ENDPOINT, + "datasets": _DATASETS, + "phases": phases, + } + } + + +def test_flat_phases_shorthand_gets_profiling_kind() -> None: + cfg = _cfg({"type": "concurrency", "requests": 2, "concurrency": 1}) + + assert len(cfg.phases) == 1 + assert cfg.phases[0].name == "profiling" + assert cfg.phases[0].kind == "profiling" + assert cfg.phases[0].exclude_from_results is False + + +def test_warmup_profiling_shorthand_gets_explicit_kinds() -> None: + cfg = BenchmarkConfig.model_validate( + { + "models": ["m"], + "endpoint": _ENDPOINT, + "datasets": _DATASETS, + "warmup": {"type": "concurrency", "requests": 1, "concurrency": 1}, + "profiling": {"type": "concurrency", "requests": 2, "concurrency": 1}, + } + ) + + assert [(p.name, p.kind, p.exclude_from_results) for p in cfg.phases] == [ + ("warmup", "warmup", True), + ("profiling", "profiling", False), + ] + + +def test_no_profiling_kind_fails_with_legacy_message() -> None: + with pytest.raises(ValidationError, match="a 'profiling' phase is required"): + _cfg([{"name": "setup", "kind": "warmup", **_PHASE}]) + + +def test_profiling_kind_allows_noncanonical_name_as_only_results_phase() -> None: + cfg = _cfg([{"name": "storm_1", "kind": "profiling", **_PHASE}]) + + assert cfg.get_profiling_phases()[0].name == "storm_1" + + +def test_legacy_canonical_names_infer_kind() -> None: + cfg = _cfg([{"name": "warmup", **_PHASE}, {"name": "profiling", **_PHASE}]) + + assert [(p.name, p.kind, p.exclude_from_results) for p in cfg.phases] == [ + ("warmup", "warmup", True), + ("profiling", "profiling", False), + ] + + +def test_custom_name_without_kind_fails() -> None: + with pytest.raises(ValidationError, match="kind is required"): + _cfg([{"name": "storm_1", **_PHASE}]) + + +def test_custom_names_with_multiple_profiling_phases_pass() -> None: + cfg = _cfg( + [ + {"name": "low_cancel_1", "kind": "profiling", **_PHASE}, + {"name": "storm_1", "kind": "profiling", **_PHASE}, + {"name": "recovery_1", "kind": "profiling", **_PHASE}, + ] + ) + + assert [p.name for p in cfg.get_profiling_phases()] == [ + "low_cancel_1", + "storm_1", + "recovery_1", + ] + + +def test_three_plus_phases_can_mix_multiple_warmups_and_profiling_phases() -> None: + cfg = _cfg( + [ + {"name": "prime_cache", "kind": "warmup", **_PHASE}, + {"name": "stabilize", "kind": "warmup", **_PHASE}, + {"name": "low", "kind": "profiling", **_PHASE}, + {"name": "storm", "kind": "profiling", **_PHASE}, + {"name": "recover", "kind": "profiling", **_PHASE}, + ] + ) + + assert [p.name for p in cfg.get_warmup_phases()] == [ + "prime_cache", + "stabilize", + ] + assert [p.exclude_from_results for p in cfg.get_warmup_phases()] == [True, True] + assert [p.name for p in cfg.get_profiling_phases()] == [ + "low", + "storm", + "recover", + ] + assert [p.exclude_from_results for p in cfg.get_profiling_phases()] == [ + False, + False, + False, + ] + + +def test_staged_cache_warmups_execute_in_order_without_profiling_indexes() -> None: + cfg = _cfg( + [ + { + "name": "cold_cache_warmup", + "kind": "warmup", + "requests": 50, + "concurrency": 4, + "type": "concurrency", + }, + { + "name": "warm_cache_warmup", + "kind": "warmup", + "requests": 100, + "concurrency": 16, + "type": "concurrency", + }, + { + "name": "steady_state", + "kind": "profiling", + "duration": "30m", + "concurrency": 128, + "type": "concurrency", + }, + ] + ) + run = BenchmarkRun( + benchmark_id="run", cfg=cfg, artifact_dir=Path("/tmp/aiperf-test-artifacts") + ) + + timing = TimingConfig.from_run(run) + + assert [p.phase_name for p in timing.phase_configs] == [ + "cold_cache_warmup", + "warm_cache_warmup", + "steady_state", + ] + assert [p.phase_kind for p in timing.phase_configs] == [ + "warmup", + "warmup", + "profiling", + ] + assert [p.phase_index for p in timing.phase_configs] == [0, 1, 2] + assert [p.profiling_index for p in timing.phase_configs] == [None, None, 0] + assert [p.exclude_from_results for p in cfg.phases] == [True, True, False] + + +def test_multiple_warmup_phases_without_profiling_still_fails() -> None: + with pytest.raises(ValidationError, match="a 'profiling' phase is required"): + _cfg( + [ + {"name": "prime_cache", "kind": "warmup", **_PHASE}, + {"name": "stabilize", "kind": "warmup", **_PHASE}, + ] + ) + + +def test_case_insensitive_duplicate_names_fail() -> None: + with pytest.raises(ValidationError, match="case-insensitively"): + _cfg( + [ + {"name": "Storm", "kind": "profiling", **_PHASE}, + {"name": "storm", "kind": "profiling", **_PHASE}, + ] + ) + + +@pytest.mark.parametrize( + "name,kind", [("warmup", "profiling"), ("profiling", "warmup")] +) +def test_reserved_canonical_names_require_matching_kind(name: str, kind: str) -> None: + with pytest.raises(ValidationError, match="reserved for kind"): + _cfg([{"name": name, "kind": kind, **_PHASE}]) + + +def test_kind_drives_exclude_from_results_validation() -> None: + with pytest.raises(ValidationError, match="exclude_from_results must be True"): + _cfg( + [ + { + "name": "setup", + "kind": "warmup", + "exclude_from_results": False, + **_PHASE, + }, + {"name": "main", "kind": "profiling", **_PHASE}, + ] + ) + + +def test_profiling_kind_cannot_be_explicitly_excluded_from_results() -> None: + with pytest.raises(ValidationError, match="exclude_from_results must be False"): + _cfg( + [ + { + "name": "storm", + "kind": "profiling", + "exclude_from_results": True, + **_PHASE, + }, + ] + ) + + +def test_strict_phase_name_regex_rejects_path_unsafe_names() -> None: + with pytest.raises(ValidationError): + _cfg([{"name": "storm.1", "kind": "profiling", **_PHASE}]) + + +@pytest.mark.parametrize("name", ["NUL", "nul", "Com1", "lpt9", "AUX"]) +def test_phase_name_rejects_windows_reserved_device_names(name: str) -> None: + with pytest.raises(ValidationError, match="reserved by Windows"): + _cfg([{"name": name, "kind": "profiling", **_PHASE}]) + + +def test_phase_name_allows_reserved_name_neighbors() -> None: + cfg = _cfg([{"name": "com10", "kind": "profiling", **_PHASE}]) + + assert cfg.phases[0].name == "com10" + + +def test_sweep_path_resolves_phase_name_and_numeric_index() -> None: + base = _envelope( + [ + {"name": "warmup", "kind": "warmup", **_PHASE}, + {"name": "storm_1", "kind": "profiling", **_PHASE}, + ] + ) + by_name = { + **base, + "sweep": {"type": "grid", "parameters": {"phases.storm_1.concurrency": [8]}}, + } + by_index = { + **base, + "sweep": {"type": "grid", "parameters": {"phases.1.concurrency": [16]}}, + } + + assert expand_sweep(by_name)[0][0]["benchmark"]["phases"][1]["concurrency"] == 8 + assert expand_sweep(by_index)[0][0]["benchmark"]["phases"][1]["concurrency"] == 16 + + +def test_legacy_phases_profiling_path_targets_unique_profiling_kind() -> None: + base = _envelope([{"name": "storm_1", "kind": "profiling", **_PHASE}]) + base["sweep"] = { + "type": "grid", + "parameters": {"phases.profiling.concurrency": [4]}, + } + + assert expand_sweep(base)[0][0]["benchmark"]["phases"][0]["concurrency"] == 4 + + +def test_legacy_phases_profiling_path_fails_when_ambiguous() -> None: + base = _envelope( + [ + {"name": "low", "kind": "profiling", **_PHASE}, + {"name": "storm", "kind": "profiling", **_PHASE}, + ] + ) + base["sweep"] = { + "type": "grid", + "parameters": {"phases.profiling.concurrency": [4]}, + } + + with pytest.raises(ValueError) as exc_info: + expand_sweep(base) + + message = str(exc_info.value) + assert "phases.profiling.concurrency" in message + assert "ambiguous" in message + assert "2 profiling phases exist" in message + assert "low, storm" in message + assert "phases.low.concurrency" in message + assert "phases.0.concurrency" in message + + +def test_cli_loadgen_overlays_unique_profiling_kind() -> None: + merged = _envelope([{"name": "storm_1", "kind": "profiling", **_PHASE}]) + cli = CLIConfig(concurrency=7) + + _apply_phase_loadgen_overrides(merged, cli) + + assert merged["benchmark"]["phases"][0]["concurrency"] == 7 + + +def test_cli_loadgen_overlay_fails_on_multiple_profiling_phases() -> None: + merged = _envelope( + [ + {"name": "low", "kind": "profiling", **_PHASE}, + {"name": "storm", "kind": "profiling", **_PHASE}, + ] + ) + cli = CLIConfig(concurrency=7) + + with pytest.raises(Exception, match="2 profiling phases"): + _apply_phase_loadgen_overrides(merged, cli) + + +def test_sweep_legacy_profiling_path_uses_kind_before_non_warmup_fallback() -> None: + base = _envelope( + [ + {"name": "setup", "kind": "warmup", **_PHASE}, + {"name": "storm_1", "kind": "profiling", **_PHASE}, + ] + ) + base["sweep"] = { + "type": "grid", + "parameters": {"phases.profiling.concurrency": [9]}, + } + + expanded = expand_sweep(base)[0][0]["benchmark"]["phases"] + + assert expanded[0]["concurrency"] == 1 + assert expanded[1]["concurrency"] == 9 + + +def test_cli_loadgen_overlay_ignores_warmup_kind_when_finding_profiling() -> None: + merged = _envelope( + [ + {"name": "setup", "kind": "warmup", **_PHASE}, + {"name": "storm_1", "kind": "profiling", **_PHASE}, + ] + ) + cli = CLIConfig(concurrency=11) + + _apply_phase_loadgen_overrides(merged, cli) + + assert merged["benchmark"]["phases"][0]["concurrency"] == 1 + assert merged["benchmark"]["phases"][1]["concurrency"] == 11 + + +def test_timing_config_preserves_order_indexes_and_phase_cancellation() -> None: + cfg = _cfg( + [ + { + "name": "low", + "kind": "profiling", + "cancellation": {"rate": 5, "delay": 0}, + **_PHASE, + }, + { + "name": "storm", + "kind": "profiling", + "cancellation": {"rate": 50, "delay": 1}, + **_PHASE, + }, + {"name": "setup", "kind": "warmup", **_PHASE}, + ] + ) + run = BenchmarkRun( + benchmark_id="run", cfg=cfg, artifact_dir=Path("/tmp/aiperf-test-artifacts") + ) + + timing = TimingConfig.from_run(run) + + assert [p.phase_name for p in timing.phase_configs] == ["low", "storm", "setup"] + assert [p.phase_index for p in timing.phase_configs] == [0, 1, 2] + assert [p.profiling_index for p in timing.phase_configs] == [0, 1, None] + assert [p.request_cancellation.rate for p in timing.phase_configs] == [ + 5.0, + 50.0, + None, + ] + + +def test_timing_config_inherits_profiling_cancellation_default_only() -> None: + cfg = _cfg( + [ + { + "name": "setup", + "kind": "warmup", + "cancellation": {"rate": 99, "delay": 0}, + **_PHASE, + }, + { + "name": "cancel_phase", + "kind": "profiling", + "cancellation": {"rate": 25, "delay": 1}, + **_PHASE, + }, + {"name": "inherit_phase", "kind": "profiling", **_PHASE}, + {"name": "warmup_omitted", "kind": "warmup", **_PHASE}, + ] + ) + run = BenchmarkRun( + benchmark_id="run", cfg=cfg, artifact_dir=Path("/tmp/aiperf-test-artifacts") + ) + + timing = TimingConfig.from_run(run) + + by_name = {phase.phase_name: phase for phase in timing.phase_configs} + assert by_name["setup"].request_cancellation.rate == 99.0 + assert by_name["cancel_phase"].request_cancellation.rate == 25.0 + assert by_name["inherit_phase"].request_cancellation.rate == 25.0 + assert by_name["inherit_phase"].request_cancellation.delay == 1.0 + assert by_name["warmup_omitted"].request_cancellation.rate is None + assert timing.request_cancellation.rate == 25.0 + + +def test_timing_config_defaults_phase_metadata_for_legacy_programmatic_phases() -> None: + cfg = _cfg([{"name": "profiling", **_PHASE}]) + run = BenchmarkRun( + benchmark_id="run", cfg=cfg, artifact_dir=Path("/tmp/aiperf-test-artifacts") + ) + + timing = TimingConfig.from_run(run) + + assert [ + (p.phase_index, p.profiling_index, p.phase_name, p.phase_kind) + for p in timing.phase_configs + ] == [(0, 0, "profiling", "profiling")] diff --git a/tests/unit/config/test_v1_resolver_streaming_override.py b/tests/unit/config/test_v1_resolver_streaming_override.py index 952f3c1227..30f7af5c04 100644 --- a/tests/unit/config/test_v1_resolver_streaming_override.py +++ b/tests/unit/config/test_v1_resolver_streaming_override.py @@ -13,7 +13,6 @@ from __future__ import annotations -import asyncio import textwrap from pathlib import Path @@ -109,73 +108,8 @@ def test_yaml_cli_dataset_magic_list_targets_existing_dataset(tmp_path: Path) -> assert "datasets.main.prompts.isl.mean" not in config.sweep.parameters -_YAML_ADVANCED_ADAPTIVE = textwrap.dedent("""\ -benchmark: - models: - - test-model - endpoint: - urls: - - http://localhost:8000/v1/chat/completions - streaming: true - datasets: - - name: default - type: synthetic - entries: 100 - prompts: - isl: 128 - osl: 64 - phases: - - name: profiling - type: concurrency - duration: 60 - concurrency: 8 - sla: - request_latency: - p95: - le: 30000 - adaptive_scale: - enabled: false - min_concurrency: 2 - max_concurrency: 8 - min_completed_requests: 3 - sustain_duration: 20 - assessment_period: 5 - strategy: - type: ramp_until_fail - step_policy: fixed_percent_step - step_percent: 50 -""") - - -def test_basic_adaptive_cli_overrides_preserve_advanced_yaml(tmp_path: Path) -> None: - cfg_file = tmp_path / "adaptive.yaml" - cfg_file.write_text(_YAML_ADVANCED_ADAPTIVE) - user = CLIConfig( - adaptive_scale=True, - adaptive_sustain_duration=40, - adaptive_assessment_period=10, - concurrency=16, - adaptive_scale_sla=["request_latency:p95:le:20000"], - ) - - config = resolve_config(user, cfg_file) - phase = config.benchmark.phases[0] - - assert phase.adaptive_scale is True - assert phase.concurrency == 16 - assert phase.adaptive_sustain_duration == 40 - assert phase.adaptive_assessment_period == 10 - assert phase.sla[0].threshold == 20000 - - assert phase.adaptive_control_min == 2 - assert phase.adaptive_control_max == 8 - assert phase.adaptive_min_completed_requests == 3 - assert phase.adaptive_scale_step_policy == "fixed_percent_step" - assert phase.adaptive_scale_step_percent == 50 - - -def test_adaptive_cli_sla_overrides_nested_yaml_sla(tmp_path: Path) -> None: - cfg_file = tmp_path / "adaptive.yaml" +def test_yaml_cli_overrides_accept_single_phase_dict_shorthand(tmp_path: Path) -> None: + cfg_file = tmp_path / "single-phase.yaml" cfg_file.write_text( textwrap.dedent( """\ @@ -189,70 +123,26 @@ def test_adaptive_cli_sla_overrides_nested_yaml_sla(tmp_path: Path) -> None: datasets: - name: default type: synthetic - entries: 100 + entries: 16 prompts: - isl: 128 - osl: 64 + isl: 32 + osl: 8 phases: - - name: profiling - type: concurrency - duration: 60 - concurrency: 8 - adaptive_scale: - enabled: true - min_concurrency: 2 - max_concurrency: 8 - sustain_duration: 20 - sla: - request_latency: - p95: - le: 0.000001 + type: concurrency + concurrency: 1 + requests: 4 """ ) ) - user = CLIConfig(adaptive_scale_sla=["request_latency:p95:le:1000000"]) + user = CLIConfig(tokenizer_name="builtin", ui="none") config = resolve_config(user, cfg_file) - phase = config.benchmark.phases[0] - - assert phase.sla[0].threshold == 1000000 - - -def test_adaptive_cli_sla_requires_adaptive_scale_for_yaml_phase( - tmp_path: Path, -) -> None: - cfg_file = _write_yaml(tmp_path) - user = CLIConfig(adaptive_scale_sla=["request_latency:p95:le:1000000"]) - - with pytest.raises(ValueError, match="--adaptive-scale-sla requires"): - resolve_config(user, cfg_file) - - -def test_adaptive_cli_compact_control_overrides_yaml(tmp_path: Path) -> None: - cfg_file = tmp_path / "adaptive.yaml" - cfg_file.write_text(_YAML_ADVANCED_ADAPTIVE) - user = CLIConfig( - adaptive_scale=True, - adaptive_scale_control="concurrency:3,16:int", - concurrency=16, - ) - config = resolve_config(user, cfg_file) + assert len(config.benchmark.phases) == 1 phase = config.benchmark.phases[0] - - assert phase.adaptive_control_variable == "concurrency" - assert phase.adaptive_control_min == 3 - assert phase.adaptive_control_max == 16 - - -async def test_adaptive_cli_sla_parse_error_names_adaptive_flag(tmp_path: Path) -> None: - cfg_file = tmp_path / "adaptive.yaml" - await asyncio.to_thread(cfg_file.write_text, _YAML_ADVANCED_ADAPTIVE) - user = CLIConfig(adaptive_scale_sla=["bad"]) - - with pytest.raises(TypeError) as exc_info: - resolve_config(user, cfg_file) - - message = str(exc_info.value) - assert "--adaptive-scale-sla" in message - assert "--search-sla" not in message + assert phase.name == "profiling" + assert phase.kind == "profiling" + assert phase.concurrency == 1 + assert phase.requests == 4 + assert config.benchmark.tokenizer.name == "builtin" + assert config.benchmark.runtime.ui == "none" diff --git a/tests/unit/credit/test_callback_handler.py b/tests/unit/credit/test_callback_handler.py index cbeb49ca0c..7c24e1cb8d 100644 --- a/tests/unit/credit/test_callback_handler.py +++ b/tests/unit/credit/test_callback_handler.py @@ -104,12 +104,14 @@ def make_credit( turn_index: int = 0, num_turns: int = 1, phase: CreditPhase = CreditPhase.PROFILING, + phase_index: int | None = None, agent_depth: int = 0, ) -> Credit: """Create a Credit for testing.""" return Credit( id=credit_id, phase=phase, + phase_index=phase_index, conversation_id=conversation_id, x_correlation_id=f"corr-{conversation_id}", turn_index=turn_index, @@ -160,6 +162,58 @@ def test_register_and_unregister_phase(self, callback_handler): callback_handler.unregister_phase(CreditPhase.PROFILING) assert CreditPhase.PROFILING not in callback_handler._phase_handlers + async def test_register_phase_same_kind_phases_uses_runtime_index( + self, + callback_handler, + mock_concurrency, + mock_lifecycle, + mock_stop_checker, + mock_strategy, + ): + """Two profiling phases must not overwrite each other's callbacks.""" + progress_0 = MagicMock() + progress_0.all_credits_returned_event = asyncio.Event() + progress_0.increment_returned.return_value = False + progress_1 = MagicMock() + progress_1.all_credits_returned_event = asyncio.Event() + progress_1.increment_returned.return_value = False + + callback_handler.register_phase( + phase=CreditPhase.PROFILING, + phase_index=0, + progress=progress_0, + lifecycle=mock_lifecycle, + stop_checker=mock_stop_checker, + strategy=mock_strategy, + ) + callback_handler.register_phase( + phase=CreditPhase.PROFILING, + phase_index=1, + progress=progress_1, + lifecycle=mock_lifecycle, + stop_checker=mock_stop_checker, + strategy=mock_strategy, + ) + + await callback_handler.on_credit_return( + "worker-1", make_credit_return(make_credit(phase_index=1)) + ) + await callback_handler.on_first_token( + FirstToken( + credit_id=1, + phase=CreditPhase.PROFILING, + phase_index=1, + ttft_ns=1000000, + ) + ) + + progress_0.increment_returned.assert_not_called() + progress_0.increment_prefill_released.assert_not_called() + progress_1.increment_returned.assert_called_once() + progress_1.increment_prefill_released.assert_called_once() + mock_concurrency.release_session_slot.assert_called_once_with(1) + mock_concurrency.release_prefill_slot.assert_called_once_with(1) + # ============================================================================= # Test: Credit Return - Basic Flow diff --git a/tests/unit/exporters/test_exporter_manager.py b/tests/unit/exporters/test_exporter_manager.py index 63e0a4399f..5866323814 100644 --- a/tests/unit/exporters/test_exporter_manager.py +++ b/tests/unit/exporters/test_exporter_manager.py @@ -1,11 +1,19 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 +import json from unittest.mock import AsyncMock, MagicMock, patch import pytest -from aiperf.common.models import MetricResult, ProfileResults +from aiperf.common.exceptions import DataExporterDisabled +from aiperf.common.models import ( + ErrorDetails, + ErrorDetailsCount, + MetricResult, + PhaseProfileResults, + ProfileResults, +) from aiperf.config.flags.cli_config import CLIConfig from aiperf.exporters.exporter_manager import ExporterManager from aiperf.plugin.enums import ( @@ -83,6 +91,158 @@ async def test_export( mock_class.assert_called_once() mock_instance.export.assert_awaited_once() + @pytest.mark.asyncio + async def test_export_writes_phase_metric_artifacts( + self, endpoint_config, output_config, mock_cfg + ) -> None: + phase_records = [ + PhaseProfileResults( + phase_index=0, + phase_name="warmup_cache", + phase_kind="warmup", + records=[ + MetricResult( + tag="request_latency", + header="Request Latency", + unit="ms", + avg=12.0, + count=2, + ) + ], + start_ns=1, + end_ns=2, + successful_request_count=2, + ), + PhaseProfileResults( + phase_index=1, + profiling_index=0, + phase_name="storm", + phase_kind="profiling", + records=[ + MetricResult( + tag="request_latency", + header="Request Latency", + unit="ms", + avg=34.0, + count=3, + ) + ], + start_ns=3, + end_ns=4, + successful_request_count=3, + error_request_count=2, + error_summary=[ + ErrorDetailsCount( + error_details=ErrorDetails( + type="RequestCancellationError", message="cancelled" + ), + count=2, + ) + ], + ), + ] + manager = ExporterManager( + results=ProfileResults( + records=[], + start_ns=1, + end_ns=4, + completed=0, + was_cancelled=False, + error_summary=[], + phase_records=phase_records, + ), + run=make_run_from_cli(mock_cfg), + telemetry_results=None, + ) + + with patch( + "aiperf.exporters.exporter_manager.plugins.iter_all", return_value=[] + ): + await manager.export_data() + + manifest_path = output_config / "phase_manifest.json" + assert manifest_path.exists() + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + storm_entry = manifest["phases"][1] + assert storm_entry["metrics_json"] == "phases/storm/profile_export_aiperf.json" + assert storm_entry["metrics_csv"] == "phases/storm/profile_export_aiperf.csv" + assert storm_entry["start_ns"] == 3 + assert storm_entry["end_ns"] == 4 + assert storm_entry["successful_request_count"] == 3 + assert storm_entry["error_request_count"] == 2 + assert storm_entry["total_request_count"] == 5 + assert storm_entry["error_summary"][0]["count"] == 2 + assert ( + output_config / "phases" / "warmup_cache" / "profile_export_aiperf.json" + ).exists() + storm_json = json.loads( + ( + output_config / "phases" / "storm" / "profile_export_aiperf.json" + ).read_text(encoding="utf-8") + ) + assert storm_json["error_summary"][0]["count"] == 2 + assert ( + output_config / "phases" / "storm" / "profile_export_aiperf.csv" + ).exists() + + @pytest.mark.asyncio + async def test_write_phase_export_handles_disabled_and_failed_exporters( + self, endpoint_config, output_config, mock_cfg + ) -> None: + class DisabledPhaseExporter: + def __init__(self, exporter_config) -> None: + raise DataExporterDisabled("phase exporter disabled") + + class FailingPhaseExporter: + def __init__(self, exporter_config) -> None: + pass + + def _generate_content(self) -> str: + raise ValueError("content boom") + + manager = ExporterManager( + results=ProfileResults( + records=[], + start_ns=1, + end_ns=2, + completed=0, + was_cancelled=False, + error_summary=[], + ), + run=make_run_from_cli(mock_cfg), + telemetry_results=None, + ) + manager.error = MagicMock() + phase_profile = ProfileResults( + records=[], + start_ns=1, + end_ns=2, + completed=0, + was_cancelled=False, + error_summary=[], + ) + manifest_entry = {"phase_name": "storm"} + + await manager._write_phase_export( + exporter_cls=DisabledPhaseExporter, + phase_profile=phase_profile, + file_path=output_config / "disabled.json", + manifest_entry=manifest_entry, + manifest_key="disabled", + ) + await manager._write_phase_export( + exporter_cls=FailingPhaseExporter, + phase_profile=phase_profile, + file_path=output_config / "failing.json", + manifest_entry=manifest_entry, + manifest_key="failing", + ) + + assert "disabled" not in manifest_entry + assert "failing" not in manifest_entry + manager.error.assert_called_once() + assert "Failed to write phase export" in manager.error.call_args.args[0] + @pytest.mark.asyncio async def test_export_runs_mlflow_after_other_data_exporters( self, endpoint_config, output_config, sample_records, mock_cfg diff --git a/tests/unit/records/test_error_tracker.py b/tests/unit/records/test_error_tracker.py index 999acb1c95..127a707892 100644 --- a/tests/unit/records/test_error_tracker.py +++ b/tests/unit/records/test_error_tracker.py @@ -174,6 +174,32 @@ def test_increment_error_creates_phase_tracker_lazily(self, sample_error): assert len(summary) == 1 assert summary[0].count == 1 + def test_indexed_phase_errors_also_roll_up_to_credit_phase(self, sample_error): + """Concrete phase errors are queryable by index and roll up by phase.""" + tracker = ErrorTracker() + + tracker.increment_error_count_for_phase( + CreditPhase.PROFILING, sample_error, phase_index=1 + ) + tracker.increment_error_count_for_phase( + CreditPhase.PROFILING, sample_error, phase_index=1 + ) + tracker.increment_error_count_for_phase( + CreditPhase.PROFILING, sample_error, phase_index=2 + ) + + phase_one_summary = tracker.get_error_summary_for_phase( + CreditPhase.PROFILING, phase_index=1 + ) + phase_two_summary = tracker.get_error_summary_for_phase( + CreditPhase.PROFILING, phase_index=2 + ) + rollup_summary = tracker.get_error_summary_for_phase(CreditPhase.PROFILING) + + assert phase_one_summary[0].count == 2 + assert phase_two_summary[0].count == 1 + assert rollup_summary[0].count == 3 + def test_errors_tracked_separately_by_phase(self, sample_error, another_error): """Errors in different phases are tracked independently.""" tracker = ErrorTracker() diff --git a/tests/unit/records/test_records_manager.py b/tests/unit/records/test_records_manager.py index 5fcef15349..93ccef323f 100644 --- a/tests/unit/records/test_records_manager.py +++ b/tests/unit/records/test_records_manager.py @@ -187,6 +187,27 @@ async def test_metric_dispatch_error_recorded_in_phase_error_summary(self) -> No tracked = ErrorDetails.from_exception(dispatch_error) assert any(e.error_details == tracked for e in summary) + @pytest.mark.asyncio + async def test_on_records_tracks_errors_by_phase_index(self) -> None: + manager = self._make_manager() + dispatch_error = RuntimeError("metric accumulator failed") + request_error = ErrorDetails( + code=499, type="RequestCancellationError", message="cancelled" + ) + manager._dispatch_record = AsyncMock(return_value=[dispatch_error]) + message = self._records_message() + message.metadata.phase_index = 2 + message.error = request_error + + await manager._on_records(message) + + indexed_summary = manager._error_tracker.get_error_summary_for_phase( + CreditPhase.PROFILING, phase_index=2 + ) + indexed_errors = {item.error_details: item.count for item in indexed_summary} + assert indexed_errors[request_error] == 1 + assert indexed_errors[ErrorDetails.from_exception(dispatch_error)] == 1 + @pytest.mark.asyncio async def test_successful_metric_dispatch_records_no_phase_error(self) -> None: manager = self._make_manager() @@ -199,6 +220,26 @@ async def test_successful_metric_dispatch_records_no_phase_error(self) -> None: == [] ) + @pytest.mark.asyncio + async def test_warmup_plus_single_profiling_does_not_build_phase_results( + self, + ) -> None: + manager = RecordsManager.__new__(RecordsManager) + manager.run = SimpleNamespace( + cfg=SimpleNamespace( + phases=[ + SimpleNamespace(kind="warmup"), + SimpleNamespace(kind="profiling"), + ] + ) + ) + + results = await RecordsManager._build_phase_profile_results( + manager, CreditPhase.PROFILING, cancelled=False + ) + + assert results is None + class TestRecordsManagerTimeslice: """ProfileResults stores accumulator-backed timeslices.""" diff --git a/tests/unit/timing/strategies/test_adaptive_scale.py b/tests/unit/timing/strategies/test_adaptive_scale.py index 9907577474..fcd5fabd38 100644 --- a/tests/unit/timing/strategies/test_adaptive_scale.py +++ b/tests/unit/timing/strategies/test_adaptive_scale.py @@ -37,6 +37,10 @@ def _strategy( ) -> AdaptiveScaleStrategy: config_kwargs = { "phase": CreditPhase.PROFILING, + "phase_index": 0, + "profiling_index": 0, + "phase_name": "profiling", + "phase_kind": "profiling", "timing_mode": TimingMode.ADAPTIVE_SCALE, "expected_duration_sec": 60.0, "concurrency": 10, @@ -75,6 +79,18 @@ def _strategy( return strategy +def _event_path(tmp_path: Path, phase_name: str = "profiling") -> Path: + return tmp_path / "phases" / phase_name / "adaptive_scale_events.jsonl" + + +def _summary_path(tmp_path: Path, phase_name: str = "profiling") -> Path: + return tmp_path / "phases" / phase_name / "adaptive_scale_summary.json" + + +def _manifest_path(tmp_path: Path) -> Path: + return tmp_path / "adaptive_scale_manifest.json" + + def _assert_event_clock_fields(event: dict) -> None: assert event["schema_version"] == 2 assert isinstance(event["timestamp"], int) @@ -89,6 +105,93 @@ def test_percentile_interpolates_p95() -> None: assert _percentile([10, 20, 30, 40, 50], 95) == pytest.approx(48.0) +@pytest.mark.asyncio +async def test_setup_phase_writes_phase_scoped_manifest(tmp_path: Path) -> None: + strategy = _strategy(tmp_path) + + await strategy.setup_phase() + await strategy._artifacts.flush() + await strategy._artifacts.close() + + manifest = orjson.loads(_manifest_path(tmp_path).read_bytes()) + assert manifest["adaptive_phases"] == [ + { + "events_path": "phases/profiling/adaptive_scale_events.jsonl", + "phase_index": 0, + "phase_kind": "profiling", + "phase_name": "profiling", + "profiling_index": 0, + "summary_path": "phases/profiling/adaptive_scale_summary.json", + } + ] + assert _event_path(tmp_path).exists() + assert not (tmp_path / "adaptive_scale_events.jsonl").exists() + assert not (tmp_path / "adaptive_scale_summary.json").exists() + + +def test_manifest_entries_are_sorted_and_replaced_by_phase_identity( + tmp_path: Path, +) -> None: + writer = AdaptiveScaleArtifactWriter() + writer._schedule_write = lambda write: write() + + writer.write_manifest_entry( + tmp_path, + { + "events_path": "phases/storm_1/adaptive_scale_events.jsonl", + "phase_index": 2, + "phase_kind": "profiling", + "phase_name": "storm_1", + "profiling_index": 1, + "summary_path": "phases/storm_1/adaptive_scale_summary.json", + }, + ) + writer.write_manifest_entry( + tmp_path, + { + "events_path": "phases/low/adaptive_scale_events.jsonl", + "phase_index": 1, + "phase_kind": "profiling", + "phase_name": "low", + "profiling_index": 0, + "summary_path": "phases/low/adaptive_scale_summary.json", + }, + ) + writer.write_manifest_entry( + tmp_path, + { + "events_path": "phases/storm_1/adaptive_scale_events.v2.jsonl", + "phase_index": 2, + "phase_kind": "profiling", + "phase_name": "storm_1", + "profiling_index": 1, + "summary_path": "phases/storm_1/adaptive_scale_summary.v2.json", + }, + ) + + manifest = orjson.loads(_manifest_path(tmp_path).read_bytes()) + + assert manifest["schema_version"] == 2 + assert manifest["adaptive_phases"] == [ + { + "events_path": "phases/low/adaptive_scale_events.jsonl", + "phase_index": 1, + "phase_kind": "profiling", + "phase_name": "low", + "profiling_index": 0, + "summary_path": "phases/low/adaptive_scale_summary.json", + }, + { + "events_path": "phases/storm_1/adaptive_scale_events.v2.jsonl", + "phase_index": 2, + "phase_kind": "profiling", + "phase_name": "storm_1", + "profiling_index": 1, + "summary_path": "phases/storm_1/adaptive_scale_summary.v2.json", + }, + ] + + @pytest.mark.asyncio async def test_handle_credit_result_buffers_record_request_latency(tmp_path) -> None: strategy = _strategy(tmp_path) @@ -216,12 +319,9 @@ def test_discover_scales_up_and_writes_event(tmp_path) -> None: strategy._assess_discover(10.0, True, stats) - strategy._concurrency_manager.set_session_limit.assert_called_with( - CreditPhase.PROFILING, 10 - ) + strategy._concurrency_manager.set_session_limit.assert_called_with(0, 10) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_decision" assert events[-1]["control_value_before"] == 2 @@ -245,9 +345,7 @@ def test_percent_step_policy_uses_current_concurrency_percent(tmp_path) -> None: strategy._assess_discover(10.0, True, stats) - strategy._concurrency_manager.set_session_limit.assert_called_with( - CreditPhase.PROFILING, 3 - ) + strategy._concurrency_manager.set_session_limit.assert_called_with(0, 3) def test_margin_step_is_capped_by_max_multiplier(tmp_path) -> None: @@ -622,8 +720,7 @@ async def test_adaptive_window_reports_itl_and_goodput_sla_values(tmp_path) -> N await strategy._assess_window() events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] window = next(event for event in events if event["event"] == "adaptive_window") assert window["sla_values"][strategy._sla_key(itl_sla)] == pytest.approx(10.0) @@ -640,12 +737,9 @@ def test_breach_enters_sustain_at_last_good_boundary(tmp_path) -> None: assert strategy._controller_phase == "sustain" assert strategy._boundary_concurrency == 4 - strategy._concurrency_manager.set_session_limit.assert_called_with( - CreditPhase.PROFILING, 4 - ) + strategy._concurrency_manager.set_session_limit.assert_called_with(0, 4) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert [event["event"] for event in events] == [ "sustain_started", @@ -668,11 +762,10 @@ def test_sustain_completion_writes_complete_event_and_summary(tmp_path) -> None: strategy._assess_sustain(50.0, True, stats) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_complete" - summary = orjson.loads((tmp_path / "adaptive_scale_summary.json").read_bytes()) + summary = orjson.loads(_summary_path(tmp_path).read_bytes()) assert summary["schema_version"] == 2 assert summary["status"] == "completed" assert summary["boundary_value"] == 4 @@ -714,12 +807,11 @@ def test_execute_finalizer_writes_summary_when_phase_stops_before_boundary( strategy._complete_controller(reason="phase_stopped") events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_complete" assert events[-1]["reason"] == "phase_stopped" - summary = orjson.loads((tmp_path / "adaptive_scale_summary.json").read_bytes()) + summary = orjson.loads(_summary_path(tmp_path).read_bytes()) assert summary["status"] == "completed" assert summary["boundary_value"] is None assert summary["result"]["boundary_value"] is None @@ -736,12 +828,9 @@ def test_all_failed_discover_window_enters_sustain(tmp_path) -> None: assert strategy._controller_phase == "sustain" assert strategy._boundary_concurrency == 4 - strategy._concurrency_manager.set_session_limit.assert_called_with( - CreditPhase.PROFILING, 4 - ) + strategy._concurrency_manager.set_session_limit.assert_called_with(0, 4) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert [event["event"] for event in events] == [ "sustain_started", @@ -758,13 +847,12 @@ def test_minimum_breach_fails_without_sustainable_concurrency(tmp_path) -> None: strategy._assess_discover(150.0, False, stats) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_failed" assert events[-1]["reason"] == "no_sustainable_concurrency_found" assert events[-1]["first_failing_value"] == 2 - summary = orjson.loads((tmp_path / "adaptive_scale_summary.json").read_bytes()) + summary = orjson.loads(_summary_path(tmp_path).read_bytes()) assert summary["status"] == "failed" assert summary["completed_reason"] == "no_sustainable_concurrency_found" assert summary["result"] == { @@ -782,8 +870,7 @@ def test_max_concurrency_passing_is_incomplete_not_boundary(tmp_path) -> None: strategy._assess_discover(50.0, True, stats) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_incomplete" assert events[-1]["reason"] == "max_control_value_reached_without_saturation" @@ -799,8 +886,7 @@ async def test_sparse_window_is_inconclusive(tmp_path) -> None: await strategy._assess_window() events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_window" assert events[-1]["adaptive_iteration"] == 0 @@ -916,12 +1002,9 @@ async def test_setup_phase_sets_initial_concurrency_and_event(tmp_path) -> None: await strategy.setup_phase() - strategy._concurrency_manager.set_session_limit.assert_called_with( - CreditPhase.PROFILING, 2 - ) + strategy._concurrency_manager.set_session_limit.assert_called_with(0, 2) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_phase_started" assert events[-1]["control_value"] == 2 @@ -944,11 +1027,10 @@ async def fail_window() -> None: assert strategy._completed_reason == "assessment_failed: bad window" strategy._lifecycle.cancel.assert_called_once() events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_failed" - summary = orjson.loads((tmp_path / "adaptive_scale_summary.json").read_bytes()) + summary = orjson.loads(_summary_path(tmp_path).read_bytes()) assert summary["status"] == "failed" assert summary["completed_reason"] == "assessment_failed: bad window" @@ -964,8 +1046,7 @@ async def test_assess_window_evaluates_sustain_phase(tmp_path) -> None: await strategy._assess_window() events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert {event["adaptive_iteration"] for event in events} == {0} assert strategy._adaptive_iteration == 1 @@ -987,8 +1068,7 @@ async def test_assess_window_all_failed_without_boundary_fails(tmp_path) -> None ) assert strategy._progress.all_credits_sent_event.is_set() events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-2]["reason"] == "no successful requests in assessment window" assert events[-1]["event"] == "adaptive_failed" @@ -1002,8 +1082,7 @@ async def test_cancellation_only_window_fails_and_reports_cancelled(tmp_path) -> await strategy._assess_window() events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-2]["event"] == "adaptive_window" assert events[-2]["reason"] == "no successful requests in assessment window" @@ -1011,7 +1090,7 @@ async def test_cancellation_only_window_fails_and_reports_cancelled(tmp_path) -> assert events[-2]["cancelled"] == 2 assert events[-1]["event"] == "adaptive_failed" assert events[-1]["cancelled"] == 2 - summary = orjson.loads((tmp_path / "adaptive_scale_summary.json").read_bytes()) + summary = orjson.loads(_summary_path(tmp_path).read_bytes()) assert summary["status"] == "failed" assert summary["totals"] == { "sent": 2, @@ -1030,12 +1109,9 @@ def test_all_failed_sustain_window_downshifts_with_reason(tmp_path) -> None: strategy._assess_failed_window(stats) - strategy._concurrency_manager.set_session_limit.assert_called_with( - CreditPhase.PROFILING, 4 - ) + strategy._concurrency_manager.set_session_limit.assert_called_with(0, 4) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["reason"] == "all requests failed in assessment window" assert events[-1]["step_size"] == 2 @@ -1055,11 +1131,10 @@ def test_second_sustain_breach_after_recovery_fails(tmp_path) -> None: assert strategy._completed_reason == "sustain_failed_after_recovery" events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["event"] == "adaptive_failed" - summary = orjson.loads((tmp_path / "adaptive_scale_summary.json").read_bytes()) + summary = orjson.loads(_summary_path(tmp_path).read_bytes()) assert summary["status"] == "failed" assert summary["completed_reason"] == "sustain_failed_after_recovery" @@ -1156,8 +1231,7 @@ def test_emit_event_preserves_zero_candidate_value(tmp_path) -> None: ) events = [ - orjson.loads(line) - for line in (tmp_path / "adaptive_scale_events.jsonl").read_text().splitlines() + orjson.loads(line) for line in _event_path(tmp_path).read_text().splitlines() ] assert events[-1]["candidate_value"] == 0 assert events[-1]["accepted_value"] == 5 @@ -1181,8 +1255,8 @@ def test_artifact_disabled_paths_do_not_write(tmp_path) -> None: strategy._complete_controller(reason="ignored") assert strategy._completed_reason == "done" - assert not (tmp_path / "adaptive_scale_events.jsonl").exists() - assert not (tmp_path / "adaptive_scale_summary.json").exists() + assert not _event_path(tmp_path).exists() + assert not _summary_path(tmp_path).exists() def test_percentile_empty_single_and_exact_rank() -> None: @@ -1331,9 +1405,7 @@ def test_control_backend_clamps_and_snapshots(tmp_path) -> None: strategy._set_control(99) assert strategy._control.current == 10 - strategy._concurrency_manager.set_session_limit.assert_called_with( - CreditPhase.PROFILING, 10 - ) + strategy._concurrency_manager.set_session_limit.assert_called_with(0, 10) assert strategy._control.snapshot() == {"target_value": 10, "actual_value": 10} diff --git a/tests/unit/timing/test_race_conditions.py b/tests/unit/timing/test_race_conditions.py index 315e96e720..d42093f0b2 100644 --- a/tests/unit/timing/test_race_conditions.py +++ b/tests/unit/timing/test_race_conditions.py @@ -190,6 +190,79 @@ def test_duplicate_completion_returns_false(self): CreditPhase.PROFILING ) + def test_same_kind_phases_track_counts_and_metadata_independently(self): + rt = RecordsTracker() + low = CreditPhaseStats( + phase=CreditPhase.PROFILING, + phase_index=0, + profiling_index=0, + phase_name="low", + phase_kind="profiling", + final_requests_completed=1, + start_ns=1000, + ) + storm = CreditPhaseStats( + phase=CreditPhase.PROFILING, + phase_index=1, + profiling_index=1, + phase_name="storm", + phase_kind="profiling", + final_requests_completed=2, + start_ns=2000, + ) + rt.update_phase_info(low) + rt.update_phase_info(storm) + rt._get_phase_tracker( + CreditPhase.PROFILING, phase_index=0 + ).increment_success_records() + for _ in range(2): + rt._get_phase_tracker( + CreditPhase.PROFILING, phase_index=1 + ).increment_success_records() + + assert rt.check_and_set_all_records_received_for_stats(low) + assert rt.check_and_set_all_records_received_for_stats(storm) + low_stats = rt.create_stats_for_phase(CreditPhase.PROFILING, phase_index=0) + storm_stats = rt.create_stats_for_phase(CreditPhase.PROFILING, phase_index=1) + + assert low_stats.total_records == 1 + assert low_stats.phase_name == "low" + assert low_stats.profiling_index == 0 + assert storm_stats.total_records == 2 + assert storm_stats.phase_name == "storm" + assert storm_stats.profiling_index == 1 + + def test_early_phase_only_stats_read_does_not_block_named_phase_completion(self): + rt = RecordsTracker() + + early_stats = rt.create_stats_for_phase(CreditPhase.PROFILING) + + assert early_stats.phase == CreditPhase.PROFILING + assert early_stats.total_records == 0 + assert rt._phase_trackers == {} + + for phase_index, phase_name in ((0, "low"), (1, "recovery")): + rt.update_phase_info( + CreditPhaseStats( + phase=CreditPhase.PROFILING, + phase_index=phase_index, + profiling_index=phase_index, + phase_name=phase_name, + phase_kind="profiling", + final_requests_completed=1, + start_ns=1000 + phase_index, + requests_end_ns=2000 + phase_index, + ) + ) + rt._get_phase_tracker( + CreditPhase.PROFILING, phase_index=phase_index + ).increment_success_records() + + assert rt.check_and_set_all_records_received_for_phase(CreditPhase.PROFILING) + aggregate = rt.create_aggregate_stats_for_phase(CreditPhase.PROFILING) + assert aggregate.final_requests_completed == 2 + assert aggregate.success_records == 2 + @pytest.mark.asyncio class TestStickyRouterWorkerRace: diff --git a/tests/unit/timing/test_timing_config.py b/tests/unit/timing/test_timing_config.py index e9789a9a12..5075e6e688 100644 --- a/tests/unit/timing/test_timing_config.py +++ b/tests/unit/timing/test_timing_config.py @@ -288,6 +288,13 @@ def test_maps_cancellation_fields(self) -> None: 25.0, 1.5, ) + profiling = next( + pc for pc in cfg.phase_configs if pc.phase == CreditPhase.PROFILING + ) + assert ( + profiling.request_cancellation.rate, + profiling.request_cancellation.delay, + ) == (25.0, 1.5) def test_uses_user_centric_rate_when_request_rate_is_none(self) -> None: # USER_CENTRIC mode requires multi-turn sessions; the v1 -> v2 resolver diff --git a/tests/unit/workers/test_inference_client_enrich_record.py b/tests/unit/workers/test_inference_client_enrich_record.py index fe36b7a340..5965cd3d9a 100644 --- a/tests/unit/workers/test_inference_client_enrich_record.py +++ b/tests/unit/workers/test_inference_client_enrich_record.py @@ -19,6 +19,10 @@ def _make_request_info(**overrides) -> RequestInfo: turn_index=0, x_request_id="r", x_correlation_id="x", + phase_index=2, + profiling_index=1, + phase_name="recovery", + phase_kind="profiling", agent_depth=2, parent_correlation_id="root", model_endpoint=ModelEndpointInfo.model_construct(), @@ -44,6 +48,20 @@ def test_dag_fields_propagate(self): assert enriched.request_info.agent_depth == 3 assert enriched.request_info.parent_correlation_id == "p" + def test_phase_identity_fields_propagate(self): + ri = _make_request_info( + phase_index=3, + profiling_index=2, + phase_name="storm", + phase_kind="profiling", + ) + record = RequestRecord() + enriched = InferenceClient._enrich_request_record(record, ri) + assert enriched.request_info.phase_index == 3 + assert enriched.request_info.profiling_index == 2 + assert enriched.request_info.phase_name == "storm" + assert enriched.request_info.phase_kind == "profiling" + def test_transport_extras_dropped(self): ri = _make_request_info() record = RequestRecord()