diff --git a/docs/resources/sample-stores.md b/docs/resources/sample-stores.md index d8173cc80..7402747f3 100644 --- a/docs/resources/sample-stores.md +++ b/docs/resources/sample-stores.md @@ -275,7 +275,7 @@ copyFrom: generatorIdentifier: "vllm-benchmark-run" identifierColumn: "config" experiments: - - experimentIdentifier: "test-deployment-v1" + - experimentIdentifier: "vllm-bench-deployment" actuatorIdentifier: "vllm_performance" # Specify the actual actuator propertyFormat: "target" # if the columns for observed properties use target property names or observed property names ``` diff --git a/docs/user-guide/actuators/vllm-performance.md b/docs/user-guide/actuators/vllm-performance.md index e5e4b4112..03c484134 100644 --- a/docs/user-guide/actuators/vllm-performance.md +++ b/docs/user-guide/actuators/vllm-performance.md @@ -50,41 +50,72 @@ The `vllm_performance` actuator implements twelve experiments: **Standard LLM Benchmarking:** -- `test-deployment-v1`: This experiment can test the full vLLM workload +- `vllm-bench-deployment`: This experiment can test the full vLLM workload configuration, including resource requests and server deployment configuration. It deploys servers with given configuration on kubernetes and runs vLLM's built-in benchmarking tool (`vllm bench serve`) on them with the given parameters. -- `test-endpoint-v1`: This experiment is equivalent to running +- `vllm-bench-endpoint`: This experiment is equivalent to running `vllm bench serve` against an endpoint. -- `test-deployment-guidellm-v1`: Similar to `test-deployment-v1`, but uses +- `guidellm-bench-deployment`: Similar to `vllm-bench-deployment`, but uses GuideLLM (`guidellm benchmark run`) for benchmarking instead of vLLM's built-in benchmarking tool. -- `test-endpoint-guidellm-v1`: Similar to `test-endpoint-v1`, but uses GuideLLM +- `guidellm-bench-endpoint`: Similar to `vllm-bench-endpoint`, but uses GuideLLM (`guidellm benchmark run`) for benchmarking instead of vLLM's built-in benchmarking tool. **Geospatial Model Benchmarking:** -- `test-geospatial-deployment-v1`: Deploy and benchmark geospatial models +- `geospatial-vllm-bench-deployment`: Deploy and benchmark geospatial models (IBM-NASA Prithvi) using pre-packaged datasets for flood detection tasks with vLLM's built-in benchmarking tool. -- `test-geospatial-endpoint-v1`: Benchmark existing geospatial model endpoints +- `geospatial-vllm-bench-endpoint`: Benchmark existing geospatial model endpoints using pre-packaged datasets with vLLM's built-in benchmarking tool. -- `test-geospatial-deployment-guidellm-v1`: Deploy and benchmark geospatial +- `geospatial-guidellm-bench-deployment`: Deploy and benchmark geospatial models using pre-packaged datasets with GuideLLM. -- `test-geospatial-endpoint-guidellm-v1`: Benchmark existing geospatial model +- `geospatial-guidellm-bench-endpoint`: Benchmark existing geospatial model endpoints using pre-packaged datasets with GuideLLM. -- `test-geospatial-deployment-custom-dataset-v1`: Deploy and benchmark +- `geospatial-vllm-bench-deployment-custom-dataset`: Deploy and benchmark geospatial models with custom datasets using vLLM's built-in benchmarking tool. -- `test-geospatial-endpoint-custom-dataset-v1`: Benchmark existing geospatial +- `geospatial-vllm-bench-endpoint-custom-dataset`: Benchmark existing geospatial model endpoints with custom datasets using vLLM's built-in benchmarking tool. -- `test-geospatial-deployment-guidellm-custom-dataset-v1`: Deploy and benchmark +- `geospatial-guidellm-bench-deployment-custom-dataset`: Deploy and benchmark geospatial models with custom datasets using GuideLLM. -- `test-geospatial-endpoint-guidellm-custom-dataset-v1`: Benchmark existing +- `geospatial-guidellm-bench-endpoint-custom-dataset`: Benchmark existing geospatial model endpoints with custom datasets using GuideLLM. +> [!NOTE] Deprecated experiment names +> +> Earlier versions of the actuator used different names for the same experiments. +> The behaviour and output are identical — only the names changed. If you have +> existing discoveryspace or operation YAML files that reference the old names, +> update them to the current names shown above. +> +> **Standard LLM experiments:** +> +> | Deprecated name | Current name | +> | --- | --- | +> | `test-deployment-v1` | `vllm-bench-deployment` | +> | `test-endpoint-v1` | `vllm-bench-endpoint` | +> | `test-deployment-guidellm-v1` | `guidellm-bench-deployment` | +> | `test-endpoint-guidellm-v1` | `guidellm-bench-endpoint` | +> +> **Geospatial experiments:** +> +> | Deprecated name | Current name | +> | --- | --- | +> | `test-geospatial-deployment-v1` | `geospatial-vllm-bench-deployment` | +> | `test-geospatial-endpoint-v1` | `geospatial-vllm-bench-endpoint` | +> | `test-geospatial-deployment-guidellm-v1` | `geospatial-guidellm-bench-deployment` | +> | `test-geospatial-endpoint-guidellm-v1` | `geospatial-guidellm-bench-endpoint` | +> | `test-geospatial-deployment-custom-dataset-v1` | `geospatial-vllm-bench-deployment-custom-dataset` | +> | `test-geospatial-endpoint-custom-dataset-v1` | `geospatial-vllm-bench-endpoint-custom-dataset` | +> | `test-geospatial-deployment-guidellm-custom-dataset-v1` | `geospatial-guidellm-bench-deployment-custom-dataset` | +> | `test-geospatial-endpoint-guidellm-custom-dataset-v1` | `geospatial-guidellm-bench-endpoint-custom-dataset` | + + + > [!NOTE] Threadpool Support for Geospatial Models > > Geospatial experiments support threadpool pre/post processing for improved @@ -149,7 +180,7 @@ entity: request_rate: 50 experiments: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-endpoint-v1 + experimentIdentifier: vllm-bench-endpoint ``` Then run: @@ -195,7 +226,7 @@ entity: request_rate: 10 experiments: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-deployment-v1 + experimentIdentifier: vllm-bench-deployment ``` Then run: @@ -281,8 +312,8 @@ ado create actuatorconfiguration -f vllm_config.yaml > [!WARNING] GPU type > > The GPU type to use in an experiment is set via the experiment itself -> (test-deployment-v1). **Do not** set this via the `node_selector` parameter of -> the configuration. +> (`vllm-bench-deployment`). **Do not** set this via the `node_selector` +> parameter of the configuration. @@ -339,10 +370,10 @@ experiments can execute even if these packages are not pre-installed in your Ray cluster or local environment. The actuator intelligently selects which tool to install based on the experiment type: -- For experiments using vLLM's built-in benchmarking (`test-deployment-v1`, - `test-endpoint-v1`), it installs `ado-vllm-performance[vllm]` -- For experiments using GuideLLM (`test-deployment-guidellm-v1`, - `test-endpoint-guidellm-v1`), it installs `ado-vllm-performance[guidellm]` +- For experiments using vLLM's built-in benchmarking (`vllm-bench-deployment`, + `vllm-bench-endpoint`), it installs `ado-vllm-performance[vllm]` +- For experiments using GuideLLM (`guidellm-bench-deployment`, + `guidellm-bench-endpoint`), it installs `ado-vllm-performance[guidellm]` > [!NOTE] Deployment vs. Benchmark environment > @@ -399,7 +430,7 @@ parameters: The `in_cluster` option in your `actuatorconfiguration` tells the `vllm_performance` actuator how to communicate with the target Kubernetes or -OpenShift cluster when running `test-deployment-v1`. +OpenShift cluster when running `vllm-bench-deployment`. If running `ado` from outside the Kubernetes/OpenShift cluster where the deployments will be created, leave `in_cluster: false` (the default). diff --git a/docs/user-guide/examples/example_yamls/geospatial_flood_detection_space.yaml b/docs/user-guide/examples/example_yamls/geospatial_flood_detection_space.yaml index e21d73cff..df4a4407e 100644 --- a/docs/user-guide/examples/example_yamls/geospatial_flood_detection_space.yaml +++ b/docs/user-guide/examples/example_yamls/geospatial_flood_detection_space.yaml @@ -45,4 +45,4 @@ entitySpace: # values: [32, 64] measurementSpace: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-geospatial-deployment-v1 + experimentIdentifier: geospatial-vllm-bench-deployment diff --git a/docs/user-guide/examples/index.md b/docs/user-guide/examples/index.md index ff103eac0..177423ae2 100644 --- a/docs/user-guide/examples/index.md +++ b/docs/user-guide/examples/index.md @@ -73,8 +73,8 @@ to full Kubernetes/OpenShift deployments and specialised geospatial models. - **[Exploring vLLM deployment configurations](vllm-performance-full.md)** — Evaluate different vLLM server deployment configurations (GPU type, batch size, memory limits) on Kubernetes/OpenShift by combining the - `test-deployment-v1` experiment with the `random_walk` operator. + `vllm-bench-deployment` experiment with the `random_walk` operator. - **[Benchmarking geospatial models with vLLM](vllm-performance-geospatial.md)** — Benchmark IBM-NASA Prithvi geospatial models for Earth observation tasks (flood detection, land-use classification) using the - `test-geospatial-deployment-v1` experiment. + `geospatial-vllm-bench-deployment` experiment. diff --git a/docs/user-guide/examples/vllm-performance-endpoint.md b/docs/user-guide/examples/vllm-performance-endpoint.md index 29cfcb8fb..ffc3952f6 100644 --- a/docs/user-guide/examples/vllm-performance-endpoint.md +++ b/docs/user-guide/examples/vllm-performance-endpoint.md @@ -92,7 +92,7 @@ entitySpace: interval: 1 experiments: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-endpoint-v1 + experimentIdentifier: vllm-bench-endpoint ``` Create the space with: @@ -227,7 +227,7 @@ and the best region is unlikely to be visited. - Try running the same operation with the [GuideLLM](https://github.com/vllm-project/guidellm) benchmarking tool by setting the `experimentIdentifier` field in the entity space definition to - `test-endpoint-guidellm-v1`. + `guidellm-bench-endpoint`. - Use `ado describe experiment vllm_performance_endpoint` to see what other parameters can be explored - Try varying **`burstiness`** or **`number_input_tokens`**, or adding them as diff --git a/docs/user-guide/examples/vllm-performance-full.md b/docs/user-guide/examples/vllm-performance-full.md index 8d78562eb..b82bf4fb0 100644 --- a/docs/user-guide/examples/vllm-performance-full.md +++ b/docs/user-guide/examples/vllm-performance-full.md @@ -16,7 +16,7 @@ > explore the deployment parameter space. In this example: > > - We will define a space of vLLM deployment configurations to test with the -> `vllm_performance` actuator's `test-deployment-v1` experiment +> `vllm_performance` actuator's `vllm-bench-deployment` experiment > - This experiment can create and characterize a vLLM deployment on > Kubernetes > - Use the [`random_walk` operator](../operators/random-walk.md) to explore the @@ -205,7 +205,7 @@ ado show measurements space --output csv --use-latest > entities.csv - Try running the same operation with the [GuideLLM](https://github.com/vllm-project/guidellm) benchmarking tool by setting the `experimentIdentifier` field in the entity space definition to - `test-deployment-guidellm-v1`. + `guidellm-bench-deployment`. - Try varying **`max_batch_tokens`** or **`gpu_memory_utilization`** to explore the impact on throughput. - Try creating a different `actuatorconfiguration` with more `max_environments` diff --git a/docs/user-guide/examples/vllm-performance-geospatial.md b/docs/user-guide/examples/vllm-performance-geospatial.md index 5e87675c8..df7afcf0b 100644 --- a/docs/user-guide/examples/vllm-performance-geospatial.md +++ b/docs/user-guide/examples/vllm-performance-geospatial.md @@ -19,7 +19,7 @@ > In this example: > > - We will define a space of geospatial model deployment configurations to test -> - Use the `test-geospatial-deployment-v1` experiment to create and benchmark +> - Use the `geospatial-vllm-bench-deployment` experiment to create and benchmark > vLLM deployments serving Prithvi models > - Explore how deployment parameters affect inference latency for flood > detection tasks @@ -65,9 +65,9 @@ experiments: ado get experiments --details ``` -You should see experiments including `test-geospatial-deployment-v1`, -`test-geospatial-endpoint-v1`, `test-geospatial-deployment-custom-dataset-v1`, -and `test-geospatial-endpoint-custom-dataset-v1`. +You should see experiments including `geospatial-vllm-bench-deployment`, +`geospatial-vllm-bench-endpoint`, `geospatial-vllm-bench-deployment-custom-dataset`, +and `geospatial-vllm-bench-endpoint-custom-dataset`. ## Create an actuator configuration @@ -212,7 +212,7 @@ Prithvi-EO-2.0 flood detection models. ## Using Custom Datasets To use your own geospatial datasets, use the -`test-geospatial-deployment-custom-dataset-v1` experiment. Your dataset should +`geospatial-vllm-bench-deployment-custom-dataset` experiment. Your dataset should be a JSONL (JSON Lines) file where each line is a JSON object with this structure: @@ -251,7 +251,7 @@ Update your space definition to use the custom dataset experiment: ```yaml measurementSpace: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-geospatial-deployment-custom-dataset-v1 + experimentIdentifier: geospatial-vllm-bench-deployment-custom-dataset ``` And add the dataset path to your entity space: @@ -298,7 +298,7 @@ versions (0.20.0+) and fail early with a clear error if there's a mismatch. - Try the **600M parameter Prithvi model** by changing the model identifier to `ibm-nasa-geospatial/Prithvi-EO-2.0-600M-TL-Sen1Floods11` - Explore different **GPU types** if your cluster has multiple options -- Test **endpoint benchmarking** with `test-geospatial-endpoint-v1` if you have +- Test **endpoint benchmarking** with `geospatial-vllm-bench-endpoint` if you have an existing deployment - **Enable threadpool rendering** with vLLM 0.20.0+ to improve inference performance for geospatial models diff --git a/plugins/actuators/vllm_performance/README.md b/plugins/actuators/vllm_performance/README.md index db57a31f5..3a732aa5f 100644 --- a/plugins/actuators/vllm_performance/README.md +++ b/plugins/actuators/vllm_performance/README.md @@ -10,12 +10,11 @@ The actuator implements a set of functionalities to deploy and run serving benchmarks for different LLMs for vLLM. This actuator deploys [vLLM](https://github.com/vllm-project/vllm) on to an [OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift) -cluster to serve -[IBM Granite-3.3-8b](https://huggingface.co/ibm-granite/granite-3.3-8b-instruct) -and runs an experiment that utilises the -[vLLM serving benchmark](https://docs.vllm.ai/en/stable/api/vllm/benchmarks/serve.html). -The the actuator is named `vllm_performance` and features two experiments: -`performance-testing-full` and `performance-testing-endpoint`. +cluster and runs benchmarks using either vLLM's built-in benchmarking tool +([`vllm bench serve`](https://docs.vllm.ai/en/stable/api/vllm/benchmarks/serve.html)) +or [GuideLLM](https://github.com/vllm-project/guidellm). The actuator is named +`vllm_performance` and provides thirteen experiments spanning standard LLM +benchmarking, agentic tool-calling, and geospatial model benchmarking. # Getting Started @@ -67,19 +66,19 @@ pip install ado-vllm-performance This will automatically install both vLLM and GuideLLM benchmarking tools, enabling all experiments: -- `test-deployment-v1` and `test-endpoint-v1` (vLLM benchmarks) -- `test-deployment-guidellm-v1` and `test-endpoint-guidellm-v1` (GuideLLM +- `vllm-bench-deployment` and `vllm-bench-endpoint` (vLLM benchmarks) +- `guidellm-bench-deployment` and `guidellm-bench-endpoint` (GuideLLM benchmarks) -- `test-geospatial-deployment-v1` and `test-geospatial-endpoint-v1` (Geospatial - model benchmarks with vLLM) -- `test-geospatial-deployment-guidellm-v1` and - `test-geospatial-endpoint-guidellm-v1` (Geospatial model benchmarks with +- `geospatial-vllm-bench-deployment` and `geospatial-vllm-bench-endpoint` + (Geospatial model benchmarks with vLLM) +- `geospatial-guidellm-bench-deployment` and + `geospatial-guidellm-bench-endpoint` (Geospatial model benchmarks with GuideLLM) -- `test-geospatial-deployment-custom-dataset-v1` and - `test-geospatial-endpoint-custom-dataset-v1` (Geospatial with custom datasets - using vLLM) -- `test-geospatial-deployment-guidellm-custom-dataset-v1` and - `test-geospatial-endpoint-guidellm-custom-dataset-v1` (Geospatial with custom +- `geospatial-vllm-bench-deployment-custom-dataset` and + `geospatial-vllm-bench-endpoint-custom-dataset` (Geospatial with custom + datasets using vLLM) +- `geospatial-guidellm-bench-deployment-custom-dataset` and + `geospatial-guidellm-bench-endpoint-custom-dataset` (Geospatial with custom datasets using GuideLLM) **For development from source:** @@ -97,47 +96,47 @@ git clone https://github.com/IBM/ado.git Confirm that the actuator is installed: ```commandline -ado get experiments --details +ado get actuators --details ``` -You should see an output like below: - - +You should see `vllm_performance` in the list. To confirm all experiments are +registered, run: ```commandline -┌──────────────────┬─────────────────────────────┬───────────────────────────────────────────────────────────────────┐ -│ ACTUATOR ID │ EXPERIMENT ID │ DESCRIPTION │ -├──────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────┤ -│ mock │ test-experiment │ │ -│ mock │ test-experiment-two │ │ -│ vllm_performance │ test-deployment-guidellm-v1 │ VLLM performance testing using GuideLLM benchmark suite across │ -│ │ │ compute resource and workload configuration │ -│ vllm_performance │ test-deployment-v1 │ VLLM performance testing across compute resource and workload │ -│ │ │ configuration │ -│ vllm_performance │ test-endpoint-guidellm-v1 │ Test inference performance of a model served by vLLM endpoint │ -│ │ │ using GuideLLM benchmark suite across inference workload │ -│ │ │ configurations │ -│ vllm_performance │ test-endpoint-v1 │ Test inference performance of a model served by vLLM endpoint │ -│ │ │ across inference workload configurations │ -└──────────────────┴─────────────────────────────┴───────────────────────────────────────────────────────────────────┘ +ado get experiments --details ``` - - -On the last two lines you can see the new actuator and the experiments. You can -understand the +The vllm_performance experiments you should see are: + +- `vllm-bench-deployment` +- `vllm-bench-endpoint` +- `guidellm-bench-deployment` +- `guidellm-bench-endpoint` +- `test-agentic-tool-calling` +- `geospatial-vllm-bench-deployment` +- `geospatial-vllm-bench-endpoint` +- `geospatial-guidellm-bench-deployment` +- `geospatial-guidellm-bench-endpoint` +- `geospatial-vllm-bench-deployment-custom-dataset` +- `geospatial-vllm-bench-endpoint-custom-dataset` +- `geospatial-guidellm-bench-deployment-custom-dataset` +- `geospatial-guidellm-bench-endpoint-custom-dataset` + +You can understand the [constitutive properties required for the experiment](https://ibm.github.io/ado/core-concepts/actuators/#experiments) and the [target and observed properties](https://ibm.github.io/ado/core-concepts/actuators/#target-and-observed-properties) measured by an experiment by running: ```commandline -ado describe experiment test-deployment-v1 +ado describe experiment vllm-bench-deployment ``` -The experiment protocol for the vLLM actuator is defined in -[this YAML file](https://github.com/IBM/ado/blob/main/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments.yaml). -You will need to update this if you want to modify the values that can be +The experiment protocols for the vLLM actuator are defined in +[`ado_actuators/vllm_performance/experiments/performance_testing.yaml`](ado_actuators/vllm_performance/experiments/performance_testing.yaml) +and +[`ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml`](ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml). +You will need to update these if you want to modify the values that can be accepted as valid for the input properties. ### Configuring the actuator @@ -150,8 +149,7 @@ ado template actuatorconfiguration --actuator-identifier vllm_performance \ -o actuatorconfiguration.yaml ``` -This will create the `vllm_performance_actuatorconfiguration.yaml` file, which -will look like: +This will create the `actuatorconfiguration.yaml` file, which will look like: ```yaml actuatorIdentifier: vllm_performance @@ -162,12 +160,14 @@ metadata: parameters: benchmark_retries: 3 deployment_template: null - hf_token: "" - image_pull_secret_name: "" + developer_mode: false + hf_token: '' + image_pull_secret_name: '' in_cluster: false max_environments: 1 namespace: null node_selector: {} + otlp_traces_endpoint: null pvc_name: null pvc_template: null retries_timeout: 5 @@ -490,11 +490,11 @@ the set of values to expand the configuration space being studied. For example, you may want to benchmark a different LLM or you may want to change the GPU type to the one installed in your cluster. In the former case, you will -add values to `model_name` and in the latter case, you will have to modify the +add values to `model` and in the latter case, you will have to modify the domain of the `gpu_type` parameter to avoid validation errors. To do this, open the -[experiment definition YAML file](ado_actuators/vllm_performance/experiments.yaml) +[experiment definition YAML file](ado_actuators/vllm_performance/experiments/performance_testing.yaml) in a text editor, and add your GPU model to the list of values of `gpu_type`. Then, reinstall this actuator by running: @@ -505,7 +505,7 @@ pip install . After that, you can use the new value of `gpu_type` in your experiments. For example, in -[the sample space definition file](yamls/discoveryspace_override_defaults.yaml), +[the sample space definition file](yamls/discoveryspace_override_defaults_small.yaml), the location to update will be: ```yaml diff --git a/plugins/actuators/vllm_performance/VERSION b/plugins/actuators/vllm_performance/VERSION index 587c5f0c7..1cac385c6 100644 --- a/plugins/actuators/vllm_performance/VERSION +++ b/plugins/actuators/vllm_performance/VERSION @@ -1 +1 @@ -1.10.3 +1.11.0 diff --git a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py index 1c3ce6c7c..a7e8d7afc 100644 --- a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py +++ b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py @@ -251,11 +251,17 @@ async def submit( if experiment.identifier in [ "test-deployment-v1", + "vllm-bench-deployment", "test-deployment-guidellm-v1", + "guidellm-bench-deployment", "test-geospatial-deployment-v1", + "geospatial-vllm-bench-deployment", "test-geospatial-deployment-custom-dataset-v1", + "geospatial-vllm-bench-deployment-custom-dataset", "test-geospatial-deployment-guidellm-v1", + "geospatial-guidellm-bench-deployment", "test-geospatial-deployment-guidellm-custom-dataset-v1", + "geospatial-guidellm-bench-deployment-custom-dataset", "test-agentic-tool-calling", ]: if not self.env_manager: diff --git a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py index 3076c3823..c5f49256f 100644 --- a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py +++ b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py @@ -532,8 +532,11 @@ def run_resource_and_workload_experiment( result: BenchmarkResult if experiment.identifier in [ "test-geospatial-deployment-v1", + "geospatial-vllm-bench-deployment", "test-geospatial-deployment-custom-dataset-v1", + "geospatial-vllm-bench-deployment-custom-dataset", "test-geospatial-endpoint-custom-dataset-v1", + "geospatial-vllm-bench-endpoint-custom-dataset", ]: logger.info("Using geospatial benchmark for deployment") result = execute_geospatial_benchmark( @@ -550,7 +553,9 @@ def run_resource_and_workload_experiment( ) elif experiment.identifier in [ "test-geospatial-deployment-guidellm-v1", + "geospatial-guidellm-bench-deployment", "test-geospatial-deployment-guidellm-custom-dataset-v1", + "geospatial-guidellm-bench-deployment-custom-dataset", ]: logger.info("Using GuideLLM geospatial benchmark for deployment") result = execute_guidellm_geospatial_benchmark( @@ -565,7 +570,10 @@ def run_resource_and_workload_experiment( retries_timeout=actuator_parameters.retries_timeout, burstiness=benchmark_parameters.burstiness, ) - elif experiment.identifier == "test-deployment-guidellm-v1": + elif experiment.identifier in [ + "test-deployment-guidellm-v1", + "guidellm-bench-deployment", + ]: logger.info("Using GuideLLM benchmark for deployment") result = execute_guidellm_benchmark( base_url=benchmark_parameters.endpoint, @@ -715,7 +723,9 @@ def run_workload_experiment( result: BenchmarkResult if experiment.identifier in [ "test-geospatial-endpoint-v1", + "geospatial-vllm-bench-endpoint", "test-geospatial-endpoint-custom-dataset-v1", + "geospatial-vllm-bench-endpoint-custom-dataset", ]: logger.info("Using geospatial benchmark for endpoint") result = execute_geospatial_benchmark( @@ -732,7 +742,9 @@ def run_workload_experiment( ) elif experiment.identifier in [ "test-geospatial-endpoint-guidellm-v1", + "geospatial-guidellm-bench-endpoint", "test-geospatial-endpoint-guidellm-custom-dataset-v1", + "geospatial-guidellm-bench-endpoint-custom-dataset", ]: logger.info("Using GuideLLM geospatial benchmark for endpoint") result = execute_guidellm_geospatial_benchmark( @@ -747,7 +759,10 @@ def run_workload_experiment( retries_timeout=actuator_parameters.retries_timeout, burstiness=benchmark_parameters.burstiness, ) - elif experiment.identifier == "test-endpoint-guidellm-v1": + elif experiment.identifier in [ + "test-endpoint-guidellm-v1", + "guidellm-bench-endpoint", + ]: logger.info("Using GuideLLM benchmark for endpoint") result = execute_guidellm_benchmark( base_url=benchmark_parameters.endpoint, diff --git a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing.yaml b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing.yaml index 3af5734c3..902ae5ba7 100644 --- a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing.yaml +++ b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing.yaml @@ -5,6 +5,7 @@ test-deployment-v1: identifier: test-deployment-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values - identifier: 'model' metadata: @@ -234,6 +235,7 @@ test-deployment-v1: test-endpoint-v1: identifier: test-endpoint-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values - identifier: 'model' metadata: @@ -357,6 +359,7 @@ test-endpoint-v1: test-deployment-guidellm-v1: identifier: test-deployment-guidellm-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: - identifier: 'model' metadata: @@ -427,7 +430,7 @@ test-deployment-guidellm-v1: values: ["64Gi", "128Gi", "256Gi"] - identifier: dtype metadata: - description: '(deployment) data type for model weights and activations. "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' + description: '(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' propertyDomain: variableType: "CATEGORICAL_VARIABLE_TYPE" values: ["auto", "half", "float16", "bfloat16", "float", "float32"] @@ -584,6 +587,7 @@ test-deployment-guidellm-v1: test-endpoint-guidellm-v1: identifier: test-endpoint-guidellm-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: - identifier: 'model' metadata: @@ -992,3 +996,707 @@ test-agentic-tool-calling: metadata: description: 'Test inference performance of an agent-style model deployed by vLLM across compute resource and workload configurations' version: 1.0.0 +vllm-bench-deployment: + identifier: vllm-bench-deployment + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values + - identifier: 'model' + metadata: + description: 'model to use for testing. Must be available through the huggingface hub (public or private). Private models require the hf_token field to be set in the actuator configuration.' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["meta-llama/Llama-3.1-8B-Instruct", "ibm-granite/granite-3.3-8b-instruct", "openai/gpt-oss-20b"] + - identifier: 'request_rate' + metadata: + description: "(benchmark) The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "(benchmark) The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "(benchmark) The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - 1.0 is a Poisson distribution with rate = request_rate. Others are gamma distributions with lambda = request_rate and shape = burstiness." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 10] + interval: 1 + - identifier: 'number_input_tokens' + metadata: + description: "(benchmark) The number of input tokens to send per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 128000] + interval: 1 + - identifier: 'max_output_tokens' + metadata: + description: "(benchmark) The maximum number of output tokens to generate per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10000] + interval: 1 + - identifier: image + metadata: + description: "(deployment) Docker image to use to create vllm deployments" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["vllm/vllm-openai:v0.14.0"] + - identifier: n_cpus + metadata: + description: "(deployment) the number of CPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 256] + interval: 1 + - identifier: memory + metadata: + description: "(deployment) the amount of memory to allocate to vLLM pod" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["64Gi", "128Gi", "256Gi"] + - identifier: dtype + metadata: + description: "(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models." + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["auto", "half", "float16", "bfloat16", "float", "float32"] + - identifier: 'gpu_memory_utilization' + metadata: + description: "(deployment) The fraction of GPU memory to be used for the model executor," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [.5, .75, .9] + - identifier: 'cpu_offload' + metadata: + description: "(deployment) The amount of model weights in GB to offload to the CPU per GPU. 0 means all weights are on GPU," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [0, 8, 16, 24, 32] + - identifier: 'max_batch_tokens' + metadata: + description: "(deployment) maximum number of batched tokens per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [8192, 32769] + interval: 1024 + - identifier: 'max_num_seq' + metadata: + description: "(deployment) Maximum number of sequences per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [32, 2049] + interval: 32 + - identifier: 'n_gpus' + metadata: + description: "(deployment) Number of GPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 9] + interval: 1 + - identifier: 'gpu_type' + metadata: + description: "(deployment) The GPU type to use" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ['NVIDIA-A100-80GB-PCIe', 'NVIDIA-A100-SXM4-80GB'] + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to use for the benchmark" + propertyDomain: + variableType: 'CATEGORICAL_VARIABLE_TYPE' + values: ['random'] + - identifier: 'kv_cache_dtype' + metadata: + description: "(deployment) KV cache data type for vLLM (e.g. fp8, turboquant_k8v4). If not set, vLLM uses the default." + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["fp8"] + - identifier: 'enable_prefix_caching' + metadata: + description: "(deployment) When 1, enables prefix caching in vLLM. If not set (0), prefix caching is not enabled." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [0, 1] + defaultParameterization: + - property: + identifier: 'image' + value: "vllm/vllm-openai:v0.14.0" + - property: + identifier: n_cpus + value: 8 + - property: + identifier: memory + value: "128Gi" + - property: + identifier: dtype + value: "auto" + - property: + identifier: 'num_prompts' + value: 500 + - property: + identifier: 'max_concurrency' + value: -1 + - property: + identifier: 'burstiness' + value: 1.0 + - property: + identifier: 'number_input_tokens' + value: 1024 + - property: + identifier: 'max_output_tokens' + value: 128 + - property: + identifier: 'dataset' + value: 'random' + - property: + identifier: 'gpu_memory_utilization' + value: .9 + - property: + identifier: 'cpu_offload' + value: 0 + - property: + identifier: 'max_batch_tokens' + value: 16384 + - property: + identifier: 'max_num_seq' + value: 256 + - property: + identifier: 'n_gpus' + value: 1 + - property: + identifier: 'gpu_type' + value: 'NVIDIA-A100-80GB-PCIe' + - property: + identifier: 'kv_cache_dtype' + value: null + - property: + identifier: 'enable_prefix_caching' + value: 0 + # measurements + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "total_input_tokens" + - identifier: "total_output_tokens" + - identifier: "request_throughput" + - identifier: "output_throughput" + - identifier: "total_token_throughput" + - identifier: "mean_ttft_ms" + - identifier: "median_ttft_ms" + - identifier: "std_ttft_ms" + - identifier: "p25_ttft_ms" + - identifier: "p50_ttft_ms" + - identifier: "p75_ttft_ms" + - identifier: "p99_ttft_ms" + - identifier: "mean_tpot_ms" + - identifier: "median_tpot_ms" + - identifier: "std_tpot_ms" + - identifier: "p25_tpot_ms" + - identifier: "p50_tpot_ms" + - identifier: "p75_tpot_ms" + - identifier: "p99_tpot_ms" + - identifier: "mean_itl_ms" + - identifier: "median_itl_ms" + - identifier: "std_itl_ms" + - identifier: "p25_itl_ms" + - identifier: "p50_itl_ms" + - identifier: "p75_itl_ms" + - identifier: "p99_itl_ms" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'VLLM performance testing across compute resource and workload configuration' +vllm-bench-endpoint: + identifier: vllm-bench-endpoint + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["meta-llama/Llama-3.1-8B-Instruct", "ibm-granite/granite-3.3-8b-instruct", "openai/gpt-oss-20b"] + - identifier: 'endpoint' + metadata: + description: 'The endpoint(s) to test' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["http://localhost:8000"] + - identifier: 'request_rate' + metadata: + description: "The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'number_input_tokens' + metadata: + description: "The number of input tokens to send per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 128000] + interval: 1 # -1 means send all requests at time 0 + - identifier: 'max_output_tokens' + metadata: + description: "The maximum number of output tokens to generate per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10000] + interval: 1 # -1 means send all requests at time 0 + - identifier: 'burstiness' + metadata: + description: "The burstiness of the requests - 1.0 is a Poisson distribution with rate = request_rate. Others are gamma distributions with lambda = request_rate and shape = burstiness." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 10] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'dataset' + metadata: + description: "The dataset to use for the benchmark" + propertyDomain: + variableType: 'CATEGORICAL_VARIABLE_TYPE' + values: ['random'] + defaultParameterization: + - value: 1000 + property: + identifier: 'num_prompts' + - value: -1 + property: + identifier: 'max_concurrency' + - value: 1.0 + property: + identifier: 'burstiness' + - value: 1024 + property: + identifier: 'number_input_tokens' + - value: 128 + property: + identifier: 'max_output_tokens' + - value: 'random' + property: + identifier: "dataset" + # measurements + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "total_input_tokens" + - identifier: "total_output_tokens" + - identifier: "request_throughput" + - identifier: "output_throughput" + - identifier: "total_token_throughput" + - identifier: "mean_ttft_ms" + - identifier: "median_ttft_ms" + - identifier: "std_ttft_ms" + - identifier: "p25_ttft_ms" + - identifier: "p50_ttft_ms" + - identifier: "p75_ttft_ms" + - identifier: "p99_ttft_ms" + - identifier: "mean_tpot_ms" + - identifier: "median_tpot_ms" + - identifier: "std_tpot_ms" + - identifier: "p25_tpot_ms" + - identifier: "p50_tpot_ms" + - identifier: "p75_tpot_ms" + - identifier: "p99_tpot_ms" + - identifier: "mean_itl_ms" + - identifier: "median_itl_ms" + - identifier: "std_itl_ms" + - identifier: "p25_itl_ms" + - identifier: "p50_itl_ms" + - identifier: "p75_itl_ms" + - identifier: "p99_itl_ms" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'Test inference performance of a model served by vLLM endpoint across inference workload configurations' +guidellm-bench-deployment: + identifier: guidellm-bench-deployment + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: + - identifier: 'model' + metadata: + description: 'model to use for testing. Must be available through the huggingface hub (public or private). Private models require the hf_token field to be set in the actuator configuration.' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["meta-llama/Llama-3.1-8B-Instruct", "ibm-granite/granite-3.3-8b-instruct", "openai/gpt-oss-20b"] + - identifier: 'request_rate' + metadata: + description: "(benchmark) The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "(benchmark) The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "(benchmark) The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'number_input_tokens' + metadata: + description: "(benchmark) The number of input tokens to send per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 128000] + interval: 1 + - identifier: 'max_output_tokens' + metadata: + description: "(benchmark) The maximum number of output tokens to generate per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10000] + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - forced to 1.0 to match the meaning of burstiness=1 in vLLM experiments (Poisson distribution). GuideLLM uses poisson profile for this behavior." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [1.0] + - identifier: image + metadata: + description: "(deployment) Docker image to use to create vllm deployments" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["vllm/vllm-openai:v0.14.0"] + - identifier: n_cpus + metadata: + description: "(deployment) the number of CPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 256] + interval: 1 + - identifier: memory + metadata: + description: "(deployment) the amount of memory to allocate to vLLM pod" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["64Gi", "128Gi", "256Gi"] + - identifier: dtype + metadata: + description: '(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["auto", "half", "float16", "bfloat16", "float", "float32"] + - identifier: 'gpu_memory_utilization' + metadata: + description: "(deployment) The fraction of GPU memory to be used for the model executor," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [.5, .75, .9] + - identifier: 'cpu_offload' + metadata: + description: "(deployment) The amount of model weights in GB to offload to the CPU per GPU. 0 means all weights are on GPU," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [0, 8, 16, 24, 32] + - identifier: 'max_batch_tokens' + metadata: + description: "(deployment) maximum number of batched tokens per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [8192, 32769] + interval: 1024 + - identifier: 'max_num_seq' + metadata: + description: "(deployment) Maximum number of sequences per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [32, 2049] + interval: 32 + - identifier: 'n_gpus' + metadata: + description: "(deployment) Number of GPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 9] + interval: 1 + - identifier: 'gpu_type' + metadata: + description: "(deployment) The GPU type to use" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ['NVIDIA-A100-80GB-PCIe', 'NVIDIA-A100-SXM4-80GB'] + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to use for the benchmark" + propertyDomain: + variableType: 'CATEGORICAL_VARIABLE_TYPE' + values: ['random'] + - identifier: 'kv_cache_dtype' + metadata: + description: "(deployment) KV cache data type for vLLM (e.g. fp8, turboquant_k8v4). If not set, vLLM uses the default." + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["fp8"] + - identifier: 'enable_prefix_caching' + metadata: + description: "(deployment) When 1, enables prefix caching in vLLM. If not set (0), prefix caching is not enabled." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [0, 1] + defaultParameterization: + - property: + identifier: 'image' + value: "vllm/vllm-openai:v0.14.0" + - property: + identifier: n_cpus + value: 8 + - property: + identifier: memory + value: "128Gi" + - property: + identifier: dtype + value: "auto" + - property: + identifier: 'num_prompts' + value: 500 + - property: + identifier: 'max_concurrency' + value: -1 + - property: + identifier: 'number_input_tokens' + value: 1024 + - property: + identifier: 'max_output_tokens' + value: 128 + - property: + identifier: 'burstiness' + value: 1.0 + - property: + identifier: 'dataset' + value: 'random' + - property: + identifier: 'gpu_memory_utilization' + value: .9 + - property: + identifier: 'cpu_offload' + value: 0 + - property: + identifier: 'max_batch_tokens' + value: 16384 + - property: + identifier: 'max_num_seq' + value: 256 + - property: + identifier: 'n_gpus' + value: 1 + - property: + identifier: 'gpu_type' + value: 'NVIDIA-A100-80GB-PCIe' + - property: + identifier: 'kv_cache_dtype' + value: null + - property: + identifier: 'enable_prefix_caching' + value: 0 + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "total_input_tokens" + - identifier: "total_output_tokens" + - identifier: "request_throughput" + - identifier: "output_throughput" + - identifier: "total_token_throughput" + - identifier: "mean_ttft_ms" + - identifier: "median_ttft_ms" + - identifier: "std_ttft_ms" + - identifier: "p25_ttft_ms" + - identifier: "p50_ttft_ms" + - identifier: "p75_ttft_ms" + - identifier: "p99_ttft_ms" + - identifier: "mean_tpot_ms" + - identifier: "median_tpot_ms" + - identifier: "std_tpot_ms" + - identifier: "p25_tpot_ms" + - identifier: "p50_tpot_ms" + - identifier: "p75_tpot_ms" + - identifier: "p99_tpot_ms" + - identifier: "mean_itl_ms" + - identifier: "median_itl_ms" + - identifier: "std_itl_ms" + - identifier: "p25_itl_ms" + - identifier: "p50_itl_ms" + - identifier: "p75_itl_ms" + - identifier: "p99_itl_ms" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'VLLM performance testing using GuideLLM benchmark suite across compute resource and workload configuration' +guidellm-bench-endpoint: + identifier: guidellm-bench-endpoint + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["meta-llama/Llama-3.1-8B-Instruct", "ibm-granite/granite-3.3-8b-instruct", "openai/gpt-oss-20b"] + - identifier: 'endpoint' + metadata: + description: 'The endpoint(s) to test' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["http://localhost:8000"] + - identifier: 'request_rate' + metadata: + description: "The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'number_input_tokens' + metadata: + description: "The number of input tokens to send per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 128000] + interval: 1 + - identifier: 'max_output_tokens' + metadata: + description: "The maximum number of output tokens to generate per request" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10000] + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - forced to 1.0 to match the meaning of burstiness=1 in vLLM experiments (Poisson distribution). GuideLLM uses poisson profile for this behavior." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [1.0] + - identifier: 'max_concurrency' + metadata: + description: "The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to use for the benchmark" + propertyDomain: + variableType: 'CATEGORICAL_VARIABLE_TYPE' + values: ['random'] + defaultParameterization: + - value: 1000 + property: + identifier: 'num_prompts' + - value: -1 + property: + identifier: 'max_concurrency' + - value: 1024 + property: + identifier: 'number_input_tokens' + - value: 128 + property: + identifier: 'max_output_tokens' + - value: 1.0 + property: + identifier: 'burstiness' + - value: 'random' + property: + identifier: 'dataset' + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "total_input_tokens" + - identifier: "total_output_tokens" + - identifier: "request_throughput" + - identifier: "output_throughput" + - identifier: "total_token_throughput" + - identifier: "mean_ttft_ms" + - identifier: "median_ttft_ms" + - identifier: "std_ttft_ms" + - identifier: "p25_ttft_ms" + - identifier: "p50_ttft_ms" + - identifier: "p75_ttft_ms" + - identifier: "p99_ttft_ms" + - identifier: "mean_tpot_ms" + - identifier: "median_tpot_ms" + - identifier: "std_tpot_ms" + - identifier: "p25_tpot_ms" + - identifier: "p50_tpot_ms" + - identifier: "p75_tpot_ms" + - identifier: "p99_tpot_ms" + - identifier: "mean_itl_ms" + - identifier: "median_itl_ms" + - identifier: "std_itl_ms" + - identifier: "p25_itl_ms" + - identifier: "p50_itl_ms" + - identifier: "p75_itl_ms" + - identifier: "p99_itl_ms" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'Test inference performance of a model served by vLLM endpoint using GuideLLM benchmark suite across inference workload configurations' diff --git a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml index d4db6ded0..516e6f05a 100644 --- a/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml +++ b/plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml @@ -5,6 +5,7 @@ performance_testing-geospatial-endpoint: identifier: test-geospatial-endpoint-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values - identifier: 'model' metadata: @@ -83,6 +84,7 @@ performance_testing-geospatial-endpoint: performance_testing-geospatial-full: identifier: test-geospatial-deployment-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values - identifier: 'model' metadata: @@ -286,6 +288,7 @@ performance_testing-geospatial-full: performance_testing-geospatial-full-custom-dataset: identifier: test-geospatial-deployment-custom-dataset-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values - identifier: 'model' metadata: @@ -488,6 +491,7 @@ performance_testing-geospatial-full-custom-dataset: performance_testing-geospatial-endpoint-custom-dataset: identifier: test-geospatial-endpoint-custom-dataset-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values - identifier: 'model' metadata: @@ -563,6 +567,7 @@ performance_testing-geospatial-endpoint-custom-dataset: performance_testing-geospatial-endpoint-guidellm: identifier: test-geospatial-endpoint-guidellm-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: - identifier: 'model' metadata: @@ -639,6 +644,7 @@ performance_testing-geospatial-endpoint-guidellm: performance_testing-geospatial-full-guidellm: identifier: test-geospatial-deployment-guidellm-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: - identifier: 'model' metadata: @@ -701,7 +707,7 @@ performance_testing-geospatial-full-guidellm: values: ["64Gi", "128Gi", "256Gi"] - identifier: dtype metadata: - description: '(deployment) data type for model weights and activations. "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' + description: '(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' propertyDomain: variableType: "CATEGORICAL_VARIABLE_TYPE" values: ["auto", "half", "float16", "bfloat16", "float", "float32"] @@ -840,6 +846,7 @@ performance_testing-geospatial-full-guidellm: performance_testing-geospatial-guidellm-deployment-custom-dataset: identifier: test-geospatial-deployment-guidellm-custom-dataset-v1 actuatorIdentifier: "vllm_performance" + deprecated: true requiredProperties: - identifier: 'model' metadata: @@ -902,7 +909,7 @@ performance_testing-geospatial-guidellm-deployment-custom-dataset: values: ["64Gi", "128Gi", "256Gi"] - identifier: dtype metadata: - description: '(deployment) data type for model weights and activations. "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' + description: '(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' propertyDomain: variableType: "CATEGORICAL_VARIABLE_TYPE" values: ["auto", "half", "float16", "bfloat16", "float", "float32"] @@ -1040,6 +1047,1123 @@ performance_testing-geospatial-guidellm-deployment-custom-dataset: performance_testing-geospatial-guidellm-endpoint-custom-dataset: identifier: test-geospatial-endpoint-guidellm-custom-dataset-v1 actuatorIdentifier: "vllm_performance" + deprecated: true + requiredProperties: + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'endpoint' + metadata: + description: 'The endpoint(s) to test' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["http://localhost:8000"] + - identifier: 'request_rate' + metadata: + description: "The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to be used for the experiment" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["custom_dataset.jsonl"] + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'burstiness' + metadata: + description: "The burstiness of the requests - must be 1.0 for GuideLLM (Poisson distribution)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [1.0] + - identifier: 'max_concurrency' + metadata: + description: "The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + defaultParameterization: + - value: 100 + property: + identifier: 'num_prompts' + - value: -1 + property: + identifier: 'max_concurrency' + - value: 1.0 + property: + identifier: 'burstiness' + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'Test inference performance of a geospatial model served by vLLM endpoint using GuideLLM benchmark suite with custom dataset' +geospatial-vllm-bench-endpoint: + identifier: geospatial-vllm-bench-endpoint + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'endpoint' + metadata: + description: 'The endpoint(s) to test' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["http://localhost:8000"] + - identifier: 'request_rate' + metadata: + description: "The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'burstiness' + metadata: + description: "The burstiness of the requests - 1.0 is a Poisson distribution with rate = request_rate. Others are gamma distributions with lambda = request_rate and shape = burstiness." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 10] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'dataset' + metadata: + description: "The dataset to be used for the experiment" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['india_url_in_b64_out', 'valencia_url_in_b64_out', 'terramind_flood_url_in_b64_out'] + defaultParameterization: + - value: 100 + property: + identifier: 'num_prompts' + - value: -1 + property: + identifier: 'max_concurrency' + - value: 1.0 + property: + identifier: 'burstiness' + - property: + identifier: 'dataset' + value: 'india_url_in_b64_out' + # measurements + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'Test inference performance of a geospatial model served by vLLM endpoint across inference workload configurations' +geospatial-vllm-bench-deployment: + identifier: geospatial-vllm-bench-deployment + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'request_rate' + metadata: + description: "(benchmark) The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "(benchmark) The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "(benchmark) The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - 1.0 is a Poisson distribution with rate = request_rate. Others are gamma distributions with lambda = request_rate and shape = burstiness." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 10] + interval: 1 + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to be used for the experiment" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['india_url_in_b64_out', 'valencia_url_in_b64_out', 'terramind_flood_url_in_b64_out'] + - identifier: image + metadata: + description: "(deployment) Docker image to use to create vllm deployments" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7"] + - identifier: n_cpus + metadata: + description: "(deployment) the number of CPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 256] + interval: 1 + - identifier: memory + metadata: + description: "(deployment) the amount of memory to allocate to vLLM pod" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["64Gi", "128Gi", "256Gi"] + - identifier: dtype + metadata: + description: "(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models." + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["auto", "half", "float16", "bfloat16", "float", "float32"] + - identifier: 'gpu_memory_utilization' + metadata: + description: "(deployment) The fraction of GPU memory to be used for the model executor," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [.5, .75, .9] + - identifier: 'cpu_offload' + metadata: + description: "(deployment) The amount of model weights in GB to offload to the CPU per GPU. 0 means all weights are on GPU," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [0, 8, 16, 24, 32] + - identifier: 'max_num_seq' + metadata: + description: "(deployment) Maximum number of sequences per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [32, 2049] + interval: 32 + - identifier: 'max_batch_tokens' + metadata: + description: "(deployment) maximum number of batched tokens per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [256, 32769] + interval: 256 + - identifier: 'n_gpus' + metadata: + description: "(deployment) Number of GPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 9] + interval: 1 + - identifier: 'gpu_type' + metadata: + description: "(deployment) The GPU type to use" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['NVIDIA-A100-80GB-PCIe', 'NVIDIA-A100-SXM4-80GB'] + - identifier: 'skip_tokenizer_init' + metadata: + description: "(deployment) skip tokenizer initialization" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'enforce_eager' + metadata: + description: "(deployment) enforce pytorch eager mode" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'io_processor_plugin' + metadata: + description: 'IO Processor plugin to load for the model' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: [None, "terratorch_segmentation"] + - identifier: 'renderer_num_workers' + metadata: + description: "Number of renderer workers for threadpool-based rendering. Set to 0 to disable threadpool (for vLLM versions < 0.20.0). Set to a positive value to enable threadpool (requires vLLM >= 0.20.0). If not specified, threadpool is disabled." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 128] + interval: 1 + defaultParameterization: + - property: + identifier: 'image' + value: "quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7" + - property: + identifier: n_cpus + value: 8 + - property: + identifier: memory + value: "128Gi" + - property: + identifier: dtype + value: "auto" + - property: + identifier: 'num_prompts' + value: 500 + - property: + identifier: 'max_concurrency' + value: -1 + - property: + identifier: 'burstiness' + value: 1.0 + - property: + identifier: 'gpu_memory_utilization' + value: .9 + - property: + identifier: 'cpu_offload' + value: 0 + - property: + identifier: 'max_num_seq' + value: 256 + - property: + identifier: 'max_batch_tokens' + value: 16384 + - property: + identifier: 'n_gpus' + value: 1 + - property: + identifier: 'gpu_type' + value: 'NVIDIA-A100-80GB-PCIe' + - property: + identifier: 'skip_tokenizer_init' + value: 1 + - property: + identifier: 'enforce_eager' + value: 1 + - property: + identifier: 'io_processor_plugin' + value: "terratorch_segmentation" + - property: + identifier: 'dataset' + value: 'india_url_in_b64_out' + - property: + identifier: 'renderer_num_workers' + value: 0 + # measurements + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'VLLM performance testing across compute resource and workload configuration' +geospatial-vllm-bench-deployment-custom-dataset: + identifier: geospatial-vllm-bench-deployment-custom-dataset + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'request_rate' + metadata: + description: "(benchmark) The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to be used for the experiment" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["custom_dataset.jsonl"] + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "(benchmark) The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "(benchmark) The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - 1.0 is a Poisson distribution with rate = request_rate. Others are gamma distributions with lambda = request_rate and shape = burstiness." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 10] + interval: 1 + - identifier: image + metadata: + description: "(deployment) Docker image to use to create vllm deployments" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: + - ["quay.io/mgazzetti/vllm-performance/vllm:v0.18.0-tt.v1.2.5", "0.18.0"] + - ["quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7", "0.20.1"] + - identifier: n_cpus + metadata: + description: "(deployment) the number of CPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 256] + interval: 1 + - identifier: memory + metadata: + description: "(deployment) the amount of memory to allocate to vLLM pod" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["64Gi", "128Gi", "256Gi"] + - identifier: dtype + metadata: + description: "(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models." + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["auto", "half", "float16", "bfloat16", "float", "float32"] + - identifier: 'gpu_memory_utilization' + metadata: + description: "(deployment) The fraction of GPU memory to be used for the model executor," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [.5, .75, .9] + - identifier: 'cpu_offload' + metadata: + description: "(deployment) The amount of model weights in GB to offload to the CPU per GPU. 0 means all weights are on GPU," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [0, 8, 16, 24, 32] + - identifier: 'max_num_seq' + metadata: + description: "(deployment) Maximum number of sequences per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [32, 2049] + interval: 32 + - identifier: 'max_batch_tokens' + metadata: + description: "(deployment) maximum number of batched tokens per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [256, 32769] + interval: 256 + - identifier: 'n_gpus' + metadata: + description: "(deployment) Number of GPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 9] + interval: 1 + - identifier: 'gpu_type' + metadata: + description: "(deployment) The GPU type to use" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['NVIDIA-A100-80GB-PCIe', 'NVIDIA-A100-SXM4-80GB', 'NVIDIA-H100-PCIe'] + - identifier: 'skip_tokenizer_init' + metadata: + description: "(deployment) skip tokenizer initialization" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'enforce_eager' + metadata: + description: "(deployment) enforce PyTorch eager mode" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'io_processor_plugin' + metadata: + description: 'IO Processor plugin to load for the model' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["terratorch_segmentation"] + - identifier: 'renderer_num_workers' + metadata: + description: "Number of renderer workers for threadpool-based rendering. Set to 0 to disable threadpool (for vLLM versions < 0.20.0). Set to a positive value to enable threadpool (requires vLLM >= 0.20.0). If not specified, threadpool is disabled." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 128] + interval: 1 + defaultParameterization: + - property: + identifier: 'image' + value: "quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7" + - property: + identifier: n_cpus + value: 8 + - property: + identifier: memory + value: "128Gi" + - property: + identifier: dtype + value: "auto" + - property: + identifier: 'num_prompts' + value: 500 + - property: + identifier: 'max_concurrency' + value: -1 + - property: + identifier: 'burstiness' + value: 1.0 + - property: + identifier: 'gpu_memory_utilization' + value: .9 + - property: + identifier: 'cpu_offload' + value: 0 + - property: + identifier: 'max_num_seq' + value: 256 + - property: + identifier: 'max_batch_tokens' + value: 16384 + - property: + identifier: 'n_gpus' + value: 1 + - property: + identifier: 'gpu_type' + value: 'NVIDIA-A100-80GB-PCIe' + - property: + identifier: 'skip_tokenizer_init' + value: 1 + - property: + identifier: 'enforce_eager' + value: 1 + - property: + identifier: 'io_processor_plugin' + value: "terratorch_segmentation" + - property: + identifier: 'renderer_num_workers' + value: 0 + # measurements + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'VLLM performance testing across compute resource and workload configuration' +geospatial-vllm-bench-endpoint-custom-dataset: + identifier: geospatial-vllm-bench-endpoint-custom-dataset + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: # Any entity passed to this experiment must have constitutive properties with these values + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'endpoint' + metadata: + description: 'The endpoint(s) to test' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["http://localhost:8000"] + - identifier: 'request_rate' + metadata: + description: "The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to be used for the experiment" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["custom_dataset.jsonl"] + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'burstiness' + metadata: + description: "The burstiness of the requests - 1.0 is a Poisson distribution with rate = request_rate. Others are gamma distributions with lambda = request_rate and shape = burstiness." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 10] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + defaultParameterization: + - value: 100 + property: + identifier: 'num_prompts' + - value: -1 + property: + identifier: 'max_concurrency' + - value: 1.0 + property: + identifier: 'burstiness' + # measurements + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'Test inference performance of a geospatial model served by vLLM endpoint across inference workload configurations' +geospatial-guidellm-bench-endpoint: + identifier: geospatial-guidellm-bench-endpoint + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested.' + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'endpoint' + metadata: + description: 'The endpoint(s) to test' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["http://localhost:8000"] + - identifier: 'request_rate' + metadata: + description: "The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - forced to 1.0 to match the meaning of burstiness=1 in vLLM experiments (Poisson distribution). GuideLLM uses poisson profile for this behavior." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [1.0] + - identifier: 'max_concurrency' + metadata: + description: "The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] + interval: 1 + - identifier: 'dataset' + metadata: + description: "The dataset to be used for the experiment" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['india_url_in_b64_out', 'valencia_url_in_b64_out'] + defaultParameterization: + - value: 100 + property: + identifier: 'num_prompts' + - value: -1 + property: + identifier: 'max_concurrency' + - value: 1.0 + property: + identifier: 'burstiness' + - property: + identifier: 'dataset' + value: 'india_url_in_b64_out' + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'Test inference performance of a geospatial model served by vLLM endpoint using GuideLLM benchmark suite across inference workload configurations' +geospatial-guidellm-bench-deployment: + identifier: geospatial-guidellm-bench-deployment + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: + - identifier: 'model' + metadata: + description: 'model to use for testing' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'request_rate' + metadata: + description: "(benchmark) The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "(benchmark) The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "(benchmark) The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - forced to 1.0 to match the meaning of burstiness=1 in vLLM experiments (Poisson distribution). GuideLLM uses poisson profile for this behavior." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [1.0] + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to be used for the experiment" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['india_url_in_b64_out', 'valencia_url_in_b64_out'] + - identifier: image + metadata: + description: "(deployment) Docker image to use to create vllm deployments" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7"] + - identifier: n_cpus + metadata: + description: "(deployment) the number of CPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 256] + interval: 1 + - identifier: memory + metadata: + description: "(deployment) the amount of memory to allocate to vLLM pod" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["64Gi", "128Gi", "256Gi"] + - identifier: dtype + metadata: + description: '(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["auto", "half", "float16", "bfloat16", "float", "float32"] + - identifier: 'gpu_memory_utilization' + metadata: + description: "(deployment) The fraction of GPU memory to be used for the model executor" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [.5, .75, .9] + - identifier: 'cpu_offload' + metadata: + description: "(deployment) The amount of model weights in GB to offload to the CPU per GPU. 0 means all weights are on GPU" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [0, 8, 16, 24, 32] + - identifier: 'max_num_seq' + metadata: + description: "(deployment) Maximum number of sequences per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [32, 2049] + interval: 32 + - identifier: 'max_batch_tokens' + metadata: + description: "(deployment) maximum number of batched tokens per iteration" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [256, 32769] + interval: 256 + - identifier: 'n_gpus' + metadata: + description: "(deployment) Number of GPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 9] + interval: 1 + - identifier: 'gpu_type' + metadata: + description: "(deployment) The GPU type to use" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['NVIDIA-A100-80GB-PCIe', 'NVIDIA-A100-SXM4-80GB'] + - identifier: 'skip_tokenizer_init' + metadata: + description: "(deployment) skip tokenizer initialization" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'enforce_eager' + metadata: + description: "(deployment) enforce pytorch eager mode" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'io_processor_plugin' + metadata: + description: 'IO Processor plugin to load for the model' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: [None, "terratorch_segmentation"] + - identifier: 'renderer_num_workers' + metadata: + description: "Number of renderer workers for threadpool-based rendering. Set to 0 to disable threadpool (for vLLM versions < 0.20.0). Set to a positive value to enable threadpool (requires vLLM >= 0.20.0). If not specified, threadpool is disabled." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 128] + interval: 1 + defaultParameterization: + - property: + identifier: 'image' + value: "quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7" + - property: + identifier: n_cpus + value: 8 + - property: + identifier: memory + value: "128Gi" + - property: + identifier: dtype + value: "auto" + - property: + identifier: 'num_prompts' + value: 500 + - property: + identifier: 'max_concurrency' + value: -1 + - property: + identifier: 'burstiness' + value: 1.0 + - property: + identifier: 'gpu_memory_utilization' + value: .9 + - property: + identifier: 'cpu_offload' + value: 0 + - property: + identifier: 'max_num_seq' + value: 256 + - property: + identifier: 'max_batch_tokens' + value: 16384 + - property: + identifier: 'n_gpus' + value: 1 + - property: + identifier: 'gpu_type' + value: 'NVIDIA-A100-80GB-PCIe' + - property: + identifier: 'skip_tokenizer_init' + value: 1 + - property: + identifier: 'enforce_eager' + value: 1 + - property: + identifier: 'io_processor_plugin' + value: "terratorch_segmentation" + - property: + identifier: 'dataset' + value: 'india_url_in_b64_out' + - property: + identifier: 'renderer_num_workers' + value: 0 + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'VLLM performance testing using GuideLLM benchmark suite across compute resource and workload configuration for geospatial models' +geospatial-guidellm-bench-deployment-custom-dataset: + identifier: geospatial-guidellm-bench-deployment-custom-dataset + actuatorIdentifier: "vllm_performance" + version: 1.0.0 + requiredProperties: + - identifier: 'model' + metadata: + description: 'model to use for testing. Assumed to be served by all endpoints tested. Required to obtain correct tokenizer for benchmarking metrics calculation' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] + - identifier: 'request_rate' + metadata: + description: "(benchmark) The number of requests to send per second" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 1000] + interval: 1 # -1 means send all requests at time 0 + - identifier: 'dataset' + metadata: + description: "(benchmark) The dataset to be used for the experiment" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["custom_dataset.jsonl"] + optionalProperties: + - identifier: 'num_prompts' + metadata: + description: "(benchmark) The number of prompts to send (total number of requests)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 10001] + interval: 1 + - identifier: 'max_concurrency' + metadata: + description: "(benchmark) The maximum number of concurrent requests to send" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [-1, 500] # -1 means no concurrency control + interval: 1 + - identifier: 'burstiness' + metadata: + description: "(benchmark) The burstiness of the requests - must be 1.0 for GuideLLM (Poisson distribution)" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + values: [1.0] + - identifier: image + metadata: + description: "(deployment) Docker image to use to create vllm deployments" + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: ["quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7"] + - identifier: n_cpus + metadata: + description: "(deployment) the number of CPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 256] + interval: 1 + - identifier: memory + metadata: + description: "(deployment) the amount of memory to allocate to vLLM pod" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["64Gi", "128Gi", "256Gi"] + - identifier: dtype + metadata: + description: '(deployment) data type for model weights and activations. “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.' + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ["auto", "half", "float16", "bfloat16", "float", "float32"] + - identifier: 'gpu_memory_utilization' + metadata: + description: "(deployment) The fraction of GPU memory to be used for the model executor," + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 1] + interval: 0.01 + - identifier: 'cpu_offload' + metadata: + description: "(deployment) The amount of CPU offload to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 1] + interval: 1 + - identifier: 'max_num_seq' + metadata: + description: "(deployment) Maximum number of sequences per iteration." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 257] + interval: 1 + - identifier: 'max_batch_tokens' + metadata: + description: "(deployment) Maximum number of tokens to be processed in a single batch." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [256, 32769] + interval: 256 + - identifier: 'n_gpus' + metadata: + description: "(deployment) the number of GPUs to use" + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [1, 9] + interval: 1 + - identifier: 'gpu_type' + metadata: + description: "(deployment) the type of GPU to use" + propertyDomain: + variableType: "CATEGORICAL_VARIABLE_TYPE" + values: ['NVIDIA-A100-80GB-PCIe', 'NVIDIA-L40S'] + - identifier: 'skip_tokenizer_init' + metadata: + description: "(deployment) skip tokenizer initialization" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'enforce_eager' + metadata: + description: "(deployment) enforce pytorch eager mode" + propertyDomain: + variableType: DISCRETE_VARIABLE_TYPE + values: [1] + - identifier: 'io_processor_plugin' + metadata: + description: 'IO Processor plugin to load for the model' + propertyDomain: + variableType: "OPEN_CATEGORICAL_VARIABLE_TYPE" + values: [None, "terratorch_segmentation"] + - identifier: 'renderer_num_workers' + metadata: + description: "Number of renderer workers for threadpool-based rendering. Set to 0 to disable threadpool (for vLLM versions < 0.20.0). Set to a positive value to enable threadpool (requires vLLM >= 0.20.0). If not specified, threadpool is disabled." + propertyDomain: + variableType: 'DISCRETE_VARIABLE_TYPE' + domainRange: [0, 128] + interval: 1 + defaultParameterization: + - property: + identifier: 'image' + value: "quay.io/mgazzetti/vllm-performance/vllm:vllm.v0.20.1-tt.v1.2.7" + - property: + identifier: n_cpus + value: 8 + - property: + identifier: memory + value: "128Gi" + - property: + identifier: dtype + value: "auto" + - property: + identifier: 'num_prompts' + value: 500 + - property: + identifier: 'max_concurrency' + value: -1 + - property: + identifier: 'burstiness' + value: 1.0 + - property: + identifier: 'gpu_memory_utilization' + value: .9 + - property: + identifier: 'cpu_offload' + value: 0 + - property: + identifier: 'max_num_seq' + value: 256 + - property: + identifier: 'max_batch_tokens' + value: 16384 + - property: + identifier: 'n_gpus' + value: 1 + - property: + identifier: 'gpu_type' + value: 'NVIDIA-A100-80GB-PCIe' + - property: + identifier: 'skip_tokenizer_init' + value: 1 + - property: + identifier: 'enforce_eager' + value: 1 + - property: + identifier: 'io_processor_plugin' + value: "terratorch_segmentation" + - property: + identifier: 'renderer_num_workers' + value: 0 + targetProperties: + - identifier: "duration" + - identifier: "completed" + - identifier: "request_throughput" + - identifier: "mean_e2el_ms" + - identifier: "median_e2el_ms" + - identifier: "std_e2el_ms" + - identifier: "p25_e2el_ms" + - identifier: "p50_e2el_ms" + - identifier: "p75_e2el_ms" + - identifier: "p99_e2el_ms" + metadata: + description: 'GuideLLM performance testing across compute resource and workload configuration for geospatial models with custom dataset' +geospatial-guidellm-bench-endpoint-custom-dataset: + identifier: geospatial-guidellm-bench-endpoint-custom-dataset + actuatorIdentifier: "vllm_performance" + version: 1.0.0 requiredProperties: - identifier: 'model' metadata: diff --git a/plugins/actuators/vllm_performance/yamls/discoveryspace_override_defaults_small.yaml b/plugins/actuators/vllm_performance/yamls/discoveryspace_override_defaults_small.yaml index fceec95ab..f9db6108c 100644 --- a/plugins/actuators/vllm_performance/yamls/discoveryspace_override_defaults_small.yaml +++ b/plugins/actuators/vllm_performance/yamls/discoveryspace_override_defaults_small.yaml @@ -48,6 +48,7 @@ entitySpace: values: ['NVIDIA-A100-80GB-PCIe'] experiments: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-deployment-v1 + experimentIdentifier: vllm-bench-deployment + experimentVersion: 1.0.0 metadata: description: Parameters for VLLM performance testing diff --git a/plugins/actuators/vllm_performance/yamls/random_walk_operation_grouped.yaml b/plugins/actuators/vllm_performance/yamls/random_walk_operation_grouped.yaml index d676f73c2..afd04ca6a 100644 --- a/plugins/actuators/vllm_performance/yamls/random_walk_operation_grouped.yaml +++ b/plugins/actuators/vllm_performance/yamls/random_walk_operation_grouped.yaml @@ -1,7 +1,7 @@ # Copyright IBM Corporation 2025, 2026 # SPDX-License-Identifier: MIT metadata: - name: randomwalk-grouped-vllm-performance-test-deployment + name: randomwalk-grouped-vllm-performance-vllm-bench-deployment spaces: - space-230d24-03b22d actuatorConfigurationIdentifiers: @@ -16,7 +16,7 @@ operation: samplerConfig: mode: 'sequentialgrouped' samplerType: 'generator' - grouping: #These are entity properties compatible with vllm_performance.test-deployment-v1 and vllm_performance.test-deployment-guidellm-v1 + grouping: #These are entity properties compatible with vllm_performance.vllm-bench-deployment and vllm_performance.guidellm-bench-deployment - model - image - n_gpus diff --git a/plugins/actuators/vllm_performance/yamls/vllm_deployment_space.yaml b/plugins/actuators/vllm_performance/yamls/vllm_deployment_space.yaml index 6bfc7746f..b99851a25 100644 --- a/plugins/actuators/vllm_performance/yamls/vllm_deployment_space.yaml +++ b/plugins/actuators/vllm_performance/yamls/vllm_deployment_space.yaml @@ -53,7 +53,8 @@ entitySpace: values: ["NVIDIA-A100-80GB-PCIe"] experiments: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-deployment-v1 + experimentIdentifier: vllm-bench-deployment + experimentVersion: 1.0.0 metadata: description: A space of vllm deployment configurations name: vllm_deployments diff --git a/plugins/actuators/vllm_performance/yamls/vllm_request_rate_space.yaml b/plugins/actuators/vllm_performance/yamls/vllm_request_rate_space.yaml index 0449f3b5f..18227e365 100644 --- a/plugins/actuators/vllm_performance/yamls/vllm_request_rate_space.yaml +++ b/plugins/actuators/vllm_performance/yamls/vllm_request_rate_space.yaml @@ -15,7 +15,8 @@ entitySpace: interval: 1 experiments: - actuatorIdentifier: vllm_performance - experimentIdentifier: test-endpoint-v1 + experimentIdentifier: vllm-bench-endpoint + experimentVersion: 1.0.0 metadata: description: "Test space for request throughput of a vllm endpoint" name: "vllm_endpoint_performance_test_space" diff --git a/tests/ado/describe/test_ado_describe.py b/tests/ado/describe/test_ado_describe.py index 5c8502ae0..a246525ec 100644 --- a/tests/ado/describe/test_ado_describe.py +++ b/tests/ado/describe/test_ado_describe.py @@ -79,11 +79,11 @@ def test_describe_calculate_density_experiment() -> None: assert "calculate_density" in result.output -def test_describe_vllm_test_deployment_experiment() -> None: +def test_describe_vllm_bench_deployment_experiment() -> None: runner = CliRunner() - result = runner.invoke(ado, ["describe", "experiment", "test-deployment-v1"]) + result = runner.invoke(ado, ["describe", "experiment", "vllm-bench-deployment"]) assert result.exit_code == 0 - assert "test-deployment-v1" in result.output + assert "vllm-bench-deployment" in result.output @requires_sqlite_3_38 diff --git a/tests/ado/template/test_ado_template_space.py b/tests/ado/template/test_ado_template_space.py index b204b9c4f..c0b6c1851 100644 --- a/tests/ado/template/test_ado_template_space.py +++ b/tests/ado/template/test_ado_template_space.py @@ -70,7 +70,7 @@ def test_template_space_from_vllm_experiment( "template", "space", "--from-experiment", - "test-deployment-v1", + "vllm-bench-deployment", "--output-file", file_name, ], @@ -80,7 +80,8 @@ def test_template_space_from_vllm_experiment( yaml.safe_load(file_name.read_text()) ) assert ( - space_configuration.experiments[0].experimentIdentifier == "test-deployment-v1" + space_configuration.experiments[0].experimentIdentifier + == "vllm-bench-deployment" ) assert space_configuration.experiments[0].actuatorIdentifier == "vllm_performance"