Skip to content

refactor(vllm-performance): update experiment naming and versioning; deprecate old experiments - #1300

Merged
AlessandroPomponio merged 4 commits into
mainfrom
cp-exp-versioning
Jul 28, 2026
Merged

refactor(vllm-performance): update experiment naming and versioning; deprecate old experiments#1300
AlessandroPomponio merged 4 commits into
mainfrom
cp-exp-versioning

Conversation

@christian-pinto

@christian-pinto christian-pinto commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

This PR introduces a versioning scheme for experiments in the vllm-performance actuator. All existing experiments have been marked as deprecated, and a new set of cleanly named, versioned replacement experiments has been added. The actuator code and executor have been updated to recognise both old and new identifiers, ensuring backward compatibility while signalling that users should migrate to the new names.

Resolves #1203

High-level Changes

  • Deprecated legacy experiments: all existing experiment identifiers (e.g. test-deployment-v1, test-endpoint-guidellm-v1, and all geospatial variants) have been marked deprecated: true in the experiment YAML catalog.
  • New versioned experiments added: a full set of replacement experiments with cleaner identifiers has been introduced (vllm-bench-deployment, vllm-bench-endpoint, guidellm-bench-deployment, guidellm-bench-endpoint, and all geospatial equivalents), each carrying an explicit version: 1.0.0 field.
  • Actuator and executor updated: the routing logic in the actuator and experiment executor now recognises both old and new experiment identifiers, so existing spaces and operations remain functional.
  • Documentation updated: the actuator README and website docs reflect the new experiment names and deprecation notices.

Impact

Users can continue using existing discovery spaces and operations without modification — all legacy experiment identifiers remain functional. New spaces should use the new versioned experiment identifiers. The deprecated: true flag allows tooling to surface warnings for spaces still referencing the old names.

@AlessandroPomponio AlessandroPomponio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to fixing the test we must make sure we can still load data from the other ones and update them. we must also consider if it's possible to add a validator that changes entity data or something from the old experiment to the new one - given that nothing but a rename changed -
We must also update the documentation and examples

@christian-pinto christian-pinto changed the title feat(vllm-performance): Enabled experiments versioning and deprecated old experiments feat(vllm-performance): Update experiment naming and deprecated old experiments Jul 28, 2026
@christian-pinto christian-pinto changed the title feat(vllm-performance): Update experiment naming and deprecated old experiments feat(vllm-performance): Update experiment naming and deprecate old experiments Jul 28, 2026
@AlessandroPomponio AlessandroPomponio changed the title feat(vllm-performance): Update experiment naming and deprecate old experiments feat(vllm-performance): update experiment naming and versioning; deprecate old experiments Jul 28, 2026

@AlessandroPomponio AlessandroPomponio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

@AlessandroPomponio
AlessandroPomponio added this pull request to the merge queue Jul 28, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 28, 2026
@AlessandroPomponio
AlessandroPomponio added this pull request to the merge queue Jul 28, 2026
@AlessandroPomponio
AlessandroPomponio removed this pull request from the merge queue due to a manual request Jul 28, 2026
@AlessandroPomponio AlessandroPomponio changed the title feat(vllm-performance): update experiment naming and versioning; deprecate old experiments refactor(vllm-performance): update experiment naming and versioning; deprecate old experiments Jul 28, 2026
@AlessandroPomponio
AlessandroPomponio added this pull request to the merge queue Jul 28, 2026
@michael-johnston

Copy link
Copy Markdown
Member

also consider if it's possible to add a validator that changes entity data or something from the old experiment to the new one

We just have to pay the price of changing the names. We shouldn't allow retroactive edit of spaces and the data collected from them as this breaks many guarantees we try to keep (spaces don't change, the schema implied by a space (observed property names etc.) doesn't change, the data associated with the space is the data collected by the ops etc.)

Merged via the queue into main with commit 7024e83 Jul 28, 2026
18 checks passed
@AlessandroPomponio
AlessandroPomponio deleted the cp-exp-versioning branch July 28, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(vllm-performance): use updated experiment versioning

3 participants