Skip to content

feat: Trim operator to use the full metric name for experiment's output properties #1225

Description

@christian-pinto

Is your feature request related to a problem? Please describe.

TRIM's documentation states:

targetOutput
Type: str (required)

Tuning Guidance: Must exactly match an output property identifier from your experiment. All of TRIM's logic revolves around this target.

So I describe my experiment and get one of the output property identifiers

$ ado describe experiment test-deployment-v1
...

Outputs:
 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   test-deployment-v1-duration
   test-deployment-v1-completed
   test-deployment-v1-total_input_tokens
   test-deployment-v1-total_output_tokens
   test-deployment-v1-request_throughput
   test-deployment-v1-output_throughput
   test-deployment-v1-total_token_throughput
   test-deployment-v1-mean_ttft_ms
   test-deployment-v1-median_ttft_ms
   test-deployment-v1-std_ttft_ms
   test-deployment-v1-p25_ttft_ms
   test-deployment-v1-p50_ttft_ms
...

Let's say I want to use TRIM for generating test-deployment-v1-p50_ttft_ms over a discovery space.

If I follow the documentation I'd set:

parameters:
  targetOutput: test-deployment-v1-p50_ttft_ms

However, when I run the operation it fails with the below error:

The current version of TRIM assumes that all measurements produce the observed target output property 'test-deployment-v1-p50_ttft_ms'. The measurements obtained with your custom experiment (or actuator) did not contain the target output property 'test-deployment-v1-p50_ttft_ms'. Additional info: This was detected during the no-priors characterization phase: 25 out of 25. This is insufficient for starting the Iterative Modeling phase, the operation will exit with an error. For more information, refer to the documentation here: `https://ibm.github.io/ado/operators/trim/`..

The fix is to remove the experiment name from the output property name.

parameters:
  targetOutput: p50_ttft_ms

However, this is not documented and the error is not explicit enough for one to figure it out quickly.
Also note that this will happen only after the initial samples are measured. I'd catch it right at the beginning!

Describe the solution you'd like. A clear and concise description of what
you want to happen.

I would like for trim to take the full output metric name so that it matches what I get from ado describe experiment.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions