Is your feature request related to a problem? Please describe.
TRIM's documentation states:
targetOutput
Type: str (required)
Tuning Guidance: Must exactly match an output property identifier from your experiment. All of TRIM's logic revolves around this target.
So I describe my experiment and get one of the output property identifiers
$ ado describe experiment test-deployment-v1
...
Outputs:
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test-deployment-v1-duration
test-deployment-v1-completed
test-deployment-v1-total_input_tokens
test-deployment-v1-total_output_tokens
test-deployment-v1-request_throughput
test-deployment-v1-output_throughput
test-deployment-v1-total_token_throughput
test-deployment-v1-mean_ttft_ms
test-deployment-v1-median_ttft_ms
test-deployment-v1-std_ttft_ms
test-deployment-v1-p25_ttft_ms
test-deployment-v1-p50_ttft_ms
...
Let's say I want to use TRIM for generating test-deployment-v1-p50_ttft_ms over a discovery space.
If I follow the documentation I'd set:
parameters:
targetOutput: test-deployment-v1-p50_ttft_ms
However, when I run the operation it fails with the below error:
The current version of TRIM assumes that all measurements produce the observed target output property 'test-deployment-v1-p50_ttft_ms'. The measurements obtained with your custom experiment (or actuator) did not contain the target output property 'test-deployment-v1-p50_ttft_ms'. Additional info: This was detected during the no-priors characterization phase: 25 out of 25. This is insufficient for starting the Iterative Modeling phase, the operation will exit with an error. For more information, refer to the documentation here: `https://ibm.github.io/ado/operators/trim/`..
The fix is to remove the experiment name from the output property name.
parameters:
targetOutput: p50_ttft_ms
However, this is not documented and the error is not explicit enough for one to figure it out quickly.
Also note that this will happen only after the initial samples are measured. I'd catch it right at the beginning!
Describe the solution you'd like. A clear and concise description of what
you want to happen.
I would like for trim to take the full output metric name so that it matches what I get from ado describe experiment.
Is your feature request related to a problem? Please describe.
TRIM's documentation states:
So I describe my experiment and get one of the output property identifiers
Let's say I want to use TRIM for generating
test-deployment-v1-p50_ttft_msover a discovery space.If I follow the documentation I'd set:
However, when I run the operation it fails with the below error:
The fix is to remove the experiment name from the output property name.
However, this is not documented and the error is not explicit enough for one to figure it out quickly.
Also note that this will happen only after the initial samples are measured. I'd catch it right at the beginning!
Describe the solution you'd like. A clear and concise description of what
you want to happen.
I would like for trim to take the full output metric name so that it matches what I get from
ado describe experiment.