Path: /api-reference/experiment/create-experiment
Reported issue:
According to the Galileo API reference, the Create Experiment endpoint indicates that scorers can be included in the request body:
https://v2docs.galileo.ai/api-reference/experiment/create-experiment#body-scorers
However, when I tried creating an experiment with scorers included at creation time, it didn’t work. What did work instead was:
- Create the experiment without scorers
- Enable metrics afterward using Update Metric Settings:
https://v2docs.galileo.ai/api-reference/experiment/update-metric-settings
Wanted to flag this in case the docs need updating to match current behavior, or there’s a bug if scorers are intended to be supported during experiment creation.
Looking further into this, documentation can be improved to include and point out the following:
scorers can be included in the request body at creation time given following are true:
trigger=true + prompt_template_version_id has a value provided for it. This is not clear with current documentation.
- The
Create Experiment endpoint accepts a scorers field in its request body, but scorers are silently ignored server-side when trigger=false (the default scenario).
Path: /api-reference/experiment/create-experiment
Reported issue:
Looking further into this, documentation can be improved to include and point out the following:
scorerscan be included in the request body at creation time given following are true:trigger=true+prompt_template_version_idhas a value provided for it. This is not clear with current documentation.Create Experimentendpoint accepts ascorersfield in its request body, butscorersare silently ignored server-side whentrigger=false(the default scenario).