Skip to content

Make usage of Scorers with create experiment endpoint API clearer #571

Description

@ckazz

Path: /api-reference/experiment/create-experiment

Reported issue:

According to the Galileo API reference, the Create Experiment endpoint indicates that scorers can be included in the request body:
https://v2docs.galileo.ai/api-reference/experiment/create-experiment#body-scorers
However, when I tried creating an experiment with scorers included at creation time, it didn’t work. What did work instead was:

  1. Create the experiment without scorers
  2. Enable metrics afterward using Update Metric Settings:
    https://v2docs.galileo.ai/api-reference/experiment/update-metric-settings
    Wanted to flag this in case the docs need updating to match current behavior, or there’s a bug if scorers are intended to be supported during experiment creation.

Looking further into this, documentation can be improved to include and point out the following:

  • scorers can be included in the request body at creation time given following are true:
    • trigger=true + prompt_template_version_id has a value provided for it. This is not clear with current documentation.
  • The Create Experiment endpoint accepts a scorers field in its request body, but scorers are silently ignored server-side when trigger=false (the default scenario).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions