Fix SDK prompt experiments using saved template model - #631
Conversation
|
Closing this for now to keep the Blackstone fix scoped to the UI path. SDK users can work around the current behavior by passing prompt_settings explicitly with the saved custom model alias. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #631 +/- ##
==========================================
+ Coverage 83.31% 83.61% +0.30%
==========================================
Files 125 124 -1
Lines 10659 11050 +391
==========================================
+ Hits 8881 9240 +359
- Misses 1778 1810 +32 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
| selected_version = getattr(prompt_template, "selected_version", None) | ||
| settings = getattr(selected_version, "settings", None) | ||
| if settings is None or isinstance(settings, Unset): | ||
| return None | ||
| if isinstance(settings, PromptRunSettings): | ||
| return settings | ||
| if isinstance(settings, dict) and settings: | ||
| return PromptRunSettings.from_dict(settings) |
There was a problem hiding this comment.
_prompt_template_settings() duplicates the selected_version -> settings -> Unset parsing already done in Experiment.get_prompt_template_settings(), so schema/coercion changes need two edits and the two paths can drift — can we centralize this in one utility reused by both callers? Separately, when template settings are truthy it returns PromptRunSettings.from_dict(settings) without merging _default_prompt_settings(), so missing required keys stay UNSET and get dropped on serialize when Experiments.run() or Experiment.create() omits prompt_settings, silently leaving the job never starting — should we overlay the template dict onto the defaults before returning?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
src/galileo/experiments.py, refactor _prompt_template_settings() (and its counterpart
Experiment.get_prompt_template_settings()) so both share a single utility for the
selected_version -> settings -> Unset parsing, avoiding duplicated logic that can drift.
Additionally, ensure _prompt_template_settings() always produces a fully populated
PromptRunSettings by starting from _default_prompt_settings() and overlaying any fields
provided by the template dict, rather than returning
PromptRunSettings.from_dict(settings) directly when settings is truthy - this prevents
missing required keys from staying UNSET and being dropped on serialization. Verify
Experiments.run around lines 215-217 uses the new behavior so prompt-template precedence
never reintroduces the silent non-start issue when prompt_settings is omitted, and apply
the same merge behavior anywhere else _prompt_template_settings() is used, e.g., within
Experiment.create().
| mock_projects_service = MagicMock() | ||
| mock_projects_class.return_value = mock_projects_service | ||
| mock_projects_service.get_with_env_fallbacks.return_value = mock_project | ||
|
|
||
| mock_dataset = MagicMock() |
There was a problem hiding this comment.
Missing Given/When/Then in new test
test_create_uses_prompt_template_settings_when_available jumps straight into setup without non-empty # Given:/# When:/# Then: comments, so the test is harder to scan and doesn't follow AGENTS.md's Given/When/Then convention — should we add phase comments only?
Want Baz to fix this for you? Activate Fixer You can also update your AI coding guidelines based on this comment by apply pr to [branch name]
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
tests/test_experiment.py around lines 629-639, within the
`test_create_uses_prompt_template_settings_when_available` test function, add non-empty
human-readable `# Given:`, `# When:`, and `# Then:` comments to match the repo’s
AGENTS.md guideline. Place the `# Given:` comment right before the initial setup/mocking
block (currently starting with `mock_projects_service = MagicMock()`), then add `#
When:` before the `Experiment(...).create()` call and `# Then:` before the assertions.
Do not change any test behavior or logic—only insert the missing comment structure.
| saved_settings = PromptRunSettings(model_alias="GPT 5.4 Mini (custom)", temperature=0.0, max_tokens=-1) | ||
|
|
||
| run_experiment( | ||
| "test_experiment", | ||
| project="awesome-new-project", |
There was a problem hiding this comment.
Missing Given/When/Then in new test
test_run_experiment_w_prompt_template_uses_template_settings has empty # Given:, # When:, and # Then: comments, so it doesn't follow the repo's test structure and is harder to scan — should we add non-empty, human-readable phase comments, as AGENTS.md asks?
Want Baz to fix this for you? Activate Fixer You can also update your AI coding guidelines based on this comment by apply pr to [branch name]
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
tests/test_experiments.py around lines 1008-1029 in
`test_run_experiment_w_prompt_template_uses_template_settings`, the test currently lacks
non-empty “Given:”, “When:”, and “Then:” comments per AGENTS.md. Add clear
phase comments: describe the Given setup for `saved_settings`, the When call to
`run_experiment(...)`, and the Then assertions that `prompt_settings` equals
`saved_settings` and has the expected `model_alias`. Do not change any logic or
assertions—only insert the missing human-readable comments.
1 participant
User description
Summary
GPT 5.4 Mini (custom)so the SDK no longer defaults to GPT-4o for saved custom-model prompts.Testing
PYENV_VERSION=3.13.13 poetry run pytest tests/test_experiments.py::TestExperiments::test_run_experiment_w_prompt_template_uses_template_settings tests/test_experiments.py::TestExperiments::test_run_experiment_w_prompt_template_and_prompt_settings tests/test_experiments.py::TestExperiments::test_run_experiment_w_prompt_template_and_metrics tests/test_experiment.py::TestExperimentCreate::test_create_uses_prompt_template_settings_when_available tests/test_experiment.py::TestExperimentCreate::test_create_fills_default_prompt_settings_for_prompt_template tests/test_experiment.py::TestExperimentCreate::test_create_preserves_user_prompt_settings_when_overriding_model_aliasPYENV_VERSION=3.13.13 poetry run ruff check --select F,I src/galileo/experiments.py src/galileo/experiment.py tests/test_experiments.py tests/test_experiment.pyPYENV_VERSION=3.13.13 poetry run ruff format --check src/galileo/experiments.py src/galileo/experiment.py tests/test_experiments.py tests/test_experiment.pyGenerated description
Below is a concise technical summary of the changes proposed in this PR:
Use the selected prompt template version settings as the default for prompt-driven SDK experiments in
Experiments.run()andExperiment.create(), while keeping explicitprompt_settingsand model overrides highest priority. Add regression coverage for saved custom-model prompt templates so custom aliases likeGPT 5.4 Mini (custom)are preserved instead of defaulting to GPT-4o.Modified files (2)
Latest Contributors(2)
Modified files (2)
Latest Contributors(2)