docs: adjust experiment pages snippets by thiagobomfin-galileo · Pull Request #850 · rungalileo/docs-official

thiagobomfin-galileo · 2026-05-06T02:32:54Z

Describe your changes

Update missing snippents in experiments pages

Shortcut ticket

For Galileo internally raised PRs only, please update this with your shortcut ticket.

Keep the formatting, replacing SC-number with the shortcut ticket, e.g. [SC-12345], and adding the link to the ticket correctly in the brackets. This format is important as it allows Shortcut to track the ticket, moving the status to in review, then merged once the ticket is merged.

For external PRs, please add the issue (just put the number after the # below, and GitHub will automatically create a link):

Issue: #number

Checklist before requesting a review

mintlify · 2026-05-06T02:32:57Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
v2galileo	🟢 Ready	View Preview	May 6, 2026, 2:35 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

github-actions · 2026-05-06T02:33:16Z

Card links check

✅ No broken Card links found. Checked external links in 1.1s

github-actions · 2026-05-06T02:33:25Z

✅ Markdown Lint Passed

All markdown files meet the linting standards! 🎉

github-actions · 2026-05-06T02:34:38Z

✅ Mintlify Dev Server Started Successfully!

The documentation can be served without errors.

fercor-cisco · 2026-05-07T01:21:41Z

@xke please review

- Rename dataset version variable for clarity (content_of_specific_version) - Remove unused 'Get an existing logger' section from running-experiments - Add Beta snippets for Custom dataset evaluation and Custom metrics - Replace GalileoMetrics with Metric.metrics.X across python-beta experiment snippets - Add Beta code for 'Run your app using an experiment' (rag-and-tools) - Add Beta version of OTEL run-experiment snippet - Drop misleading 'works unchanged on the Beta SDK' notes from python snippets - Guard Dataset.get() None returns in python-beta function-based experiments

fercor-cisco · 2026-05-12T23:47:57Z

Optional cosmetic nits (not blockers)

custom-dataset.mdx uses model="gpt-4" while the rest of the suite uses gpt-4o.
python/concepts/experiments/prompt.mdx has a pre-existing double-space typo in "If the prompt already exists" (not introduced by this PR).
All python-beta snippets correctly carry the comment about Experiment not yet supporting function= — matches experiment.py:177 (# TODO: Function-based experiments temporarily disabled).

fercor-cisco · 2026-05-12T23:48:07Z

@xke please review

Add Python (Beta) snippet siblings for the observability nav group, mirroring the experiments-pages rollout (PR #850). Beta variants demonstrate the new `LogStream` class for routing logs via `LogStream.context()` instead of `galileo_context(project=..., log_stream=...)`.

fercor-cisco · 2026-05-14T16:54:24Z

/astra review

xke · 2026-05-14T16:55:36Z


 You will need to change your code to use this instead of creating a new logger and starting a new trace.

-#### Get an existing logger and check for an existing trace


why did this section get removed? https://docs.galileo.ai/sdk-api/experiments/running-experiments#get-an-existing-logger-and-check-for-an-existing-trace

Good catch — that removal wasn't intentional. Restored the section, its 3 CodeGroup snippets, the 6 Python/TypeScript imports, and the <Note> link to the section anchor in 0407be5.

galileo-astra

⚠️ This review was generated by an AI agent (Astra) and may contain mistakes. Please verify all suggestions independently.

Verdict: approve — Docs-only PR adding Python (Beta) snippets; all SDK APIs verified correct against galileo==2.2.0, no broken anchors or imports. Only minor consistency/style issues remain.

General Comments

🟡 minor (design): The Beta snippets aren't consistent about how they obtain an existing dataset before calling run_experiment / Experiment(...):
existing-dataset.mdx and existing-dataset-log.mdx (Beta) fetch the dataset, guard against None, then pass dataset_name=dataset.name (which makes the SDK look the dataset up a second time).
prompt.mdx (Beta) inlines dataset=Dataset.get(name="countries") with no None check, so if the dataset doesn't exist it silently passes None into Experiment(...) and the failure surfaces only at .create() with a less helpful error.

Picking one pattern across all four Beta snippets would make the examples easier to copy-paste correctly. The cleanest version, IMO, is to fetch once, guard for None, then pass dataset=dataset — both the legacy run_experiment and the new Experiment accept a Dataset object directly, so there's no need for the second name lookup.

Follow-ups

Suggested follow-up work that could be tracked as Shortcut stories:

snippets/code/python/concepts/experiments/get-logger.mdx:1-1: After this PR, the snippet files snippets/code/python/concepts/experiments/{get-logger,get-current-parent,logging-with-parent-check}.mdx and their TypeScript counterparts (snippets/code/typescript/concepts/experiments/{get-logger,get-current-parent,logging-with-parent-check}.mdx) are no longer imported anywhere in the docs. Worth deleting in a follow-up cleanup PR so they don't drift out of date.
snippets/code/python/concepts/experiments/prompt.mdx:26-26: Pre-existing double-space typo # 1b. (OPTIONAL) If the prompt already exists, fetch it: — this PR fixed the same typo on line 13 (1a) but the matching typo on line 26 (1b) is left in place. Trivially fixable in a follow-up.

galileo-astra · 2026-05-14T17:03:18Z

+experiment.create()
+results = experiment.run()


🔵 nit (documentation): Per the SDK (experiment.py:1060-1101), Experiment.create() already triggers the run via trigger=True; Experiment.run() is documented as existing only for backward compatibility with the create().run() chain pattern. Calling both back-to-back here isn't wrong, but it suggests .run() is what kicks off the experiment, which is misleading for a Beta-introduction snippet. Consider either dropping the .run() line or merging the two: results = experiment.create().run(). Same comment applies to snippets/code/python-beta/concepts/experiments/generated-output-inline.mdx:44-45.

@xke

…' section Reverts the inadvertent removal of the section (and its 3 CodeGroup snippets + Python/TypeScript imports) flagged by @xke in PR review. Also restores the <Note> link to the section anchor.

fercor-cisco · 2026-05-14T23:05:40Z

/astra review

galileo-astra

⚠️ This review was generated by an AI agent (Astra) and may contain mistakes. Please verify all suggestions independently.

Verdict: request_changes — Three of four prior review comments (Dataset.get None-guard, redundant .run() after .create(), and dataset=dataset vs dataset_name=dataset.name) remain unaddressed in the latest push, and the same Dataset.get None-return pitfall also applies to the commented Prompt.get example in prompt.mdx.

General Comments

🟠 major (documentation): Of the four existing review comments, only xke's question about the removed ### Get an existing logger and check for an existing trace section has been addressed (restored in 0407be5). The three galileo-astra[bot] findings are still applicable to HEAD and I independently verified them against the installed galileo SDK source — they are correct, not noise:
python-beta/concepts/experiments/prompt.mdx:30 — Dataset.get(name=...) returns None when the dataset isn't found (galileo/dataset.py:237-238). Inlining it inside Experiment(dataset=...) silently sets no dataset (constructor at galileo/experiment.py:320 skips when dataset is None); the failure surfaces later with a less obvious error. The peer Beta snippets in this PR (existing-dataset.mdx, existing-dataset-log.mdx) already guard with if dataset is None: raise ValueError(...). Make prompt.mdx match.
python-beta/concepts/experiments/prompt.mdx:44 and python-beta/concepts/experiments/generated-output-inline.mdx:44 — Experiment.create() already triggers the run (galileo/experiment.py:476, trigger=True). Experiment.run() only exists for backward compat with the create().run() chain (docstring at experiment.py:1060-1066). The snippet works, but presenting experiment.create(); results = experiment.run() as the canonical Beta pattern is misleading for new users. Pick one: drop .run() (and assign experiment.create() if you need the return), or use the chain results = experiment.create().run().
python-beta/concepts/experiments/existing-dataset-log.mdx:18 and python-beta/concepts/experiments/existing-dataset.mdx:33 — you have a Dataset object in scope, so passing dataset_name=dataset.name triggers a redundant name lookup inside the SDK. Use dataset=dataset.

Either apply the fixes or reply with the reasoning so the thread can be resolved.

Follow-ups

Suggested follow-up work that could be tracked as Shortcut stories:

snippets/code/python-beta/concepts/experiments/custom-dataset.mdx:15-15: Uses model="gpt-4" while every other snippet in this Beta suite uses gpt-4o. The legacy Python snippet has the same gpt-4 already, so this is consistency cleanup across both — out of scope for this PR but worth a follow-up.
snippets/code/python-beta/concepts/experiments/existing-dataset.mdx:7-7: client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"]) is at module top level, so importing this snippet (e.g. when copy-pasted into a notebook cell that re-runs) raises immediately if OPENAI_API_KEY is unset. The legacy Python snippet has the same pattern, so this is a pre-existing pattern in the docs — consider moving the client into the function in a follow-up.
snippets/code/python-beta/concepts/experiments/custom-dataset.mdx:26-27: The # Function-based experiments still go through run_experiment; the Experiment class doesn't support function= yet. comment is repeated verbatim across six new Beta snippets (custom-dataset, custom-metrics, existing-dataset, existing-dataset-log, otel-experiment/run-experiment, rag-and-tools/experiment-main). Consider hoisting this caveat into one of the parent .mdx pages rather than echoing it in every code snippet.
snippets/code/python-beta/concepts/experiments/existing-dataset-log.mdx:9-12: def llm_call(input): ... return result references an undefined result. Matches the legacy Python snippet (also pre-existing), but readers who copy this code verbatim will get a NameError before they ever see the Galileo behavior. Replace with a more explicit placeholder like return "<your-llm-output>" or raise NotImplementedError(...) in a follow-up.

galileo-astra · 2026-05-14T23:10:50Z

+)
+prompt.create()


🟡 minor (documentation): Prompt.get(name=...) has the same silent-None behavior as Dataset.get: it returns None when the prompt isn't found (verified at galileo/prompt.py:413-414), it does not raise. If a reader uncomments this line they will hit the same swallowed-error class the existing bot comment is calling out on line 30. Add a similar guard in the commented snippet so the example is honest about the contract.

Suggested change

)

prompt.create()

# 1b. (OPTIONAL) If the prompt already exists, fetch it:

# prompt = Prompt.get(name="geography-prompt")

# if prompt is None:

# raise ValueError("Prompt 'geography-prompt' not found")

galileo-astra · 2026-05-14T23:10:51Z

@@ -127,7 +135,11 @@ This is ideal for:

 <Note>This flow is currently supported in the Python SDK (v1.50.1+). TypeScript support uses the same API — omit `promptTemplate` to use this flow.</Note>


🟡 minor (documentation): This Note still says the flow is "currently supported in the Python SDK (v1.50.1+)." Now that a Python (Beta) snippet sits next to the legacy Python snippet, it is unclear whether the Beta variant has the same lower bound, a different one, or no constraint. Either drop the version-specific phrasing or add a parenthetical for the Beta SDK version that introduced support.

galileo-astra · 2026-05-14T23:10:51Z

+import SnippetExperimentsGeneratedOutputInlinePythonBeta from "/snippets/code/python-beta/concepts/experiments/generated-output-inline.mdx";
+



🔵 nit (other): Extra blank line between the new Beta import and the closing markdownlint comment — the file uses a single blank line everywhere else.

Suggested change

import SnippetExperimentsGeneratedOutputInlinePythonBeta from "/snippets/code/python-beta/concepts/experiments/generated-output-inline.mdx";

import SnippetExperimentsGeneratedOutputInlinePythonBeta from "/snippets/code/python-beta/concepts/experiments/generated-output-inline.mdx";

{/*  */}

…t-log.mdx Co-authored-by: galileo-astra[bot] <275138225+galileo-astra[bot]@users.noreply.github.com>

Co-authored-by: galileo-astra[bot] <275138225+galileo-astra[bot]@users.noreply.github.com>

galileo-automation · 2026-06-27T02:23:08Z

No activity for 30 days — this PR will be closed in 5 days unless updated.

galileo-automation · 2026-07-28T02:19:10Z

No activity for 30 days — this PR will be closed in 5 days unless updated.

galileo-automation · 2026-08-28T04:18:57Z

No activity for 30 days — this PR will be closed in 5 days unless updated.

thiagobomfin-galileo requested review from Murike and fercor-cisco May 6, 2026 02:32

thiagobomfin-galileo self-assigned this May 6, 2026

mintlify Bot deployed to staging May 6, 2026 02:35 View deployment

fercor-cisco requested a review from xke May 7, 2026 01:21

thiagobomfin-galileo added 2 commits May 12, 2026 12:30

docs: adjust experiment pages snippets

84afc83

thiagobomfin-galileo force-pushed the fix/sc-63600-experiment-pages branch from 85a9f85 to 8020c02 Compare May 12, 2026 15:31

mintlify Bot deployed to staging May 12, 2026 15:33 View deployment

thiagobomfin-galileo mentioned this pull request May 13, 2026

docs: adjust observability pages snippets #897

Open

10 tasks

Merge branch 'main' into fix/sc-63600-experiment-pages

3cba43a

mintlify Bot deployed to staging May 13, 2026 23:31 View deployment

xke requested changes May 14, 2026

View reviewed changes

galileo-astra Bot reviewed May 14, 2026

View reviewed changes

docs: restore 'Get an existing logger and check for an existing trace…

0407be5

…' section Reverts the inadvertent removal of the section (and its 3 CodeGroup snippets + Python/TypeScript imports) flagged by @xke in PR review. Also restores the <Note> link to the section anchor.

mintlify Bot deployed to staging May 14, 2026 20:17 View deployment

galileo-astra Bot reviewed May 14, 2026

View reviewed changes

Merge branch 'main' into fix/sc-63600-experiment-pages

456867f

mintlify Bot deployed to staging May 27, 2026 06:23 View deployment

Update snippets/code/python-beta/concepts/experiments/existing-datase…

66d9516

…t-log.mdx Co-authored-by: galileo-astra[bot] <275138225+galileo-astra[bot]@users.noreply.github.com>

mintlify Bot deployed to staging May 27, 2026 06:32 View deployment

Update snippets/code/python-beta/concepts/experiments/prompt.mdx

6b2ec7c

Co-authored-by: galileo-astra[bot] <275138225+galileo-astra[bot]@users.noreply.github.com>

mintlify Bot deployed to staging May 27, 2026 06:35 View deployment

xke self-requested a review May 27, 2026 06:36

xke approved these changes May 27, 2026

View reviewed changes


		You will need to change your code to use this instead of creating a new logger and starting a new trace.

		#### Get an existing logger and check for an existing trace

-)
-prompt.create()
+# 1b. (OPTIONAL) If the prompt already exists, fetch it:
+# prompt = Prompt.get(name="geography-prompt")
+# if prompt is None:
+#     raise ValueError("Prompt 'geography-prompt' not found")

		@@ -127,7 +135,11 @@ This is ideal for:

		<Note>This flow is currently supported in the Python SDK (v1.50.1+). TypeScript support uses the same API — omit `promptTemplate` to use this flow.</Note>

		import SnippetExperimentsGeneratedOutputInlinePythonBeta from "/snippets/code/python-beta/concepts/experiments/generated-output-inline.mdx";

		experiment.create()
		results = experiment.run()

Uh oh!

Conversation

thiagobomfin-galileo commented May 6, 2026

Describe your changes

Shortcut ticket

Checklist before requesting a review

Uh oh!

mintlify Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Card links check

Uh oh!

github-actions Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Markdown Lint Passed

Uh oh!

github-actions Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fercor-cisco commented May 7, 2026

Uh oh!

fercor-cisco commented May 12, 2026

Uh oh!

fercor-cisco commented May 12, 2026

Uh oh!

fercor-cisco commented May 14, 2026

Uh oh!

xke May 14, 2026

Choose a reason for hiding this comment

Uh oh!

thiagobomfin-galileo May 14, 2026

Choose a reason for hiding this comment

Uh oh!

galileo-astra Bot left a comment

Choose a reason for hiding this comment

General Comments

Follow-ups

Uh oh!

Uh oh!

galileo-astra Bot May 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fercor-cisco commented May 14, 2026

Uh oh!

galileo-astra Bot left a comment

Choose a reason for hiding this comment

General Comments

Follow-ups

Uh oh!

galileo-astra Bot May 14, 2026

Choose a reason for hiding this comment

Uh oh!

galileo-astra Bot May 14, 2026

Choose a reason for hiding this comment

Uh oh!

galileo-astra Bot May 14, 2026

Choose a reason for hiding this comment

Uh oh!

galileo-automation commented Jun 27, 2026

Uh oh!

galileo-automation commented Jul 28, 2026

Uh oh!

galileo-automation commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mintlify Bot commented May 6, 2026 •

edited

Loading

github-actions Bot commented May 6, 2026 •

edited

Loading

github-actions Bot commented May 6, 2026 •

edited

Loading

github-actions Bot commented May 6, 2026 •

edited

Loading