feat: sync Python SDK with latest Hub API - #115
Open
cursor[bot] wants to merge 3 commits into
Open
Conversation
Add red team audits, dataset import preview, and evaluation upload. Drop evaluation include params, return dicts from test_connection, and document generate_completion as GenerateCompletionOutput.
henchaves
marked this pull request as ready for review
September 8, 2026 15:21
Leave those Hub endpoints out of the SDK for now: red team audits are still behind a feature flag, and dataset import preview is not needed.
Accept the removed include argument on evaluations.retrieve/list so existing callers do not TypeError, and map filters chat_test_case_id to scenario_id. Both emit a DeprecationWarning and are not sent as removed API fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the Python SDK to match the latest Hub OpenAPI spec.
client.evaluations.uploadand stop sending the removedincludequery on evaluation list/retrieveevaluations.run_singleat/v2/evaluations/run-interaction-checksgenerate_completionasGenerateCompletionOutputandtest_connectionas a dict (already the v3.2.0 runtime types)target_pathto document-based dataset generation andscenario_idto result filtersBackward compatibility:
evaluations.retrieve/evaluations.liststill acceptinclude=["agent"|"dataset"]so existing callers do notTypeError; the argument is ignored with aDeprecationWarningbecause the Hub no longer returns included payloadsresults.searchstill acceptsfilters["chat_test_case_id"]and maps it toscenario_idwith aDeprecationWarningRed team audits and
datasets.preview_importare intentionally omitted: audits are still behind a feature flag, and import preview is not needed in the SDK yet.Existing v3.2.0 aliases (prompt presets, dataset scenarios,
rerun_scenario,generate_preset_based) are unchanged.Test plan
uv run ruff check . --fixanduv run ruff formatuv run pytest— 579 passed (Prism tests remain disabled)uv run pyright— 0 errorsuv run mypy .— no issues