Skip to content

reasoningEffort: executor-level guard for deprecated sampling params is Python-only (cross-runtime parity gap) #523

Open

Description

Summary

reasoningEffort was added to ModelOptions in TypeSpec (fc07e78) and flows to all seven runtimes as a generated model field. However, the executor-level behavior that acts on it exists only in the Python runtime. The other runtimes carry the field but do nothing with it at call time, which is a cross-runtime parity gap.

What Python does (the reference behavior)

In runtime/python/prompty/prompty/providers/openai/executor.py:

  • _is_reasoning_model(model_id) — name-based detection of OpenAI reasoning models (o1/o3/o4/o5…, GPT-5 family).
  • _OPENAI_REASONING_UNSUPPORTED = ("temperature", "top_p", "top_k", "frequency_penalty", "presence_penalty").
  • A live-call capability guard applied in the executor arg builders just before the SDK call:
    • Reasoning models: drop the scalar sampling params above, each with a DeprecationWarning ("… does not support '{key}'; dropping it. Use reasoningEffort instead.").
    • Non-reasoning models: drop reasoning_effort (only valid on reasoning models).

This is deliberately in the executor (a live-call concern), not in the pure to_wire option mappers.

The gap

reasoningEffort appears only in generated model files (model_options, session_start_payload, generated tests) in Go, Rust, C#, Java, TypeScript, and Swift — none of their execution layers translate reasoningEffort into the provider request or guard the deprecated sampling params. Verified via git grep: no executor/provider references outside Python.

Runtimes with execution layers that should reach parity: Rust (src/pipeline, src/engine), C# (Prompty.Foundry executors), TypeScript (packages/foundry). Model-only runtimes are out of scope until they grow a live executor.

Why it matters

This is exactly the class of gap the live-integration vectors are meant to surface: newer reasoning models reject temperature/top_p, so a runtime that forwards them will get live API errors while Python silently adapts. Cross-runtime behavior should match.

Proposed resolution

  1. Define the reasoning-model detection + unsupported-sampling-param set once (ideally data-driven from TypeSpec metadata so it isn't re-hardcoded per runtime — see note below).
  2. Port the executor-level guard to Rust, C#, and TypeScript executors (drop unsupported sampling params on reasoning models; drop reasoningEffort on non-reasoning models).
  3. Add shared conformance coverage so the guard's behavior is asserted cross-runtime rather than per-runtime.

Note on enforcement

The per-runtime name-based detection is duplicated logic. Consider whether the reasoning-model classification and its unsupported-param set belong in the TypeSpec model surface (so all runtimes derive the same table) rather than each runtime hardcoding OpenAI naming conventions. Tracked thought, not a blocker.

Acceptance criteria

  • Rust, C#, and TypeScript executors guard deprecated sampling params for reasoning models and translate reasoningEffort, matching Python.
  • Shared conformance asserts the behavior across runtimes.

Related: introduced in PR #522 (Python-only guard 09f5ce2). Turn-engine coverage backlog is separate (#519).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions