fix(planning): make auto model defaults runtime-aware; surface empty harness completions distinctly#108
Open
AbirAbbas wants to merge 3 commits into
Open
fix(planning): make auto model defaults runtime-aware; surface empty harness completions distinctly#108AbirAbbas wants to merge 3 commits into
AbirAbbas wants to merge 3 commits into
Conversation
The planning pipeline resolved its default model from env keys alone via
_default_planning_model(), ignoring the caller's resolved runtime. A caller
that pinned ai_provider="codex" in an OpenRouter-only environment therefore
had the codex CLI spawned with an "openrouter/..." model its OpenAI backend
cannot resolve — completing in ~1s with a null message.
Thread the resolved runtime into _default_planning_model() and delegate to
the already-runtime-aware resolve_runtime_models() cascade for the high-tier
pm role. The auto default is now gated on runtime:
- codex -> a codex-native model (never openrouter/...-prefixed)
- open_code -> the OpenRouter auto default (OpenRouter-only env) or the
open_code base default otherwise
- claude_code -> the "sonnet" historical default
Explicit args and deployer env (SWE_DEFAULT_MODEL / AI_MODEL / HARNESS_MODEL,
SWE_MODEL_HIGH) still win verbatim — only the auto default became
runtime-aware. Omitting the runtime arg preserves the prior env-only behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…der+model An empty harness completion (no parsed object, no text — the signature of a provider/model mismatch or bad auth) was collapsed into the generic "failed to produce a valid <artifact>" message, indistinguishable from a genuine schema-quality failure. Two distinct root causes, one opaque error. Add EmptyHarnessCompletionError and check_empty_harness_completion(), and wire them into the PM, architect, tech-lead and sprint-planner reasoners between the fatal-error check and the parsed-None check. The empty case now raises an error naming the provider and model; the generic schema-invalid message is kept only for non-empty-but-unparseable output and also gains provider+model context. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n error Matrix over (env: openrouter-only / anthropic / SWE_DEFAULT_MODEL) x (runtime: open_code / codex / claude_code) asserting the resolved planning default, including that codex never yields an openrouter/-prefixed id. Plus unit and reasoner-level tests that an empty harness completion raises the distinct EmptyHarnessCompletionError naming provider and model, separate from the schema-invalid message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
A caller invoked
swe-planner.planwithai_provider="codex"explicitly and nomodel overrides, in an environment where only
OPENROUTER_API_KEYwas set (noAnthropic/OpenAI keys, no
SWE_DEFAULT_*vars). Two independent failurescollapsed into one opaque error:
Cross-runtime model-id leakage. The planning-model default cascade
(
_default_planning_model()) resolved from env keys alone, ignoring thecaller's pinned runtime. With only an OpenRouter key present it returned the
OpenRouter auto-default model id — so the codex CLI was spawned with an
openrouter/…-prefixed model its OpenAI backend cannot resolve. Everyattempt completed in ~1s with a null/empty message.
Opaque empty completions. The pipeline surfaced only the generic
"Product manager failed to produce a valid PRD" — indistinguishable from a
genuine schema-quality failure (a weak model emitting unparseable output).
One message, two very different root causes.
What changed
Runtime-aware auto default (
fix(planning)) —_default_planning_model()now takes the resolved runtime and delegates to the already-runtime-aware
resolve_runtime_models()cascade for the high-tierpmrole.plan()threadsthe resolved
ai_providerinto it. The auto default is gated on runtime:codex→ a codex-native model (never anopenrouter/…-prefixed id)open_code→ the OpenRouter auto default (OpenRouter-only env) or theopen_codebase default otherwiseclaude_code→ the historicalsonnetdefaultExplicitly passed models and deployer env (
SWE_DEFAULT_MODEL/AI_MODEL/HARNESS_MODEL,SWE_MODEL_HIGH) still win verbatim — only the autodefault became runtime-aware. Omitting the runtime arg preserves the prior
env-only behavior, so existing call sites and tests are unaffected.
Distinct empty-completion error (
fix(runtime)) — newEmptyHarnessCompletionError+check_empty_harness_completion(), wired intothe PM, architect, tech-lead and sprint-planner reasoners between the
fatal-error check and the
parsed is Nonecheck. An empty completion (no parsedobject and no text — the signature of a provider/model mismatch or bad auth)
now raises an error naming the provider and model, e.g.:
The generic "failed to produce a valid …" message is kept only for genuinely
non-empty-but-unparseable output, and now also includes provider+model context.
Validation Contract
codex→ auto default is NOTopenrouter/-prefixed (codex-native)test_default_planning_model_matrix[openrouter_only-codex],test_codex_default_is_never_openrouter_prefixedopen_code→ OpenRouter auto default preservedtest_default_planning_model_matrix[openrouter_only-open_code]claude_code→sonnetdefault preservedtest_default_planning_model_matrix[*-claude_code]SWE_DEFAULT_MODEL/ explicit arg wins in all casestest_default_planning_model_matrix[swe_default_model-*], existingtest_plan_explicit_args_override_env,test_plan_swe_default_model_overrides_openrouter_autoTestCheckEmptyHarnessCompletion::*,test_run_product_manager_empty_completion_surfaces_provider_and_modelAlso covered: runtime-alias normalization, the omitted-runtime env fallback, and
that non-empty-but-unparseable output is a no-op for the empty-completion check
(so the generic schema-invalid path still fires).
Test results
make check(full pytest suite +compileall, Python 3.12,AGENTFIELD_SERVERset as in CI): green.and reasoner-level tests) passing.
ruff checkon all changed/new files: clean.module.
🤖 Generated with Claude Code