feat: allow templating per-agent reasoning.effort and context_tier (#262)

[franklixuefei]

Summary

Allows per-agent reasoning.effort and context_tier to be templated with {{ workflow.input.* }} (and {% %} statement) Jinja2 expressions, resolved at runtime. Closes #262.

Both fields were strict pydantic Literal enums, so a templated value like effort: "{{ workflow.input.eff }}" was rejected at YAML load — before --input / Jinja2 templates are resolved. model: already escaped this because it is a free-form str resolved at runtime.

agents:
  - name: thinker
    reasoning:
      effort: "{{ workflow.input.effort }}"   # now accepted; resolved + validated at runtime
    context_tier: "{{ workflow.input.tier }}"

Approach: defer enum validation to runtime (issue Approach 2)

The issue marks Approach 1 ("resolve templates before enum validation") as preferred, but after reading the code Approach 2 is the idiomatic, lower-risk fit — and it mirrors a shipped, load-bearing precedent in the same file: the wait step's duration (templatable-but-validated via _validate_wait_duration + runtime render in executor/wait.py).

Why Approach 1 was rejected (grounded in the code):

• Inputs aren't available at load. --input values + workflow.input defaults are merged later in the engine, not in config/loader.py.
• conductor validate has no inputs at all, so a {{ workflow.input.x }} could never resolve during validate — you'd still need a "defer unresolved template" path.
• Chicken-and-egg: workflow.input defaults live inside the same YAML that model_validate parses.
• Scoping a pre-resolution to "only workflow.input, only these enum fields" converges right back to Approach 2's surgery anyway.

Changes

Schema (config/schema.py)

• Widen ReasoningConfig.effort → ReasoningEffort | str + a @model_validator(mode="after") that defers {{/{% templates and validates literals via get_args.
• Widen AgentDef.context_tier → ContextTier | str | None + _validate_context_tier() dispatched from the regular-agent branch of validate_agent_type. Non-agent step types still reject the field outright (a template string is still "not None").

Runtime (executor/agent.py)

• After the existing model render, render + validate templated reasoning.effort / context_tier with the full agent context. Every execution path (main / retry / dialog / for_each) routes through AgentExecutor.execute, so this single chokepoint covers them all. A _render_enum_field helper renders, strips trailing whitespace, validates the resolved literal, and raises a clear ValidationError on a bad resolution. The provider then sees a concrete enum — no provider changes.

Providers (providers/reasoning.py, providers/context_tier.py)

• resolve_* cast(...) the widened field back to the strict enum — by the time they run (provider execute, after the executor resolves the field) the value is concrete and validated.

Validator (config/validator.py)

• The reasoning.effort capability cross-check defers only the value-dependent membership check for templates. The value-independent "provider supports no reasoning at all" check (reasoning_effort=None) still fires for templated efforts — otherwise a templated effort on a no-reasoning provider would pass validate while a literal errors.

Rigor

Pre-impl + dual-model post-impl rubber-duck (Claude Opus 4.8 + GPT-5.5, run in parallel). Both independently flagged the validator capability gap above (fixed). Opus additionally caught a schema/executor asymmetry — the schema deferred only on {{ while the executor renders {{ or {%, so a {% if %} template was rejected at load though the executor would render it; both schema validators now mirror the executor guard. Verified end-to-end: {% if workflow.input.heavy %}xhigh{% else %}low{% endif %} resolves to xhigh/low per input.

Tests

• Schema: templated effort/context_tier deferred (incl. {% %} statements); literal-invalid still rejected; human_gate/script/workflow/etc still reject templated fields.
• Runtime: render for both fields, static pass-through, no-mutation of the original templated agent, bad-resolution raises, trailing-newline strip.
• Validator: templated effort defers the membership check; templated effort on a no-reasoning provider still errors.
• CLI: conductor validate accepts a templated-enum workflow (exit 0), new fixture valid_templated_enums.yaml.

Verification

• ruff check + ruff format --check: clean
• ty check src: 12 diagnostics, == baseline (0 introduced; all pre-existing Windows termios / dialog stubs)
• pytest: 3526 passed; the 13 failures are identical on a clean upstream/main checkout (pre-existing Windows-env: registry TOML, event_log non-serializable, script stdin tty) — 0 new failures.

Out of scope (deliberate non-goals)

• Workflow-level runtime.default_reasoning_effort / default_context_tier templating — the AC is per-agent only; these remain strict enums (a templated workflow-level default is rejected with a clear message). Can follow up separately.
• A generalized Annotated[...] "templatable enum" type — starting with | str + the duration-style validator to match existing code; generalize only if a third enum field needs it.

A minor cosmetic note: the workflow_started telemetry event reports the raw (unrendered) per-agent template, since per-for_each-item resolutions differ. This dict is telemetry-only (never sent to a provider), so there's no correctness impact — flagging it as a conscious choice.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@5d769eb). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #263   +/-   ##
=======================================
  Coverage        ?   86.41%           
=======================================
  Files           ?       69           
  Lines           ?    12149           
  Branches        ?        0           
=======================================
  Hits            ?    10498           
  Misses          ?     1651           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[franklixuefei]

@jrob5756 kindly requesting your review on this, thanks Jason!

[franklixuefei]

@jrob5756 kindly requesting your review on this, thanks Jason!

[jrob5756]

I ran a full pass over this: code quality, tests, types, comments, and error handling. The implementation is correct. The widen-at-schema, defer, render-in-executor, cast-in-resolver chain holds on every path I traced, and the new tests pass. No blocking issues.

A few non-blocking things, in rough priority:

Harden the two resolver casts (see the inline notes on reasoning.py and context_tier.py). The cast is sound today only because the executor always renders first, and the one direct provider.execute caller (the output validator) builds a synthetic agent without these fields. Nothing enforces that, so a guard is cheap insurance against a future caller.

The ("{{" in v or "{%" in v) check is now duplicated across five sites: both schema validators, the two executor blocks, and the capability cross-check. A shared is_jinja_template() helper keeps them from drifting. One catch worth flagging: it has to return TypeGuard[str], not bool, or the executor loses the narrowing it depends on and ty fails. A leaf module such as conductor/templating.py avoids an import cycle. Leave _validate_wait_duration as-is; its {{-only check is deliberate.

Two test gaps. Nothing renders a {% %} statement template through the executor, which is the only layer that actually renders it; if that guard regressed, the schema and validator tests would still pass. Separately, an empty resolution (a {% if %} with no else) raises instead of falling back to the runtime default, which is a real behavior choice worth pinning with a test.

Pre-existing, not introduced here: a context_tier set on a non-Copilot provider is silently dropped, with no capability check and no warning, unlike reasoning effort which is rejected at validate time. Templating extends that silent drop to the templated path. Probably a follow-up.

[franklixuefei]

I ran a full pass over this: code quality, tests, types, comments, and error handling. The implementation is correct. The widen-at-schema, defer, render-in-executor, cast-in-resolver chain holds on every path I traced, and the new tests pass. No blocking issues.

A few non-blocking things, in rough priority:

Harden the two resolver casts (see the inline notes on reasoning.py and context_tier.py). The cast is sound today only because the executor always renders first, and the one direct provider.execute caller (the output validator) builds a synthetic agent without these fields. Nothing enforces that, so a guard is cheap insurance against a future caller.

The ("{{" in v or "{%" in v) check is now duplicated across five sites: both schema validators, the two executor blocks, and the capability cross-check. A shared is_jinja_template() helper keeps them from drifting. One catch worth flagging: it has to return TypeGuard[str], not bool, or the executor loses the narrowing it depends on and ty fails. A leaf module such as conductor/templating.py avoids an import cycle. Leave _validate_wait_duration as-is; its {{-only check is deliberate.

Two test gaps. Nothing renders a {% %} statement template through the executor, which is the only layer that actually renders it; if that guard regressed, the schema and validator tests would still pass. Separately, an empty resolution (a {% if %} with no else) raises instead of falling back to the runtime default, which is a real behavior choice worth pinning with a test.

Pre-existing, not introduced here: a context_tier set on a non-Copilot provider is silently dropped, with no capability check and no warning, unlike reasoning effort which is rejected at validate time. Templating extends that silent drop to the templated path. Probably a follow-up.

Thanks @jrob5756! I addressed all comments.

re: shared is_jinja_template() helper (TypeGuard[str], leaf module)

Done (a4c698c) — added conductor/templating.py with is_jinja_template(value) -> TypeGuard[str], a leaf module with no intra-conductor imports so there's no cycle. Routed the duplicated sites through it: both schema validators, the two executor blocks, and the capability cross-check — plus the model render block (same predicate, same function, so leaving it inline would invite exactly the drift you're guarding against) and the two new resolver guards. _validate_wait_duration stays as-is per your note (its {{-only check is deliberate), as does router.py's {{+}} when check (a different predicate). Confirmed it returns TypeGuard[str], not bool — ty passes and the executor keeps its narrowing into _render_enum_field. Added a small unit test for the helper itself (detects {{ and {%, rejects plain strings and non-strings — None in particular, so the model guard could drop its old agent.model and … short-circuit).

re: test gap ({% %} statement through the executor)

Added (a4c698c) — test_statement_template_effort_is_rendered drives a {% if … %}…{% else %}…{% endif %} statement template through the executor (the only layer that renders), plus a matching one for context_tier. As you said, the schema/validator tests would have stayed green if that guard regressed, so these pin the actual render path.

re: review summary: empty-resolution behavior choice

Pinned (a4c698c) — test_effort_resolving_to_empty_raises covers a {% if … %}…{% endif %} with no matching branch (resolves to "").

On the behavior itself, I made a deliberate call to keep it fail-closed (raise) rather than fall back to the runtime default, and gave the empty case its own actionable message ("…resolved to an empty value" + a hint to add an else-branch or omit the field). The reasoning: it's consistent with the surrounding contracts — _render_enum_field already rejects any value not in allowed, and the new B1/B2 resolver guards are fail-closed too — so treating empty as a special "unset → default" would add a third, silent meaning for empty that could mask a genuinely broken conditional. It also keeps this a pure pin rather than a behavior change with a type wrinkle: ReasoningConfig.effort is ReasoningEffort | str (not None-able), so "fall back to default" would mean making the override itself droppable, not just resolving it.

If you'd rather it fall back to the runtime default (treat an empty conditional as "no override"), I'm happy to flip it — I mainly wanted to pin the current behavior and surface the choice explicitly, as you suggested.

re: context_tier silently dropped on non-Copilot providers (pre-existing)

Agreed this is worth a follow-up, and that it's pre-existing — templating just extends the existing silent drop to the templated path. It's a real asymmetry vs reasoning effort (which is rejected at validate time via the capability cross-check). I'd keep it out of this PR to keep the scope tight; happy to open an issue to add a capability-style check (or at least a warning) for context_tier on providers that don't consume it. Let me know if you'd like me to file that.

[jrob5756]

Thanks for turning these around fast. One snag: commit a4c698c that all five replies point to isn't on the branch.

The head is still f5d5ee52 from June 19 (2 commits, both pre-review), and when I pull the live files none of the changes are there:

• reasoning.py / context_tier.py resolvers still do a bare cast with no guard
• the shared is_jinja_template() helper and templating.py module 404
• the agent.py comment still says Enum | str | None
• the schema docstrings still claim they mirror _validate_wait_duration

Looks like the push landed on a different branch, or got dropped. Could you push a4c698c to xuefl/templatable-reasoning-enums? Once the head moves I'll re-verify. None of it's blocking; I just want the branch history to match the replies.

[franklixuefei]

Thanks for turning these around fast. One snag: commit a4c698c that all five replies point to isn't on the branch.

The head is still f5d5ee52 from June 19 (2 commits, both pre-review), and when I pull the live files none of the changes are there:

• reasoning.py / context_tier.py resolvers still do a bare cast with no guard
• the shared is_jinja_template() helper and templating.py module 404
• the agent.py comment still says Enum | str | None
• the schema docstrings still claim they mirror _validate_wait_duration

Looks like the push landed on a different branch, or got dropped. Could you push a4c698c to xuefl/templatable-reasoning-enums? Once the head moves I'll re-verify. None of it's blocking; I just want the branch history to match the replies.

Ah sorry about that... now my local commit is pushed, please check again. Thanks Jason!

[jrob5756]

LGTM. Re-verified at 7d42fb06: all eight points are in, the shared is_jinja_template helper is wired through every site, both resolver guards are there, and the docstrings/comments read correctly now. Lint and format clean, ty is at baseline (the lone dialog_evaluator diagnostic is pre-existing), and the affected suites are green (357 passed). Thanks for the thorough turnaround.