feat(evaluation): add an auditable eval-optimize pipeline

[neutronstar238]

Closes #91

Supersedes #169, which was closed so the verified current head could be submitted as a fresh pull request.

Summary

• Adds a reproducible evaluation and prompt-optimization loop under examples/optimization/eval_optimize_loop.
• Separates baseline train/dev/validation evaluation and records per-case metrics, pass/fail status, traces, and failure attribution.
• Optimizes auditable TargetPrompt candidates through AgentOptimizer, then re-evaluates every candidate on the validation set.
• Produces schema-validated JSON and Markdown reports with scores, case deltas, gate decisions, costs, durations, seeds, rejection reasons, and prompt artifacts.
• Enforces fail-closed acceptance gates for validation improvement, hard failures, critical regressions, malformed evidence, and budget constraints.
• Includes deterministic fake and trace modes without API keys, plus explicit opt-in online execution.
• Covers accepted optimization, no-op rejection, and validation-regression rejection.
• Handles DeepSeek JSON response-format compatibility and closes asynchronous judge runs cleanly.

Validation

• GitHub CI: 9,667 passed, 2 skipped, 1 xfailed.
• Total coverage: 87.98%.
• Codecov patch: all modified and coverable lines are covered.
• Lint, build, CLA, and open-source scan passed.
• Fake and trace pipelines generate complete schema-valid reports.
• Online execution with configured credentials completed successfully without the targeted response-format or async stream cleanup warnings.

[Copilot]

Pull request overview

This PR adds a new, offline-first evaluation + prompt-optimization loop example with auditable artifacts and hard gates, while also hardening core model/agent streaming behavior and improving judge/model compatibility for providers (notably DeepSeek) that don’t support native JSON schemas.

Changes:

• Introduces examples/optimization/eval_optimize_loop/ with evalsets, fixtures, schema-validated reports, and documentation for fake/trace/online modes.
• Adds provider capability plumbing for “native response schema” support (DeepSeek → JSON-object mode without schema) and updates LLM judge behavior accordingly.
• Improves non-streaming execution paths and streaming cleanup (closing streams and disabling streaming for judge runs), plus makes evalset file.json:case_id parsing safer.

Reviewed changes

Copilot reviewed 30 out of 34 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
trpc_agent_sdk/models/openai_adapter/_deepseek.py	Declares DeepSeek schema capability (JSON mode only) via supports_response_schema().
trpc_agent_sdk/models/openai_adapter/_base.py	Adds default supports_response_schema() capability hook for adapters.
trpc_agent_sdk/models/_openai_model.py	Delegates schema capability to adapter; ensures streaming responses are closed before the client.
trpc_agent_sdk/evaluation/_llm_judge.py	Disables model streaming for judge runs; closes judge agent runs; avoids sending schemas to schema-incapable models.
trpc_agent_sdk/evaluation/_agent_evaluator.py	Makes file.json:case_id selector parsing safer (avoids Windows drive-letter conflicts).
trpc_agent_sdk/agents/core/_llm_processor.py	Buffers events when stream=False so non-streaming runs don’t emit partial events mid-call.
trpc_agent_sdk/agents/_llm_agent.py	Threads RunConfig.streaming through to model invocation (streaming opt-out).
tests/models/test_openai_model_ext.py	Adds regression test ensuring stream cleanup happens before client cleanup.
tests/evaluation/test_llm_judge_think.py	Adds tests for judge text collection, run closure, and streaming disabled via RunConfig.
tests/evaluation/test_llm_judge_multi_model.py	Validates DeepSeek judge uses JSON mode without native schema; schema-capable models keep schema.
tests/evaluation/test_agent_evaluator.py	Tests evalset path suffix selection (path.json:case_id).
tests/agents/test_llm_agent_ext.py	Tests that disabling streaming in RunConfig prevents provider streaming.
tests/agents/core/test_llm_processor.py	Tests that non-streaming drains/finishes the model generator before yielding.
pyproject.toml	Adds jsonschema dependency to extras used by eval/dev/all.
examples/optimization/eval_optimize_loop/README.md	Documents the eval/optimize loop, modes, inputs/outputs, and design notes.
examples/optimization/eval_optimize_loop/train.evalset.json	Adds training evalset (3 cases).
examples/optimization/eval_optimize_loop/optimizer_dev.evalset.json	Adds optimizer-dev evalset (3 cases) separated from train/val.
examples/optimization/eval_optimize_loop/val.evalset.json	Adds validation evalset (3 cases, including a tagged critical case).
examples/optimization/eval_optimize_loop/optimizer.json	Adds bounded optimizer config for online mode.
examples/optimization/eval_optimize_loop/optimization_report.schema.json	Adds JSON Schema contract for the optimization report.
examples/optimization/eval_optimize_loop/fixtures/fake_outputs.json	Deterministic fake-mode fixture outputs for baseline + candidates.
examples/optimization/eval_optimize_loop/fixtures/trace_outputs.json	Deterministic trace-mode fixture outputs for replay.
examples/optimization/eval_optimize_loop/fixtures/offline_metrics.sample.json	Sample offline metrics config used in fixtures/sample report.
examples/optimization/eval_optimize_loop/fixtures/optimization_report.sample.json	Checked-in sample optimization report artifact.
examples/optimization/eval_optimize_loop/agent/prompts/system.md	Sample system prompt artifact for the example agent.
examples/optimization/eval_optimize_loop/agent/prompts/router.md	Sample router prompt artifact for the example agent.
examples/optimization/eval_optimize_loop/agent/config.py	Online-mode env var validation helper for example.
examples/optimization/eval_optimize_loop/agent/agent.py	Example LlmAgent factory reading prompt files.
examples/optimization/eval_optimize_loop/agent/init.py	Package marker for example agent.
docs/superpowers/plans/2026-07-10-eval-optimize-loop-hardening.md	Adds an implementation/hardening plan doc for the example pipeline.
AGENTS.md	Adds repo-wide agent/eval/optimization guidelines (offline-first, opt-in online, audit outputs).
.gitignore	Ignores runs/ and other generated eval/trace metrics artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.