feat(evaluation): add an auditable eval-optimize pipeline#169
Closed
neutronstar238 wants to merge 39 commits into
Closed
feat(evaluation): add an auditable eval-optimize pipeline#169neutronstar238 wants to merge 39 commits into
neutronstar238 wants to merge 39 commits into
Conversation
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
There was a problem hiding this comment.
Pull request overview
Adds an offline-first, auditable evaluation + prompt-optimization loop example (examples/optimization/eval_optimize_loop) and hardens SDK plumbing used by the pipeline (judge execution, provider schema capability detection, and streaming resource cleanup).
Changes:
- Introduces a reproducible eval/optimize pipeline example with evalsets, fixtures, JSON Schema contract, and documentation.
- Adds provider capability hooks for “native response schema” support (DeepSeek disables schemas; judge falls back to JSON mode).
- Improves runtime correctness: disables model streaming when requested and closes async streaming resources / agent runs deterministically (with new regression tests).
Reviewed changes
Copilot reviewed 29 out of 33 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| trpc_agent_sdk/models/openai_adapter/_deepseek.py | DeepSeek schema capability + JSON mode |
| trpc_agent_sdk/models/openai_adapter/_base.py | Default provider schema capability hook |
| trpc_agent_sdk/models/_openai_model.py | Stream cleanup + exposes schema capability |
| trpc_agent_sdk/evaluation/_llm_judge.py | Non-streaming judge runs + schema fallback |
| trpc_agent_sdk/evaluation/_agent_evaluator.py | Safer file.json:case_id parsing |
| trpc_agent_sdk/agents/core/_llm_processor.py | Buffer events for non-streaming runs |
| trpc_agent_sdk/agents/_llm_agent.py | Honor RunConfig.streaming |
| tests/models/test_openai_model_ext.py | Verifies stream closed before client |
| tests/evaluation/test_llm_judge_think.py | Verifies judge closes agent run + disables streaming |
| tests/evaluation/test_llm_judge_multi_model.py | DeepSeek judge uses JSON mode w/o schema |
| tests/agents/test_llm_agent_ext.py | Verifies non-streaming run disables model streaming |
| tests/agents/core/test_llm_processor.py | Verifies non-streaming drains before yielding |
| pyproject.toml | Adds jsonschema to extras |
| examples/optimization/eval_optimize_loop/val.evalset.json | Example validation evalset |
| examples/optimization/eval_optimize_loop/train.evalset.json | Example training evalset |
| examples/optimization/eval_optimize_loop/README.md | Example usage + design notes |
| examples/optimization/eval_optimize_loop/optimizer.json | Example optimizer configuration |
| examples/optimization/eval_optimize_loop/optimizer_dev.evalset.json | Optimizer holdout evalset |
| examples/optimization/eval_optimize_loop/optimization_report.schema.json | JSON Schema contract for reports |
| examples/optimization/eval_optimize_loop/fixtures/trace_outputs.json | Trace-mode fixture outputs |
| examples/optimization/eval_optimize_loop/fixtures/optimization_report.sample.json | Sample report artifact |
| examples/optimization/eval_optimize_loop/fixtures/offline_metrics.sample.json | Offline metric config sample |
| examples/optimization/eval_optimize_loop/fixtures/fake_outputs.json | Fake-mode fixture outputs |
| examples/optimization/eval_optimize_loop/agent/prompts/system.md | Example system prompt |
| examples/optimization/eval_optimize_loop/agent/prompts/router.md | Example routing prompt |
| examples/optimization/eval_optimize_loop/agent/config.py | Online mode env config helper |
| examples/optimization/eval_optimize_loop/agent/agent.py | Online agent factory |
| examples/optimization/eval_optimize_loop/agent/init.py | Package marker for example agent |
| docs/superpowers/plans/2026-07-10-eval-optimize-loop-hardening.md | Implementation plan doc (new) |
| AGENTS.md | Repo-wide agent contribution guidelines |
| .gitignore | Ignore generated runs/ and metrics artifacts |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1790
to
+1794
| try: | ||
| close_stream = getattr(response, "close", None) | ||
| if not callable(close_stream): | ||
| close_stream = getattr(response, "aclose", None) | ||
| if callable(close_stream): |
Comment on lines
+1
to
+3
| # Eval Optimize Loop Hidden-Test Hardening Implementation Plan | ||
|
|
||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #169 +/- ##
==========================================
Coverage ? 87.97662%
==========================================
Files ? 467
Lines ? 44139
Branches ? 0
==========================================
Hits ? 38832
Misses ? 5307
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Author
|
I have read the CLA Document and I hereby sign the CLA |
Rook1ex
added a commit
to trpc-group/cla-database
that referenced
this pull request
Jul 12, 2026
Author
|
Coverage follow-up for
A maintainer must either configure |
Author
|
Closing this PR so the verified current head can be submitted as a fresh PR and retrigger the external pull-request pipeline. A replacement link will follow. |
Author
|
Replacement pull request: #171 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #91
Summary Adds a reproducible evaluation and prompt-optimization loop under
examples/optimization/eval_optimize_loop. - Evaluates baseline, optimizer-dev, and validation sets separately with per-case metrics, pass/fail status, traces, and failure attribution. - Runs candidate prompt optimization throughAgentOptimizer/TargetPrompt, then re-evaluates each candidate against the validation set. - Produces schema-validated JSON and Markdown reports with baseline/candidate scores, case deltas, gate decisions, rejection reasons, costs, durations, seeds, and prompt artifacts. - Adds fail-closed acceptance gates for validation improvement, hard failures, critical-case regressions, malformed evidence, and budget constraints. - Includes deterministicfakeandtracemodes that run without API keys, plus explicit opt-in online mode. - Covers accepted optimization, no-op rejection, and training-only improvement with validation regression (overfitting) rejection. - Handles DeepSeek JSON-object response formatting without sending unsupported native schemas and closes judge execution paths cleanly. ## Validation - Ran the evaluation suite plus affected agent/model regression tests. - Ran the full pipeline in fake and trace modes; both completed in about 3.3 seconds. - Ran the online pipeline with configured credentials; the optimizer completed successfully and generated a schema-valid report. - Verified that the online run no longer emits the targeted DeepSeek response-format or async stream cleanup warnings.