feat(examples): add evaluation optimization closed-loop example#119
Open
YAO-001 wants to merge 35 commits into
Open
feat(examples): add evaluation optimization closed-loop example#119YAO-001 wants to merge 35 commits into
YAO-001 wants to merge 35 commits into
Conversation
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #119 +/- ##
==========================================
Coverage ? 87.56402%
==========================================
Files ? 467
Lines ? 44130
Branches ? 0
==========================================
Hits ? 38642
Misses ? 5488
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Author
|
I have read the CLA Document and I hereby sign the CLA |
Rook1ex
added a commit
to trpc-group/cla-database
that referenced
this pull request
Jul 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
English
Closes #91.
This PR adds a reproducible Evaluation + Optimization closed loop under
examples/optimization/eval_optimize_loop.What changed
AgentOptimizer,TargetPrompt, and post-optimizationAgentEvaluatorruns.best_prompts, then fully reevaluates each candidate on train and validation. SDK optimizer aggregates remain audit context, not gate evidence.AgentOptimizer(update_source=False)internally.--update-sourcecommits only the selected candidate after full gate acceptance, input-integrity checks, and audit preparation, with transactional rollback/conflict reporting.max_tokensandtoken_budget.Validation
python -m pytest examples/optimization/eval_optimize_loop/tests --tb=short— 349 passed, 2 Windows symlink-capability skipspython -m pytest examples/optimization/eval_optimize_loop/tests/test_acceptance_thresholds.py examples/optimization/eval_optimize_loop/tests/test_sdk_integration.py -q— 7 passedpython -m yapf --diff -r examples/optimization/eval_optimize_looppython -m flake8 examples/optimization/eval_optimize_looptrpc_agent_sdk/YAPF + flake8 checkspython -m build— builttrpc_agent_py-1.1.12sdist and wheel; isolated wheel import succeeded中文
本 PR 完成 Issue #91:新增可复现的 Evaluation + Optimization 自动闭环。fake 与 SDK 模式统一执行 baseline、失败归因、候选 train/validation 完整复评、逐 case delta、完整 gate 和审计落盘;SDK 内部始终禁止提前回写,只有候选通过全部门禁与完整性检查后,流水线才可事务性更新源 prompt。运行目录采用不可覆盖的原子发布,记录哈希清单、成本、耗时、轮次、评测结果和复现命令;配置快照会递归脱敏常见及 camelCase 密钥。独立验收测试覆盖 ≥80% gate 决策准确率、≥75% 归因准确率、过拟合拒绝和三分钟性能上限。