Skip to content

feat(examples): add evaluation optimization closed-loop example#119

Open
YAO-001 wants to merge 35 commits into
trpc-group:mainfrom
YAO-001:codex/issue-91-eval-optimize-loop
Open

feat(examples): add evaluation optimization closed-loop example#119
YAO-001 wants to merge 35 commits into
trpc-group:mainfrom
YAO-001:codex/issue-91-eval-optimize-loop

Conversation

@YAO-001

@YAO-001 YAO-001 commented Jul 4, 2026

Copy link
Copy Markdown

English

Closes #91.

This PR adds a reproducible Evaluation + Optimization closed loop under examples/optimization/eval_optimize_loop.

What changed

  • Runs baseline evaluation on the complete train and validation EvalSets, preserving per-case metrics, pass/fail, failure evidence, and trace availability.
  • Classifies failures across response, format, tool, parameter, rubric, and knowledge-recall categories.
  • Supports deterministic offline fake mode and a real SDK path using AgentOptimizer, TargetPrompt, and post-optimization AgentEvaluator runs.
  • Deduplicates every round candidate plus best_prompts, then fully reevaluates each candidate on train and validation. SDK optimizer aggregates remain audit context, not gate evidence.
  • Applies one complete gate in both modes: validation improvement, overfit rejection, new hard failures, protected-case regressions, per-case score drops, and fail-closed cost budgets.
  • Keeps AgentOptimizer(update_source=False) internally. --update-source commits only the selected candidate after full gate acceptance, input-integrity checks, and audit preparation, with transactional rollback/conflict reporting.
  • Publishes each run once to an immutable run-specific directory using atomic no-replace semantics, with SHA-256/size manifest, input hashes, rounds, case results, deltas, gate decisions, costs, durations, and reproducibility command.
  • Recursively redacts credential-shaped optimizer keys, including separator/camelCase token variants, from all public audit snapshots while retaining safe fields such as max_tokens and token_budget.
  • Includes 3 train + 3 validation public cases, stable example JSON/Markdown reports, a 300–500 Chinese-character design note, independent hidden-style gate/attribution fixtures, and real-SDK integration coverage without live API calls.

Validation

  • python -m pytest examples/optimization/eval_optimize_loop/tests --tb=short — 349 passed, 2 Windows symlink-capability skips
  • python -m pytest examples/optimization/eval_optimize_loop/tests/test_acceptance_thresholds.py examples/optimization/eval_optimize_loop/tests/test_sdk_integration.py -q — 7 passed
  • python -m yapf --diff -r examples/optimization/eval_optimize_loop
  • python -m flake8 examples/optimization/eval_optimize_loop
  • changed trpc_agent_sdk/ YAPF + flake8 checks
  • python -m build — built trpc_agent_py-1.1.12 sdist and wheel; isolated wheel import succeeded
  • Python 3.12 repository run reached 87.35% coverage. The local Windows-only failures are the repository's existing POSIX/Bash/permission/symlink cases; the PR's Linux CI is the merge authority.

中文

本 PR 完成 Issue #91:新增可复现的 Evaluation + Optimization 自动闭环。fake 与 SDK 模式统一执行 baseline、失败归因、候选 train/validation 完整复评、逐 case delta、完整 gate 和审计落盘;SDK 内部始终禁止提前回写,只有候选通过全部门禁与完整性检查后,流水线才可事务性更新源 prompt。运行目录采用不可覆盖的原子发布,记录哈希清单、成本、耗时、轮次、评测结果和复现命令;配置快照会递归脱敏常见及 camelCase 密钥。独立验收测试覆盖 ≥80% gate 决策准确率、≥75% 归因准确率、过拟合拒绝和三分钟性能上限。

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@e113610). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #119   +/-   ##
==========================================
  Coverage        ?   87.56402%           
==========================================
  Files           ?         467           
  Lines           ?       44130           
  Branches        ?           0           
==========================================
  Hits            ?       38642           
  Misses          ?        5488           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@YAO-001

YAO-001 commented Jul 4, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Rook1ex added a commit to trpc-group/cla-database that referenced this pull request Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Evaluation + Optimization 的自动回归与提示词优化闭环

1 participant