Skip to content

feat(examples): add auditable AgentOptimizer evaluation loop#159

Open
quan020406 wants to merge 15 commits into
trpc-group:mainfrom
quan020406:issue-91
Open

feat(examples): add auditable AgentOptimizer evaluation loop#159
quan020406 wants to merge 15 commits into
trpc-group:mainfrom
quan020406:issue-91

Conversation

@quan020406

Copy link
Copy Markdown

Summary

Closes #91.

Implements a self-contained, auditable evaluation and optimization loop under
examples/optimization/eval_optimize_loop without changing SDK public APIs.

What is included

  • Deterministic no-Key fake pipeline with AgentEvaluator-backed rubric scoring.
  • Offline trace mode with rule-first failure attribution.
  • Safe live AgentOptimizer backend:
    • validates required environment variables before startup;
    • always uses update_source=False and verbose=0;
    • independently evaluates every candidate on full train and validation splits;
    • only permits optional prompt write-back after Gate acceptance and CAS checks.
  • Conservative Gate, deterministic winner selection, and complete audit artifacts.
  • Stable, secret-redacted reports and sample output.
  • README and DESIGN documentation.

Safety decisions

  • Fake and trace modes require no API key, network, or LLM judge.
  • Live mode exits with code 2 before optimizer/network work when required variables are absent.
  • Optimizer success is not treated as acceptance; Gate decisions are based on independent regression results.
  • Runtime JSON artifacts recursively redact sensitive fields.

Verification

  • python -m pytest tests/examples/optimization/eval_optimize_loop -q
    • 85 passed
  • Fake CLI: exit 0, selects candidate_general_fix.
  • Trace CLI: exit 0, produces an auditable reject decision.
  • Live CLI with credentials absent: exit 2 before optimizer startup.
  • git diff --check passed.

Notes

Existing third-party LangChain and requests dependency warnings remain unchanged and do not affect the test results.

Deliverables

  • Added examples/optimization/eval_optimize_loop/, including the pipeline entrypoint, evalsets, prompt samples, optimizer configuration, README, and DESIGN.md.
  • Added 3 training and 3 validation evaluation cases.
  • Added deterministic fixture candidates covering an accepted optimization, a no-op candidate, and an overfitting/regression candidate.
  • Added committed, secret-free sample_output/optimization_report.json and supporting audit artifacts.
  • Added a 300–500 Chinese-character design note covering failure attribution, Gate acceptance, overfitting prevention, and artifact auditing.

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@4336564). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #159   +/-   ##
==========================================
  Coverage        ?   87.89198%           
==========================================
  Files           ?         467           
  Lines           ?       44103           
  Branches        ?           0           
==========================================
  Hits            ?       38763           
  Misses          ?        5340           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Evaluation + Optimization 的自动回归与提示词优化闭环

1 participant