examples: add evaluation optimization regression loop

[16yunH]

Summary

• Add a one-command, no-API-key Evaluation + Optimization regression loop under examples/optimization/eval_optimize_loop.
• Run the real AgentOptimizer + GEPA + TargetPrompt path, capture every unique proposal (including optimizer-rejected proposals), and independently replay each candidate through AgentEvaluator on train and validation splits.
• Produce per-case score/pass deltas, six evidence-backed failure categories, paired bootstrap confidence intervals, configurable safety/resource gates, Pareto selection, and atomic JSON/Markdown audit reports.
• Include 3 train cases, 3 validation cases, ineffective/overfit/robust prompt candidates, a reproducible sample report, a 300–500 Chinese-character design note, and fail-closed regression tests.
• Harden hidden-case behavior for ambiguous replay queries, key-case pass regressions, non-finite metrics, common tool-error payloads, cross-platform artifact paths, and concurrent runs.

Why

Issue #91 requires a reproducible closed loop that turns prompt optimization into an auditable promotion decision. Optimizer aggregate scores alone cannot show validation regressions, newly failed hard cases, failure causes, or whether a candidate stayed within its resource budget. This example keeps optimization and release gating separate: every proposal is re-evaluated from materialized traces before it can be selected or applied.

Design and safety

• The public seam is EvalOptimizePipeline.run(); production SDK code is unchanged.
• Offline agent, judge, and reflector models are registered through ModelRegistry, so the flow still exercises LlmAgent, rubric parsing, GEPA callbacks, and AgentEvaluator without monkeypatching or network access.
• Train/validation overlap, near-duplicates, prompt leakage, partial candidate evaluation, unknown cost, key-case regression, new hard failures, and train-up/validation-down overfitting all fail closed.
• Format attribution prioritizes evaluator rubric details, then replay evidence, then explicit positive JSON/single-line/Markdown instructions. Negated requirements and unevaluated judges are not misreported as semantic failures.
• Optimizer writes use an output-local prompt workspace, so source prompts remain untouched by default; applying an accepted prompt still requires the explicit --apply-if-accepted option and uses atomic TargetPrompt.write_all().

Validation

• pytest -q tests/evaluation/test_eval_optimize_loop_example.py — 41 passed.
• pytest -q tests/evaluation — 930 passed.
• GitHub CI pytest --cov=trpc_agent_sdk --cov-fail-under=80 tests/ — 9,386 passed, 1 expected XFAIL, 88.13% coverage.
• flake8 examples/optimization/eval_optimize_loop tests/evaluation/test_eval_optimize_loop_example.py --max-line-length=120 — passed.
• yapf --diff --recursive examples/optimization/eval_optimize_loop tests/evaluation/test_eval_optimize_loop_example.py — clean.
• python examples/optimization/eval_optimize_loop/run_pipeline.py --output-dir /tmp/issue91-audit — accepted robust, 100% failed-case attribution coverage, no API key, and 1.80 seconds wall time.
• Two simultaneous subprocess runs with separate output directories both accepted robust without changing the source prompt; same-process concurrency fails closed.

The single XFAIL is an existing expected repository test and is unrelated to this branch.

Requested PR type: type/feature.

Fixes #91

RELEASE NOTES: Add a reproducible, auditable Evaluation + Optimization regression-loop example that runs without external model credentials.

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@e113610). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff             @@
##             main        #161   +/-   ##
==========================================
  Coverage        ?   88.12552%           
==========================================
  Files           ?         467           
  Lines           ?       44103           
  Branches        ?           0           
==========================================
  Hits            ?       38866           
  Misses          ?        5237           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[16yunH]

I have read the CLA Document and I hereby sign the CLA

[Copilot]

Pull request overview

Adds a new end-to-end Evaluation + Optimization regression-loop example under examples/optimization/eval_optimize_loop, exercising the real AgentOptimizer (GEPA) + TargetPrompt path while independently re-evaluating every unique candidate with AgentEvaluator trace replay and producing auditable JSON/Markdown reports and gates.

Changes:

• Introduces an offline, no-API-key pipeline (EvalOptimizePipeline.run()) that runs baseline eval → optimizer proposals capture → per-candidate independent replay eval → deltas/gates → atomic reports.
• Adds deterministic offline agent/judge/reflector adapters via ModelRegistry, plus replay trace materialization and evidence-backed failure attribution.
• Adds comprehensive regression tests validating deliverables, data-quality protections, gating behavior, and report outputs.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/evaluation/test_eval_optimize_loop_example.py	Public-interface regression tests covering deliverables, gates, replay attribution, cost handling, and concurrency fail-closed behavior.
examples/optimization/eval_optimize_loop/README.md	Usage guide and design rationale for the closed-loop example, including inputs/outputs and gate semantics.
examples/optimization/eval_optimize_loop/DESIGN.md	300–500 Chinese-character design note describing attribution, gates, anti-overfit, and auditing.
examples/optimization/eval_optimize_loop/run_pipeline.py	One-command CLI entrypoint to run the pipeline and emit decision + report paths.
examples/optimization/eval_optimize_loop/pipeline.json	Example manifest wiring datasets/configs/prompts/candidates and run controls (seed/bootstrap/etc.).
examples/optimization/eval_optimize_loop/optimizer.json	Offline-validated GEPA optimizer configuration used by AgentOptimizer.
examples/optimization/eval_optimize_loop/regression.metrics.json	Trace-only regression metrics config for independent candidate replay evaluation.
examples/optimization/eval_optimize_loop/gate.json	Gate policy config for validation gain, CI floor, regression protections, and budgets.
examples/optimization/eval_optimize_loop/sample_output/optimization_report.md	Reproducible sample Markdown report output for the example.
examples/optimization/eval_optimize_loop/data/train.evalset.json	Offline train evalset with deterministic per-variant replay traces and usage accounting.
examples/optimization/eval_optimize_loop/data/val.evalset.json	Offline validation evalset including a key hard-fail format regression case.
examples/optimization/eval_optimize_loop/agent/init.py	Declares prompt assets package for the example.
examples/optimization/eval_optimize_loop/agent/prompts/system.md	Baseline system prompt (includes [variant: baseline] tag).
examples/optimization/eval_optimize_loop/agent/prompts/candidates/ineffective.md	Deterministic candidate prompt fixture for offline reflection.
examples/optimization/eval_optimize_loop/agent/prompts/candidates/overfit.md	Deterministic overfit candidate prompt fixture for regression rejection scenarios.
examples/optimization/eval_optimize_loop/agent/prompts/candidates/robust.md	Deterministic robust candidate prompt fixture expected to be accepted.
examples/optimization/eval_optimize_loop/loop/init.py	Public seam exports (EvalOptimizePipeline, PipelineSpec, OptimizationReport).
examples/optimization/eval_optimize_loop/loop/models.py	Pydantic models for manifests, audits, deltas, gate checks, and report schema.
examples/optimization/eval_optimize_loop/loop/offline.py	Patch-free offline LLMModel provider (agent/judge/reflector) and real LlmAgent call path.
examples/optimization/eval_optimize_loop/loop/trace.py	Replay trace materialization, SDK evaluation, normalization, and deterministic failure attribution.
examples/optimization/eval_optimize_loop/loop/analysis.py	Data-quality checks, paired bootstrap CI, resource accounting, gates, and Pareto marking.
examples/optimization/eval_optimize_loop/loop/pipeline.py	Orchestrates baseline, optimization, proposal capture, independent candidate evaluation, gating, and reporting.
examples/optimization/eval_optimize_loop/loop/reporting.py	Atomic JSON/Markdown report persistence and Markdown rendering.
examples/optimization/eval_optimize_loop/.gitignore	Ignores local runs and selected sample artifacts.
examples/optimization/eval_optimize_loop/init.py	Declares the example package.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.