feat(eval): 增加 Agent 轨迹评测回归门

[mirror29]

Summary

Orchestration seams

• Extract createOrchestrator() so production and evaluation share the same Mastra Agent construction without loading production providers, MCP, memory, or service processors in offline evals.
• Extract side-effect-free tool wiring while preserving the production hook order, permission YAML, shared runner/engine, dynamic MCP tools, and pending-plan processor.
• Lazy-load the default approval stores only when an ask path actually needs them.

Agent Eval

• Add versioned, fail-closed Golden Task fixtures with strict schema validation.
• Run scripted AI SDK v2 models and synthetic tools through the real Mastra multi-step loop, hooks, and permissions.
• Normalize FullOutput.steps into attempted/executed trajectories and grade required calls, ordering, input subsets, result classes, forbidden paths, budgets, and final text.
• Add structured failure classes, network isolation, per-trial approval-cache isolation, item timeouts, deterministic report ordering, live pass-rate/variance statistics, and recursive credential redaction.
• Keep real-model eval explicit and bounded: one selected case, explicit provider/model, opt-in environment flag, 3–5 serial trials, and no automatic retry.

CI

• Add a zero-secret PR Agent Eval job with an always-uploaded JSON report.
• Add a daily offline nightly workflow and manual dispatch entry.
• Keep paid live-model evaluation out of automatic workflows by design.

Test Coverage

CODE PATHS                                             CI / OPERATION
[★★★] Orchestrator factory and stateless eval Agent     [★★★] PR offline lane
[★★★] Production memory / processor / dynamic tools     [★★★] nightly artifact lane
[★★★] Strict golden schema and loader                    [★★★] live 3–5 trial guard
[★★★] Scripted generate / stream protocol                [★★★] report redaction
[★★★] Synthetic return / throw / delay / abort            [★★★] stable report + variance
[★★★] Permission deny / ask / raw-execute=0
[★★★] Timeout / network / cache isolation
[★★★] Trajectory association and result classes
[★★★] Multiset / subsequence / subset / forbidden / budget / outcome

COVERAGE: 61/64 behavior paths (95%)

Remaining gaps are intentionally outside the required offline lane: paid live-provider behavior, dynamic references from one fixture result into a later scripted call, and a provider that ignores its AbortSignal.

Tests: 38 → 57 files (+19).

Pre-Landing Review

• Two fix-first review rounds found and auto-fixed 19 test, isolation, reporting, and maintainability issues.
• Final Testing, Maintainability, Security, and Performance specialist passes reported no findings.
• Main safety checklist found no SQL, authorization-bypass, LLM trust-boundary, or CI configuration issues.
• Independent adversarial review correctly noted that scripted CI evaluates the harness and deterministic policy contracts, not real-model judgment. The live lane remains explicit because automatic paid evals require separate environment approval, secrets, and a spending policy.
• Codex was unavailable locally. The final red-team agent did not return before timeout; its focus areas were covered by the four specialist passes and the independent adversarial pass.

Design Review

No frontend files changed — design review skipped.

Eval Results

• eval:pr: 4/4 passed, scripted, no provider secrets or model cost.
• eval:nightly: 4/4 passed, scripted, no provider secrets or model cost.
• Report artifact scan: approval tokens, authorization headers, generic token fields, provider metadata, stack/cause, and credential-shaped strings are redacted.
• eval:live was not run; no paid model calls were made.

Scope Drift

CLEAN — the diff contains only the H0 Agent Eval bootstrap, its orchestration seams, tests, and CI. Existing E1 design work was not included.

Plan Completion

• 36 DONE, 2 CHANGED, 0 deferred, 0 unverifiable.
• Timeout recovery is split between a recoverable tool-error golden case and a hard runner-timeout test.
• The approval golden case proves first-attempt fail-closed behavior; same-turn retry hardening remains with the E1 integration branch before its later rebase.

Verification Results

• TypeScript typecheck
• Full Vitest suite: 51 files, 531 tests
• Agent Eval PR and nightly lanes: 4/4 each
• Ruff for data, paper, and research services
• Repository consistency check: 10 passed, 0 failed
• Workflow YAML parse
• JSON report redaction scan
• All added files remain below the repository's 4000-character limit

The full Vitest suite used one local worker because unrelated timing tests were unstable under heavy host CPU contention. GitHub CI still runs the repository's default parallel command.

Test plan

• Invalid, incomplete, duplicate, and undeclared Golden Task inputs fail closed.
• Direct-order and first-ask cases never enter raw tool execution.
• Plan calls preserve create → approve → execute order and approval-token input.
• Tool errors, hard timeouts, unexpected network access, and parallel trials remain isolated.
• CLI failures still emit redacted, machine-readable artifacts.
• Production Agent memory, processor, and dynamic tool wiring remain intact under mocked dependencies.

🤖 Generated with Claude Code

[github-actions]

Review base: 7482495

🤖 DeepSeek V4 Pro PR Review

⚠️ review 未完成：DeepSeek API 调用失败：DeepSeek 未返回可发布的 review 正文