Skip to content

feat: add evidence-backed evaluation workflow - #39

Merged
regenrek merged 8 commits into
mainfrom
codex/eval-security-evidence
Jul 22, 2026
Merged

feat: add evidence-backed evaluation workflow#39
regenrek merged 8 commits into
mainfrom
codex/eval-security-evidence

Conversation

@regenrek

Copy link
Copy Markdown
Contributor

Why

Planr needs stable, replayable evaluation evidence so quality, speed, and cost-per-task improvements can be promoted without trusting one-off runs or token prices alone.

What

  • define and implement the durable eval contract, storage, CLI/MCP surfaces, comparison gates, rescoring, invalidation, resume lineage, and cost-per-verified-success evidence
  • add deterministic and live lifecycle coverage, including failure, stale, noisy, incompatible, interrupted, imported, and metering cases
  • harden secret/privacy checks, GitHub Actions, dependency policy, and packaging boundaries
  • remove ignored local workspace state from the committed pnpm lockfile while preserving main version 1.6.0 and its Dependabot policy
  • sanitize live-oracle output, clean temporary artifacts, and reduce startup-jitter flakiness without weakening the 10% regression threshold

Verification

  • pnpm install --frozen-lockfile
  • scripts/ci-local.sh: 246 Rust tests, clippy/fmt, release build, npm dry-run, shellcheck, cargo audit
  • pnpm --filter @planr/docs build
  • scripts/security-local.sh: BetterLeaks clean; Trivy reports 0 Cargo and pnpm vulnerabilities
  • scripts/verify-eval-contract-live.sh passed twice consecutively with unchanged local config fingerprints and automatic temp cleanup

No release or tag is included in this PR.

@regenrek
regenrek merged commit 78250d2 into main Jul 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant