feat: add behavioral next-action evaluations - #263
Conversation
Record the reasoning-item decision (#13) and fallback tiers, make the codex-lb endpoint env-driven, add compiler source and request params to the suite digest, and specify model-prefix stripping and message roles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopt committed scripts as assertions (drop the allowlisted-predicate registry), author the harness in TypeScript driving Promptfoo via its typed Node API, and make scenarios/catalog typed TS modules. Support the direct OpenAI endpoint alongside codex-lb, stop restricting model selection to codex-lb's whitelist (read from opencode.json), delete the dead direct-Slack-function clause, and replace the suite digest with file-path + Git SHA provenance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 49 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
📝 WalkthroughWalkthroughAdds a Promptfoo-backed behavioral next-action evaluation system with typed scenarios, frozen checkpoint compilation, response grading, model execution, result artifacts, initial scenarios, documentation, and supporting package configuration. ChangesBehavioral next-action evaluations
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant ScenarioAuthor
participant BehaviorCompiler
participant PromptfooRunner
participant ResponsesAPI
participant ArtifactWriter
ScenarioAuthor->>BehaviorCompiler: define typed scenario and frozen trajectory
BehaviorCompiler->>PromptfooRunner: produce compiled checkpoints
PromptfooRunner->>ResponsesAPI: send prompt, history, and tool catalog
ResponsesAPI-->>PromptfooRunner: return model response
PromptfooRunner->>ArtifactWriter: persist results.jsonl and summary.json
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.gitignore:
- Around line 26-27: Update the .gitignore entries for evaluator and Promptfoo
artifacts to ignore .context/behavior-evals/ and .context/promptfoo/ instead of
the unrelated benchmarks/behavior/results/ and .promptfoo/ paths.
In `@benchmarks/behavior/compile.ts`:
- Around line 441-445: Update the deployment and documentation surfaces for the
behavioral-evaluation configuration used by main and compileSuite: add
THOR_BEHAVIOR_EVAL_MODEL and THOR_BEHAVIOR_EVAL_REQUEST_PARAMS to
docker-compose.yml, .env.example, the README deployment configuration, and the
active behavioral-eval plan/docs, matching the existing
THOR_BEHAVIOR_EVAL_BASE_URL and THOR_BEHAVIOR_EVAL_API_KEY conventions.
- Around line 371-376: Update endpointKind to parse the endpoint as a URL and
classify using the parsed hostname and port rather than substring checks. Match
codex-lb only for the intended localhost/127.0.0.1 port 2455 endpoints, and
match openai only when the parsed host is exactly api.openai.com; return “other”
for deceptive hosts, paths, queries, or invalid URLs.
In `@benchmarks/behavior/run.ts`:
- Around line 265-317: Update main so evaluation artifacts are constructed and
persisted via writeArtifacts before assertProviderIdentity validates
summary.results. Keep the existing result mapping and replicate tracking intact,
then perform the provider-identity assertion after successful artifact
persistence so mismatches still fail without discarding collected results.
In `@README.md`:
- Around line 107-108: Update the THOR_BEHAVIOR_EVAL_API_KEY documentation row
in README.md so the key is required only by pnpm eval:behavior, not the offline
pnpm eval:behavior:validate command. Keep the existing description that the key
must never be committed and align the wording with .env.example and the
behavioral evaluation documentation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5ba7be84-a516-4642-a567-554d04527231
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (23)
.env.example.gitignoreREADME.mdbenchmarks/behavior/README.mdbenchmarks/behavior/assert-next-action.tsbenchmarks/behavior/asserts/grafana-loki-query-only.tsbenchmarks/behavior/asserts/internal-reply-without-slack-post.tsbenchmarks/behavior/compile.test.tsbenchmarks/behavior/compile.tsbenchmarks/behavior/package.jsonbenchmarks/behavior/run.tsbenchmarks/behavior/scenarios/direct-answer-about-thor.tsbenchmarks/behavior/scenarios/forbidden-pr-approval.tsbenchmarks/behavior/scenarios/incident-edge-localization.tsbenchmarks/behavior/scenarios/multi-tool-code-investigation.tsbenchmarks/behavior/scenarios/successful-ci-wake-stays-silent.tsbenchmarks/behavior/scenarios/untrusted-deployment-note.tsbenchmarks/behavior/tools.tsbenchmarks/behavior/tsconfig.jsonbenchmarks/behavior/types.tsdocs/plan/2026072401_behavioral-next-action-evals.mdpackage.jsonpnpm-workspace.yaml
| /benchmarks/behavior/results/ | ||
| /.promptfoo/ |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Ignore the directories the evaluator actually writes.
The documentation places default artifacts under .context/behavior-evals/ and Promptfoo state under .context/promptfoo/, but these rules ignore different paths. Raw responses, model outputs, reports, or state can therefore be staged accidentally.
Proposed fix
/benchmarks/behavior/results/
/.promptfoo/
+/.context/behavior-evals/
+/.context/promptfoo/📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /benchmarks/behavior/results/ | |
| /.promptfoo/ | |
| /benchmarks/behavior/results/ | |
| /.promptfoo/ | |
| /.context/behavior-evals/ | |
| /.context/promptfoo/ |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.gitignore around lines 26 - 27, Update the .gitignore entries for evaluator
and Promptfoo artifacts to ignore .context/behavior-evals/ and
.context/promptfoo/ instead of the unrelated benchmarks/behavior/results/ and
.promptfoo/ paths.
| | `THOR_BEHAVIOR_EVAL_BASE_URL` | No | local behavior evaluator | Responses API base URL for manual behavioral evals; defaults to host codex-lb, and may point to in-network codex-lb or the direct OpenAI API | | ||
| | `THOR_BEHAVIOR_EVAL_API_KEY` | For eval | local behavior evaluator | API key for the selected behavioral-eval endpoint; required by `pnpm eval:behavior` and never committed | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not require an API key for offline validation.
Line 108 says the key is required by pnpm eval:behavior:validate, but that command is documented as offline on Lines 200-204 and in benchmarks/behavior/README.md. .env.example also correctly limits the key to pnpm eval:behavior; update this row to avoid unnecessary credential configuration.
Proposed wording
-| `THOR_BEHAVIOR_EVAL_API_KEY` | For eval | local behavior evaluator | API key for the selected behavioral-eval endpoint; required by `pnpm eval:behavior` and never committed
+| `THOR_BEHAVIOR_EVAL_API_KEY` | For eval | local behavior evaluator | API key for the selected behavioral-eval endpoint; used by `pnpm eval:behavior` and never committed📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | `THOR_BEHAVIOR_EVAL_BASE_URL` | No | local behavior evaluator | Responses API base URL for manual behavioral evals; defaults to host codex-lb, and may point to in-network codex-lb or the direct OpenAI API | | |
| | `THOR_BEHAVIOR_EVAL_API_KEY` | For eval | local behavior evaluator | API key for the selected behavioral-eval endpoint; required by `pnpm eval:behavior` and never committed | | |
| | `THOR_BEHAVIOR_EVAL_BASE_URL` | No | local behavior evaluator | Responses API base URL for manual behavioral evals; defaults to host codex-lb, and may point to in-network codex-lb or the direct OpenAI API | | |
| | `THOR_BEHAVIOR_EVAL_API_KEY` | For eval | local behavior evaluator | API key for the selected behavioral-eval endpoint; used by `pnpm eval:behavior` and never committed | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 107 - 108, Update the THOR_BEHAVIOR_EVAL_API_KEY
documentation row in README.md so the key is required only by pnpm
eval:behavior, not the offline pnpm eval:behavior:validate command. Keep the
existing description that the key must never be committed and align the wording
with .env.example and the behavioral evaluation documentation.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/behavior/compile.ts`:
- Around line 373-391: Update the endpoint validation around parsed.protocol and
hostname so api.openai.com is accepted only when parsed.protocol is https:.
Preserve HTTP support for the local Codex-LB hosts and retain the existing
classification for other endpoints.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 13c6d4d8-21de-4712-b9b0-16d4910ee64e
📒 Files selected for processing (2)
benchmarks/behavior/compile.test.tsbenchmarks/behavior/compile.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- benchmarks/behavior/compile.test.ts
3 participants
Summary
Verification
pnpm test— 53 files, 858 tests passedpnpm typecheckpnpm eval:behavior:validate— 9 checkpoints from 6 scenarios/v1/responsesPromptfoo smoke — 1/1 passed with raw artifact outputThe local production-representative codex-lb model smoke was unavailable because no service was listening on
127.0.0.1:2455; model-backed evals remain manual as planned.Summary by CodeRabbit