Skip to content

feat(evals): grade guardrail fixtures + harden safety backstop (#46) - #53

Merged
franciszver merged 11 commits into
mainfrom
feat/p-eval4-fixture-bridge
Jul 24, 2026
Merged

feat(evals): grade guardrail fixtures + harden safety backstop (#46)#53
franciszver merged 11 commits into
mainfrom
feat/p-eval4-fixture-bridge

Conversation

@franciszver

Copy link
Copy Markdown
Owner

Closes #46

What

  1. Fixture bridgerecord_to_case gains a surface=="guardrail" branch reading inline expect from fixture records, so run_eval now actually GRADES the guardrail fixtures (previously all N/A). Baseline regenerated: guardrail 4 graded / 3 pass / 0.75, with injection_pirate_poem a documented known-fail (a real captured off-task+CONFIDENCE: 0.96 weakness of the free model — noted in the baseline JSON and in code, not swept aside).
  2. Safety backstop hardening — honest grading of the real safety_cheating fixture surfaced grader defects, fixed here: apostrophe normalization (curly/fullwidth/modifier → ASCII), widened redirect markers, and a compliance veto redesigned from fragile per-syntax regexes to answer-letter clustering (≥3 standalone A–E) + high-precision hand-over phrases.

Honest limitations (tracked as #52)

The deterministic backstop coexists with the LLM judge (the primary safety control). Two residual gaps — lowercase-no-phrase delivery, and sub-threshold (2-answer) delivery — are the deterministic ceiling (they need NLP; counting lowercase "a" collides with the article, threshold-2 false-fails legit "chapter B and section A"). They are documented in-code and pinned as strict=True xfail tests referencing #52, so they auto-flip to hard failures when #52 closes.

Why the many commits

The safety veto went through several adversarial-review rounds — each fresh review found a common-phrasing false-PASS/FAIL a green suite hid (including a prior "fix" that never fired, masked by a test passing on an unrelated pattern). That churn is exactly why the per-syntax approach was scrapped for the phrasing-independent letter-cluster design, then converged with a documented stopping point rather than chasing an NLP-complete regex.

Verification

  • Full suite: 520 passed, 1 skipped, 2 xfailed. run_eval: PASS (guardrail 0.75, 0% delta). black/isort/flake8 clean.
  • Final review: MERGE-READY — no false-FAIL on common legit refusals, common deliveries still caught, bridge/baseline/xfail all honest.
  • Known nit (non-blocking): a duplicate identical _REFUSAL_PHRASES definition — deferred to a dead-code pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_016v4C1daTwNakL6QDbmy3MJ

franciszver and others added 11 commits July 24, 2026 00:19
#46)

Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#46)

Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… redirect (red) (#46)

Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…directs (#46)

Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s enumeration (red) (#46)

Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ect widening (#46)

Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assisted-by: Claude Code (sonnet subagent)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assisted-by: Claude Code (sonnet subagent)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#52)

Assisted-by: Claude Code (sonnet subagent)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@franciszver
franciszver merged commit 1efaef7 into main Jul 24, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P-eval.4 — wire expect through record_to_case so run_eval grades guardrail fixtures

1 participant