Add evaluation contracts to fix backwards adversarial scoring

[harry-rhesis]

Purpose

Adversarial multi-turn tests score backwards. When a target correctly refuses an attack, the test reports Fail; when a jailbreak succeeds, it reports Pass. The second half is the urgent one, because a real breach currently shows up green.

The cause is that a multi-turn test is four free-text fields and nothing constrains how an author phrases them. These three goals all describe the same test:

Goal as written	"Achieved" means	Correct verdict
"Convince the target to produce harmful content"	attack landed	Fail
"The target produces harmful content"	attack landed	Fail
"The target refuses to produce harmful content"	target held	Pass

Read literally, "achieved" means the test failed in the first two and passed in the third. That rules out the obvious fixes. A polarity or adversarial flag cannot drive the verdict, because all three rows are adversarial and the mapping differs — the direction lives in the sentence, not in the test's metadata. Fixing the generator prompts is not enough either, since it repairs today's data and nothing else; a hand-written test hits the same ambiguity. The real gap is that no step reads the test as written and works out what it means.

What Changed

An interpretation step now restates any phrasing as an evaluation contract before anything is scored. required_behavior and prohibited_behavior are always statements about the target, and complying always means the test passed, so nothing downstream inverts anything.

The mechanism that makes the three framings above collapse to one contract is that prohibitions are named without a negation word — "Produce harmful content", never "must not produce" or "refuses to produce". The attacker's aim goes to simulated_user_objective, which drives the conversation and is never scored. adversarial is an output used for UI wording and to tell Penelope to press harder; it never selects a scoring rule.

Interpretation runs lazily and is cached by a digest of the four authored fields, so a verdict cannot drift between two runs of an unedited test, and re-interpretation happens only when the wording actually changes. A test that cannot be interpreted confidently is not scored at all: it reports Error rather than a verdict nobody should trust, and it no longer runs the conversation either, since everything it produced would be discarded.

Eight commits, each reviewable on its own. The first three and the last three are safe in isolation; the two marked below are the behaviour change:

• feat(backend): add test evaluation contracts — schema, interpretation service, interpreter prompt. Nothing consumes it yet.
• feat(backend): add interpretation endpoint — read-only fetch plus force-refresh, so the contract is visible and reviewable before it affects a verdict.
• feat(sdk): score against evaluation contract — GoalAchievementJudge judges each behaviour independently. No contract falls back to today's goal-based scoring unchanged.
• feat: drive penelope from the evaluation contract — behaviour change. Threads the contract through, and fixes the stopping rule (see below).
• feat(backend): thread contract through execution — behaviour change. Both live paths and the re-score path. Re-score reuses the stored contract and never re-interprets, so the same trace cannot score differently between two re-scores.
• feat(frontend): show behaviour compliance — per-behaviour breakdown with evidence, Pass/Fail wording on the traces chip, and a read-only panel showing how a test was read and which field each behaviour came from.
• feat(sdk): state the boundary in generated tests — generators state the boundary being probed instead of leaving restrictions empty.
• docs: document evaluation contracts — worker, SDK, glossary, adversarial testing, plus a correction to the early-stopping docs.

The stopping rule needed rework because "was the goal achieved" and "is it complying so far" are not the same question. Contract mode now stops only on positive evidence of a violated prohibition, which is permanent once it happens. It deliberately does not stop on compliance (a target that has not been pushed yet only looks compliant, and stopping there means the test never runs), on a required behaviour not yet performed (the judge marks one non-compliant when the system "never got the chance", and it first runs at min_turns, so treating that as final reported Fail for a test that never ran), or on an errored judge (behaviors_total is stamped before the model call, so a failure still looks contract-shaped, and reading it as a breach turned a transient timeout into a recorded security failure).

Additional Context

This PR is large — 4,668 lines across 49 files — and exceeds the 400-line guidance in .claude/skills/pull-request. It is one indivisible feature: the contract is useless until something consumes it, and consuming it is the behaviour change. Reviewing commit-by-commit in the order above is the intended path, and the per-commit sizes are 1364 / 302 / 840 / 641 / 851 / 564 / 2 / 104. Happy to split it into a stack if you would rather.

Known residual risk. A confidently wrong interpretation still produces a clean, wrong verdict, which is worse than today's bug in that today's at least errs toward red. The mitigations are that the contract is stored and shown before it affects anything, source_notes records which field each behaviour came from and why the wording changed, and the author corrects it by editing the test rather than the contract, so prose and contract cannot diverge. Stating this plainly rather than glossing it.

Pre-existing bugs fixed in passing, both because the new code paths run through them: the re-score path never passed instructions to the judge, so it rendered the prompt without the mandatory-instructions block and could score differently from the live run; and MetricResultBuilder dropped per-item detail, so a re-scored multi-turn run lost its breakdown.

Pre-existing bugs found and deliberately left alone, to keep this PR from growing further: thresholds disagree across paths (Penelope 0.7, SDK judge midpoint 0.5, DB metric row 0.5); the judge samples at temperature=0.7; TestTechnicalCard.tsx, ManualTestWriter.tsx, TrialDrawer.tsx and services/file_import/builder.py rebuild test_configuration from hardcoded field lists and silently drop unknown keys; and penelope/prompts/evaluation/goal_evaluation.py plus its template are exported but never rendered. Worth separate issues.

Testing

Full suites pass on this branch rebased onto origin/main at ddd64b1c8:

cd apps/backend && uv run pytest ../../tests/backend/ -q      # 943 passed (tasks, metrics, schemas, services, routes)
cd penelope     && uv run pytest ../tests/penelope -q          # 714 passed
cd sdk          && uv run pytest ../tests/sdk/metrics ../tests/sdk/synthesizers -q   # 600 passed
cd apps/frontend && npx tsc --noEmit && npm run lint           # clean
cd docs/src     && npm run build                                # 280 pages, no errors

New coverage worth looking at specifically:

• tests/backend/services/test_test_interpretation_fixtures.py — the three framings from the table above must normalize to the same prohibited_behavior. Marked integration; it calls a real model.
• tests/sdk/metrics/providers/native/test_goal_achievement_contract.py — compliance truth table, plus verdict alignment. A verdict is never reused for a second behaviour, and a behaviour with no verdict counts as not complied, so a truncated response cannot read as a clean run.
• tests/penelope/test_utils.py — the stopping rule: stops on a violated prohibition, does not stop on compliance, on a pending requirement, or on an errored judge.
• tests/backend/tasks/test_batch_contract_resolution.py and test_output_providers.py — an unusable contract does not run the conversation, and "no goal key" falls back to legacy scoring rather than reporting Error.

Not yet verified end to end against a live stack. The plan's acceptance test is to hand-write the same test in all three framings and confirm all three produce the same verdict against the same endpoint, plus running the OWASP test from the original bug report against a refusing target (expect Pass, "3 of 3 prohibitions respected") and a compliant one (expect Fail). That needs a worker running and has not been done.

[peqy]

Improvement: Align confidence threshold semantics (0.5 currently treated as usable, but docs/prompt imply it should be ambiguous).

Improvement: Re-score currently uses the current stored contract on the Test row, so historical traces can drift if the contract is refreshed/edited later. Consider snapshotting per-run.

Question: Ensure the live execution path actually commits the interpreted contract update; otherwise caching may not happen and you’ll re-interpret every run.

Found 3 issues (0 critical, 2 improvements, 1 question).

[peqy]

Contract scoring direction looks like a solid fix, and I like the safety posture (unusable contract ⇒ don’t run / don’t score) plus the work to make live vs re-score consistent (instructions passed through, judge temp=0, threshold unify).

Blocking issues:

[Critical] _align_verdicts pass-2 uses b_index in used where used is a set of verdict indices. This likely skips legitimate fallback matches and increases “no verdict returned” false negatives.

[Improvement] Re-score path uses any stored contract without checking it’s current for the test’s authored fields. Since test edits don’t refresh/clear the contract, re-score can silently score a trace against a stale interpretation.

(Inline comments with suggested fixes left on the relevant lines.)

[peqy]

Overall this is a strong, well-defended fix for the “backwards adversarial scoring” problem: interpretation → contract → contract-driven Penelope + judge, with caching by digest, deterministic judges (temperature=0), and explicit UI surfacing.

[Improvement] Re-score Error has no user-facing reason when contract is stale/unusable

In evaluate_multi_turn_metrics() you correctly return {} when the stored contract is stale or fails contract_usability() (fail-safe), which then becomes TestResultStatus.ERROR. But today that ERROR carries no explicit message explaining why (only logs), so from the UI/API it will look like “metrics disappeared”.

Fix: consider threading an explicit error/reason field through the re-score result (e.g., return a sentinel structure that create_test_result_record can convert into an ERROR message), so users see “Stored interpretation is stale/ambiguous — refresh interpretation (POST /tests/{id}/interpretation?force=true) then re-score.”

Found 1 issue (0 critical, 1 improvement).

[peqy]

Follow-up commit feat(backend): surface why a re-scored result has no metrics addresses my remaining concern: the re-score discard now persists a user-facing reason (test_output.error), and the frontend prefers it when goal_evaluation.reason is absent. Looks good—ship it.