Recipe 54: Question and policy optimization #54

Recipe 54 of 60 · Level 5 (Expert) · Evaluation & optimization · Choice + Noul + Score

Use case

Optimize Jev question wording, criteria, and decision thresholds with an external search loop, then compare frozen variants on an untouched test set.

What to build

The folder recipes/54-question-and-policy-optimization/ with notebook.ipynb, README.md, and fixtures/, meeting the recipe contract. Start from python tools/new_recipe.py 54 and the template in recipes/_template/.

Build notes

• An external search loop varies question wording, criteria, and decision thresholds.
• Frozen variants are compared on a test split that the search never touches. Guard the split with a hash check.
• TypeWright in this organisation takes a compiler approach to the same problem and is related reading.

Decision design

• Choice. Options are a fixed set supplied by Python, including the fallback outcome this use case names. Show the choice, the per-option probabilities, and the confidence for individual examples before aggregating.
• Noul. Write each proposition as a statement that is true or false. One independent question per label or concern, sent together in one request. Noul has no confidence field, so each threshold is chosen from examples and evaluated.
• Score. Describe every rubric level in words, in order. Show the score and the distribution over levels. Mapping, weighting, and arithmetic on scores happen in Python.
• Composition. Independent questions share one request. A question that depends on an earlier answer goes in a later request, and the notebook makes that boundary visible.

Level 5 scope

Controlled optimization studies or closed-loop simulations requiring budgets, baselines, and held-out or trajectory evaluation.

Acceptance criteria

• recipes/54-question-and-policy-optimization/notebook.ipynb runs top to bottom offline, with no key and no network, and its outputs are committed
• Live mode is opt-in (JEV_COOKBOOK_LIVE=1 with TYPESAFE_API_KEY) and uses the same question definitions as the offline run
• Fixtures are synthetic, include the hard cases named in the build notes, carry gold labels, and pass the fixture validator
• Every response fixture records its provenance, and the notebook states which mode it ran in
• Evaluation: Frozen-variant comparison on the untouched test split, with the search budget and split hashes recorded.
• The offline run is presented as a harness check. The recipe README has a section on how to produce real results with a live run, and reports none unless one was recorded
• Budgets, seeds, and frozen configurations are recorded in the notebook output
• A baseline is run under the same conditions, and any held-out split is used exactly once after choices are frozen
• Results are reproducible from the committed seeds and fixtures
• No statement about Jev's quality, latency, or cost appears without recorded live inference; in a synthetic run, metrics are labelled as pipeline checks
• recipes/54-question-and-policy-optimization/README.md says what the recipe teaches, how to run it, and what was and was not measured live
• The pull request changes only recipes/54-question-and-policy-optimization/ and CI is green

Dependencies

• Blocked by Foundation: CI executes every notebook offline and validates fixtures #69 (CI executes every notebook offline and validates fixtures)
• Blocked by Foundation: opt-in live Jev backend and response recorder #64 (opt-in live Jev backend and response recorder)
• Builds on Recipe 38: Abstention and review queues #38 (Abstention and review queues). Merge those first and reuse their patterns; do not copy their fixtures.
• Tracked in Epic: Level 5 · Expert recipes #79

Sources

Read these before writing the questions.

• S02 · TypeSafe AI: Primitives (Questions): Choice, Noul, Score, independent questions, and dependent follow-up calls.
• S03 · TypeSafe AI: Confidence: Distribution-derived confidence and task-specific thresholds informed by observed outcomes.
• S06 · TypeSafe AI: Cookbooks: Worked patterns for extraction, citation checking, skill selection, hierarchical classification, and feature discovery.

Out of scope

• Changes to shared code, the catalog, or the root README. If the shared package is missing something, open a foundation issue and link it here.
• Real side effects of any kind. Actions are simulated.