From 3ce53019935a40e096b2191e21848f2395a246d1 Mon Sep 17 00:00:00 2001 From: Timothy Wayne Gregg Date: Sat, 31 Oct 2026 18:33:58 -0400 Subject: [PATCH 1/2] Foundation: image-aware scaffold, tools/new_recipe.py --mode replay-images (Refs #315) Slice 2 of #315: the scaffolder items only. `python tools/new_recipe.py NN --mode replay-images` scaffolds a replay recipe whose requests carry images. After writing the replay files it runs the new folder's own build_fixtures.py once, which draws 14 example images with jev_cookbook.synthetic_images, writes assets/ and assets-manifest.json (origin generated, licence, attestations) and a complete fixtures/ whose Box labels are derived from the drawn geometry. helpers.py loads each image with image_asset(path, root=ROOT), derives image_reference and keys the replay with replay_key(state, questions, assets=assets). The notebook cells are generated from Python source in the tool: one image answer up close, draw_boxes, the presence/review rule (locate) and localization_report. request_budget.json declares the 14 requests and the two image quantities of docs/backends.md, so the scaffold passes the request audit with no hand edit. tests/test_build_fixtures.py for an image recipe is a variant that also copies assets/ into the temporary folder it rebuilds in, so the committed picture bytes (part of every replay key that carries a picture) are compared, not regenerated. build_fixtures_test_text(recipe) still returns the text variant; build_fixtures_test_text(recipe, "replay-images") returns the image one. The text scaffolds are byte-unchanged: all 100 catalog slugs in both existing modes were scaffolded before and after and the trees compared. MODES keeps its two text values; ALL_MODES adds the image mode. Tests: tests/test_new_recipe.py and the template/scaffolder agreement tests cover the image variant. Deferred to other slices: README/backends/fixtures links, the experimental label, docs/live.md, the docs-link test, a frame variant, and the recipe guard's byte comparison of test_build_fixtures.py for a recipe that has assets/. Co-Authored-By: Claude Sonnet 5.5 Co-Authored-By: Claude Haiku 5.5 --- docs/recipe-template.md | 81 +- tests/test_new_recipe.py | 708 +++++++++++- tests/test_template_scaffolder_agreement.py | 81 +- tools/new_recipe.py | 1094 ++++++++++++++++++- 4 files changed, 1926 insertions(+), 38 deletions(-) diff --git a/docs/recipe-template.md b/docs/recipe-template.md index ffbdf6f9..f945446b 100644 --- a/docs/recipe-template.md +++ b/docs/recipe-template.md @@ -8,6 +8,7 @@ folder for it. This page is the reference and the walkthrough; the template's ow ```bash python tools/new_recipe.py NN # create recipes/NN-slug/ from the catalog (NN is 1 to 100) python tools/new_recipe.py NN --mode scripted # the same for a scripted or simulator recipe +python tools/new_recipe.py NN --mode replay-images # the same for a recipe whose requests carry images python tools/execute_notebook.py recipes/X # run X/notebook.ipynb offline, write the outputs back ``` @@ -38,13 +39,13 @@ live inference explicitly opt-in under the [contract](../CONTRIBUTING.md). | `README.md` | The one-page recipe README, plus the builder's walkthrough (a real recipe's README has only the first part). | | `helpers.py` | `build_state`, `build_questions`, and the rule Python enforces (`route`). | | `build_fixtures.py` | Writes `fixtures/`; the replay keys come from `helpers.py`, so they cannot drift from the questions. It lives next to the notebook, outside `fixtures/` ([fixtures.md](fixtures.md)). Generating inputs and labels is separate from generating responses: `--force` is needed to overwrite a `responses.json` that already holds a `recorded` answer. | -| `assets/` | Multimodal recipes only (absent from the template): the image files the recipe sends to Jev, PNG or baseline JPEG, each at most 1 MiB, 8 MiB in all, with no metadata, and nowhere else in the recipe. The contract and the check that enforces it are in [fixtures.md](fixtures.md#image-fixtures); how a recipe loads them is in [vision.md](vision.md). | -| `assets-manifest.json` | Multimodal recipes only, beside `assets/`: one entry per image with its hash, size, generator and parameters, licence and the attestations that it is synthetic and holds no real photograph, brand, person or third-party content. Required exactly when `assets/` exists; `build_fixtures.py` should write it with the images. Format, worked example and rules: [fixtures.md](fixtures.md#image-provenance-manifest). | +| `assets/` | Multimodal recipes only (absent from the template): the image files the recipe sends to Jev, PNG or baseline JPEG, each at most 1 MiB, 8 MiB in all, with no metadata, and nowhere else in the recipe. The contract and the check that enforces it are in [fixtures.md](fixtures.md#image-fixtures); how a recipe loads them is in [vision.md](vision.md). `python tools/new_recipe.py NN --mode replay-images` creates it with generated example images ([Image recipes](#image-recipes)). | +| `assets-manifest.json` | Multimodal recipes only, beside `assets/`: one entry per image with its hash, size, generator and parameters, licence and the attestations that it is synthetic and holds no real photograph, brand, person or third-party content. Required exactly when `assets/` exists; `build_fixtures.py` should write it with the images. Format, worked example and rules: [fixtures.md](fixtures.md#image-provenance-manifest). `--mode replay-images` writes it through the scaffolded `build_fixtures.py`. | | `fixtures/` | `inputs.jsonl`, `labels.jsonl`, `responses.json`: 22 examples (10 `validation`, 10 `test`, 2 `demo`), synthetic and deliberately imperfect. | | `request_budget.json` | The recipe's declared live request count (#213, #368), the one place it is stated: one audited backend, `main`, declaring `requests: 22` and `fixtures: "fixtures/responses.json"`. The setup cell's one `get_backend(...)` is wrapped in `AuditedBackend(..., label="main")`, and a final standalone cell (nested under "What was and was not measured" as a `###` subsection, never a new `##` heading) prints `backend.audit_json()`. The README's `## Switch to live` paragraph is *rendered* from this file by `jev_cookbook.accounting.render_recipe_note`, never typed by hand, and the recipe guard's `readme_budget_note_matches_request_audit` check (#213) compares the declaration, the audited execution and the rendered paragraph against each other for every published recipe. The template's own agreement is checked by `tests/test_template.py` using the same audit. [backends.md](backends.md#the-declared-count-request_budgetjson) is the full contract; this is the one route a recipe's live request count is stated through. | | `tests/test_helpers.py` | Tests for the rule, a check that the stored keys match the current question, `test_stored_answers_are_not_all_right` (replay recipes only): re-derives the frozen threshold from `validation` (the `select_confidence_threshold` call written out, or `gate = helpers.(...)` through a function of the recipe's own `helpers.py` that calls it; the guard finds the test by its name, so keep `test_stored_answers_are_not_all_right` as the prefix) and requires a wrong `test` answer at or above it, so a fixture set that is merely "not all correct" (but has every mistake caught by the gate) still fails, and a test that pins the threshold-selection call's own **objective** keyword against the committed cell source (see "Pin the threshold's OBJECTIVE" above) — a value-only re-derivation does not pin which objective ran. `tools/new_recipe.py` scaffolds a working copy of the stored-answers test, so no recipe has to hand-write it from scratch; the objective pin is adapted per recipe from the template's own worked example. | | `side_effects.json` | The recipe's statement of its simulated side effects (a `ReviewQueue`, `ActionLog`, `Store`, `Transactor` or `Simulator` write): `{"schema": 1, "sites": {"helpers.submit_reviews": 2}}`, the exact number of places the notebook calls each side-effect function or method, or `{"schema": 1, "none": "why"}`. `tools/new_recipe.py` scaffolds it as a `none` declaration whose reason is a `TODO` placeholder (the template recipe has no simulated side effect, so `none` is what it would declare): rewrite the reason, or replace the file with `sites`, before the recipe is done. The recipe guard rejects a reason that still carries the `TODO`, rejects `none` for a recipe that has a side effect or needs a simulator, and fails a notebook that calls fewer, more or none of what it declares. Schema, keys, `not_side_effects` (bookkeeping only, with a reason per entry), factory-built simulators, the guard's known limits and the two function shapes (a batch fill, a decide-and-apply function) with their test rules: [simulation.md](simulation.md). | -| `tests/test_build_fixtures.py` | Guards `build_fixtures.py` (replay recipes only): refusal without `--force`, `inputs.jsonl`/`labels.jsonl` regenerated from `ROWS` even on a refused run, and the committed `responses.json` byte-identical to what `jev_cookbook.dump_responses` returns (the public form of the serialization the recorder writes with; `jev_cookbook.live._dump` is kept only as a transitional alias, and recipes do not import it). A scripted recipe has no `responses.json` and nothing for these tests to guard, so the scaffolder does not emit this file for `--mode scripted`. | +| `tests/test_build_fixtures.py` | Guards `build_fixtures.py` (replay recipes only): refusal without `--force`, `inputs.jsonl`/`labels.jsonl` regenerated from `ROWS` even on a refused run, and the committed `responses.json` byte-identical to what `jev_cookbook.dump_responses` returns (the public form of the serialization the recorder writes with; `jev_cookbook.live._dump` is kept only as a transitional alias, and recipes do not import it). A scripted recipe has no `responses.json` and nothing for these tests to guard, so the scaffolder does not emit this file for `--mode scripted`. `--mode replay-images` emits a variant of it that also copies `assets/` into the temporary folder it rebuilds in ([Image recipes](#image-recipes)). | The notebook sections, in order: **What you will build**, **Setup and run mode**, **The state**, **The questions**, **One answer up close**, **Python's part**, **Evaluation**, **What was and was not @@ -524,6 +525,80 @@ describes, with the simulator owning the state and the script standing in for th a pipeline check, not a Jev result; the README and notebook say so under their `TODO`s. A scripted recipe that later records real answers becomes a replay recipe: add the keys and `responses.json`. +## Image recipes + +A recipe whose requests carry images (the multimodal track, ranks 61–100: +[multimodal-track.md](multimodal-track.md)) is a replay recipe with an `assets/` folder. Scaffold it +with: + +```bash +python tools/new_recipe.py NN --mode replay-images +``` + +`--mode` is the option that already chooses a variant of the scaffold, so the image variant is one +more value of it (read it as "replay, with images") rather than a separate flag; a flag could be +combined with `--mode scripted`, and no value of `--mode` can ask for an image recipe that is not a +replay recipe. Everything in [vision.md](vision.md) applies to what it writes. + +**It writes a working example, not a skeleton.** The text scaffold raises `NotImplementedError` +until you write `helpers.py` and `ROWS`; the image scaffold ships a small recipe that runs end to +end, so that the image plumbing (the files, the references, the budget) is correct before you change +anything, and `TODO` markers say what to replace. The example asks of fourteen drawings of an +invented sign-in window (`jev_cookbook.synthetic_images.login_screen`; 6 `validation`, 7 `test` and +1 `demo` example) whether an error banner is shown (a `Noul`) and where (a `Box`). + +The command writes the files of the replay scaffold, then runs the new folder's own +`build_fixtures.py` once. That script draws the images, saves them under the byte policy of +[synthetic-images.md](synthetic-images.md#byte-stability-and-regeneration), and writes +`assets-manifest.json` and `fixtures/`. `tools/new_recipe.py` itself needs only the standard +library; that one step needs the installed package and Pillow, which the scaffolded +`build_fixtures.py` imports (`pip install -e ".[dev]"`). If it fails, the folder is removed and the +command exits with status 1. + +| File | What the image scaffold adds to the replay scaffold | +| - | - | +| `assets/`, `assets-manifest.json` | Fourteen PNG files, `assets/screen-.png`, and their manifest ([fixtures.md](fixtures.md#image-provenance-manifest)): `origin: generated`, the generator `jev_cookbook.synthetic_images:login_screen` with the arguments that redraw each file, licence `CC0-1.0` and the five attestations. | +| `fixtures/` | Complete. A fixture row's `fields` hold a `capture` identifier (Python keeps it) and the image's path, `assets/screen-.png`, as [fixtures.md](fixtures.md#image-fixtures) requires. Gold labels are Box labels ([fixtures.md](fixtures.md#box-labels)) derived from what was drawn: the banner's `Drawn.label()` where the screen has one, `{"present": false}` where not. The stored answers are synthetic and deliberately imperfect (one rectangle that misses the banner, two wrong yes answers and one wrong no, one rectangle without a confidence). | +| `helpers.py` | `build_request(fields)` returns `(state, assets)`: it loads the image with `image_asset(path, root=ROOT)` because the path comes from fixture data, derives `image_reference`, and puts the reference in the state once (every place a reference appears adds a copy of the image to the request: [vision.md](vision.md#image-copies-and-the-32-mib-caps)). `build_questions()` asks `present` (a `Noul`) and `location` (a `Box`). `locate(...)` is the rule that holds whatever the model answers: a presence answer below the threshold is a review, not an absence; a probability of yes above one half means the target is there; a rectangle with no confidence, or below its threshold, is a review. `open_image` and `png_bytes` serve the notebook's drawing cell. | +| `build_fixtures.py` | `ROWS`, then `draw`, `write_images`, `build_inputs_and_labels`, `answers_for` and `build_responses`. Images are written first, then the keys are computed with `replay_key(state, questions, assets=assets)` from the files on disk. `--force` behaves as in the text scaffold; `--overwrite-assets` accepts a picture that differs from the committed file, which changes its replay keys. Without it, a changed picture stops the script before anything else is written. | +| notebook | The cells of the replay notebook, with `get_backend(fixtures=responses_path(), images=True)` in the audited setup cell, and: `state` (builds the request for the demo example and shows its image), `answer` and `answer-boxes` (one answer up close, and `draw_boxes` on the image with the returned rectangle), `python-rule` (freezes the presence and rectangle-confidence thresholds on `validation` with `select_confidence_threshold`, then applies `locate` to the demo answer), `evaluation` (every `test` example once through `localization_report`, printed with `check` beside each line) and `evaluation-review` (what was not located, and why). Every `decide()` passes `assets=assets`. The setup cell defines `selection`, because these cells print `validation` numbers. | +| `request_budget.json` | `requests: 14`, `fixtures: "fixtures/responses.json"`, and `profile: {"images": 14, "distinct_images": 14}` ([backends.md](backends.md#actual-calls-and-distinct-requests)): the notebook decides each of the 14 examples once, and the reference sits once in each state, so each request is one copy of one distinct image. A recipe that repeats a reference or sends more than one image per request changes these two numbers with the count. The README's paragraph is rendered from this file, as for every scaffold. | +| `tests/test_helpers.py` | Tests of `locate` (each outcome and each reason, and a grid of probabilities and thresholds), the replay-key test with `assets=`, a test that redraws every shared-scene image from its manifest entry (`check_reproducible`), and `test_stored_answers_are_not_all_right`, which re-derives the presence threshold and requires a wrong `test` answer at or above it. | +| `tests/test_build_fixtures.py` | The text scaffold's three guards, run in a temporary folder that also holds a copy of `assets/` (the third also checks that the images and `assets-manifest.json` come back unchanged), and a fourth: a picture that differs from the committed file is refused, not rewritten. | + +**Why `tests/test_build_fixtures.py` differs.** A PNG's bytes depend on the Pillow and zlib that +wrote them, and the bytes are part of the replay key of each request that carries a picture. The +text scaffold's test copies only +`helpers.py` and `build_fixtures.py` into a temporary folder and runs the script there; for an +image recipe that would draw every picture afresh with the encoder of whatever environment runs the +test, and the regenerated fixtures could disagree with the committed ones although the pixels are +the same. The image variant copies the committed `assets/` too, so the script compares each picture +it draws with the committed file under that byte policy. That file must stay byte-identical to the +scaffold's, so the choice cannot be left to the builder: +`build_fixtures_test_text(recipe)` returns the text variant and +`build_fixtures_test_text(recipe, "replay-images")` the image one, which is the file an image +recipe commits. + +**What the scaffold passes offline.** `python -m jev_cookbook.fixtures validate`, +`python tools/check_recipe_assets.py`, `python tools/check_hygiene.py`, `ruff check`, +`ruff format --check`, and its own tests (`pytest recipes/NN-slug`), with no network, no key and no +hand-edit. `tests/test_new_recipe.py` checks that its declaration, with the two image quantities, agrees with +the request audit of the calls its notebook makes. The notebook has no outputs when it is +scaffolded: execute it with `tools/execute_notebook.py` (step 6 above) before the freshness check +can compare anything. + +**From scaffold to your recipe.** Replace the sign-in windows, the two questions and the answers +with your own, keeping the pattern: draw with `jev_cookbook.synthetic_images` (a scene of your own +calls `canvas.provenance("build_fixtures.py:your_function", {...})`, so the manifest names a +generator that exists), derive every gold Box label from a `Drawn`, put the image path in `fields`, +and keep the reference out of the questions. Grow `ROWS` to the size your level needs +([fixtures.md](fixtures.md)), update `request_budget.json` and its README paragraph together as +Evaluation adds `decide()` calls, and recreate the images under the notebook constraints pin +(`pip install -e ".[ml]" -c .github/constraints-notebooks.txt`) before you commit them, so that the +committed bytes are the pin's. The checklist in +[synthetic-images.md](synthetic-images.md#a-checklist-for-a-recipe) is the same list from the +drawing side. A frame sequence (ranks 85–92) has no scaffold of its own yet. + ## Executing a notebook `tools/execute_notebook.py ` runs `notebook.ipynb` with `nbclient` in a fresh kernel and diff --git a/tests/test_new_recipe.py b/tests/test_new_recipe.py index 1034b774..d8178910 100644 --- a/tests/test_new_recipe.py +++ b/tests/test_new_recipe.py @@ -4,6 +4,7 @@ directory and a temporary copy of the catalog. """ +import ast import importlib.util import json import os @@ -16,12 +17,22 @@ import nbformat import pytest from nbclient.exceptions import DeadKernelError +from PIL import Image import jev_cookbook -from jev_cookbook import DecisionResult, Provenance, dump_responses, load_helpers, replay_key +from jev_cookbook import ( + DecisionResult, + Provenance, + dump_responses, + get_backend, + load_helpers, + replay_key, +) from jev_cookbook.accounting import parse_budget, render_budget_note from jev_cookbook.answers import ChoiceAnswer -from jev_cookbook.fixtures import validate_recipe +from jev_cookbook.evaluation import noul_confidence, select_confidence_threshold +from jev_cookbook.fixtures import load_inputs, load_labels, select_split, validate_recipe +from jev_cookbook.vision import box_confidence, box_iou, gold_box, localization_report REPO = Path(__file__).resolve().parent.parent TEMPLATE = REPO / "recipes" / "_template" @@ -113,7 +124,7 @@ def test_it_creates_the_folder_named_by_the_catalog(catalog, recipes): } -@pytest.mark.parametrize("mode", new_recipe.MODES) +@pytest.mark.parametrize("mode", new_recipe.ALL_MODES) def test_the_scaffold_writes_a_side_effects_declaration_with_a_placeholder_reason( catalog, recipes, mode ): @@ -1573,7 +1584,7 @@ def test_the_scaffold_readme_live_section_states_the_budget_and_env_var(catalog, assert "TODO" in section -@pytest.mark.parametrize("mode", new_recipe.MODES) +@pytest.mark.parametrize("mode", new_recipe.ALL_MODES) def test_the_local_budget_renderer_and_default_cap_agree_byte_for_byte_with_the_shared_package( mode, ): @@ -1841,3 +1852,692 @@ def test_the_scaffolded_build_fixtures_test_file_passes_against_a_filled_in_reci result = run_generated_tests(folder) assert result.returncode == 0, result.stdout + result.stderr assert "passed" in result.stdout and "failed" not in result.stdout + + +# -------------------------------------------------------------------------------------------- +# --mode replay-images (#315): the multimodal scaffold +# -------------------------------------------------------------------------------------------- +# +# Nothing below executes a notebook except test_a_fresh_image_scaffold_passes_the_request_audit_ +# with_no_hand_edit, which needs a kernel exactly like its text counterpart above. Everything else +# builds and checks the scaffold in plain Python and subprocesses. The scaffold draws its own +# images, by running the new folder's build_fixtures.py once. + +IMAGE = new_recipe.IMAGE_MODE +IMAGE_IDS = [ + "v01", + "v02", + "v03", + "v04", + "v05", + "v06", + "t01", + "t02", + "t03", + "t04", + "t05", + "t06", + "t07", + "d01", +] +# The drawn error banner of login_screen is the pixel rectangle (80, 66, 560, 98) of a 640 x 400 +# image, so its gold box is 66/400, 98/400, 80/640 and 560/640, computed by hand. +GOLD_BANNER = {"top": 0.165, "bottom": 0.245, "left": 0.125, "right": 0.875} +IMAGE_FILES = { + "notebook.ipynb", + "README.md", + "helpers.py", + "build_fixtures.py", + "side_effects.json", + "request_budget.json", + "assets-manifest.json", + "fixtures/inputs.jsonl", + "fixtures/labels.jsonl", + "fixtures/responses.json", + "tests/test_helpers.py", + "tests/test_build_fixtures.py", + *(f"assets/screen-{ident}.png" for ident in IMAGE_IDS), +} + + +def scaffold_images(catalog, recipes, number=77): + """Scaffold ``--mode replay-images`` into ``recipes`` and return the new recipe folder.""" + _, recipe = entry(catalog, number) + assert run(number, catalog, recipes, mode=IMAGE) == 0 + return recipes / recipe["slug"] + + +@pytest.fixture(scope="module") +def image_scaffold(tmp_path_factory): + """One image scaffold shared by the tests that only read it; a test that changes anything + works on ``copy_of`` it instead.""" + base = tmp_path_factory.mktemp("image-scaffold") + catalog = base / "recipes.json" + shutil.copy(REPO / "catalog" / "recipes.json", catalog) + return scaffold_images(catalog, base / "recipes") + + +def entry_for_folder(folder): + data = json.loads((REPO / "catalog" / "recipes.json").read_text("utf-8")) + return data, next(r for r in data["recipes"] if r["slug"] == folder.name) + + +def copy_of(folder, tmp_path): + target = tmp_path / folder.name + shutil.copytree(folder, target, ignore=shutil.ignore_patterns("__pycache__")) + return target + + +def files_of(folder): + return { + p.relative_to(folder).as_posix() + for p in folder.rglob("*") + if p.is_file() and "__pycache__" not in p.parts + } + + +def run_generated(folder, *targets): + """pytest over some of a scaffold's own tests (``tests/test_helpers.py``, or a node id), + where ``run_generated_tests`` runs them all.""" + return subprocess.run( + [sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "--rootdir", str(folder)] + + [str(folder / target) for target in targets], + cwd=folder, + capture_output=True, + text=True, + timeout=180, + ) + + +def test_the_modes_the_text_scaffolds_walk_are_unchanged_and_the_image_mode_is_one_more(): + """MODES is what the tests and the recipe guard iterate over to walk the text scaffolds; the + image mode is a further value of --mode that they do not walk.""" + assert new_recipe.MODES == ("replay", "scripted") + assert new_recipe.IMAGE_MODE == "replay-images" + assert new_recipe.ALL_MODES == ("replay", "scripted", "replay-images") + + +def test_the_image_scaffold_writes_the_text_files_the_assets_the_manifest_and_the_fixtures( + image_scaffold, +): + assert files_of(image_scaffold) == IMAGE_FILES + assert not list(image_scaffold.rglob("*.part")), "no temporary file is left in assets/" + + +def test_the_image_scaffold_passes_the_fixture_validator_offline(image_scaffold): + report = validate_recipe(image_scaffold) + assert list(report) == [] and report.mode == "replay" + done = subprocess.run( + [sys.executable, "-m", "jev_cookbook.fixtures", "validate", str(image_scaffold)], + capture_output=True, + text=True, + ) + assert done.returncode == 0, done.stdout + done.stderr + assert "fixtures valid (mode replay)" in done.stdout + + +def test_the_image_scaffold_passes_the_asset_checker_and_the_hygiene_check( + image_scaffold, monkeypatch +): + spec = importlib.util.spec_from_file_location( + "check_recipe_assets_for_image_test", REPO / "tools" / "check_recipe_assets.py" + ) + checker = importlib.util.module_from_spec(spec) + monkeypatch.setitem(sys.modules, spec.name, checker) # a dataclass looks its module up here + spec.loader.exec_module(checker) + entries = [ + checker.Entry( + f"recipes/{image_scaffold.name}/{name}", "100644", (image_scaffold / name).read_bytes() + ) + for name in sorted(files_of(image_scaffold)) + ] + assert checker.check_entries(entries) == [] + done = subprocess.run( + [sys.executable, "tools/check_hygiene.py"] + + [str(image_scaffold / name) for name in sorted(files_of(image_scaffold))], + cwd=REPO, + capture_output=True, + text=True, + ) + assert done.returncode == 0, done.stdout + done.stderr + + +def test_the_image_scaffold_manifest_describes_drawn_images_with_a_shared_generator( + image_scaffold, +): + manifest = json.loads((image_scaffold / "assets-manifest.json").read_text("utf-8")) + assert manifest["schema"] == 1 + assert [e["path"] for e in manifest["assets"]] == sorted( + f"assets/screen-{ident}.png" for ident in IMAGE_IDS + ) + states = {"v01": "error", "v03": "normal", "v04": "loading", "v06": "crash", "d01": "error"} + for item in manifest["assets"]: + ident = item["path"].split("screen-")[1].split(".")[0] + assert item["origin"] == "generated" and item["licence"] == "CC0-1.0" + assert item["generator"] == "jev_cookbook.synthetic_images:login_screen" + assert item["attestations"] == { + "synthetic": True, + "real_photo": False, + "real_brand": False, + "real_person": False, + "third_party_content": False, + } + assert (item["width"], item["height"]) == (640, 400) and item["format"] == "PNG" + if ident in states: + assert item["parameters"]["state"] == states[ident] + + +def test_the_image_scaffold_labels_are_the_drawn_geometry_not_typed_numbers(image_scaffold): + labels = { + row["id"]: row["label"] + for row in map( + json.loads, + (image_scaffold / "fixtures" / "labels.jsonl").read_text("utf-8").splitlines(), + ) + } + assert "d01" not in labels, "the demo example carries no label" + banner = {"present": True, "box": GOLD_BANNER} + absent = {"present": False} + assert labels == { + "v01": banner, + "v02": banner, + "v03": absent, + "v04": absent, + "v05": banner, + "v06": absent, + "t01": banner, + "t02": banner, + "t03": banner, + "t04": absent, + "t05": absent, + "t06": absent, + "t07": banner, + } + + +def test_the_image_scaffold_inputs_keep_the_identifier_and_the_path_in_fields(image_scaffold): + rows = [ + json.loads(line) + for line in (image_scaffold / "fixtures" / "inputs.jsonl").read_text("utf-8").splitlines() + ] + assert [row["id"] for row in rows] == IMAGE_IDS + for row in rows: + assert set(row["fields"]) == {"capture", "image"} + assert row["fields"]["image"] == f"assets/screen-{row['id']}.png" + assert len(row["replay_keys"]) == 1 + assert [row["split"] for row in rows].count("demo") == 1 + + +def test_the_image_scaffold_declares_one_image_per_request_and_the_readme_renders_it( + image_scaffold, +): + declared = json.loads((image_scaffold / "request_budget.json").read_text("utf-8")) + count = len(IMAGE_IDS) + assert declared == { + "schema": 1, + "backends": { + "main": { + "requests": count, + "fixtures": "fixtures/responses.json", + "profile": {"images": count, "distinct_images": count}, + } + }, + } + budget = parse_budget(declared) + readme = (image_scaffold / "README.md").read_text("utf-8") + assert render_budget_note(budget) in readme + assert f"**{count}** requests" in readme and f"JEV_COOKBOOK_LIVE_MAX_REQUESTS={count}" in readme + responses = json.loads((image_scaffold / "fixtures" / "responses.json").read_text("utf-8")) + assert len(responses) == count, "every request has exactly one stored response" + + +def test_the_image_scaffold_notebook_has_the_nine_sections_the_image_cells_and_no_outputs( + image_scaffold, +): + nb = nbformat.read(image_scaffold / "notebook.ipynb", as_version=4) + nbformat.validate(nb) + headings = [ + line[3:] + for c in nb.cells + if c.cell_type == "markdown" + for line in c.source.splitlines() + if line.startswith("## ") + ] + assert headings == new_recipe.SECTIONS + assert all( + not c.outputs and c.execution_count is None for c in nb.cells if c.cell_type == "code" + ) + ids = [c.id for c in nb.cells] + assert ids == [ + "intro", + "setup-md", + "setup", + "state-md", + "state", + "questions-md", + "questions", + "answer-md", + "answer", + "answer-boxes", + "python-md", + "python-rule", + "evaluation-md", + "evaluation", + "evaluation-review", + "measured-md", + "measured", + "audit-md", + "audit", + "next-md", + ] + source = {c.id: c.source for c in nb.cells} + assert ( + source["setup"].count("get_backend(") == 1 and source["setup"].count("AuditedBackend(") == 1 + ) + assert 'get_backend(fixtures=responses_path(), images=True), label="main"' in source["setup"] + assert "draw_boxes(picture," in source["answer-boxes"] + assert "localization_report(test_gold," in source["evaluation"] + assert "select_confidence_threshold(" in source["python-rule"] + assert [c.source for c in nb.cells if c.cell_type == "code"][ + -1 + ] == "print(backend.audit_json())" + + +def test_every_decide_call_in_the_image_notebook_passes_the_assets(): + """A decide() without assets= would send the references as plain text and find no stored + answer. Every cell that decides is checked from its parsed source, not by a text search.""" + catalog_data = json.loads((REPO / "catalog" / "recipes.json").read_text("utf-8")) + recipe = next(r for r in catalog_data["recipes"] if r["rank"] == 77) + sites = {} + for cell in new_recipe.notebook_cells(catalog_data, recipe, IMAGE): + if cell["cell_type"] != "code": + continue + for node in ast.walk(ast.parse(cell["source"])): + if isinstance(node, ast.Call) and getattr(node.func, "attr", None) == "decide": + sites[cell["id"]] = [k.arg for k in node.keywords] + assert sites == { + "answer": ["assets"], + "python-rule": ["assets"], + "evaluation": ["assets"], + } + + +def test_the_image_example_numbers_are_the_ones_the_notebook_will_print(image_scaffold): + """The notebook's python-rule and evaluation cells, written out in plain Python against the + scaffold's own fixtures (no kernel), and compared with values worked out by hand: the + thresholds are the lowest confidences at which every call above them is right, and the + report counts each of the seven test examples once.""" + helpers = load_helpers(image_scaffold) + questions = helpers.build_questions() + backend = get_backend(fixtures=image_scaffold / "fixtures" / "responses.json", images=True) + examples = load_inputs(image_scaffold) + labels = load_labels(image_scaffold) + + def decide(example): + state, assets = helpers.build_request(example.fields) + return backend.decide(state, questions, assets=assets) + + validation = select_split(examples, "validation") + results = [decide(e) for e in validation] + gold = [labels[e.id] for e in validation] + right = [ + (r["present"].noul > 0.5) == (gold_box(g) is not None) + for r, g in zip(results, gold, strict=True) + ] + # v06 says yes (0.58) about a screen with no banner, the only wrong call: its confidence 0.16 + # is the lowest, and the next lowest is v05's 2 * 0.64 - 1 = 0.28. + assert right == [True, True, True, True, True, False] + presence_threshold = select_confidence_threshold( + right, noul_confidence([r["present"].noul for r in results]), target_accuracy=1.0 + ) + assert presence_threshold == pytest.approx(0.28) + # Rectangles of the three present validation targets: v01 and v02 overlap the gold box by 1.0 + # and 0.872 (a hit each, at confidences 0.90 and 0.86), v05 by 0.374 (a miss, at 0.55). + sized = [ + (box_iou(gold_box(g), r["location"]) >= 0.5, box_confidence(r["location"])) + for r, g in zip(results, gold, strict=True) + if gold_box(g) is not None and box_confidence(r["location"]) is not None + ] + assert [ok for ok, _ in sized] == [True, True, False] + box_threshold = select_confidence_threshold( + [ok for ok, _ in sized], [c for _, c in sized], target_accuracy=1.0 + ) + assert box_threshold == pytest.approx(0.86) + + test = select_split(examples, "test") + outcomes = [ + helpers.locate( + e.fields["capture"], r["present"], r["location"], presence_threshold, box_threshold + ) + for e, r in zip(test, (decide(e) for e in test), strict=True) + ] + assert [(o.kind, o.reason) for o in outcomes] == [ + ("located", "confident rectangle"), # t01 + ("located", "confident rectangle"), # t02 + ("review", "the rectangle has no confidence"), # t03: a present target in review + ("absent", "no error banner"), # t04 + ("located", "confident rectangle"), # t05: a box on a screen with no banner + ("review", "presence confidence below the threshold"), # t06 + ("absent", "no error banner"), # t07: a present target said to be absent + ] + report = localization_report([labels[e.id] for e in test], [o.outcome for o in outcomes]) + assert (report.n_total, report.n_present, report.n_absent) == (7, 4, 3) + assert report.present_table == {"located": 2, "absent": 1, "review": 1} + assert report.absent_table == {"located": 1, "absent": 1, "review": 1} + # t01 overlaps by 1.0 and t02 by 12366 / 15360 = 0.805078125 (a 458 x 27 rectangle inside the + # 480 x 32 banner), so the mean over the 2 located targets is 0.9025390625 and over all 4 + # present targets half of that, 0.45126953125. Both located targets are hits. + assert report.mean_iou_located == pytest.approx(0.9025390625) + assert report.mean_iou_present == pytest.approx(0.45126953125) + assert report.hit_rate == 0.5 and report.hits == 2 + assert ( + report.denominators["mean_iou_present"] == 4 + and report.denominators["mean_iou_located"] == 2 + ) + + +def test_the_text_scaffolds_mention_neither_images_nor_assets(catalog, recipes): + """The text scaffolds mention neither images nor assets, in any file (their bytes are also + compared before and after this change in the pull request).""" + for mode in new_recipe.MODES: + folder = recipes / mode + new_recipe.scaffold(1, catalog, folder, mode) + for path in folder.rglob("*"): + if path.is_file(): + text = path.read_text("utf-8").lower() + assert "assets" not in text and "images" not in text, (mode, path.name) + + +def test_the_image_scaffold_emits_no_private_shared_name(image_scaffold): + detector = load_module_at( + Path(__file__).with_name("test_no_private_shared_names.py"), "private_names_for_image_test" + ) + emitted = list(detector.python_sources(image_scaffold)) + assert {p.name for p, _ in emitted} >= { + "helpers.py", + "build_fixtures.py", + "notebook.ipynb", + "test_helpers.py", + "test_build_fixtures.py", + } + for path, source in emitted: + assert detector.private_shared_uses(source) == [], path + + +def test_the_image_scaffold_is_lint_clean_at_several_ranks(catalog, recipes): + """ruff check and ruff format --check pass on the files and the notebook with no hand-edit. + Run from the repository root, as the CI job does: the import order depends on jev_cookbook + being found under src/ as first-party code.""" + for number in (61, 77, 100): + scaffold_images(catalog, recipes, number) + ruff = [sys.executable, "-m", "ruff"] + config = ["--config", str(REPO / "pyproject.toml"), "--no-cache"] + subprocess.run([*ruff, "check", *config, str(recipes)], check=True, cwd=REPO) + subprocess.run([*ruff, "format", "--check", *config, str(recipes)], check=True, cwd=REPO) + + +def test_the_image_scaffold_runs_its_own_tests_offline(image_scaffold, tmp_path): + folder = copy_of(image_scaffold, tmp_path) + result = run_generated_tests(folder) + assert result.returncode == 0, result.stdout + result.stderr + assert "passed" in result.stdout and "failed" not in result.stdout + + +def test_a_fresh_image_scaffold_passes_the_request_audit_with_no_hand_edit(catalog, recipes): + """The image counterpart of ``test_a_fresh_scaffold_passes_the_request_audit_with_no_hand_edit``: + the scaffold is executed as it is written (its helpers and fixtures are already complete), + and the real ``audit_recipe`` agrees with the untouched declaration, including the two image + quantities of docs/backends.md, and with the untouched README. This is the one test here that + executes the notebook, so it needs a kernel.""" + folder = scaffold_images(catalog, recipes, 1) + assert execute_notebook.main([str(folder)]) == 0 + cells = json.loads((folder / "notebook.ipynb").read_text("utf-8"))["cells"] + assert _request_audit.audit_recipe(folder, cells) == len(IMAGE_IDS) + + +def emulated_cells(folder): + """The scaffold's cells with the committed-execution evidence the audit reads: execution + counts in order, and the audit line a backend produces for the ``decide()`` calls the + notebook makes (the demo example, then ``validation``, then ``test``), made here in plain + Python rather than by a kernel.""" + helpers = load_helpers(folder) + questions = helpers.build_questions() + backend = jev_cookbook.AuditedBackend( + get_backend(fixtures=folder / "fixtures" / "responses.json", images=True), label="main" + ) + examples = load_inputs(folder) + for split in ("demo", "validation", "test"): + for example in select_split(examples, split): + state, assets = helpers.build_request(example.fields) + backend.decide(state, questions, assets=assets) + cells = json.loads((folder / "notebook.ipynb").read_text("utf-8"))["cells"] + count = 0 + for cell in cells: + if cell["cell_type"] == "code": + count += 1 + cell["execution_count"] = count + if cell["id"] == "audit": + cell["outputs"] = [ + {"output_type": "stream", "name": "stdout", "text": backend.audit_json() + "\n"} + ] + return cells + + +def test_the_declared_image_budget_agrees_with_the_audit_of_the_calls_the_notebook_makes( + image_scaffold, tmp_path +): + """The same audit as above without a kernel (so it runs where a kernel does not), plus the + mutations that show each part of the declaration is read: a wrong request count, a missing + profile, a wrong profile and a README paragraph that is not the rendered one all fail.""" + folder = copy_of(image_scaffold, tmp_path) + cells = emulated_cells(folder) + assert _request_audit.audit_recipe(folder, cells) == len(IMAGE_IDS) + + budget_path = folder / "request_budget.json" + declared = json.loads(budget_path.read_text("utf-8")) + main = declared["backends"]["main"] + readme_path = folder / "README.md" + readme = readme_path.read_text("utf-8") + + def mutated(change, match): + data = json.loads(json.dumps(declared)) + change(data["backends"]["main"]) + budget_path.write_text(json.dumps(data), encoding="utf-8") + with pytest.raises(AssertionError, match=match): + _request_audit.audit_recipe(folder, cells) + budget_path.write_text(json.dumps(declared), encoding="utf-8") + + mutated(lambda body: body.update(requests=main["requests"] - 1), "request") + mutated(lambda body: body.pop("profile"), "profile") + mutated(lambda body: body["profile"].update(images=main["requests"] + 1), "profile") + mutated(lambda body: body["profile"].update(distinct_images=main["requests"] - 1), "profile") + readme_path.write_text(readme.replace(f"**{main['requests']}**", "**1**"), encoding="utf-8") + with pytest.raises(AssertionError): + _request_audit.audit_recipe(folder, cells) + readme_path.write_text(readme, encoding="utf-8") + assert _request_audit.audit_recipe(folder, cells) == len(IMAGE_IDS) + + +def test_the_image_build_fixtures_tests_are_the_ones_the_recipe_guard_can_compare( + catalog, recipes, image_scaffold +): + """``build_fixtures_test_text(recipe)`` keeps returning the text variant, so the recipe guard's + byte comparison for a text recipe is unchanged; ``mode='replay-images'`` returns the variant + the image scaffold writes, which a recipe with ``assets/`` is compared with.""" + _, recipe = entry(catalog, 77) + text_variant = new_recipe.build_fixtures_test_text(recipe) + image_variant = new_recipe.build_fixtures_test_text(recipe, IMAGE) + assert (image_scaffold / "tests" / "test_build_fixtures.py").read_text("utf-8") == image_variant + assert text_variant != image_variant + assert "copytree" in image_variant and "copytree" not in text_variant + folder = scaffold_replay_with_build_script(catalog, recipes, number=11) + written = (folder / "tests" / "test_build_fixtures.py").read_text("utf-8") + assert written == new_recipe.build_fixtures_test_text(entry(catalog, 11)[1]) + + +def test_the_image_scaffold_fails_cleanly_and_leaves_no_folder_when_the_drawing_step_fails( + catalog, recipes, monkeypatch, capsys +): + """If the new folder's build_fixtures.py cannot run (the package or Pillow is missing, say), + the folder is removed and the command exits 1 with the reason, never a half-written recipe.""" + _, recipe = entry(catalog, 77) + + def failing(*args, **kwargs): + return subprocess.CompletedProcess(args, 1, stdout="", stderr="ImportError: no Pillow\n") + + monkeypatch.setattr(new_recipe.subprocess, "run", failing) + assert run(77, catalog, recipes, mode=IMAGE) == 1 + error = capsys.readouterr().err + assert "ImportError: no Pillow" in error and "nothing was created" in error + assert not (recipes / recipe["slug"]).exists() + + def missing(*args, **kwargs): + raise FileNotFoundError("python") + + monkeypatch.setattr(new_recipe.subprocess, "run", missing) + assert run(77, catalog, recipes, mode=IMAGE) == 1 + assert not (recipes / recipe["slug"]).exists() + + def silent(*args, **kwargs): + return subprocess.CompletedProcess(args, 3, stdout="", stderr="") + + monkeypatch.setattr(new_recipe.subprocess, "run", silent) + assert run(77, catalog, recipes, mode=IMAGE) == 1 + assert "exit status 3" in capsys.readouterr().err + assert not (recipes / recipe["slug"]).exists() + + +def test_the_image_scaffold_never_overwrites_either(catalog, recipes, capsys): + folder = scaffold_images(catalog, recipes, 62) + before = {p: p.read_bytes() for p in folder.rglob("*") if p.is_file()} + assert run(62, catalog, recipes, mode=IMAGE) == 1 + assert "already exists" in capsys.readouterr().err + assert {p: p.read_bytes() for p in folder.rglob("*") if p.is_file()} == before + + +def test_the_generated_test_catches_a_review_that_becomes_an_absence(image_scaffold, tmp_path): + """Mutation check: an unconfident presence answer must be a review, never an absence. Change + the rule so that it answers absent, and the generated tests fail on exactly that.""" + folder = copy_of(image_scaffold, tmp_path) + helpers_path = folder / "helpers.py" + text = helpers_path.read_text("utf-8") + mutated = text.replace( + 'return Localized(capture, REVIEW, None, "presence confidence below the threshold")', + 'return Localized(capture, ABSENT, None, "presence confidence below the threshold")', + ) + assert mutated != text + helpers_path.write_text(mutated, encoding="utf-8", newline="\n") + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_an_unconfident_presence_answer_is_a_review_never_an_absence" in failed.stdout + assert "test_a_review_is_never_an_absence_over_a_grid" in failed.stdout + + +def test_the_generated_test_catches_a_rectangle_gate_that_lets_a_missing_confidence_through( + image_scaffold, tmp_path +): + folder = copy_of(image_scaffold, tmp_path) + helpers_path = folder / "helpers.py" + text = helpers_path.read_text("utf-8") + mutated = text.replace(" if confidence is None:\n", " if confidence is None and False:\n") + assert mutated != text + helpers_path.write_text(mutated, encoding="utf-8", newline="\n") + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_a_rectangle_without_enough_confidence_is_a_review" in failed.stdout + + +def test_the_generated_stored_answers_test_fails_when_no_wrong_answer_is_confident( + image_scaffold, tmp_path +): + """The guard must discriminate, as the text scaffold's does: make the two wrong test answers + (t05 and t07) unconfident and the generated test fails instead of passing vacuously.""" + folder = copy_of(image_scaffold, tmp_path) + script = folder / "build_fixtures.py" + text = script.read_text("utf-8") + weakened = text.replace('"loading", 25, 0.81,', '"loading", 25, 0.51,').replace( + '"error", 27, 0.18,', '"error", 27, 0.49,' + ) + assert weakened.count("0.51") == 1 and weakened.count("0.49") == 1 + script.write_text(weakened, encoding="utf-8", newline="\n") + subprocess.run([sys.executable, str(script)], check=True, cwd=tmp_path, capture_output=True) + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_stored_answers_are_not_all_right" in failed.stdout + + +def test_a_changed_picture_fails_the_generated_replay_key_test(image_scaffold, tmp_path): + """Mutation check: the bytes of a picture are part of its replay key, so a different picture + under the same name is caught by the generated test (and by the validator's hash).""" + folder = copy_of(image_scaffold, tmp_path) + picture = folder / "assets" / "screen-t01.png" + Image.new("RGB", (640, 400), "white").save(picture) + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_every_replay_key_in_the_fixtures_matches_the_current_question" in failed.stdout + assert any("hash-mismatch" in str(problem) for problem in validate_recipe(folder)) + + +def test_the_image_build_fixtures_tests_pass_under_a_different_png_encoder_and_the_text_ones_do_not( + image_scaffold, tmp_path +): + """Why the image variant copies ``assets/``. Pillow and zlib versions decide the bytes of a + PNG, and the bytes are part of every replay key, so a rerun that draws the pictures afresh can + disagree with the committed fixtures although every pixel is the same. ``sitecustomize`` below + makes this environment another encoder: the same pixels, different bytes, and not canonical. + The image variant, whose copy holds the committed pictures, still passes (the script keeps + the committed bytes); the text variant, which copies only the two scripts, fails on exactly + the comparison of the regenerated fixtures with the committed ones. In the encoder that wrote + the committed files, the text variant passes too, which is why this needs the second encoder.""" + folder = copy_of(image_scaffold, tmp_path) + shim = tmp_path / "shim" + shim.mkdir() + (shim / "sitecustomize.py").write_text( + "import io\n" + "from PIL import Image\n" + "import jev_cookbook.synthetic_images as si\n" + "_original = si.encode_png\n" + "def encode_png(image):\n" + " out = io.BytesIO()\n" + " with Image.open(io.BytesIO(_original(image))) as decoded:\n" + " decoded.save(out, format='PNG', compress_level=1)\n" + " return out.getvalue()\n" + "si.encode_png = encode_png\n" + "si.bytes_are_canonical = lambda: False\n", + encoding="utf-8", + ) + paths = [str(shim)] + [p for p in [os.environ.get("PYTHONPATH")] if p] + environment = {**os.environ, "PYTHONPATH": os.pathsep.join(paths)} + target = folder / "tests" / "test_build_fixtures.py" + + def run_it(): + return subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "-q", + "-p", + "no:cacheprovider", + "--rootdir", + str(folder), + str(target), + "-k", + "reproduces", + ], + cwd=folder, + capture_output=True, + text=True, + timeout=180, + env=environment, + ) + + image_variant = run_it() + assert image_variant.returncode == 0, image_variant.stdout + image_variant.stderr + + _, recipe = entry_for_folder(folder) + target.write_text(new_recipe.build_fixtures_test_text(recipe), encoding="utf-8", newline="\n") + text_variant = run_it() + assert text_variant.returncode != 0 + assert "FAILED" in text_variant.stdout + assert "test_the_generator_reproduces_the_committed_fixtures" in text_variant.stdout diff --git a/tests/test_template_scaffolder_agreement.py b/tests/test_template_scaffolder_agreement.py index f3062126..17f3f189 100644 --- a/tests/test_template_scaffolder_agreement.py +++ b/tests/test_template_scaffolder_agreement.py @@ -40,6 +40,14 @@ that direction and ``test_the_setup_comparison_catches_a_changed_offline_definition`` for a reword. +The image scaffold (``--mode replay-images``) is compared the same way, with one difference that +makes the comparison stricter, not looser: its ``setup`` cell defines ``selection`` (the cells +below it print ``validation`` numbers), so it needs no allowance and is compared with the +template's whole ``setup`` cell, ``selection`` included. Its ``backend = ...`` assignment is masked +like the other modes' (``images=True`` is a different call shape), so the arguments of that call +are asserted separately, in ``test_the_image_scaffold_asks_for_the_images_header_and_the_text_ +scaffolds_do_not``. + This is one assertion per cell, not a list of remembered rules, and it is blind to the specific property #215 found: deleting the offline ``N`` print (or any other statement, from either side) changes that side's statement sequence, which the comparison catches without this module's code @@ -161,14 +169,19 @@ def mode_neutral_statements(source: str) -> list[str]: TEMPLATE_ONLY_SETUP_STATEMENTS = ("selection = ' (a selection step, not a reported result)'",) -def _setup_template_stmts(source: str) -> list[str]: +def _setup_template_stmts(source: str, mode: str = "replay") -> list[str]: """The template's ``setup`` cell, normalised, with the one named template-only statement in - ``TEMPLATE_ONLY_SETUP_STATEMENTS`` filtered out -- what the scaffold's ``setup`` cell must - equal.""" - return [s for s in mode_neutral_statements(source) if s not in TEMPLATE_ONLY_SETUP_STATEMENTS] + ``TEMPLATE_ONLY_SETUP_STATEMENTS`` filtered out of the template's side -- what the scaffold's + ``setup`` cell must equal. The image scaffold defines ``selection`` itself, so for + ``replay-images`` nothing is filtered out and its ``setup`` cell must equal the template's + whole one.""" + statements = mode_neutral_statements(source) + if mode == "replay-images": + return statements + return [s for s in statements if s not in TEMPLATE_ONLY_SETUP_STATEMENTS] -@pytest.mark.parametrize("mode", ["replay", "scripted"]) +@pytest.mark.parametrize("mode", ["replay", "scripted", "replay-images"]) def test_the_scaffolded_setup_cell_matches_the_template_modulo_the_named_allowance(mode): """Every mode-neutral statement the scaffold's ``setup`` cell emits -- ``scored``, ``offline``, ``check``, and the ``run_header(...)`` call with its ``n_examples=len(scored)`` @@ -184,7 +197,7 @@ def test_the_scaffolded_setup_cell_matches_the_template_modulo_the_named_allowan generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup") template = _cell(TEMPLATE_NOTEBOOK["cells"], "setup") generated_stmts = mode_neutral_statements(_source(generated)) - template_stmts = _setup_template_stmts(_source(template)) + template_stmts = _setup_template_stmts(_source(template), mode) assert generated_stmts == template_stmts, ( f"scaffolded setup cell ({mode}) does not match the template's after normalisation " f"(modulo {TEMPLATE_ONLY_SETUP_STATEMENTS}):\ngenerated: {generated_stmts}\n" @@ -192,7 +205,7 @@ def test_the_scaffolded_setup_cell_matches_the_template_modulo_the_named_allowan ) -@pytest.mark.parametrize("mode", ["replay", "scripted"]) +@pytest.mark.parametrize("mode", ["replay", "scripted", "replay-images"]) def test_the_scaffolded_measured_cell_matches_the_template(mode): """The scaffold's ``measured`` cell, normalised, must equal the template's ``measured`` cell, normalised -- both branches, not only the offline one: this is the structural check that @@ -211,13 +224,14 @@ def test_the_scaffolded_measured_cell_matches_the_template(mode): ) -def test_the_measured_comparison_is_blind_to_n_and_still_catches_215(): +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) +def test_the_measured_comparison_is_blind_to_n_and_still_catches_215(mode): """Reproduces the historical #215 defect directly against the real generated cell -- delete the offline branch's own ``N:`` print -- and shows the comparison disagrees, without this module or ``mode_neutral_statements`` ever naming "N": the two sides simply no longer have the same number of statements in the offline branch. This is the "blind to any rule about N" standard both #219 reviewers asked for criterion 2 to meet.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "measured") + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "measured") template_stmts = mode_neutral_statements(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "measured"))) source = _source(generated) broken = re.sub( @@ -227,12 +241,13 @@ def test_the_measured_comparison_is_blind_to_n_and_still_catches_215(): assert mode_neutral_statements(broken) != template_stmts -def test_the_measured_comparison_catches_a_dropped_live_branch_print_too(): +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) +def test_the_measured_comparison_catches_a_dropped_live_branch_print_too(mode): """The same mutation on the *live* (``else``) branch's ``N:`` print, which #219 left uncovered by any assertion on the scaffolder side (issue #215 comment 6107125765, "Restore the live-branch coverage"): this comparison fails on it too, by construction, with no branch-specific code anywhere in this module.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "measured") + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "measured") template_stmts = mode_neutral_statements(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "measured"))) source = _source(generated) broken = source.replace( @@ -243,12 +258,15 @@ def test_the_measured_comparison_catches_a_dropped_live_branch_print_too(): assert mode_neutral_statements(broken) != template_stmts -def test_the_setup_comparison_catches_a_changed_offline_definition(): +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) +def test_the_setup_comparison_catches_a_changed_offline_definition(mode): """A ``setup``-cell regression unrelated to ``N`` -- rewording ``offline``'s definition -- is also caught, proving the structural comparison is not special-cased to the measured cell's own property.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "setup") - template_stmts = _setup_template_stmts(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup"))) + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup") + template_stmts = _setup_template_stmts( + _source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup")), mode + ) source = _source(generated) broken = source.replace( 'offline = backend.mode in ("synthetic", "scripted")', @@ -258,6 +276,7 @@ def test_the_setup_comparison_catches_a_changed_offline_definition(): assert mode_neutral_statements(broken) != template_stmts +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) @pytest.mark.parametrize( "removed_line", [ @@ -272,7 +291,7 @@ def test_the_setup_comparison_catches_a_changed_offline_definition(): ), ], ) -def test_the_setup_comparison_catches_a_dropped_statement(removed_line): +def test_the_setup_comparison_catches_a_dropped_statement(removed_line, mode): """Issue #215 was the scaffolder silently *dropping* a statement the template has -- not adding or rewording one. This reproduces that direction directly against the real generated ``setup`` cell: delete one of its own statements and show the comparison disagrees. A @@ -280,9 +299,37 @@ def test_the_setup_comparison_catches_a_dropped_statement(removed_line): would not catch any of these three; equality modulo ``TEMPLATE_ONLY_SETUP_STATEMENTS`` catches all three, because dropping a statement changes the length, and the two normalised lists are no longer the same list.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "setup") - template_stmts = _setup_template_stmts(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup"))) + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup") + template_stmts = _setup_template_stmts( + _source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup")), mode + ) source = _source(generated) broken = source.replace(removed_line, "", 1) assert broken != source, "the mutation did not change anything -- the line was not found" assert mode_neutral_statements(broken) != template_stmts + + +def test_the_image_scaffold_setup_comparison_catches_a_dropped_selection(): + """The image scaffold's ``setup`` cell is compared with the template's whole cell, so it is + the one cell where dropping ``selection`` (which the text scaffolds are allowed to omit) is a + difference: the cells below it print ``validation`` numbers with ``{selection}{check}``.""" + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay-images"), "setup") + template_stmts = _setup_template_stmts( + _source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup")), "replay-images" + ) + assert TEMPLATE_ONLY_SETUP_STATEMENTS[0] in template_stmts + source = _source(generated) + dropped = re.sub(r"\nselection = .*", "", source, count=1) + assert dropped != source, "the mutation did not change anything -- the regex did not match" + assert mode_neutral_statements(dropped) != template_stmts + + +def test_the_image_scaffold_asks_for_the_images_header_and_the_text_scaffolds_do_not(): + """``backend = ...`` is masked by the comparison above, so what the image scaffold changes in + that one call is asserted here: ``images=True``, the option that enables the experimental + images request header in a live run (docs/backends.md) and is ignored offline.""" + wanted = 'AuditedBackend(get_backend(fixtures=responses_path(), images=True), label="main")' + for mode in ("replay", "scripted", "replay-images"): + source = _source(_cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup")) + assert ("images=True" in source) == (mode == "replay-images"), mode + assert (wanted in source) == (mode == "replay-images"), mode diff --git a/tools/new_recipe.py b/tools/new_recipe.py index 2289de00..93bd7a33 100644 --- a/tools/new_recipe.py +++ b/tools/new_recipe.py @@ -1,7 +1,8 @@ """Create ``recipes/NN-slug/`` for recipe NN from ``catalog/recipes.json``. - python tools/new_recipe.py 7 # a replay recipe (the default) - python tools/new_recipe.py 36 --mode scripted # a scripted or simulator recipe + python tools/new_recipe.py 7 # a replay recipe (the default) + python tools/new_recipe.py 36 --mode scripted # a scripted or simulator recipe + python tools/new_recipe.py 77 --mode replay-images # a replay recipe whose requests carry images The folder gets a skeleton that follows ``recipes/_template/``: the notebook sections in order, a README, ``helpers.py``, ``build_fixtures.py``, ``side_effects.json`` (the recipe's @@ -15,10 +16,19 @@ what is left. The command never overwrites: if the folder exists it stops with a message and exit status 1. +``--mode replay-images`` is the multimodal variant of replay (docs/vision.md). It writes the files +above and then runs the new folder's own ``build_fixtures.py`` once, which draws the example +images with ``jev_cookbook.synthetic_images`` and writes ``assets/``, ``assets-manifest.json`` and +``fixtures/``. This tool itself stays standard library only; that one step needs the installed +package and Pillow, which the scaffolded ``build_fixtures.py`` imports. If it fails, the folder is +removed and the command exits with status 1. + The scaffold is born on the audited request-count route (#213, #368): the notebook's one ``get_backend(...)`` is wrapped in ``AuditedBackend(..., label="main")``, a final standalone cell prints ``backend.audit_json()``, and ``request_budget.json`` declares the one ``decide()`` call the -skeleton's own ``answer`` cell makes -- matching, out of the box, with no hand-edit. The README's +skeleton's own ``answer`` cell makes (``replay-images``: the ``decide()`` calls of its example +notebook, with the two image quantities of docs/backends.md) -- matching, out of the box, with no +hand-edit. The README's ``## Switch to live`` paragraph is rendered from that declaration, never hand-typed; as the builder adds more ``decide()`` calls while writing Evaluation, ``request_budget.json`` and the rendered paragraph are updated together (the README TODO there says how). @@ -31,6 +41,8 @@ import argparse import json import re +import shutil +import subprocess import sys from pathlib import Path @@ -65,6 +77,39 @@ "names." ) MODES = ("replay", "scripted") # --mode; replay is the default and the one most recipes use +# --mode replay-images (docs/vision.md): replay, with images. It is a value of --mode rather than a +# separate flag so that no combination of options can ask for an image recipe that is not a replay +# recipe. MODES stays the two text modes: code that walks the text scaffolds (the tests, the recipe +# guard) keeps walking exactly those. +IMAGE_MODE = "replay-images" +ALL_MODES = (*MODES, IMAGE_MODE) + +# The example the image scaffold ships, so that it runs end to end the moment it is written: an +# invented sign-in window (jev_cookbook.synthetic_images.login_screen) and two questions about it, +# "is an error banner shown?" (a Noul) and "where is it?" (a Box). TODO markers in the files that +# carry it say what to replace. One row per example: +# (id, split, capture, screen state, drawing seed, probability of yes, the object Jev's +# rectangle is drawn around, that rectangle's pixel offsets from the object (left, top, right, +# bottom), the box confidence or None) +# The stored answers are deliberately imperfect: v05's rectangle overlaps the gold box by less than +# half (IoU 0.374), v06 and t05 answer yes about screens with no banner, t07 answers no about one +# with a banner, and t03's rectangle reports no confidence. +IMAGE_ROWS = ( + ("v01", "validation", "C-1001", "error", 11, 0.93, "error_banner", (0, 0, 0, 0), 0.90), + ("v02", "validation", "C-1002", "error", 12, 0.88, "error_banner", (4, -2, -6, 2), 0.86), + ("v03", "validation", "C-1003", "normal", 13, 0.06, "submit", (0, 0, 0, 0), 0.40), + ("v04", "validation", "C-1004", "loading", 14, 0.12, "spinner", (0, 0, 0, 0), 0.35), + ("v05", "validation", "C-1005", "error", 15, 0.64, "error_banner", (30, 10, -40, -8), 0.55), + ("v06", "validation", "C-1006", "crash", 16, 0.58, "dialog", (0, 0, 0, 0), 0.30), + ("t01", "test", "C-2001", "error", 21, 0.91, "error_banner", (0, 0, 0, 0), 0.92), + ("t02", "test", "C-2002", "error", 22, 0.84, "error_banner", (10, 3, -12, -2), 0.90), + ("t03", "test", "C-2003", "error", 23, 0.72, "error_banner", (4, -2, -6, 2), None), + ("t04", "test", "C-2004", "normal", 24, 0.09, "submit", (0, 0, 0, 0), 0.45), + ("t05", "test", "C-2005", "loading", 25, 0.81, "spinner", (0, 0, 0, 0), 0.93), + ("t06", "test", "C-2006", "crash", 26, 0.45, "dialog", (0, 0, 0, 0), 0.38), + ("t07", "test", "C-2007", "error", 27, 0.18, "error_banner", (0, 0, 0, 0), 0.50), + ("d01", "demo", "C-3001", "error", 31, 0.90, "error_banner", (0, 0, 0, 0), 0.91), +) # The audited request-count route (#213, #368): a recipe declares its live request count once, in # request_budget.json, and the README's budget paragraph is rendered from it by @@ -88,10 +133,24 @@ def request_budget_text(mode: str = "replay") -> str: exactly the one ``decide()`` call the skeleton notebook's ``answer`` cell makes (see ``notebook_cells``) -- so a fresh scaffold's declaration, audited notebook and rendered README paragraph already agree, with no hand-edit. ``fixtures`` is ``null`` for a scripted recipe - (nothing to replay) and the recipe-relative replay file otherwise. As the builder adds + (nothing to replay) and the recipe-relative replay file otherwise. ``replay-images`` is the + exception to "one call": its example notebook decides each of ``IMAGE_ROWS`` once, so it + declares that many requests and the two image quantities that go with them + (docs/backends.md, "Actual calls and distinct requests"). As the builder adds ``decide()`` calls while writing Evaluation, this file's declared count (and ``distinct`` or ``at_most`` where they apply) must grow to match, and the README paragraph is regenerated from it -- the ``## Switch to live`` TODO in ``readme_text`` says how.""" + if mode == IMAGE_MODE: + # The example notebook decides every example once, and each request carries one image + # copy of one distinct image (the reference sits once in the state and in no + # question), so the two image quantities of docs/backends.md equal the request count. + count = len(IMAGE_ROWS) + body = { + "requests": count, + "fixtures": "fixtures/responses.json", + "profile": {"images": count, "distinct_images": count}, + } + return json.dumps({"schema": 1, "backends": {"main": body}}, indent=2) + "\n" fixtures = None if mode == "scripted" else "fixtures/responses.json" body = {"requests": 1, "fixtures": fixtures} return json.dumps({"schema": 1, "backends": {"main": body}}, indent=2) + "\n" @@ -159,8 +218,11 @@ def _source_lines(catalog: dict, recipe: dict) -> str: def notebook_cells(catalog: dict, recipe: dict, mode: str = "replay") -> list[dict]: """The skeleton notebook's cells, with the catalog fields filled in. ``mode`` is ``replay`` - (the backend replays ``fixtures/responses.json``) or ``scripted`` (a seeded script in - ``helpers.py`` answers, and there are no stored responses).""" + (the backend replays ``fixtures/responses.json``), ``scripted`` (a seeded script in + ``helpers.py`` answers, and there are no stored responses) or ``replay-images`` (replay, with + images; see ``image_notebook_cells``).""" + if mode == IMAGE_MODE: + return image_notebook_cells(catalog, recipe) scripted = mode == "scripted" number = recipe["rank"] title = recipe["title"] @@ -363,6 +425,236 @@ def code(cell_id: str, text: str) -> dict: ] +def image_notebook_cells(catalog: dict, recipe: dict) -> list[dict]: + """The ``--mode replay-images`` notebook: the replay notebook's cells (so the intro, the + run-mode paragraph, the ``questions``, ``measured`` and audit cells are the ones a replay + recipe has), with the cells that touch images or a Box answer replaced, and one added. + + * ``setup`` turns the images header on (``get_backend(..., images=True)``), imports what the + cells below use and defines ``selection``, because the cells below print ``validation`` + numbers; + * ``state`` builds the request (state and assets) for the demo example and shows its image; + * ``answer`` decides it with ``assets=`` and ``answer-boxes`` (added) draws the Box answer on + the image with ``draw_boxes``; + * ``python-rule`` freezes the presence and box-confidence thresholds on ``validation``; + * ``evaluation`` reports the ``test`` split through ``localization_report``, and + ``evaluation-review`` lists what Python did not locate and why. + + The example is marked ``TODO`` wherever it is the builder's to replace.""" + cells = notebook_cells(catalog, recipe, "replay") + number, title = recipe["rank"], recipe["title"] + + def md(cell_id: str, text: str) -> dict: + return _cell("markdown", cell_id, text) + + def code(cell_id: str, text: str) -> dict: + return _cell("code", cell_id, text) + + replacements = { + "setup-md": md( + "setup-md", + f"## {SECTIONS[1]}\n\n" + "The notebook runs from its own folder with `jev_cookbook` and the packages it " + "installs; Pillow reads and draws the images. The header states which mode ran. " + + RUN_MODE_PARAGRAPH + + f" {TODO_MARK}: keep this paragraph as it is; the setup cell defines `selection` " + "and `check`, so print `{selection}{check}` on every `validation` number.\n\n" + "The images are in `assets/`, drawn by `build_fixtures.py`; `assets-manifest.json` " + "records where each came from ([docs/vision.md](../../docs/vision.md), " + "[docs/fixtures.md](../../docs/fixtures.md#image-provenance-manifest)). " + "`get_backend(..., images=True)` is the option that enables the experimental images " + "request header in a live run; offline backends ignore it " + "([docs/backends.md](../../docs/backends.md#get_backend)).\n\n" + f"{TODO_MARK}: add the imports your later sections use, one sentence on the fixture " + "set and its size, and what the images show.", + ), + "setup": code( + "setup", + f""" +from IPython.display import Image, display + +from jev_cookbook import AuditedBackend, get_backend, load_helpers +from jev_cookbook.evaluation import noul_confidence, select_confidence_threshold +from jev_cookbook.fixtures import load_inputs, load_labels, responses_path, select_split +from jev_cookbook.style import apply_style, run_header, show_answer +from jev_cookbook.vision import box_confidence, box_iou, draw_boxes, gold_box, localization_report + +apply_style() +helpers = load_helpers() # this recipe's helpers.py, loaded by path +examples = load_inputs() +labels = load_labels() +backend = AuditedBackend(get_backend(fixtures=responses_path(), images=True), label="main") +# scored excludes the demo example: it carries no gold label, so no metric below is about it. +scored = [e for e in examples if e.split not in ("train", "demo")] + +offline = backend.mode in ("synthetic", "scripted") +check = " (a pipeline check, not a Jev result)" if offline else "" +# A number from the validation split is a selection step, not a result, in every run mode. +selection = " (a selection step, not a reported result)" + +run_header( + {number}, + {json.dumps(title)}, + backend=backend, + n_examples=len(scored), +) +""", + ), + "state-md": md( + "state-md", + f"## {SECTIONS[2]}\n\n" + f"{TODO_MARK}: say how Python builds the request from one example and what Jev sees: " + "the fixture names the image by a path in `fields`, Python loads it with " + "`image_asset(path, root=...)` and gives Jev a reference to its bytes, once, in the " + "state. Name what Python keeps for itself (the capture identifier, the path, the gold " + "label).", + ), + "state": code( + "state", + """ +import json + +demo = [e for e in examples if e.split == "demo"] +if not demo: + raise ValueError("add at least one demo example to ROWS in build_fixtures.py") +example = demo[0] +print("Python has:") +print(json.dumps(example.fields, indent=2)) +state, assets = helpers.build_request(example.fields) +print("Jev sees:") +print(json.dumps(state, indent=2)) +display(Image(filename=helpers.ROOT / example.fields["image"])) +""", + ), + "answer-md": md( + "answer-md", + f"## {SECTIONS[4]}\n\n" + f"{TODO_MARK}: show one image answer before any aggregate and say what its fields " + "mean: the probability of yes for presence, and the rectangle's four edges " + "(normalized to the image) with its confidence, which can be missing. A Box returns a " + "rectangle even when the target is absent, so a returned rectangle cannot establish " + "that the target exists.", + ), + "answer": code( + "answer", + """ +result = backend.decide(state, questions, assets=assets) +for answer in result.answers.values(): + show_answer(answer) +""", + ), + "python-md": md( + "python-md", + f"## {SECTIONS[5]}\n\n" + f"{TODO_MARK}: describe the rule in `helpers.py` that holds whatever the model " + "answers (`locate`): a rectangle counts only after the presence question says the " + "target is there, an uncertain presence answer or a rectangle without enough " + "confidence goes to an explicit review outcome, and a review is never an absence. " + "Say how each threshold is chosen on `validation` and frozen. Then apply it here.", + ), + "python-rule": code( + "python-rule", + """ +val_examples = select_split(examples, "validation") +val_results = [ + backend.decide(state, questions, assets=assets) + for state, assets in (helpers.build_request(e.fields) for e in val_examples) +] +val_gold = [labels[e.id] for e in val_examples] +# A yes is a probability above one half; only a call at or above the frozen threshold is acted on, +# so the threshold is the lowest confidence at which every call is right. +presence_right = [ + (r["present"].noul > 0.5) == (gold_box(g) is not None) + for r, g in zip(val_results, val_gold, strict=True) +] +presence_threshold = select_confidence_threshold( + presence_right, noul_confidence([r["present"].noul for r in val_results]), target_accuracy=1.0 +) +# The same rule for the rectangle: among present targets whose rectangle has a confidence, a +# rectangle is right when its IoU with the gold box is at least one half. +sized = [ + (box_iou(gold_box(g), r["location"]) >= 0.5, box_confidence(r["location"])) + for r, g in zip(val_results, val_gold, strict=True) + if gold_box(g) is not None and box_confidence(r["location"]) is not None +] +box_threshold = select_confidence_threshold( + [ok for ok, _ in sized], [c for _, c in sized], target_accuracy=1.0 +) +print(f"presence threshold, frozen from here on: {presence_threshold:.2f}{selection}{check}") +print(f"box threshold, frozen from here on: {box_threshold:.2f}{selection}{check}") + +shown = helpers.locate( + example.fields["capture"], + result["present"], + result["location"], + presence_threshold, + box_threshold, +) +print(f"{shown.capture}: {shown.kind} ({shown.reason})") +""", + ), + "evaluation-md": md( + "evaluation-md", + f"## {SECTIONS[6]}\n\n" + f"{TODO_MARK}: choose every setting on `validation` (above: the presence and box " + "thresholds), freeze it, and report once on `test` through `localization_report`, " + "which counts every example once and names each denominator. In a synthetic or " + "scripted run print each metric with `check` beside it; an individual example listed " + "below a metric is not a metric and needs no label. Say which `validation` numbers " + "your selection forces, or that none is. Compare with the baseline the issue names, " + "if there is one.", + ), + "evaluation": code( + "evaluation", + """ +test_examples = select_split(examples, "test") +test_results = [ + backend.decide(state, questions, assets=assets) + for state, assets in (helpers.build_request(e.fields) for e in test_examples) +] +test_gold = [labels[e.id] for e in test_examples] +test_outcomes = [ + helpers.locate( + e.fields["capture"], r["present"], r["location"], presence_threshold, box_threshold + ) + for e, r in zip(test_examples, test_results, strict=True) +] +report = localization_report(test_gold, [o.outcome for o in test_outcomes]) +print(f"test, {len(test_examples)} examples, thresholds frozen{check}") +for line in str(report).splitlines(): + print(f"{line}{check}") +""", + ), + } + out: list[dict] = [] + for cell in cells: + out.append(replacements.get(cell["id"], cell)) + if cell["id"] == "answer": + out.append( + code( + "answer-boxes", + """ +picture = helpers.open_image(example.fields["image"]) +marked = draw_boxes(picture, {"Jev": result["location"]}) +display(Image(data=helpers.png_bytes(marked), format="png")) +""", + ) + ) + if cell["id"] == "evaluation": + out.append( + code( + "evaluation-review", + """ +print("Test examples Python did not locate, and why:") +for item, decided in zip(test_examples, test_outcomes, strict=True): + if decided.kind != "located": + print(f" {item.id}: {decided.kind} ({decided.reason})") +""", + ) + ) + return out + + def notebook_json(catalog: dict, recipe: dict, mode: str = "replay") -> str: notebook = { "cells": notebook_cells(catalog, recipe, mode), @@ -402,6 +694,29 @@ def readme_text(catalog: dict, recipe: dict, mode: str = "replay") -> str: else "the offline run replays synthetic answers written for this recipe, so its numbers " "check that the pipeline works." ) + if mode == IMAGE_MODE: + run_note = ( + "The notebook runs from this folder with no network and no API key, replaying " + "`fixtures/` and reading the images in `assets/`. `build_fixtures.py` draws those " + "images and writes `assets-manifest.json`, which records where each came from, its " + "licence and what it is attested not to be " + "([docs/fixtures.md](../../docs/fixtures.md#image-provenance-manifest)); " + "[docs/vision.md](../../docs/vision.md) covers loading images and Box answers.\n\n" + f"{TODO_MARK}: before you commit, recreate the images under the notebook constraints " + "pin (the install in the paragraph below) with `python build_fixtures.py`: the PNG " + "bytes are part of the replay key of each request that carries a picture, and the " + "committed bytes are meant to be the ones " + "that pin writes " + "([docs/synthetic-images.md](../../docs/synthetic-images.md#byte-stability-and-regeneration))." + ) + mode_line = ( + "synthetic (offline replay of hand-written answers about drawn images). " + "Not measured live." + ) + closing = ( + "the offline run replays synthetic answers written for this recipe, about images " + "drawn for it, so its numbers check that the pipeline works." + ) types = " + ".join(f"`{t}`" for t in recipe["decision_types"]) return f"""# {recipe["title"]} @@ -490,6 +805,8 @@ def readme_text(catalog: dict, recipe: dict, mode: str = "replay") -> str: def helpers_text(recipe: dict, mode: str = "replay") -> str: + if mode == IMAGE_MODE: + return image_helpers_text(recipe) script_block = "" if mode == "scripted": script_block = f''' @@ -544,6 +861,8 @@ def rows_hint(level: int) -> str: def build_fixtures_text(recipe: dict, mode: str = "replay") -> str: + if mode == IMAGE_MODE: + return image_build_fixtures_text(recipe) if mode == "scripted": return build_scripted_fixtures_text(recipe) return f'''"""Write fixtures/inputs.jsonl, labels.jsonl and responses.json for recipe {recipe["rank"]:02d}. @@ -744,6 +1063,8 @@ def tests_text(recipe: dict, mode: str = "replay") -> str: # The scaffolder knows the mode when it writes this file, so each mode gets only the # replay-key test that applies to it: a replay recipe never defines helpers.script or # helpers.SEED, so a scripted branch in its test file would be dead code, copied sixty times. + if mode == IMAGE_MODE: + return _fill(IMAGE_TESTS, recipe) if mode == "scripted": return f'''"""Tests for recipe {recipe["rank"]:02d}'s helpers. They load the helpers by file path.""" @@ -845,13 +1166,21 @@ def predicted(answer): ''' -def build_fixtures_test_text(recipe: dict) -> str: - """Guard tests for ``build_fixtures.py`` (replay mode only; a scripted recipe has no +def build_fixtures_test_text(recipe: dict, mode: str = "replay") -> str: + """Guard tests for ``build_fixtures.py`` (replay modes only; a scripted recipe has no ``responses.json`` and nothing for these tests to guard, see ``build_scripted_fixtures_text``): refusal without ``--force``, ``inputs.jsonl``/``labels.jsonl`` regenerated from ``ROWS`` even on a refused run, and the committed ``responses.json`` byte-identical to what ``jev_cookbook.live``'s recorder would write. See - ``recipes/_template/tests/test_build_fixtures.py``, which this follows.""" + ``recipes/_template/tests/test_build_fixtures.py``, which this follows. + + ``mode="replay-images"`` returns the variant for a recipe with ``assets/``: the temporary + copy it rebuilds in also holds the committed images (their bytes are part of the replay + key of each request that carries them), and it adds the checks that the images and ``assets-manifest.json`` are reproduced and + that a changed picture is refused. A recipe that has ``assets/`` commits this variant; called + with one argument, the function returns the text variant.""" + if mode == IMAGE_MODE: + return _fill(IMAGE_BUILD_FIXTURES_TEST, recipe) slug = recipe["slug"] module_name = f"recipe{recipe['rank']:02d}_build_fixtures_for_test" return f'''"""build_fixtures.py follows the recipes/_template/build_fixtures.py pattern: inputs and @@ -960,6 +1289,705 @@ def test_the_generator_reproduces_the_committed_fixtures(tmp_path): ''' +# ------------------------------------------------------------------------ the image variant +# +# The three files below (and the image tests after them) are written from plain templates with +# @TOKEN@ placeholders rather than f-strings: they are full of braces and backslashes, and a +# generated file must read exactly as written here. + +IMAGE_HELPERS = r'''"""Python's half of recipe @RANK@: the request Jev sees, the questions and the rules. + +The fixture generator, the notebook and the tests all load this file with +``jev_cookbook.load_helpers``, so the images, the questions and the rules come from one place. + +@TODO@: the example below asks whether an invented sign-in window shows an error banner, and where. +Replace the questions, the request and the rule with your recipe's, and keep the pattern. A path +read from a fixture is data, so an image is loaded with ``image_asset(path, root=ROOT)``; its +reference comes from its bytes; the reference goes into the state once, because every place it +appears adds another copy of the image to the request (docs/vision.md); and a rectangle counts +only after a presence question says the target is there. +""" + +import io +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from PIL import Image + +from jev_cookbook import Box, Noul +from jev_cookbook.evaluation import noul_confidence +from jev_cookbook.vision import ( + box_confidence, + image_asset, + image_reference, + normalized_box, + resolve_asset_path, +) + +# No ``from __future__ import annotations`` here: load_helpers removes this module from +# ``sys.modules``, so typing.get_type_hints cannot resolve postponed annotations on a dataclass. + +# The recipe folder. The image paths in fixtures/inputs.jsonl are relative to it. +ROOT = Path(__file__).resolve().parent + +LOCATED, ABSENT, REVIEW = "located", "absent", "review" # what localization_report counts + + +def build_request(fields: dict[str, Any]) -> tuple[dict[str, str], dict[str, dict[str, str]]]: + """``(state, assets)`` for one example: what Jev sees and the images that go with it. + + ``fields`` names the image by a path relative to the recipe and also carries the capture + identifier. Python keeps the identifier and attaches it to the outcome, so it never needs to + pass through the model. The state holds only the image's reference. + """ + asset = image_asset(fields["image"], root=ROOT) + reference = image_reference(asset) + return {"screenshot": reference}, {reference: asset} + + +def build_questions() -> dict[str, Any]: + """The questions asked about every example: is the target there, and where is it?""" + return { + "present": Noul(instructions="The sign-in window in `screenshot` shows an error banner."), + "location": Box(instructions="The error banner in the sign-in window in `screenshot`."), + } + + +def open_image(path: str) -> Image.Image: + """The image at ``path`` (relative to the recipe) as an RGB Pillow image to draw on.""" + with Image.open(resolve_asset_path(path, root=ROOT)) as picture: + return picture.convert("RGB") + + +def png_bytes(image: Image.Image) -> bytes: + """``image`` as PNG bytes, for ``IPython.display.Image(data=..., format="png")``.""" + buffer = io.BytesIO() + image.save(buffer, format="PNG") + return buffer.getvalue() + + +@dataclass(frozen=True) +class Localized: + """What Python decided: a rectangle, an absence, or a review, with the reason.""" + + capture: str + kind: str + box: dict[str, float] | None + reason: str + + @property + def outcome(self) -> tuple[str, dict[str, float] | None]: + """The pair ``jev_cookbook.vision.localization_report`` takes for this example.""" + return (self.kind, self.box) + + +def locate( + capture: str, + presence: Any, + location: Any, + presence_threshold: float, + box_threshold: float, +) -> Localized: + """Accept a rectangle only when the presence answer is confident and says the target is there. + + ``presence`` is the Noul answer and ``location`` the Box answer. A presence answer whose + confidence is below ``presence_threshold`` goes to review, not to absent. Otherwise a + probability of yes above one half means the target is there, and anything else means it is + not (the Box's rectangle is then ignored). A present target is located only when its + rectangle carries a confidence of at least ``box_threshold``; a rectangle with no confidence + or a lower one goes to review. The rule is code, so it holds whatever the model answers. + """ + for name, value in ( + ("presence_threshold", presence_threshold), + ("box_threshold", box_threshold), + ): + if not 0.0 <= value <= 1.0: + raise ValueError(f"{name} must be between 0 and 1, got {value!r}") + if noul_confidence([presence.noul])[0] < presence_threshold: + return Localized(capture, REVIEW, None, "presence confidence below the threshold") + if presence.noul <= 0.5: + return Localized(capture, ABSENT, None, "no error banner") + confidence = box_confidence(location) + if confidence is None: + return Localized(capture, REVIEW, None, "the rectangle has no confidence") + if confidence < box_threshold: + return Localized(capture, REVIEW, None, "rectangle confidence below the threshold") + return Localized(capture, LOCATED, normalized_box(location), "confident rectangle") + + +# @TODO@: the other rules Python enforces whatever the model answers, with tests. +''' + +IMAGE_BUILD_FIXTURES = r'''"""Write assets/, assets-manifest.json and fixtures/ for recipe @RANK@. + + python build_fixtures.py # from anywhere; it writes next to this file + python build_fixtures.py --force # also overwrite a responses.json holding a recorded answer + python build_fixtures.py --overwrite-assets # accept a picture that changed (replay keys change too) + +The images are drawn here with jev_cookbook.synthetic_images (docs/synthetic-images.md): no +photograph, brand, person or third-party picture, and the gold Box label of each is derived from +what was drawn, not typed in. The responses are synthetic and deliberately imperfect. The replay +keys come from the same ``build_request`` and ``build_questions`` the notebook uses, and cover the +images' bytes. See docs/fixtures.md and docs/vision.md. + +@TODO@: the sign-in windows, the two questions and the answers below are an example. Replace them +with your recipe's images (draw them with ``jev_cookbook.synthetic_images`` primitives, or give +``canvas.provenance("build_fixtures.py:your_function", {...})`` to a scene of your own so the +manifest names a generator that exists here), and keep ROWS' split names and demo example. + +Images are saved under the byte policy of docs/synthetic-images.md. Their bytes are part of the +replay key of each request that carries them, so if a regenerated picture differs from the file that is already there this script +stops and writes nothing more, unless ``--overwrite-assets`` accepts the change. + +Generating inputs and labels is kept separate from generating responses, on purpose: once +responses.json holds even one recorded answer (provenance "recorded", captured from a real Jev +call), running this script again must not silently replace it with a synthetic probability. +inputs.jsonl and labels.jsonl are always rewritten from ROWS, because neither ever holds a +model's answer; responses.json is rewritten only when it does not yet exist, holds only +synthetic answers, or --force is given. +""" + +import argparse +import json +from pathlib import Path + +from jev_cookbook import ( + BoxAnswer, + DecisionResult, + NoulAnswer, + Provenance, + dump_responses, + load_helpers, + replay_key, +) +from jev_cookbook import synthetic_images as si + +HERE = Path(__file__).resolve().parent +helpers = load_helpers(HERE) +QUESTIONS = helpers.build_questions() + +TARGET = "error_banner" # the drawn object whose rectangle is the gold box, when the screen has it + +# (id, split, capture, screen state, drawing seed, probability of yes, the drawn object Jev's +# rectangle is around, that rectangle's pixel offsets from the object (left, top, right, bottom), +# the rectangle's confidence or None for a Box answer that reports none) +ROWS = [ +@ROWS@ +] # @TODO@: your examples, across validation and test, and at least one demo + + +def image_path(ident): + """Where the image of example ``ident`` is saved, relative to the recipe.""" + return f"assets/screen-{ident}.png" + + +def draw(row): + """The canvas of one row. ``login_screen`` is a shared scene, so the manifest entry names it + as the generator and ``si.check_reproducible`` can redraw it from the entry.""" + _ident, _split, _capture, state, seed, *_answer = row + return si.login_screen(state=state, seed=seed) + + +def write_images(rows, overwrite): + """Draw and save every image under the byte policy; return the manifest entries by path. + + Raises ``si.RegenerationMismatch`` (and saves nothing more) for a picture that differs from + the committed file, unless ``overwrite`` is true.""" + entries = [] + for row in rows: + path = image_path(row[0]) + canvas = draw(row) + status = canvas.write(HERE, path, overwrite=overwrite) + print(f"{status}: {path}") + description = f"An invented sign-in window in the {row[3]!r} state (seed {row[4]})." + entries.append(canvas.manifest_entry(HERE, path, description)) + return sorted(entries, key=lambda entry: entry["path"]) + + +def _fields(row): + return {"capture": row[2], "image": image_path(row[0])} + + +def build_inputs_and_labels(rows): + """``(inputs, labels)`` from ``rows``, with the images already saved. Neither ever holds a + model's answer, so both are always safe to regenerate, even after ``responses.json`` has + been recorded. A gold Box label is derived from what was drawn.""" + inputs, labels = [], [] + for row in rows: + ident, split = row[0], row[1] + fields = _fields(row) + state, assets = helpers.build_request(fields) + key = replay_key(state, QUESTIONS, assets=assets) + inputs.append({"id": ident, "split": split, "fields": fields, "replay_keys": [key]}) + if split != "demo": + drawn = draw(row).objects.get(TARGET) + labels.append({"id": ident, "label": drawn.label() if drawn else si.absent_label()}) + return inputs, labels + + +def answers_for(row, provenance: Provenance): + """{question name: answer} for one row: the presence probability and a rectangle around the + object ``row`` names, moved by its offsets (the rectangle a model returns even when the + target is absent).""" + _ident, _split, _capture, _state, _seed, p_yes, shown, offsets, confidence = row + canvas = draw(row) + width, height = canvas.size + left, top, right, bottom = ( + edge + offset for edge, offset in zip(canvas.objects[shown].pixels, offsets, strict=True) + ) + box = { + "top": top / height, + "bottom": bottom / height, + "left": left / width, + "right": right / width, + } + return { + "present": NoulAnswer(p_yes, provenance), + "location": BoxAnswer(box, confidence, provenance), + } + + +def build_responses(rows): + """``{replay_key: stored response}`` from ``rows``, with the images already saved. Always + synthetic: this script never calls Jev, so it can never produce a recorded response.""" + responses = {} + for row in rows: + state, assets = helpers.build_request(_fields(row)) + key = replay_key(state, QUESTIONS, assets=assets) + answers = answers_for(row, Provenance.synthetic()) + responses[key] = DecisionResult(answers, "synthetic").to_dict() + return responses + + +def _is_recorded(path: Path) -> bool: + """True if ``path`` exists and holds at least one response whose model is not + ``"synthetic"`` (a recorded, or otherwise real, answer). A file that fails to parse, or + whose top level is not a JSON object, cannot hold a valid synthetic response either, so it + is treated as not recorded rather than raising; inside an object, an entry that is itself + not an object is treated as if it were recorded, so it blocks an overwrite instead of being + silently skipped.""" + if not path.exists(): + return False + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return False + if not isinstance(data, dict): + return False + return any( + not isinstance(entry, dict) or entry.get("model") != "synthetic" for entry in data.values() + ) + + +def _unresolved_keys(inputs, responses_file: Path) -> list[str]: + """Replay keys the just-rewritten ``inputs`` ask for that ``responses_file`` does not have, + used only to warn when a ROWS edit has desynchronised the two.""" + if not responses_file.exists(): + return [] + try: + data = json.loads(responses_file.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return [] + if not isinstance(data, dict): + return [] + return [key for row in inputs for key in row["replay_keys"] if key not in data] + + +def main() -> None: + parser = argparse.ArgumentParser(description="Write this recipe's assets/ and fixtures/.") + parser.add_argument( + "--force", + action="store_true", + help="overwrite responses.json even if it holds a recorded (non-synthetic) answer", + ) + parser.add_argument( + "--overwrite-assets", + action="store_true", + help="accept an image that differs from the committed file (its replay keys change)", + ) + args = parser.parse_args() + if not ROWS: + raise SystemExit("@TODO@: add examples to ROWS in build_fixtures.py") + try: + entries = write_images(ROWS, args.overwrite_assets) + except si.RegenerationMismatch as error: + raise SystemExit(f"refused: {error}\nnothing else was written") from None + manifest = {"schema": 1, "assets": entries} + (HERE / "assets-manifest.json").write_text( + json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="\n" + ) + folder = HERE / "fixtures" + folder.mkdir(exist_ok=True) + inputs, labels = build_inputs_and_labels(ROWS) + for name, rows in (("inputs.jsonl", inputs), ("labels.jsonl", labels)): + text = "".join(json.dumps(row) + "\n" for row in rows) + (folder / name).write_text(text, encoding="utf-8", newline="\n") + responses_file = folder / "responses.json" + if _is_recorded(responses_file) and not args.force: + message = ( + f"refusing to overwrite {responses_file}: it holds a recorded response " + "(pass --force to overwrite it anyway)" + ) + if _unresolved_keys(inputs, responses_file): + message += ( + "\ninputs.jsonl and labels.jsonl above were rewritten from ROWS; " + "responses.json was not, and at least one of the keys the rewritten inputs " + "ask for is missing from it. The three files are desynchronised until you " + "--force a rewrite or record the missing answers." + ) + raise SystemExit(message) + responses = build_responses(ROWS) + # dump_responses is the serialization the recorder writes with (top-level keys sorted, each + # response keeping the field order DecisionResult.to_dict() emits), so a recording's diff + # stays limited to the values that actually changed. + responses_file.write_text(dump_responses(responses), encoding="utf-8", newline="\n") + print(f"wrote {len(inputs)} examples, {len(labels)} labels, {len(responses)} responses") + + +if __name__ == "__main__": + main() +''' + + +IMAGE_TESTS = r'''"""Tests for recipe @RANK@'s helpers. They load the helpers by file path.""" + +import json +from pathlib import Path + +import pytest + +from jev_cookbook import BoxAnswer, NoulAnswer, Provenance, get_backend, load_helpers, replay_key +from jev_cookbook import synthetic_images as si +from jev_cookbook.evaluation import noul_confidence, select_confidence_threshold +from jev_cookbook.fixtures import load_inputs, load_labels, responses_path +from jev_cookbook.vision import gold_box + +RECIPE = Path(__file__).resolve().parent.parent +helpers = load_helpers(RECIPE) + +BOX = {"top": 0.25, "bottom": 0.5, "left": 0.125, "right": 0.5} + + +def noul(probability): + return NoulAnswer(probability, Provenance.synthetic()) + + +def box(confidence): + return BoxAnswer(BOX, confidence, Provenance.synthetic()) + + +def test_questions_are_built_by_python(): + # @TODO@: assert your questions' instructions and criteria against literal expected values + # (not helpers' own data, which would make the assertion self-referential), then test every + # rule in helpers.py + questions = helpers.build_questions() + assert list(questions) == ["present", "location"] + assert [question.to_dict()["type"] for question in questions.values()] == ["noul", "box"] + + +def test_every_replay_key_in_the_fixtures_matches_the_current_question(): + # The fixture validator cannot see question drift; this test can. One request per example; + # adapt this when an example needs a dependent second request (CONTRIBUTING: a question that + # depends on an earlier answer goes in a later request), where the example lists a key for + # each request in order. The key covers the image's bytes, so a changed picture fails here. + questions = helpers.build_questions() + for example in load_inputs(RECIPE): + state, assets = helpers.build_request(example.fields) + assert example.replay_keys == (replay_key(state, questions, assets=assets),) + + +def test_every_scene_can_be_redrawn_from_its_manifest_entry(): + # A shared scene is rebuilt from the generator and parameters its entry records and compared + # with the committed file; a recipe's own generator is covered by test_build_fixtures.py. + manifest = json.loads((RECIPE / "assets-manifest.json").read_text("utf-8")) + shared = [ + entry + for entry in manifest["assets"] + if entry["generator"].startswith("jev_cookbook.synthetic_images:") + ] + if not shared: + pytest.skip("no entry names a shared scene; test_build_fixtures.py covers the rest") + for entry in shared: + assert si.check_reproducible(RECIPE, entry) in ("unchanged", "kept-committed-bytes") + + +def test_a_confident_yes_with_a_confident_rectangle_is_located(): + found = helpers.locate("C-1", noul(0.9), box(0.9), 0.3, 0.8) + assert (found.kind, found.reason) == ("located", "confident rectangle") + assert found.box == BOX and found.outcome == ("located", BOX) + + +def test_a_confident_no_is_absent_and_ignores_the_rectangle(): + found = helpers.locate("C-1", noul(0.1), box(0.99), 0.3, 0.8) + assert found.outcome == ("absent", None) + + +def test_an_unconfident_presence_answer_is_a_review_never_an_absence(): + # confidence 2 * max(p, 1 - p) - 1: 0.6 for p = 0.8, 0.2 for p = 0.4 + assert helpers.locate("C-1", noul(0.4), box(0.99), 0.3, 0.8).outcome == ("review", None) + assert helpers.locate("C-1", noul(0.8), box(0.99), 0.7, 0.8).outcome == ("review", None) + assert helpers.locate("C-1", noul(0.8), box(0.99), 0.6, 0.8).kind == "located" + + +def test_a_probability_of_exactly_one_half_is_not_a_yes(): + assert helpers.locate("C-1", noul(0.5), box(0.99), 0.0, 0.8).outcome == ("absent", None) + + +def test_a_review_is_never_an_absence_over_a_grid_of_probabilities_and_thresholds(): + # 101 probabilities (0.00 to 1.00) by 5 thresholds: below the threshold the outcome is a + # review; at or above it, an absence is the answer "no" and nothing else. + for step in range(101): + probability = step / 100 + confidence = noul_confidence([probability])[0] + for threshold in (0.0, 0.25, 0.5, 0.75, 1.0): + found = helpers.locate("C-1", noul(probability), box(0.99), threshold, 0.0) + if confidence < threshold: + assert found.kind == "review", (probability, threshold) + elif probability <= 0.5: + assert found.kind == "absent", (probability, threshold) + else: + assert found.kind == "located", (probability, threshold) + + +def test_a_rectangle_without_enough_confidence_is_a_review(): + missing = helpers.locate("C-1", noul(0.9), box(None), 0.3, 0.0) + assert (missing.kind, missing.reason) == ("review", "the rectangle has no confidence") + low = helpers.locate("C-1", noul(0.9), box(0.5), 0.3, 0.8) + assert (low.kind, low.reason) == ("review", "rectangle confidence below the threshold") + assert helpers.locate("C-1", noul(0.9), box(0.8), 0.3, 0.8).kind == "located" + + +def test_a_threshold_outside_zero_to_one_is_an_error(): + with pytest.raises(ValueError, match="presence_threshold"): + helpers.locate("C-1", noul(0.9), box(0.9), 1.1, 0.8) + with pytest.raises(ValueError, match="box_threshold"): + helpers.locate("C-1", noul(0.9), box(0.9), 0.3, -0.1) + + +def test_stored_answers_are_not_all_right(): + # A wrong answer anywhere is a weak guard: it would still pass even if a confidence gate + # caught every mistake. Mirror your notebook's own threshold selection here (the lowest + # confidence at which every validation answer that counts is right) and require a wrong + # `test` answer at or above it: a mistake the gate would still let through, which is what + # evaluating on `test` exists to catch. This checks the presence question; repeat it for the + # rectangle's confidence if that gate matters to your recipe. + backend = get_backend(fixtures=responses_path(RECIPE), images=True) + questions = helpers.build_questions() + labels = load_labels(RECIPE) + examples = load_inputs(RECIPE) + + def answer_of(example): + state, assets = helpers.build_request(example.fields) + return backend.decide(state, questions, assets=assets)["present"] + + def right(example, answer): + return (answer.noul > 0.5) == (gold_box(labels[example.id]) is not None) + + validation = [e for e in examples if e.split == "validation" and e.id in labels] + val_answers = [answer_of(e) for e in validation] + threshold = select_confidence_threshold( + [right(e, a) for e, a in zip(validation, val_answers, strict=True)], + noul_confidence([a.noul for a in val_answers]), + target_accuracy=1.0, + ) + + test = [e for e in examples if e.split == "test" and e.id in labels] + test_answers = [answer_of(e) for e in test] + wrong_and_confident = [ + e.id + for e, a in zip(test, test_answers, strict=True) + if not right(e, a) and noul_confidence([a.noul])[0] >= threshold + ] + assert wrong_and_confident, "expected at least one confidently wrong test answer" +''' + +IMAGE_BUILD_FIXTURES_TEST = r'''"""build_fixtures.py follows the recipes/_template/build_fixtures.py pattern, with images: inputs +and labels generation is separate from responses generation, a recorded responses.json is never +silently overwritten, a picture that differs from the committed file is refused rather than +rewritten, and the committed files match what the script writes: the images and +assets-manifest.json unchanged, and responses.json byte for byte what jev_cookbook.live's writer +produces.""" + +import importlib.util +import json +import shutil +import subprocess +import sys +from pathlib import Path + +from PIL import Image + +from jev_cookbook import dump_responses +from jev_cookbook import synthetic_images as si + +RECIPE = Path(__file__).resolve().parent.parent +SCRIPT = RECIPE / "build_fixtures.py" + + +def load_module_at(path, name): + """Import build_fixtures.py by file path so its functions and module-level values (ROWS, + build_responses) are plain Python objects, not subprocess output or a re-dump of the file + it wrote.""" + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _jsonl(rows): + return "".join(json.dumps(row) + "\n" for row in rows) + + +def _copy_of_the_recipe(folder): + """A folder holding what a rerun needs: the two scripts and the committed images. The images + are copied because their bytes are part of the replay key of each request that carries + them: with the committed files beside it the script compares each picture it draws with the + committed one, and without them it would create the pictures afresh with whatever PNG + encoder this environment has.""" + folder.mkdir() + for name in ("helpers.py", "build_fixtures.py"): + (folder / name).write_bytes((RECIPE / name).read_bytes()) + shutil.copytree(RECIPE / "assets", folder / "assets") + return folder / "build_fixtures.py" + + +def test_the_committed_responses_are_byte_identical_to_the_recorders_writer(): + """jev_cookbook.dump_responses sorts only the top-level keys; json.dumps(..., sort_keys=True) + sorts every nested dict too and so disagrees with it on every response's field order. The + committed file must match dump_responses exactly, not just agree with it on top-level order. + + Comparing against dump_responses(json.loads(raw)) (re-dumping the file's own parsed content) + cannot catch a sort_keys=True regression: json.loads preserves whatever nested order the file + already has, and dump_responses only re-sorts the top level, so that round trip would pass no + matter which writer produced the file. Instead, import build_fixtures.py and compare against + dump_responses of what build_responses computes directly from ROWS, independent of what + main() actually wrote to disk.""" + raw = (RECIPE / "fixtures" / "responses.json").read_text("utf-8") + module = load_module_at(SCRIPT, "recipe@RANK@_build_fixtures_for_test") + assert len(module.ROWS) > 1 + assert raw == dump_responses(module.build_responses(module.ROWS)) + + +def test_build_fixtures_separates_inputs_labels_from_responses(tmp_path): + """Mutation check: regenerating must never silently overwrite a recorded responses.json, + and --force must restore the exact synthetic bytes. inputs.jsonl and labels.jsonl are + rewritten from ROWS on every run, refused or not: only responses.json is guarded, and this + also proves it by corrupting both files before the refused run and checking that they come + back exactly as build_inputs_and_labels(ROWS) computes them, not merely "changed".""" + script = _copy_of_the_recipe(tmp_path / "@SLUG@") + copy = script.parent + first = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert first.returncode == 0, first.stderr + + responses = copy / "fixtures" / "responses.json" + data = json.loads(responses.read_text("utf-8")) + for value in data.values(): + value["model"] = "jev-1.13.0" + responses.write_text( + json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8", newline="\n" + ) + before = responses.read_bytes() + + inputs_path = copy / "fixtures" / "inputs.jsonl" + labels_path = copy / "fixtures" / "labels.jsonl" + inputs_path.write_bytes(b"corrupted") + labels_path.write_bytes(b"corrupted") + + refused = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert refused.returncode != 0 + assert "recorded" in refused.stderr and "--force" in refused.stderr + assert responses.read_bytes() == before + module = load_module_at(script, "recipe@RANK@_build_fixtures_for_test_refused") + expected_inputs, expected_labels = module.build_inputs_and_labels(module.ROWS) + assert inputs_path.read_text("utf-8") == _jsonl(expected_inputs) + assert labels_path.read_text("utf-8") == _jsonl(expected_labels) + + forced = subprocess.run( + [sys.executable, str(script), "--force"], capture_output=True, text=True + ) + assert forced.returncode == 0, forced.stderr + after = json.loads(responses.read_text("utf-8")) + assert all(v["model"] == "synthetic" for v in after.values()) + + +def test_the_generator_reproduces_the_committed_fixtures_and_images(tmp_path): + script = _copy_of_the_recipe(tmp_path / "@SLUG@") + copy = script.parent + done = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert done.returncode == 0, done.stderr + for name in ("inputs.jsonl", "labels.jsonl", "responses.json"): + assert (copy / "fixtures" / name).read_bytes() == (RECIPE / "fixtures" / name).read_bytes() + assert (copy / "assets-manifest.json").read_bytes() == ( + RECIPE / "assets-manifest.json" + ).read_bytes() + committed = sorted(p.relative_to(RECIPE) for p in (RECIPE / "assets").rglob("*") if p.is_file()) + assert committed + assert sorted(p.relative_to(copy) for p in (copy / "assets").rglob("*") if p.is_file()) == ( + committed + ) + for path in committed: + assert (copy / path).read_bytes() == (RECIPE / path).read_bytes(), path + + +def test_a_picture_that_differs_from_the_committed_file_is_refused_not_rewritten(tmp_path): + """The byte policy of docs/synthetic-images.md: a changed picture is a new hash and so new + replay keys, which is a decision. The script stops, names the problem and writes nothing more; + --overwrite-assets accepts the redrawn picture and restores the committed one.""" + script = _copy_of_the_recipe(tmp_path / "@SLUG@") + copy = script.parent + picture = sorted((copy / "assets").glob("*.png"))[0] + committed = picture.read_bytes() + with Image.open(picture) as original: + size = original.size + Image.new("RGB", size, "white").save(picture) + changed = picture.read_bytes() + assert changed != committed + manifest = copy / "assets-manifest.json" + assert not manifest.exists() + + refused = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert refused.returncode != 0 + assert "refused" in refused.stderr and "nothing else was written" in refused.stderr + assert picture.read_bytes() == changed + assert not manifest.exists() and not (copy / "fixtures").exists() + + accepted = subprocess.run( + [sys.executable, str(script), "--overwrite-assets"], capture_output=True, text=True + ) + assert accepted.returncode == 0, accepted.stderr + # The same pixels as the committed picture. Whether the bytes are the committed ones too + # depends on the PNG encoder of the environment running this test (docs/synthetic-images.md). + assert picture.read_bytes() != changed + assert si.png_pixels_equal(picture.read_bytes(), committed) +''' + + +def image_rows_text() -> str: + """``IMAGE_ROWS`` as the lines of ``ROWS`` in the image ``build_fixtures.py``.""" + + def literal(value: object) -> str: + if isinstance(value, tuple): + return "(" + ", ".join(literal(item) for item in value) + ")" + return json.dumps(value) if isinstance(value, str) else repr(value) + + return "\n".join(f" {literal(row)}," for row in IMAGE_ROWS) + + +def _fill(template: str, recipe: dict) -> str: + """``template`` with its ``@TOKEN@`` placeholders replaced.""" + filled = template.replace("@RANK@", f"{recipe['rank']:02d}").replace("@TODO@", TODO_MARK) + return filled.replace("@SLUG@", recipe["slug"]).replace("@ROWS@", image_rows_text()) + + +def image_helpers_text(recipe: dict) -> str: + return _fill(IMAGE_HELPERS, recipe) + + +def image_build_fixtures_text(recipe: dict) -> str: + return _fill(IMAGE_BUILD_FIXTURES, recipe) + + def files_for(catalog: dict, recipe: dict, mode: str = "replay") -> dict[str, str]: """``{relative path: text}`` of everything the scaffold creates.""" files = { @@ -976,16 +2004,50 @@ def files_for(catalog: dict, recipe: dict, mode: str = "replay") -> dict[str, st # logic for these tests to guard (build_scripted_fixtures_text), so the file is emitted # for replay recipes only. files["tests/test_build_fixtures.py"] = build_fixtures_test_text(recipe) + if mode == IMAGE_MODE: + files["tests/test_build_fixtures.py"] = build_fixtures_test_text(recipe, mode) return files +BUILD_TIMEOUT = 300 # seconds the image scaffold waits for the new folder's build_fixtures.py + + +def _draw_images(target: Path) -> None: + """Run ``target``'s own ``build_fixtures.py`` once, which draws ``assets/`` and writes + ``assets-manifest.json`` and ``fixtures/`` (``--mode replay-images``). If it cannot run or + exits with an error, ``target`` is removed and ``ScaffoldError`` carries what it printed, so + the command leaves no half-written recipe.""" + try: + done = subprocess.run( + [sys.executable, "build_fixtures.py"], + cwd=target, + capture_output=True, + text=True, + timeout=BUILD_TIMEOUT, + ) + problem = "" if done.returncode == 0 else "\n".join(done.stderr.strip().splitlines()[-8:]) + if done.returncode != 0 and not problem: + problem = f"exit status {done.returncode}" + except (OSError, subprocess.SubprocessError) as error: + problem = f"{type(error).__name__}: {error}" + if problem: + shutil.rmtree(target, ignore_errors=True) + raise ScaffoldError( + "--mode replay-images draws the example images by running the new folder's " + "build_fixtures.py, which needs the installed package and Pillow " + f'(pip install -e ".[dev]"); it failed, so nothing was created:\n{problem}' + ) + + def scaffold(number: int, catalog_path: Path, recipes_dir: Path, mode: str = "replay") -> Path: """Create ``recipes_dir/NN-slug/`` and return it. Raises ``ScaffoldError`` and writes - nothing when the number is out of range, is not in the catalog, or the folder exists.""" + nothing when the number is out of range, is not in the catalog, or the folder exists. For + ``--mode replay-images`` it then runs the folder's ``build_fixtures.py`` once (see + ``_draw_images``); if that fails the folder is removed again.""" if not FIRST <= number <= LAST: raise ScaffoldError(f"recipe number must be {FIRST} to {LAST}, got {number}") - if mode not in MODES: - raise ScaffoldError(f"mode must be one of {', '.join(MODES)}, got {mode!r}") + if mode not in ALL_MODES: + raise ScaffoldError(f"mode must be one of {', '.join(ALL_MODES)}, got {mode!r}") catalog = _load_catalog(catalog_path) recipe = find_recipe(catalog, number) slug = recipe.get("slug") @@ -1004,6 +2066,8 @@ def scaffold(number: int, catalog_path: Path, recipes_dir: Path, mode: str = "re path = target / name path.parent.mkdir(parents=True, exist_ok=True) path.write_text(text, encoding="utf-8", newline="\n") + if mode == IMAGE_MODE: + _draw_images(target) return target @@ -1020,10 +2084,12 @@ def main(argv: list[str] | None = None) -> int: ) parser.add_argument( "--mode", - choices=MODES, + choices=ALL_MODES, default="replay", help="replay (default): fixtures/responses.json is replayed; scripted: a seeded script " - "in helpers.py answers, for simulator and ScriptedBackend recipes", + "in helpers.py answers, for simulator and ScriptedBackend recipes; replay-images: replay " + "with images, which also draws assets/ and a working example by running the new " + "folder's build_fixtures.py (needs the installed package and Pillow)", ) args = parser.parse_args(argv) if not args.number.isascii() or not args.number.isdigit(): From cf80140a9dcfbeb216913acf7b7b690d4c303844 Mon Sep 17 00:00:00 2001 From: Timothy Wayne Gregg Date: Sat, 31 Oct 2026 18:44:43 -0400 Subject: [PATCH 2/2] Foundation: give the image scaffold's example screens names that are clearly invented (Refs #315) Reading the product names that login_screen draws for the first set of seeds showed two that match real brands (MAFEX and ROMER), and several short ones that could. The manifest of an image recipe attests real_brand false, and docs/synthetic-images.md asks the author to read every drawn name, so the example uses other seeds: v04, t02, t03, t04, t06, t07 and d01 change (14 -> 17, 22 -> 32, 23 -> 34, 24 -> 35, 26 -> 48, 27 -> 54, 31 -> 60), giving longer names that were read by eye (this is not a trademark search). Only the drawing seed changes. The screen states, the gold boxes, the stored answers and every number the notebook prints are unchanged, so the tests' hand-computed values still hold; the one test that edits a row by its text follows the new seed of t07. Co-Authored-By: Claude Sonnet 5.5 Co-Authored-By: Claude Haiku 5.5 --- tests/test_new_recipe.py | 2 +- tools/new_recipe.py | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/tests/test_new_recipe.py b/tests/test_new_recipe.py index d8178910..c12e93ff 100644 --- a/tests/test_new_recipe.py +++ b/tests/test_new_recipe.py @@ -2456,7 +2456,7 @@ def test_the_generated_stored_answers_test_fails_when_no_wrong_answer_is_confide script = folder / "build_fixtures.py" text = script.read_text("utf-8") weakened = text.replace('"loading", 25, 0.81,', '"loading", 25, 0.51,').replace( - '"error", 27, 0.18,', '"error", 27, 0.49,' + '"error", 54, 0.18,', '"error", 54, 0.49,' ) assert weakened.count("0.51") == 1 and weakened.count("0.49") == 1 script.write_text(weakened, encoding="utf-8", newline="\n") diff --git a/tools/new_recipe.py b/tools/new_recipe.py index 93bd7a33..2a2b3d4d 100644 --- a/tools/new_recipe.py +++ b/tools/new_recipe.py @@ -98,17 +98,17 @@ ("v01", "validation", "C-1001", "error", 11, 0.93, "error_banner", (0, 0, 0, 0), 0.90), ("v02", "validation", "C-1002", "error", 12, 0.88, "error_banner", (4, -2, -6, 2), 0.86), ("v03", "validation", "C-1003", "normal", 13, 0.06, "submit", (0, 0, 0, 0), 0.40), - ("v04", "validation", "C-1004", "loading", 14, 0.12, "spinner", (0, 0, 0, 0), 0.35), + ("v04", "validation", "C-1004", "loading", 17, 0.12, "spinner", (0, 0, 0, 0), 0.35), ("v05", "validation", "C-1005", "error", 15, 0.64, "error_banner", (30, 10, -40, -8), 0.55), ("v06", "validation", "C-1006", "crash", 16, 0.58, "dialog", (0, 0, 0, 0), 0.30), ("t01", "test", "C-2001", "error", 21, 0.91, "error_banner", (0, 0, 0, 0), 0.92), - ("t02", "test", "C-2002", "error", 22, 0.84, "error_banner", (10, 3, -12, -2), 0.90), - ("t03", "test", "C-2003", "error", 23, 0.72, "error_banner", (4, -2, -6, 2), None), - ("t04", "test", "C-2004", "normal", 24, 0.09, "submit", (0, 0, 0, 0), 0.45), + ("t02", "test", "C-2002", "error", 32, 0.84, "error_banner", (10, 3, -12, -2), 0.90), + ("t03", "test", "C-2003", "error", 34, 0.72, "error_banner", (4, -2, -6, 2), None), + ("t04", "test", "C-2004", "normal", 35, 0.09, "submit", (0, 0, 0, 0), 0.45), ("t05", "test", "C-2005", "loading", 25, 0.81, "spinner", (0, 0, 0, 0), 0.93), - ("t06", "test", "C-2006", "crash", 26, 0.45, "dialog", (0, 0, 0, 0), 0.38), - ("t07", "test", "C-2007", "error", 27, 0.18, "error_banner", (0, 0, 0, 0), 0.50), - ("d01", "demo", "C-3001", "error", 31, 0.90, "error_banner", (0, 0, 0, 0), 0.91), + ("t06", "test", "C-2006", "crash", 48, 0.45, "dialog", (0, 0, 0, 0), 0.38), + ("t07", "test", "C-2007", "error", 54, 0.18, "error_banner", (0, 0, 0, 0), 0.50), + ("d01", "demo", "C-3001", "error", 60, 0.90, "error_banner", (0, 0, 0, 0), 0.91), ) # The audited request-count route (#213, #368): a recipe declares its live request count once, in