diff --git a/docs/recipe-template.md b/docs/recipe-template.md index ffbdf6f..f945446 100644 --- a/docs/recipe-template.md +++ b/docs/recipe-template.md @@ -8,6 +8,7 @@ folder for it. This page is the reference and the walkthrough; the template's ow ```bash python tools/new_recipe.py NN # create recipes/NN-slug/ from the catalog (NN is 1 to 100) python tools/new_recipe.py NN --mode scripted # the same for a scripted or simulator recipe +python tools/new_recipe.py NN --mode replay-images # the same for a recipe whose requests carry images python tools/execute_notebook.py recipes/X # run X/notebook.ipynb offline, write the outputs back ``` @@ -38,13 +39,13 @@ live inference explicitly opt-in under the [contract](../CONTRIBUTING.md). | `README.md` | The one-page recipe README, plus the builder's walkthrough (a real recipe's README has only the first part). | | `helpers.py` | `build_state`, `build_questions`, and the rule Python enforces (`route`). | | `build_fixtures.py` | Writes `fixtures/`; the replay keys come from `helpers.py`, so they cannot drift from the questions. It lives next to the notebook, outside `fixtures/` ([fixtures.md](fixtures.md)). Generating inputs and labels is separate from generating responses: `--force` is needed to overwrite a `responses.json` that already holds a `recorded` answer. | -| `assets/` | Multimodal recipes only (absent from the template): the image files the recipe sends to Jev, PNG or baseline JPEG, each at most 1 MiB, 8 MiB in all, with no metadata, and nowhere else in the recipe. The contract and the check that enforces it are in [fixtures.md](fixtures.md#image-fixtures); how a recipe loads them is in [vision.md](vision.md). | -| `assets-manifest.json` | Multimodal recipes only, beside `assets/`: one entry per image with its hash, size, generator and parameters, licence and the attestations that it is synthetic and holds no real photograph, brand, person or third-party content. Required exactly when `assets/` exists; `build_fixtures.py` should write it with the images. Format, worked example and rules: [fixtures.md](fixtures.md#image-provenance-manifest). | +| `assets/` | Multimodal recipes only (absent from the template): the image files the recipe sends to Jev, PNG or baseline JPEG, each at most 1 MiB, 8 MiB in all, with no metadata, and nowhere else in the recipe. The contract and the check that enforces it are in [fixtures.md](fixtures.md#image-fixtures); how a recipe loads them is in [vision.md](vision.md). `python tools/new_recipe.py NN --mode replay-images` creates it with generated example images ([Image recipes](#image-recipes)). | +| `assets-manifest.json` | Multimodal recipes only, beside `assets/`: one entry per image with its hash, size, generator and parameters, licence and the attestations that it is synthetic and holds no real photograph, brand, person or third-party content. Required exactly when `assets/` exists; `build_fixtures.py` should write it with the images. Format, worked example and rules: [fixtures.md](fixtures.md#image-provenance-manifest). `--mode replay-images` writes it through the scaffolded `build_fixtures.py`. | | `fixtures/` | `inputs.jsonl`, `labels.jsonl`, `responses.json`: 22 examples (10 `validation`, 10 `test`, 2 `demo`), synthetic and deliberately imperfect. | | `request_budget.json` | The recipe's declared live request count (#213, #368), the one place it is stated: one audited backend, `main`, declaring `requests: 22` and `fixtures: "fixtures/responses.json"`. The setup cell's one `get_backend(...)` is wrapped in `AuditedBackend(..., label="main")`, and a final standalone cell (nested under "What was and was not measured" as a `###` subsection, never a new `##` heading) prints `backend.audit_json()`. The README's `## Switch to live` paragraph is *rendered* from this file by `jev_cookbook.accounting.render_recipe_note`, never typed by hand, and the recipe guard's `readme_budget_note_matches_request_audit` check (#213) compares the declaration, the audited execution and the rendered paragraph against each other for every published recipe. The template's own agreement is checked by `tests/test_template.py` using the same audit. [backends.md](backends.md#the-declared-count-request_budgetjson) is the full contract; this is the one route a recipe's live request count is stated through. | | `tests/test_helpers.py` | Tests for the rule, a check that the stored keys match the current question, `test_stored_answers_are_not_all_right` (replay recipes only): re-derives the frozen threshold from `validation` (the `select_confidence_threshold` call written out, or `gate = helpers.(...)` through a function of the recipe's own `helpers.py` that calls it; the guard finds the test by its name, so keep `test_stored_answers_are_not_all_right` as the prefix) and requires a wrong `test` answer at or above it, so a fixture set that is merely "not all correct" (but has every mistake caught by the gate) still fails, and a test that pins the threshold-selection call's own **objective** keyword against the committed cell source (see "Pin the threshold's OBJECTIVE" above) — a value-only re-derivation does not pin which objective ran. `tools/new_recipe.py` scaffolds a working copy of the stored-answers test, so no recipe has to hand-write it from scratch; the objective pin is adapted per recipe from the template's own worked example. | | `side_effects.json` | The recipe's statement of its simulated side effects (a `ReviewQueue`, `ActionLog`, `Store`, `Transactor` or `Simulator` write): `{"schema": 1, "sites": {"helpers.submit_reviews": 2}}`, the exact number of places the notebook calls each side-effect function or method, or `{"schema": 1, "none": "why"}`. `tools/new_recipe.py` scaffolds it as a `none` declaration whose reason is a `TODO` placeholder (the template recipe has no simulated side effect, so `none` is what it would declare): rewrite the reason, or replace the file with `sites`, before the recipe is done. The recipe guard rejects a reason that still carries the `TODO`, rejects `none` for a recipe that has a side effect or needs a simulator, and fails a notebook that calls fewer, more or none of what it declares. Schema, keys, `not_side_effects` (bookkeeping only, with a reason per entry), factory-built simulators, the guard's known limits and the two function shapes (a batch fill, a decide-and-apply function) with their test rules: [simulation.md](simulation.md). | -| `tests/test_build_fixtures.py` | Guards `build_fixtures.py` (replay recipes only): refusal without `--force`, `inputs.jsonl`/`labels.jsonl` regenerated from `ROWS` even on a refused run, and the committed `responses.json` byte-identical to what `jev_cookbook.dump_responses` returns (the public form of the serialization the recorder writes with; `jev_cookbook.live._dump` is kept only as a transitional alias, and recipes do not import it). A scripted recipe has no `responses.json` and nothing for these tests to guard, so the scaffolder does not emit this file for `--mode scripted`. | +| `tests/test_build_fixtures.py` | Guards `build_fixtures.py` (replay recipes only): refusal without `--force`, `inputs.jsonl`/`labels.jsonl` regenerated from `ROWS` even on a refused run, and the committed `responses.json` byte-identical to what `jev_cookbook.dump_responses` returns (the public form of the serialization the recorder writes with; `jev_cookbook.live._dump` is kept only as a transitional alias, and recipes do not import it). A scripted recipe has no `responses.json` and nothing for these tests to guard, so the scaffolder does not emit this file for `--mode scripted`. `--mode replay-images` emits a variant of it that also copies `assets/` into the temporary folder it rebuilds in ([Image recipes](#image-recipes)). | The notebook sections, in order: **What you will build**, **Setup and run mode**, **The state**, **The questions**, **One answer up close**, **Python's part**, **Evaluation**, **What was and was not @@ -524,6 +525,80 @@ describes, with the simulator owning the state and the script standing in for th a pipeline check, not a Jev result; the README and notebook say so under their `TODO`s. A scripted recipe that later records real answers becomes a replay recipe: add the keys and `responses.json`. +## Image recipes + +A recipe whose requests carry images (the multimodal track, ranks 61–100: +[multimodal-track.md](multimodal-track.md)) is a replay recipe with an `assets/` folder. Scaffold it +with: + +```bash +python tools/new_recipe.py NN --mode replay-images +``` + +`--mode` is the option that already chooses a variant of the scaffold, so the image variant is one +more value of it (read it as "replay, with images") rather than a separate flag; a flag could be +combined with `--mode scripted`, and no value of `--mode` can ask for an image recipe that is not a +replay recipe. Everything in [vision.md](vision.md) applies to what it writes. + +**It writes a working example, not a skeleton.** The text scaffold raises `NotImplementedError` +until you write `helpers.py` and `ROWS`; the image scaffold ships a small recipe that runs end to +end, so that the image plumbing (the files, the references, the budget) is correct before you change +anything, and `TODO` markers say what to replace. The example asks of fourteen drawings of an +invented sign-in window (`jev_cookbook.synthetic_images.login_screen`; 6 `validation`, 7 `test` and +1 `demo` example) whether an error banner is shown (a `Noul`) and where (a `Box`). + +The command writes the files of the replay scaffold, then runs the new folder's own +`build_fixtures.py` once. That script draws the images, saves them under the byte policy of +[synthetic-images.md](synthetic-images.md#byte-stability-and-regeneration), and writes +`assets-manifest.json` and `fixtures/`. `tools/new_recipe.py` itself needs only the standard +library; that one step needs the installed package and Pillow, which the scaffolded +`build_fixtures.py` imports (`pip install -e ".[dev]"`). If it fails, the folder is removed and the +command exits with status 1. + +| File | What the image scaffold adds to the replay scaffold | +| - | - | +| `assets/`, `assets-manifest.json` | Fourteen PNG files, `assets/screen-.png`, and their manifest ([fixtures.md](fixtures.md#image-provenance-manifest)): `origin: generated`, the generator `jev_cookbook.synthetic_images:login_screen` with the arguments that redraw each file, licence `CC0-1.0` and the five attestations. | +| `fixtures/` | Complete. A fixture row's `fields` hold a `capture` identifier (Python keeps it) and the image's path, `assets/screen-.png`, as [fixtures.md](fixtures.md#image-fixtures) requires. Gold labels are Box labels ([fixtures.md](fixtures.md#box-labels)) derived from what was drawn: the banner's `Drawn.label()` where the screen has one, `{"present": false}` where not. The stored answers are synthetic and deliberately imperfect (one rectangle that misses the banner, two wrong yes answers and one wrong no, one rectangle without a confidence). | +| `helpers.py` | `build_request(fields)` returns `(state, assets)`: it loads the image with `image_asset(path, root=ROOT)` because the path comes from fixture data, derives `image_reference`, and puts the reference in the state once (every place a reference appears adds a copy of the image to the request: [vision.md](vision.md#image-copies-and-the-32-mib-caps)). `build_questions()` asks `present` (a `Noul`) and `location` (a `Box`). `locate(...)` is the rule that holds whatever the model answers: a presence answer below the threshold is a review, not an absence; a probability of yes above one half means the target is there; a rectangle with no confidence, or below its threshold, is a review. `open_image` and `png_bytes` serve the notebook's drawing cell. | +| `build_fixtures.py` | `ROWS`, then `draw`, `write_images`, `build_inputs_and_labels`, `answers_for` and `build_responses`. Images are written first, then the keys are computed with `replay_key(state, questions, assets=assets)` from the files on disk. `--force` behaves as in the text scaffold; `--overwrite-assets` accepts a picture that differs from the committed file, which changes its replay keys. Without it, a changed picture stops the script before anything else is written. | +| notebook | The cells of the replay notebook, with `get_backend(fixtures=responses_path(), images=True)` in the audited setup cell, and: `state` (builds the request for the demo example and shows its image), `answer` and `answer-boxes` (one answer up close, and `draw_boxes` on the image with the returned rectangle), `python-rule` (freezes the presence and rectangle-confidence thresholds on `validation` with `select_confidence_threshold`, then applies `locate` to the demo answer), `evaluation` (every `test` example once through `localization_report`, printed with `check` beside each line) and `evaluation-review` (what was not located, and why). Every `decide()` passes `assets=assets`. The setup cell defines `selection`, because these cells print `validation` numbers. | +| `request_budget.json` | `requests: 14`, `fixtures: "fixtures/responses.json"`, and `profile: {"images": 14, "distinct_images": 14}` ([backends.md](backends.md#actual-calls-and-distinct-requests)): the notebook decides each of the 14 examples once, and the reference sits once in each state, so each request is one copy of one distinct image. A recipe that repeats a reference or sends more than one image per request changes these two numbers with the count. The README's paragraph is rendered from this file, as for every scaffold. | +| `tests/test_helpers.py` | Tests of `locate` (each outcome and each reason, and a grid of probabilities and thresholds), the replay-key test with `assets=`, a test that redraws every shared-scene image from its manifest entry (`check_reproducible`), and `test_stored_answers_are_not_all_right`, which re-derives the presence threshold and requires a wrong `test` answer at or above it. | +| `tests/test_build_fixtures.py` | The text scaffold's three guards, run in a temporary folder that also holds a copy of `assets/` (the third also checks that the images and `assets-manifest.json` come back unchanged), and a fourth: a picture that differs from the committed file is refused, not rewritten. | + +**Why `tests/test_build_fixtures.py` differs.** A PNG's bytes depend on the Pillow and zlib that +wrote them, and the bytes are part of the replay key of each request that carries a picture. The +text scaffold's test copies only +`helpers.py` and `build_fixtures.py` into a temporary folder and runs the script there; for an +image recipe that would draw every picture afresh with the encoder of whatever environment runs the +test, and the regenerated fixtures could disagree with the committed ones although the pixels are +the same. The image variant copies the committed `assets/` too, so the script compares each picture +it draws with the committed file under that byte policy. That file must stay byte-identical to the +scaffold's, so the choice cannot be left to the builder: +`build_fixtures_test_text(recipe)` returns the text variant and +`build_fixtures_test_text(recipe, "replay-images")` the image one, which is the file an image +recipe commits. + +**What the scaffold passes offline.** `python -m jev_cookbook.fixtures validate`, +`python tools/check_recipe_assets.py`, `python tools/check_hygiene.py`, `ruff check`, +`ruff format --check`, and its own tests (`pytest recipes/NN-slug`), with no network, no key and no +hand-edit. `tests/test_new_recipe.py` checks that its declaration, with the two image quantities, agrees with +the request audit of the calls its notebook makes. The notebook has no outputs when it is +scaffolded: execute it with `tools/execute_notebook.py` (step 6 above) before the freshness check +can compare anything. + +**From scaffold to your recipe.** Replace the sign-in windows, the two questions and the answers +with your own, keeping the pattern: draw with `jev_cookbook.synthetic_images` (a scene of your own +calls `canvas.provenance("build_fixtures.py:your_function", {...})`, so the manifest names a +generator that exists), derive every gold Box label from a `Drawn`, put the image path in `fields`, +and keep the reference out of the questions. Grow `ROWS` to the size your level needs +([fixtures.md](fixtures.md)), update `request_budget.json` and its README paragraph together as +Evaluation adds `decide()` calls, and recreate the images under the notebook constraints pin +(`pip install -e ".[ml]" -c .github/constraints-notebooks.txt`) before you commit them, so that the +committed bytes are the pin's. The checklist in +[synthetic-images.md](synthetic-images.md#a-checklist-for-a-recipe) is the same list from the +drawing side. A frame sequence (ranks 85–92) has no scaffold of its own yet. + ## Executing a notebook `tools/execute_notebook.py ` runs `notebook.ipynb` with `nbclient` in a fresh kernel and diff --git a/tests/test_new_recipe.py b/tests/test_new_recipe.py index 1034b77..c12e93f 100644 --- a/tests/test_new_recipe.py +++ b/tests/test_new_recipe.py @@ -4,6 +4,7 @@ directory and a temporary copy of the catalog. """ +import ast import importlib.util import json import os @@ -16,12 +17,22 @@ import nbformat import pytest from nbclient.exceptions import DeadKernelError +from PIL import Image import jev_cookbook -from jev_cookbook import DecisionResult, Provenance, dump_responses, load_helpers, replay_key +from jev_cookbook import ( + DecisionResult, + Provenance, + dump_responses, + get_backend, + load_helpers, + replay_key, +) from jev_cookbook.accounting import parse_budget, render_budget_note from jev_cookbook.answers import ChoiceAnswer -from jev_cookbook.fixtures import validate_recipe +from jev_cookbook.evaluation import noul_confidence, select_confidence_threshold +from jev_cookbook.fixtures import load_inputs, load_labels, select_split, validate_recipe +from jev_cookbook.vision import box_confidence, box_iou, gold_box, localization_report REPO = Path(__file__).resolve().parent.parent TEMPLATE = REPO / "recipes" / "_template" @@ -113,7 +124,7 @@ def test_it_creates_the_folder_named_by_the_catalog(catalog, recipes): } -@pytest.mark.parametrize("mode", new_recipe.MODES) +@pytest.mark.parametrize("mode", new_recipe.ALL_MODES) def test_the_scaffold_writes_a_side_effects_declaration_with_a_placeholder_reason( catalog, recipes, mode ): @@ -1573,7 +1584,7 @@ def test_the_scaffold_readme_live_section_states_the_budget_and_env_var(catalog, assert "TODO" in section -@pytest.mark.parametrize("mode", new_recipe.MODES) +@pytest.mark.parametrize("mode", new_recipe.ALL_MODES) def test_the_local_budget_renderer_and_default_cap_agree_byte_for_byte_with_the_shared_package( mode, ): @@ -1841,3 +1852,692 @@ def test_the_scaffolded_build_fixtures_test_file_passes_against_a_filled_in_reci result = run_generated_tests(folder) assert result.returncode == 0, result.stdout + result.stderr assert "passed" in result.stdout and "failed" not in result.stdout + + +# -------------------------------------------------------------------------------------------- +# --mode replay-images (#315): the multimodal scaffold +# -------------------------------------------------------------------------------------------- +# +# Nothing below executes a notebook except test_a_fresh_image_scaffold_passes_the_request_audit_ +# with_no_hand_edit, which needs a kernel exactly like its text counterpart above. Everything else +# builds and checks the scaffold in plain Python and subprocesses. The scaffold draws its own +# images, by running the new folder's build_fixtures.py once. + +IMAGE = new_recipe.IMAGE_MODE +IMAGE_IDS = [ + "v01", + "v02", + "v03", + "v04", + "v05", + "v06", + "t01", + "t02", + "t03", + "t04", + "t05", + "t06", + "t07", + "d01", +] +# The drawn error banner of login_screen is the pixel rectangle (80, 66, 560, 98) of a 640 x 400 +# image, so its gold box is 66/400, 98/400, 80/640 and 560/640, computed by hand. +GOLD_BANNER = {"top": 0.165, "bottom": 0.245, "left": 0.125, "right": 0.875} +IMAGE_FILES = { + "notebook.ipynb", + "README.md", + "helpers.py", + "build_fixtures.py", + "side_effects.json", + "request_budget.json", + "assets-manifest.json", + "fixtures/inputs.jsonl", + "fixtures/labels.jsonl", + "fixtures/responses.json", + "tests/test_helpers.py", + "tests/test_build_fixtures.py", + *(f"assets/screen-{ident}.png" for ident in IMAGE_IDS), +} + + +def scaffold_images(catalog, recipes, number=77): + """Scaffold ``--mode replay-images`` into ``recipes`` and return the new recipe folder.""" + _, recipe = entry(catalog, number) + assert run(number, catalog, recipes, mode=IMAGE) == 0 + return recipes / recipe["slug"] + + +@pytest.fixture(scope="module") +def image_scaffold(tmp_path_factory): + """One image scaffold shared by the tests that only read it; a test that changes anything + works on ``copy_of`` it instead.""" + base = tmp_path_factory.mktemp("image-scaffold") + catalog = base / "recipes.json" + shutil.copy(REPO / "catalog" / "recipes.json", catalog) + return scaffold_images(catalog, base / "recipes") + + +def entry_for_folder(folder): + data = json.loads((REPO / "catalog" / "recipes.json").read_text("utf-8")) + return data, next(r for r in data["recipes"] if r["slug"] == folder.name) + + +def copy_of(folder, tmp_path): + target = tmp_path / folder.name + shutil.copytree(folder, target, ignore=shutil.ignore_patterns("__pycache__")) + return target + + +def files_of(folder): + return { + p.relative_to(folder).as_posix() + for p in folder.rglob("*") + if p.is_file() and "__pycache__" not in p.parts + } + + +def run_generated(folder, *targets): + """pytest over some of a scaffold's own tests (``tests/test_helpers.py``, or a node id), + where ``run_generated_tests`` runs them all.""" + return subprocess.run( + [sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "--rootdir", str(folder)] + + [str(folder / target) for target in targets], + cwd=folder, + capture_output=True, + text=True, + timeout=180, + ) + + +def test_the_modes_the_text_scaffolds_walk_are_unchanged_and_the_image_mode_is_one_more(): + """MODES is what the tests and the recipe guard iterate over to walk the text scaffolds; the + image mode is a further value of --mode that they do not walk.""" + assert new_recipe.MODES == ("replay", "scripted") + assert new_recipe.IMAGE_MODE == "replay-images" + assert new_recipe.ALL_MODES == ("replay", "scripted", "replay-images") + + +def test_the_image_scaffold_writes_the_text_files_the_assets_the_manifest_and_the_fixtures( + image_scaffold, +): + assert files_of(image_scaffold) == IMAGE_FILES + assert not list(image_scaffold.rglob("*.part")), "no temporary file is left in assets/" + + +def test_the_image_scaffold_passes_the_fixture_validator_offline(image_scaffold): + report = validate_recipe(image_scaffold) + assert list(report) == [] and report.mode == "replay" + done = subprocess.run( + [sys.executable, "-m", "jev_cookbook.fixtures", "validate", str(image_scaffold)], + capture_output=True, + text=True, + ) + assert done.returncode == 0, done.stdout + done.stderr + assert "fixtures valid (mode replay)" in done.stdout + + +def test_the_image_scaffold_passes_the_asset_checker_and_the_hygiene_check( + image_scaffold, monkeypatch +): + spec = importlib.util.spec_from_file_location( + "check_recipe_assets_for_image_test", REPO / "tools" / "check_recipe_assets.py" + ) + checker = importlib.util.module_from_spec(spec) + monkeypatch.setitem(sys.modules, spec.name, checker) # a dataclass looks its module up here + spec.loader.exec_module(checker) + entries = [ + checker.Entry( + f"recipes/{image_scaffold.name}/{name}", "100644", (image_scaffold / name).read_bytes() + ) + for name in sorted(files_of(image_scaffold)) + ] + assert checker.check_entries(entries) == [] + done = subprocess.run( + [sys.executable, "tools/check_hygiene.py"] + + [str(image_scaffold / name) for name in sorted(files_of(image_scaffold))], + cwd=REPO, + capture_output=True, + text=True, + ) + assert done.returncode == 0, done.stdout + done.stderr + + +def test_the_image_scaffold_manifest_describes_drawn_images_with_a_shared_generator( + image_scaffold, +): + manifest = json.loads((image_scaffold / "assets-manifest.json").read_text("utf-8")) + assert manifest["schema"] == 1 + assert [e["path"] for e in manifest["assets"]] == sorted( + f"assets/screen-{ident}.png" for ident in IMAGE_IDS + ) + states = {"v01": "error", "v03": "normal", "v04": "loading", "v06": "crash", "d01": "error"} + for item in manifest["assets"]: + ident = item["path"].split("screen-")[1].split(".")[0] + assert item["origin"] == "generated" and item["licence"] == "CC0-1.0" + assert item["generator"] == "jev_cookbook.synthetic_images:login_screen" + assert item["attestations"] == { + "synthetic": True, + "real_photo": False, + "real_brand": False, + "real_person": False, + "third_party_content": False, + } + assert (item["width"], item["height"]) == (640, 400) and item["format"] == "PNG" + if ident in states: + assert item["parameters"]["state"] == states[ident] + + +def test_the_image_scaffold_labels_are_the_drawn_geometry_not_typed_numbers(image_scaffold): + labels = { + row["id"]: row["label"] + for row in map( + json.loads, + (image_scaffold / "fixtures" / "labels.jsonl").read_text("utf-8").splitlines(), + ) + } + assert "d01" not in labels, "the demo example carries no label" + banner = {"present": True, "box": GOLD_BANNER} + absent = {"present": False} + assert labels == { + "v01": banner, + "v02": banner, + "v03": absent, + "v04": absent, + "v05": banner, + "v06": absent, + "t01": banner, + "t02": banner, + "t03": banner, + "t04": absent, + "t05": absent, + "t06": absent, + "t07": banner, + } + + +def test_the_image_scaffold_inputs_keep_the_identifier_and_the_path_in_fields(image_scaffold): + rows = [ + json.loads(line) + for line in (image_scaffold / "fixtures" / "inputs.jsonl").read_text("utf-8").splitlines() + ] + assert [row["id"] for row in rows] == IMAGE_IDS + for row in rows: + assert set(row["fields"]) == {"capture", "image"} + assert row["fields"]["image"] == f"assets/screen-{row['id']}.png" + assert len(row["replay_keys"]) == 1 + assert [row["split"] for row in rows].count("demo") == 1 + + +def test_the_image_scaffold_declares_one_image_per_request_and_the_readme_renders_it( + image_scaffold, +): + declared = json.loads((image_scaffold / "request_budget.json").read_text("utf-8")) + count = len(IMAGE_IDS) + assert declared == { + "schema": 1, + "backends": { + "main": { + "requests": count, + "fixtures": "fixtures/responses.json", + "profile": {"images": count, "distinct_images": count}, + } + }, + } + budget = parse_budget(declared) + readme = (image_scaffold / "README.md").read_text("utf-8") + assert render_budget_note(budget) in readme + assert f"**{count}** requests" in readme and f"JEV_COOKBOOK_LIVE_MAX_REQUESTS={count}" in readme + responses = json.loads((image_scaffold / "fixtures" / "responses.json").read_text("utf-8")) + assert len(responses) == count, "every request has exactly one stored response" + + +def test_the_image_scaffold_notebook_has_the_nine_sections_the_image_cells_and_no_outputs( + image_scaffold, +): + nb = nbformat.read(image_scaffold / "notebook.ipynb", as_version=4) + nbformat.validate(nb) + headings = [ + line[3:] + for c in nb.cells + if c.cell_type == "markdown" + for line in c.source.splitlines() + if line.startswith("## ") + ] + assert headings == new_recipe.SECTIONS + assert all( + not c.outputs and c.execution_count is None for c in nb.cells if c.cell_type == "code" + ) + ids = [c.id for c in nb.cells] + assert ids == [ + "intro", + "setup-md", + "setup", + "state-md", + "state", + "questions-md", + "questions", + "answer-md", + "answer", + "answer-boxes", + "python-md", + "python-rule", + "evaluation-md", + "evaluation", + "evaluation-review", + "measured-md", + "measured", + "audit-md", + "audit", + "next-md", + ] + source = {c.id: c.source for c in nb.cells} + assert ( + source["setup"].count("get_backend(") == 1 and source["setup"].count("AuditedBackend(") == 1 + ) + assert 'get_backend(fixtures=responses_path(), images=True), label="main"' in source["setup"] + assert "draw_boxes(picture," in source["answer-boxes"] + assert "localization_report(test_gold," in source["evaluation"] + assert "select_confidence_threshold(" in source["python-rule"] + assert [c.source for c in nb.cells if c.cell_type == "code"][ + -1 + ] == "print(backend.audit_json())" + + +def test_every_decide_call_in_the_image_notebook_passes_the_assets(): + """A decide() without assets= would send the references as plain text and find no stored + answer. Every cell that decides is checked from its parsed source, not by a text search.""" + catalog_data = json.loads((REPO / "catalog" / "recipes.json").read_text("utf-8")) + recipe = next(r for r in catalog_data["recipes"] if r["rank"] == 77) + sites = {} + for cell in new_recipe.notebook_cells(catalog_data, recipe, IMAGE): + if cell["cell_type"] != "code": + continue + for node in ast.walk(ast.parse(cell["source"])): + if isinstance(node, ast.Call) and getattr(node.func, "attr", None) == "decide": + sites[cell["id"]] = [k.arg for k in node.keywords] + assert sites == { + "answer": ["assets"], + "python-rule": ["assets"], + "evaluation": ["assets"], + } + + +def test_the_image_example_numbers_are_the_ones_the_notebook_will_print(image_scaffold): + """The notebook's python-rule and evaluation cells, written out in plain Python against the + scaffold's own fixtures (no kernel), and compared with values worked out by hand: the + thresholds are the lowest confidences at which every call above them is right, and the + report counts each of the seven test examples once.""" + helpers = load_helpers(image_scaffold) + questions = helpers.build_questions() + backend = get_backend(fixtures=image_scaffold / "fixtures" / "responses.json", images=True) + examples = load_inputs(image_scaffold) + labels = load_labels(image_scaffold) + + def decide(example): + state, assets = helpers.build_request(example.fields) + return backend.decide(state, questions, assets=assets) + + validation = select_split(examples, "validation") + results = [decide(e) for e in validation] + gold = [labels[e.id] for e in validation] + right = [ + (r["present"].noul > 0.5) == (gold_box(g) is not None) + for r, g in zip(results, gold, strict=True) + ] + # v06 says yes (0.58) about a screen with no banner, the only wrong call: its confidence 0.16 + # is the lowest, and the next lowest is v05's 2 * 0.64 - 1 = 0.28. + assert right == [True, True, True, True, True, False] + presence_threshold = select_confidence_threshold( + right, noul_confidence([r["present"].noul for r in results]), target_accuracy=1.0 + ) + assert presence_threshold == pytest.approx(0.28) + # Rectangles of the three present validation targets: v01 and v02 overlap the gold box by 1.0 + # and 0.872 (a hit each, at confidences 0.90 and 0.86), v05 by 0.374 (a miss, at 0.55). + sized = [ + (box_iou(gold_box(g), r["location"]) >= 0.5, box_confidence(r["location"])) + for r, g in zip(results, gold, strict=True) + if gold_box(g) is not None and box_confidence(r["location"]) is not None + ] + assert [ok for ok, _ in sized] == [True, True, False] + box_threshold = select_confidence_threshold( + [ok for ok, _ in sized], [c for _, c in sized], target_accuracy=1.0 + ) + assert box_threshold == pytest.approx(0.86) + + test = select_split(examples, "test") + outcomes = [ + helpers.locate( + e.fields["capture"], r["present"], r["location"], presence_threshold, box_threshold + ) + for e, r in zip(test, (decide(e) for e in test), strict=True) + ] + assert [(o.kind, o.reason) for o in outcomes] == [ + ("located", "confident rectangle"), # t01 + ("located", "confident rectangle"), # t02 + ("review", "the rectangle has no confidence"), # t03: a present target in review + ("absent", "no error banner"), # t04 + ("located", "confident rectangle"), # t05: a box on a screen with no banner + ("review", "presence confidence below the threshold"), # t06 + ("absent", "no error banner"), # t07: a present target said to be absent + ] + report = localization_report([labels[e.id] for e in test], [o.outcome for o in outcomes]) + assert (report.n_total, report.n_present, report.n_absent) == (7, 4, 3) + assert report.present_table == {"located": 2, "absent": 1, "review": 1} + assert report.absent_table == {"located": 1, "absent": 1, "review": 1} + # t01 overlaps by 1.0 and t02 by 12366 / 15360 = 0.805078125 (a 458 x 27 rectangle inside the + # 480 x 32 banner), so the mean over the 2 located targets is 0.9025390625 and over all 4 + # present targets half of that, 0.45126953125. Both located targets are hits. + assert report.mean_iou_located == pytest.approx(0.9025390625) + assert report.mean_iou_present == pytest.approx(0.45126953125) + assert report.hit_rate == 0.5 and report.hits == 2 + assert ( + report.denominators["mean_iou_present"] == 4 + and report.denominators["mean_iou_located"] == 2 + ) + + +def test_the_text_scaffolds_mention_neither_images_nor_assets(catalog, recipes): + """The text scaffolds mention neither images nor assets, in any file (their bytes are also + compared before and after this change in the pull request).""" + for mode in new_recipe.MODES: + folder = recipes / mode + new_recipe.scaffold(1, catalog, folder, mode) + for path in folder.rglob("*"): + if path.is_file(): + text = path.read_text("utf-8").lower() + assert "assets" not in text and "images" not in text, (mode, path.name) + + +def test_the_image_scaffold_emits_no_private_shared_name(image_scaffold): + detector = load_module_at( + Path(__file__).with_name("test_no_private_shared_names.py"), "private_names_for_image_test" + ) + emitted = list(detector.python_sources(image_scaffold)) + assert {p.name for p, _ in emitted} >= { + "helpers.py", + "build_fixtures.py", + "notebook.ipynb", + "test_helpers.py", + "test_build_fixtures.py", + } + for path, source in emitted: + assert detector.private_shared_uses(source) == [], path + + +def test_the_image_scaffold_is_lint_clean_at_several_ranks(catalog, recipes): + """ruff check and ruff format --check pass on the files and the notebook with no hand-edit. + Run from the repository root, as the CI job does: the import order depends on jev_cookbook + being found under src/ as first-party code.""" + for number in (61, 77, 100): + scaffold_images(catalog, recipes, number) + ruff = [sys.executable, "-m", "ruff"] + config = ["--config", str(REPO / "pyproject.toml"), "--no-cache"] + subprocess.run([*ruff, "check", *config, str(recipes)], check=True, cwd=REPO) + subprocess.run([*ruff, "format", "--check", *config, str(recipes)], check=True, cwd=REPO) + + +def test_the_image_scaffold_runs_its_own_tests_offline(image_scaffold, tmp_path): + folder = copy_of(image_scaffold, tmp_path) + result = run_generated_tests(folder) + assert result.returncode == 0, result.stdout + result.stderr + assert "passed" in result.stdout and "failed" not in result.stdout + + +def test_a_fresh_image_scaffold_passes_the_request_audit_with_no_hand_edit(catalog, recipes): + """The image counterpart of ``test_a_fresh_scaffold_passes_the_request_audit_with_no_hand_edit``: + the scaffold is executed as it is written (its helpers and fixtures are already complete), + and the real ``audit_recipe`` agrees with the untouched declaration, including the two image + quantities of docs/backends.md, and with the untouched README. This is the one test here that + executes the notebook, so it needs a kernel.""" + folder = scaffold_images(catalog, recipes, 1) + assert execute_notebook.main([str(folder)]) == 0 + cells = json.loads((folder / "notebook.ipynb").read_text("utf-8"))["cells"] + assert _request_audit.audit_recipe(folder, cells) == len(IMAGE_IDS) + + +def emulated_cells(folder): + """The scaffold's cells with the committed-execution evidence the audit reads: execution + counts in order, and the audit line a backend produces for the ``decide()`` calls the + notebook makes (the demo example, then ``validation``, then ``test``), made here in plain + Python rather than by a kernel.""" + helpers = load_helpers(folder) + questions = helpers.build_questions() + backend = jev_cookbook.AuditedBackend( + get_backend(fixtures=folder / "fixtures" / "responses.json", images=True), label="main" + ) + examples = load_inputs(folder) + for split in ("demo", "validation", "test"): + for example in select_split(examples, split): + state, assets = helpers.build_request(example.fields) + backend.decide(state, questions, assets=assets) + cells = json.loads((folder / "notebook.ipynb").read_text("utf-8"))["cells"] + count = 0 + for cell in cells: + if cell["cell_type"] == "code": + count += 1 + cell["execution_count"] = count + if cell["id"] == "audit": + cell["outputs"] = [ + {"output_type": "stream", "name": "stdout", "text": backend.audit_json() + "\n"} + ] + return cells + + +def test_the_declared_image_budget_agrees_with_the_audit_of_the_calls_the_notebook_makes( + image_scaffold, tmp_path +): + """The same audit as above without a kernel (so it runs where a kernel does not), plus the + mutations that show each part of the declaration is read: a wrong request count, a missing + profile, a wrong profile and a README paragraph that is not the rendered one all fail.""" + folder = copy_of(image_scaffold, tmp_path) + cells = emulated_cells(folder) + assert _request_audit.audit_recipe(folder, cells) == len(IMAGE_IDS) + + budget_path = folder / "request_budget.json" + declared = json.loads(budget_path.read_text("utf-8")) + main = declared["backends"]["main"] + readme_path = folder / "README.md" + readme = readme_path.read_text("utf-8") + + def mutated(change, match): + data = json.loads(json.dumps(declared)) + change(data["backends"]["main"]) + budget_path.write_text(json.dumps(data), encoding="utf-8") + with pytest.raises(AssertionError, match=match): + _request_audit.audit_recipe(folder, cells) + budget_path.write_text(json.dumps(declared), encoding="utf-8") + + mutated(lambda body: body.update(requests=main["requests"] - 1), "request") + mutated(lambda body: body.pop("profile"), "profile") + mutated(lambda body: body["profile"].update(images=main["requests"] + 1), "profile") + mutated(lambda body: body["profile"].update(distinct_images=main["requests"] - 1), "profile") + readme_path.write_text(readme.replace(f"**{main['requests']}**", "**1**"), encoding="utf-8") + with pytest.raises(AssertionError): + _request_audit.audit_recipe(folder, cells) + readme_path.write_text(readme, encoding="utf-8") + assert _request_audit.audit_recipe(folder, cells) == len(IMAGE_IDS) + + +def test_the_image_build_fixtures_tests_are_the_ones_the_recipe_guard_can_compare( + catalog, recipes, image_scaffold +): + """``build_fixtures_test_text(recipe)`` keeps returning the text variant, so the recipe guard's + byte comparison for a text recipe is unchanged; ``mode='replay-images'`` returns the variant + the image scaffold writes, which a recipe with ``assets/`` is compared with.""" + _, recipe = entry(catalog, 77) + text_variant = new_recipe.build_fixtures_test_text(recipe) + image_variant = new_recipe.build_fixtures_test_text(recipe, IMAGE) + assert (image_scaffold / "tests" / "test_build_fixtures.py").read_text("utf-8") == image_variant + assert text_variant != image_variant + assert "copytree" in image_variant and "copytree" not in text_variant + folder = scaffold_replay_with_build_script(catalog, recipes, number=11) + written = (folder / "tests" / "test_build_fixtures.py").read_text("utf-8") + assert written == new_recipe.build_fixtures_test_text(entry(catalog, 11)[1]) + + +def test_the_image_scaffold_fails_cleanly_and_leaves_no_folder_when_the_drawing_step_fails( + catalog, recipes, monkeypatch, capsys +): + """If the new folder's build_fixtures.py cannot run (the package or Pillow is missing, say), + the folder is removed and the command exits 1 with the reason, never a half-written recipe.""" + _, recipe = entry(catalog, 77) + + def failing(*args, **kwargs): + return subprocess.CompletedProcess(args, 1, stdout="", stderr="ImportError: no Pillow\n") + + monkeypatch.setattr(new_recipe.subprocess, "run", failing) + assert run(77, catalog, recipes, mode=IMAGE) == 1 + error = capsys.readouterr().err + assert "ImportError: no Pillow" in error and "nothing was created" in error + assert not (recipes / recipe["slug"]).exists() + + def missing(*args, **kwargs): + raise FileNotFoundError("python") + + monkeypatch.setattr(new_recipe.subprocess, "run", missing) + assert run(77, catalog, recipes, mode=IMAGE) == 1 + assert not (recipes / recipe["slug"]).exists() + + def silent(*args, **kwargs): + return subprocess.CompletedProcess(args, 3, stdout="", stderr="") + + monkeypatch.setattr(new_recipe.subprocess, "run", silent) + assert run(77, catalog, recipes, mode=IMAGE) == 1 + assert "exit status 3" in capsys.readouterr().err + assert not (recipes / recipe["slug"]).exists() + + +def test_the_image_scaffold_never_overwrites_either(catalog, recipes, capsys): + folder = scaffold_images(catalog, recipes, 62) + before = {p: p.read_bytes() for p in folder.rglob("*") if p.is_file()} + assert run(62, catalog, recipes, mode=IMAGE) == 1 + assert "already exists" in capsys.readouterr().err + assert {p: p.read_bytes() for p in folder.rglob("*") if p.is_file()} == before + + +def test_the_generated_test_catches_a_review_that_becomes_an_absence(image_scaffold, tmp_path): + """Mutation check: an unconfident presence answer must be a review, never an absence. Change + the rule so that it answers absent, and the generated tests fail on exactly that.""" + folder = copy_of(image_scaffold, tmp_path) + helpers_path = folder / "helpers.py" + text = helpers_path.read_text("utf-8") + mutated = text.replace( + 'return Localized(capture, REVIEW, None, "presence confidence below the threshold")', + 'return Localized(capture, ABSENT, None, "presence confidence below the threshold")', + ) + assert mutated != text + helpers_path.write_text(mutated, encoding="utf-8", newline="\n") + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_an_unconfident_presence_answer_is_a_review_never_an_absence" in failed.stdout + assert "test_a_review_is_never_an_absence_over_a_grid" in failed.stdout + + +def test_the_generated_test_catches_a_rectangle_gate_that_lets_a_missing_confidence_through( + image_scaffold, tmp_path +): + folder = copy_of(image_scaffold, tmp_path) + helpers_path = folder / "helpers.py" + text = helpers_path.read_text("utf-8") + mutated = text.replace(" if confidence is None:\n", " if confidence is None and False:\n") + assert mutated != text + helpers_path.write_text(mutated, encoding="utf-8", newline="\n") + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_a_rectangle_without_enough_confidence_is_a_review" in failed.stdout + + +def test_the_generated_stored_answers_test_fails_when_no_wrong_answer_is_confident( + image_scaffold, tmp_path +): + """The guard must discriminate, as the text scaffold's does: make the two wrong test answers + (t05 and t07) unconfident and the generated test fails instead of passing vacuously.""" + folder = copy_of(image_scaffold, tmp_path) + script = folder / "build_fixtures.py" + text = script.read_text("utf-8") + weakened = text.replace('"loading", 25, 0.81,', '"loading", 25, 0.51,').replace( + '"error", 54, 0.18,', '"error", 54, 0.49,' + ) + assert weakened.count("0.51") == 1 and weakened.count("0.49") == 1 + script.write_text(weakened, encoding="utf-8", newline="\n") + subprocess.run([sys.executable, str(script)], check=True, cwd=tmp_path, capture_output=True) + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_stored_answers_are_not_all_right" in failed.stdout + + +def test_a_changed_picture_fails_the_generated_replay_key_test(image_scaffold, tmp_path): + """Mutation check: the bytes of a picture are part of its replay key, so a different picture + under the same name is caught by the generated test (and by the validator's hash).""" + folder = copy_of(image_scaffold, tmp_path) + picture = folder / "assets" / "screen-t01.png" + Image.new("RGB", (640, 400), "white").save(picture) + failed = run_generated(folder, "tests/test_helpers.py") + assert failed.returncode != 0 + assert "test_every_replay_key_in_the_fixtures_matches_the_current_question" in failed.stdout + assert any("hash-mismatch" in str(problem) for problem in validate_recipe(folder)) + + +def test_the_image_build_fixtures_tests_pass_under_a_different_png_encoder_and_the_text_ones_do_not( + image_scaffold, tmp_path +): + """Why the image variant copies ``assets/``. Pillow and zlib versions decide the bytes of a + PNG, and the bytes are part of every replay key, so a rerun that draws the pictures afresh can + disagree with the committed fixtures although every pixel is the same. ``sitecustomize`` below + makes this environment another encoder: the same pixels, different bytes, and not canonical. + The image variant, whose copy holds the committed pictures, still passes (the script keeps + the committed bytes); the text variant, which copies only the two scripts, fails on exactly + the comparison of the regenerated fixtures with the committed ones. In the encoder that wrote + the committed files, the text variant passes too, which is why this needs the second encoder.""" + folder = copy_of(image_scaffold, tmp_path) + shim = tmp_path / "shim" + shim.mkdir() + (shim / "sitecustomize.py").write_text( + "import io\n" + "from PIL import Image\n" + "import jev_cookbook.synthetic_images as si\n" + "_original = si.encode_png\n" + "def encode_png(image):\n" + " out = io.BytesIO()\n" + " with Image.open(io.BytesIO(_original(image))) as decoded:\n" + " decoded.save(out, format='PNG', compress_level=1)\n" + " return out.getvalue()\n" + "si.encode_png = encode_png\n" + "si.bytes_are_canonical = lambda: False\n", + encoding="utf-8", + ) + paths = [str(shim)] + [p for p in [os.environ.get("PYTHONPATH")] if p] + environment = {**os.environ, "PYTHONPATH": os.pathsep.join(paths)} + target = folder / "tests" / "test_build_fixtures.py" + + def run_it(): + return subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "-q", + "-p", + "no:cacheprovider", + "--rootdir", + str(folder), + str(target), + "-k", + "reproduces", + ], + cwd=folder, + capture_output=True, + text=True, + timeout=180, + env=environment, + ) + + image_variant = run_it() + assert image_variant.returncode == 0, image_variant.stdout + image_variant.stderr + + _, recipe = entry_for_folder(folder) + target.write_text(new_recipe.build_fixtures_test_text(recipe), encoding="utf-8", newline="\n") + text_variant = run_it() + assert text_variant.returncode != 0 + assert "FAILED" in text_variant.stdout + assert "test_the_generator_reproduces_the_committed_fixtures" in text_variant.stdout diff --git a/tests/test_template_scaffolder_agreement.py b/tests/test_template_scaffolder_agreement.py index f306212..17f3f18 100644 --- a/tests/test_template_scaffolder_agreement.py +++ b/tests/test_template_scaffolder_agreement.py @@ -40,6 +40,14 @@ that direction and ``test_the_setup_comparison_catches_a_changed_offline_definition`` for a reword. +The image scaffold (``--mode replay-images``) is compared the same way, with one difference that +makes the comparison stricter, not looser: its ``setup`` cell defines ``selection`` (the cells +below it print ``validation`` numbers), so it needs no allowance and is compared with the +template's whole ``setup`` cell, ``selection`` included. Its ``backend = ...`` assignment is masked +like the other modes' (``images=True`` is a different call shape), so the arguments of that call +are asserted separately, in ``test_the_image_scaffold_asks_for_the_images_header_and_the_text_ +scaffolds_do_not``. + This is one assertion per cell, not a list of remembered rules, and it is blind to the specific property #215 found: deleting the offline ``N`` print (or any other statement, from either side) changes that side's statement sequence, which the comparison catches without this module's code @@ -161,14 +169,19 @@ def mode_neutral_statements(source: str) -> list[str]: TEMPLATE_ONLY_SETUP_STATEMENTS = ("selection = ' (a selection step, not a reported result)'",) -def _setup_template_stmts(source: str) -> list[str]: +def _setup_template_stmts(source: str, mode: str = "replay") -> list[str]: """The template's ``setup`` cell, normalised, with the one named template-only statement in - ``TEMPLATE_ONLY_SETUP_STATEMENTS`` filtered out -- what the scaffold's ``setup`` cell must - equal.""" - return [s for s in mode_neutral_statements(source) if s not in TEMPLATE_ONLY_SETUP_STATEMENTS] + ``TEMPLATE_ONLY_SETUP_STATEMENTS`` filtered out of the template's side -- what the scaffold's + ``setup`` cell must equal. The image scaffold defines ``selection`` itself, so for + ``replay-images`` nothing is filtered out and its ``setup`` cell must equal the template's + whole one.""" + statements = mode_neutral_statements(source) + if mode == "replay-images": + return statements + return [s for s in statements if s not in TEMPLATE_ONLY_SETUP_STATEMENTS] -@pytest.mark.parametrize("mode", ["replay", "scripted"]) +@pytest.mark.parametrize("mode", ["replay", "scripted", "replay-images"]) def test_the_scaffolded_setup_cell_matches_the_template_modulo_the_named_allowance(mode): """Every mode-neutral statement the scaffold's ``setup`` cell emits -- ``scored``, ``offline``, ``check``, and the ``run_header(...)`` call with its ``n_examples=len(scored)`` @@ -184,7 +197,7 @@ def test_the_scaffolded_setup_cell_matches_the_template_modulo_the_named_allowan generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup") template = _cell(TEMPLATE_NOTEBOOK["cells"], "setup") generated_stmts = mode_neutral_statements(_source(generated)) - template_stmts = _setup_template_stmts(_source(template)) + template_stmts = _setup_template_stmts(_source(template), mode) assert generated_stmts == template_stmts, ( f"scaffolded setup cell ({mode}) does not match the template's after normalisation " f"(modulo {TEMPLATE_ONLY_SETUP_STATEMENTS}):\ngenerated: {generated_stmts}\n" @@ -192,7 +205,7 @@ def test_the_scaffolded_setup_cell_matches_the_template_modulo_the_named_allowan ) -@pytest.mark.parametrize("mode", ["replay", "scripted"]) +@pytest.mark.parametrize("mode", ["replay", "scripted", "replay-images"]) def test_the_scaffolded_measured_cell_matches_the_template(mode): """The scaffold's ``measured`` cell, normalised, must equal the template's ``measured`` cell, normalised -- both branches, not only the offline one: this is the structural check that @@ -211,13 +224,14 @@ def test_the_scaffolded_measured_cell_matches_the_template(mode): ) -def test_the_measured_comparison_is_blind_to_n_and_still_catches_215(): +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) +def test_the_measured_comparison_is_blind_to_n_and_still_catches_215(mode): """Reproduces the historical #215 defect directly against the real generated cell -- delete the offline branch's own ``N:`` print -- and shows the comparison disagrees, without this module or ``mode_neutral_statements`` ever naming "N": the two sides simply no longer have the same number of statements in the offline branch. This is the "blind to any rule about N" standard both #219 reviewers asked for criterion 2 to meet.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "measured") + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "measured") template_stmts = mode_neutral_statements(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "measured"))) source = _source(generated) broken = re.sub( @@ -227,12 +241,13 @@ def test_the_measured_comparison_is_blind_to_n_and_still_catches_215(): assert mode_neutral_statements(broken) != template_stmts -def test_the_measured_comparison_catches_a_dropped_live_branch_print_too(): +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) +def test_the_measured_comparison_catches_a_dropped_live_branch_print_too(mode): """The same mutation on the *live* (``else``) branch's ``N:`` print, which #219 left uncovered by any assertion on the scaffolder side (issue #215 comment 6107125765, "Restore the live-branch coverage"): this comparison fails on it too, by construction, with no branch-specific code anywhere in this module.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "measured") + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "measured") template_stmts = mode_neutral_statements(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "measured"))) source = _source(generated) broken = source.replace( @@ -243,12 +258,15 @@ def test_the_measured_comparison_catches_a_dropped_live_branch_print_too(): assert mode_neutral_statements(broken) != template_stmts -def test_the_setup_comparison_catches_a_changed_offline_definition(): +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) +def test_the_setup_comparison_catches_a_changed_offline_definition(mode): """A ``setup``-cell regression unrelated to ``N`` -- rewording ``offline``'s definition -- is also caught, proving the structural comparison is not special-cased to the measured cell's own property.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "setup") - template_stmts = _setup_template_stmts(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup"))) + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup") + template_stmts = _setup_template_stmts( + _source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup")), mode + ) source = _source(generated) broken = source.replace( 'offline = backend.mode in ("synthetic", "scripted")', @@ -258,6 +276,7 @@ def test_the_setup_comparison_catches_a_changed_offline_definition(): assert mode_neutral_statements(broken) != template_stmts +@pytest.mark.parametrize("mode", ["replay", "replay-images"]) @pytest.mark.parametrize( "removed_line", [ @@ -272,7 +291,7 @@ def test_the_setup_comparison_catches_a_changed_offline_definition(): ), ], ) -def test_the_setup_comparison_catches_a_dropped_statement(removed_line): +def test_the_setup_comparison_catches_a_dropped_statement(removed_line, mode): """Issue #215 was the scaffolder silently *dropping* a statement the template has -- not adding or rewording one. This reproduces that direction directly against the real generated ``setup`` cell: delete one of its own statements and show the comparison disagrees. A @@ -280,9 +299,37 @@ def test_the_setup_comparison_catches_a_dropped_statement(removed_line): would not catch any of these three; equality modulo ``TEMPLATE_ONLY_SETUP_STATEMENTS`` catches all three, because dropping a statement changes the length, and the two normalised lists are no longer the same list.""" - generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay"), "setup") - template_stmts = _setup_template_stmts(_source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup"))) + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup") + template_stmts = _setup_template_stmts( + _source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup")), mode + ) source = _source(generated) broken = source.replace(removed_line, "", 1) assert broken != source, "the mutation did not change anything -- the line was not found" assert mode_neutral_statements(broken) != template_stmts + + +def test_the_image_scaffold_setup_comparison_catches_a_dropped_selection(): + """The image scaffold's ``setup`` cell is compared with the template's whole cell, so it is + the one cell where dropping ``selection`` (which the text scaffolds are allowed to omit) is a + difference: the cells below it print ``validation`` numbers with ``{selection}{check}``.""" + generated = _cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode="replay-images"), "setup") + template_stmts = _setup_template_stmts( + _source(_cell(TEMPLATE_NOTEBOOK["cells"], "setup")), "replay-images" + ) + assert TEMPLATE_ONLY_SETUP_STATEMENTS[0] in template_stmts + source = _source(generated) + dropped = re.sub(r"\nselection = .*", "", source, count=1) + assert dropped != source, "the mutation did not change anything -- the regex did not match" + assert mode_neutral_statements(dropped) != template_stmts + + +def test_the_image_scaffold_asks_for_the_images_header_and_the_text_scaffolds_do_not(): + """``backend = ...`` is masked by the comparison above, so what the image scaffold changes in + that one call is asserted here: ``images=True``, the option that enables the experimental + images request header in a live run (docs/backends.md) and is ignored offline.""" + wanted = 'AuditedBackend(get_backend(fixtures=responses_path(), images=True), label="main")' + for mode in ("replay", "scripted", "replay-images"): + source = _source(_cell(new_recipe.notebook_cells(CATALOG, RECIPE, mode=mode), "setup")) + assert ("images=True" in source) == (mode == "replay-images"), mode + assert (wanted in source) == (mode == "replay-images"), mode diff --git a/tools/new_recipe.py b/tools/new_recipe.py index 2289de0..2a2b3d4 100644 --- a/tools/new_recipe.py +++ b/tools/new_recipe.py @@ -1,7 +1,8 @@ """Create ``recipes/NN-slug/`` for recipe NN from ``catalog/recipes.json``. - python tools/new_recipe.py 7 # a replay recipe (the default) - python tools/new_recipe.py 36 --mode scripted # a scripted or simulator recipe + python tools/new_recipe.py 7 # a replay recipe (the default) + python tools/new_recipe.py 36 --mode scripted # a scripted or simulator recipe + python tools/new_recipe.py 77 --mode replay-images # a replay recipe whose requests carry images The folder gets a skeleton that follows ``recipes/_template/``: the notebook sections in order, a README, ``helpers.py``, ``build_fixtures.py``, ``side_effects.json`` (the recipe's @@ -15,10 +16,19 @@ what is left. The command never overwrites: if the folder exists it stops with a message and exit status 1. +``--mode replay-images`` is the multimodal variant of replay (docs/vision.md). It writes the files +above and then runs the new folder's own ``build_fixtures.py`` once, which draws the example +images with ``jev_cookbook.synthetic_images`` and writes ``assets/``, ``assets-manifest.json`` and +``fixtures/``. This tool itself stays standard library only; that one step needs the installed +package and Pillow, which the scaffolded ``build_fixtures.py`` imports. If it fails, the folder is +removed and the command exits with status 1. + The scaffold is born on the audited request-count route (#213, #368): the notebook's one ``get_backend(...)`` is wrapped in ``AuditedBackend(..., label="main")``, a final standalone cell prints ``backend.audit_json()``, and ``request_budget.json`` declares the one ``decide()`` call the -skeleton's own ``answer`` cell makes -- matching, out of the box, with no hand-edit. The README's +skeleton's own ``answer`` cell makes (``replay-images``: the ``decide()`` calls of its example +notebook, with the two image quantities of docs/backends.md) -- matching, out of the box, with no +hand-edit. The README's ``## Switch to live`` paragraph is rendered from that declaration, never hand-typed; as the builder adds more ``decide()`` calls while writing Evaluation, ``request_budget.json`` and the rendered paragraph are updated together (the README TODO there says how). @@ -31,6 +41,8 @@ import argparse import json import re +import shutil +import subprocess import sys from pathlib import Path @@ -65,6 +77,39 @@ "names." ) MODES = ("replay", "scripted") # --mode; replay is the default and the one most recipes use +# --mode replay-images (docs/vision.md): replay, with images. It is a value of --mode rather than a +# separate flag so that no combination of options can ask for an image recipe that is not a replay +# recipe. MODES stays the two text modes: code that walks the text scaffolds (the tests, the recipe +# guard) keeps walking exactly those. +IMAGE_MODE = "replay-images" +ALL_MODES = (*MODES, IMAGE_MODE) + +# The example the image scaffold ships, so that it runs end to end the moment it is written: an +# invented sign-in window (jev_cookbook.synthetic_images.login_screen) and two questions about it, +# "is an error banner shown?" (a Noul) and "where is it?" (a Box). TODO markers in the files that +# carry it say what to replace. One row per example: +# (id, split, capture, screen state, drawing seed, probability of yes, the object Jev's +# rectangle is drawn around, that rectangle's pixel offsets from the object (left, top, right, +# bottom), the box confidence or None) +# The stored answers are deliberately imperfect: v05's rectangle overlaps the gold box by less than +# half (IoU 0.374), v06 and t05 answer yes about screens with no banner, t07 answers no about one +# with a banner, and t03's rectangle reports no confidence. +IMAGE_ROWS = ( + ("v01", "validation", "C-1001", "error", 11, 0.93, "error_banner", (0, 0, 0, 0), 0.90), + ("v02", "validation", "C-1002", "error", 12, 0.88, "error_banner", (4, -2, -6, 2), 0.86), + ("v03", "validation", "C-1003", "normal", 13, 0.06, "submit", (0, 0, 0, 0), 0.40), + ("v04", "validation", "C-1004", "loading", 17, 0.12, "spinner", (0, 0, 0, 0), 0.35), + ("v05", "validation", "C-1005", "error", 15, 0.64, "error_banner", (30, 10, -40, -8), 0.55), + ("v06", "validation", "C-1006", "crash", 16, 0.58, "dialog", (0, 0, 0, 0), 0.30), + ("t01", "test", "C-2001", "error", 21, 0.91, "error_banner", (0, 0, 0, 0), 0.92), + ("t02", "test", "C-2002", "error", 32, 0.84, "error_banner", (10, 3, -12, -2), 0.90), + ("t03", "test", "C-2003", "error", 34, 0.72, "error_banner", (4, -2, -6, 2), None), + ("t04", "test", "C-2004", "normal", 35, 0.09, "submit", (0, 0, 0, 0), 0.45), + ("t05", "test", "C-2005", "loading", 25, 0.81, "spinner", (0, 0, 0, 0), 0.93), + ("t06", "test", "C-2006", "crash", 48, 0.45, "dialog", (0, 0, 0, 0), 0.38), + ("t07", "test", "C-2007", "error", 54, 0.18, "error_banner", (0, 0, 0, 0), 0.50), + ("d01", "demo", "C-3001", "error", 60, 0.90, "error_banner", (0, 0, 0, 0), 0.91), +) # The audited request-count route (#213, #368): a recipe declares its live request count once, in # request_budget.json, and the README's budget paragraph is rendered from it by @@ -88,10 +133,24 @@ def request_budget_text(mode: str = "replay") -> str: exactly the one ``decide()`` call the skeleton notebook's ``answer`` cell makes (see ``notebook_cells``) -- so a fresh scaffold's declaration, audited notebook and rendered README paragraph already agree, with no hand-edit. ``fixtures`` is ``null`` for a scripted recipe - (nothing to replay) and the recipe-relative replay file otherwise. As the builder adds + (nothing to replay) and the recipe-relative replay file otherwise. ``replay-images`` is the + exception to "one call": its example notebook decides each of ``IMAGE_ROWS`` once, so it + declares that many requests and the two image quantities that go with them + (docs/backends.md, "Actual calls and distinct requests"). As the builder adds ``decide()`` calls while writing Evaluation, this file's declared count (and ``distinct`` or ``at_most`` where they apply) must grow to match, and the README paragraph is regenerated from it -- the ``## Switch to live`` TODO in ``readme_text`` says how.""" + if mode == IMAGE_MODE: + # The example notebook decides every example once, and each request carries one image + # copy of one distinct image (the reference sits once in the state and in no + # question), so the two image quantities of docs/backends.md equal the request count. + count = len(IMAGE_ROWS) + body = { + "requests": count, + "fixtures": "fixtures/responses.json", + "profile": {"images": count, "distinct_images": count}, + } + return json.dumps({"schema": 1, "backends": {"main": body}}, indent=2) + "\n" fixtures = None if mode == "scripted" else "fixtures/responses.json" body = {"requests": 1, "fixtures": fixtures} return json.dumps({"schema": 1, "backends": {"main": body}}, indent=2) + "\n" @@ -159,8 +218,11 @@ def _source_lines(catalog: dict, recipe: dict) -> str: def notebook_cells(catalog: dict, recipe: dict, mode: str = "replay") -> list[dict]: """The skeleton notebook's cells, with the catalog fields filled in. ``mode`` is ``replay`` - (the backend replays ``fixtures/responses.json``) or ``scripted`` (a seeded script in - ``helpers.py`` answers, and there are no stored responses).""" + (the backend replays ``fixtures/responses.json``), ``scripted`` (a seeded script in + ``helpers.py`` answers, and there are no stored responses) or ``replay-images`` (replay, with + images; see ``image_notebook_cells``).""" + if mode == IMAGE_MODE: + return image_notebook_cells(catalog, recipe) scripted = mode == "scripted" number = recipe["rank"] title = recipe["title"] @@ -363,6 +425,236 @@ def code(cell_id: str, text: str) -> dict: ] +def image_notebook_cells(catalog: dict, recipe: dict) -> list[dict]: + """The ``--mode replay-images`` notebook: the replay notebook's cells (so the intro, the + run-mode paragraph, the ``questions``, ``measured`` and audit cells are the ones a replay + recipe has), with the cells that touch images or a Box answer replaced, and one added. + + * ``setup`` turns the images header on (``get_backend(..., images=True)``), imports what the + cells below use and defines ``selection``, because the cells below print ``validation`` + numbers; + * ``state`` builds the request (state and assets) for the demo example and shows its image; + * ``answer`` decides it with ``assets=`` and ``answer-boxes`` (added) draws the Box answer on + the image with ``draw_boxes``; + * ``python-rule`` freezes the presence and box-confidence thresholds on ``validation``; + * ``evaluation`` reports the ``test`` split through ``localization_report``, and + ``evaluation-review`` lists what Python did not locate and why. + + The example is marked ``TODO`` wherever it is the builder's to replace.""" + cells = notebook_cells(catalog, recipe, "replay") + number, title = recipe["rank"], recipe["title"] + + def md(cell_id: str, text: str) -> dict: + return _cell("markdown", cell_id, text) + + def code(cell_id: str, text: str) -> dict: + return _cell("code", cell_id, text) + + replacements = { + "setup-md": md( + "setup-md", + f"## {SECTIONS[1]}\n\n" + "The notebook runs from its own folder with `jev_cookbook` and the packages it " + "installs; Pillow reads and draws the images. The header states which mode ran. " + + RUN_MODE_PARAGRAPH + + f" {TODO_MARK}: keep this paragraph as it is; the setup cell defines `selection` " + "and `check`, so print `{selection}{check}` on every `validation` number.\n\n" + "The images are in `assets/`, drawn by `build_fixtures.py`; `assets-manifest.json` " + "records where each came from ([docs/vision.md](../../docs/vision.md), " + "[docs/fixtures.md](../../docs/fixtures.md#image-provenance-manifest)). " + "`get_backend(..., images=True)` is the option that enables the experimental images " + "request header in a live run; offline backends ignore it " + "([docs/backends.md](../../docs/backends.md#get_backend)).\n\n" + f"{TODO_MARK}: add the imports your later sections use, one sentence on the fixture " + "set and its size, and what the images show.", + ), + "setup": code( + "setup", + f""" +from IPython.display import Image, display + +from jev_cookbook import AuditedBackend, get_backend, load_helpers +from jev_cookbook.evaluation import noul_confidence, select_confidence_threshold +from jev_cookbook.fixtures import load_inputs, load_labels, responses_path, select_split +from jev_cookbook.style import apply_style, run_header, show_answer +from jev_cookbook.vision import box_confidence, box_iou, draw_boxes, gold_box, localization_report + +apply_style() +helpers = load_helpers() # this recipe's helpers.py, loaded by path +examples = load_inputs() +labels = load_labels() +backend = AuditedBackend(get_backend(fixtures=responses_path(), images=True), label="main") +# scored excludes the demo example: it carries no gold label, so no metric below is about it. +scored = [e for e in examples if e.split not in ("train", "demo")] + +offline = backend.mode in ("synthetic", "scripted") +check = " (a pipeline check, not a Jev result)" if offline else "" +# A number from the validation split is a selection step, not a result, in every run mode. +selection = " (a selection step, not a reported result)" + +run_header( + {number}, + {json.dumps(title)}, + backend=backend, + n_examples=len(scored), +) +""", + ), + "state-md": md( + "state-md", + f"## {SECTIONS[2]}\n\n" + f"{TODO_MARK}: say how Python builds the request from one example and what Jev sees: " + "the fixture names the image by a path in `fields`, Python loads it with " + "`image_asset(path, root=...)` and gives Jev a reference to its bytes, once, in the " + "state. Name what Python keeps for itself (the capture identifier, the path, the gold " + "label).", + ), + "state": code( + "state", + """ +import json + +demo = [e for e in examples if e.split == "demo"] +if not demo: + raise ValueError("add at least one demo example to ROWS in build_fixtures.py") +example = demo[0] +print("Python has:") +print(json.dumps(example.fields, indent=2)) +state, assets = helpers.build_request(example.fields) +print("Jev sees:") +print(json.dumps(state, indent=2)) +display(Image(filename=helpers.ROOT / example.fields["image"])) +""", + ), + "answer-md": md( + "answer-md", + f"## {SECTIONS[4]}\n\n" + f"{TODO_MARK}: show one image answer before any aggregate and say what its fields " + "mean: the probability of yes for presence, and the rectangle's four edges " + "(normalized to the image) with its confidence, which can be missing. A Box returns a " + "rectangle even when the target is absent, so a returned rectangle cannot establish " + "that the target exists.", + ), + "answer": code( + "answer", + """ +result = backend.decide(state, questions, assets=assets) +for answer in result.answers.values(): + show_answer(answer) +""", + ), + "python-md": md( + "python-md", + f"## {SECTIONS[5]}\n\n" + f"{TODO_MARK}: describe the rule in `helpers.py` that holds whatever the model " + "answers (`locate`): a rectangle counts only after the presence question says the " + "target is there, an uncertain presence answer or a rectangle without enough " + "confidence goes to an explicit review outcome, and a review is never an absence. " + "Say how each threshold is chosen on `validation` and frozen. Then apply it here.", + ), + "python-rule": code( + "python-rule", + """ +val_examples = select_split(examples, "validation") +val_results = [ + backend.decide(state, questions, assets=assets) + for state, assets in (helpers.build_request(e.fields) for e in val_examples) +] +val_gold = [labels[e.id] for e in val_examples] +# A yes is a probability above one half; only a call at or above the frozen threshold is acted on, +# so the threshold is the lowest confidence at which every call is right. +presence_right = [ + (r["present"].noul > 0.5) == (gold_box(g) is not None) + for r, g in zip(val_results, val_gold, strict=True) +] +presence_threshold = select_confidence_threshold( + presence_right, noul_confidence([r["present"].noul for r in val_results]), target_accuracy=1.0 +) +# The same rule for the rectangle: among present targets whose rectangle has a confidence, a +# rectangle is right when its IoU with the gold box is at least one half. +sized = [ + (box_iou(gold_box(g), r["location"]) >= 0.5, box_confidence(r["location"])) + for r, g in zip(val_results, val_gold, strict=True) + if gold_box(g) is not None and box_confidence(r["location"]) is not None +] +box_threshold = select_confidence_threshold( + [ok for ok, _ in sized], [c for _, c in sized], target_accuracy=1.0 +) +print(f"presence threshold, frozen from here on: {presence_threshold:.2f}{selection}{check}") +print(f"box threshold, frozen from here on: {box_threshold:.2f}{selection}{check}") + +shown = helpers.locate( + example.fields["capture"], + result["present"], + result["location"], + presence_threshold, + box_threshold, +) +print(f"{shown.capture}: {shown.kind} ({shown.reason})") +""", + ), + "evaluation-md": md( + "evaluation-md", + f"## {SECTIONS[6]}\n\n" + f"{TODO_MARK}: choose every setting on `validation` (above: the presence and box " + "thresholds), freeze it, and report once on `test` through `localization_report`, " + "which counts every example once and names each denominator. In a synthetic or " + "scripted run print each metric with `check` beside it; an individual example listed " + "below a metric is not a metric and needs no label. Say which `validation` numbers " + "your selection forces, or that none is. Compare with the baseline the issue names, " + "if there is one.", + ), + "evaluation": code( + "evaluation", + """ +test_examples = select_split(examples, "test") +test_results = [ + backend.decide(state, questions, assets=assets) + for state, assets in (helpers.build_request(e.fields) for e in test_examples) +] +test_gold = [labels[e.id] for e in test_examples] +test_outcomes = [ + helpers.locate( + e.fields["capture"], r["present"], r["location"], presence_threshold, box_threshold + ) + for e, r in zip(test_examples, test_results, strict=True) +] +report = localization_report(test_gold, [o.outcome for o in test_outcomes]) +print(f"test, {len(test_examples)} examples, thresholds frozen{check}") +for line in str(report).splitlines(): + print(f"{line}{check}") +""", + ), + } + out: list[dict] = [] + for cell in cells: + out.append(replacements.get(cell["id"], cell)) + if cell["id"] == "answer": + out.append( + code( + "answer-boxes", + """ +picture = helpers.open_image(example.fields["image"]) +marked = draw_boxes(picture, {"Jev": result["location"]}) +display(Image(data=helpers.png_bytes(marked), format="png")) +""", + ) + ) + if cell["id"] == "evaluation": + out.append( + code( + "evaluation-review", + """ +print("Test examples Python did not locate, and why:") +for item, decided in zip(test_examples, test_outcomes, strict=True): + if decided.kind != "located": + print(f" {item.id}: {decided.kind} ({decided.reason})") +""", + ) + ) + return out + + def notebook_json(catalog: dict, recipe: dict, mode: str = "replay") -> str: notebook = { "cells": notebook_cells(catalog, recipe, mode), @@ -402,6 +694,29 @@ def readme_text(catalog: dict, recipe: dict, mode: str = "replay") -> str: else "the offline run replays synthetic answers written for this recipe, so its numbers " "check that the pipeline works." ) + if mode == IMAGE_MODE: + run_note = ( + "The notebook runs from this folder with no network and no API key, replaying " + "`fixtures/` and reading the images in `assets/`. `build_fixtures.py` draws those " + "images and writes `assets-manifest.json`, which records where each came from, its " + "licence and what it is attested not to be " + "([docs/fixtures.md](../../docs/fixtures.md#image-provenance-manifest)); " + "[docs/vision.md](../../docs/vision.md) covers loading images and Box answers.\n\n" + f"{TODO_MARK}: before you commit, recreate the images under the notebook constraints " + "pin (the install in the paragraph below) with `python build_fixtures.py`: the PNG " + "bytes are part of the replay key of each request that carries a picture, and the " + "committed bytes are meant to be the ones " + "that pin writes " + "([docs/synthetic-images.md](../../docs/synthetic-images.md#byte-stability-and-regeneration))." + ) + mode_line = ( + "synthetic (offline replay of hand-written answers about drawn images). " + "Not measured live." + ) + closing = ( + "the offline run replays synthetic answers written for this recipe, about images " + "drawn for it, so its numbers check that the pipeline works." + ) types = " + ".join(f"`{t}`" for t in recipe["decision_types"]) return f"""# {recipe["title"]} @@ -490,6 +805,8 @@ def readme_text(catalog: dict, recipe: dict, mode: str = "replay") -> str: def helpers_text(recipe: dict, mode: str = "replay") -> str: + if mode == IMAGE_MODE: + return image_helpers_text(recipe) script_block = "" if mode == "scripted": script_block = f''' @@ -544,6 +861,8 @@ def rows_hint(level: int) -> str: def build_fixtures_text(recipe: dict, mode: str = "replay") -> str: + if mode == IMAGE_MODE: + return image_build_fixtures_text(recipe) if mode == "scripted": return build_scripted_fixtures_text(recipe) return f'''"""Write fixtures/inputs.jsonl, labels.jsonl and responses.json for recipe {recipe["rank"]:02d}. @@ -744,6 +1063,8 @@ def tests_text(recipe: dict, mode: str = "replay") -> str: # The scaffolder knows the mode when it writes this file, so each mode gets only the # replay-key test that applies to it: a replay recipe never defines helpers.script or # helpers.SEED, so a scripted branch in its test file would be dead code, copied sixty times. + if mode == IMAGE_MODE: + return _fill(IMAGE_TESTS, recipe) if mode == "scripted": return f'''"""Tests for recipe {recipe["rank"]:02d}'s helpers. They load the helpers by file path.""" @@ -845,13 +1166,21 @@ def predicted(answer): ''' -def build_fixtures_test_text(recipe: dict) -> str: - """Guard tests for ``build_fixtures.py`` (replay mode only; a scripted recipe has no +def build_fixtures_test_text(recipe: dict, mode: str = "replay") -> str: + """Guard tests for ``build_fixtures.py`` (replay modes only; a scripted recipe has no ``responses.json`` and nothing for these tests to guard, see ``build_scripted_fixtures_text``): refusal without ``--force``, ``inputs.jsonl``/``labels.jsonl`` regenerated from ``ROWS`` even on a refused run, and the committed ``responses.json`` byte-identical to what ``jev_cookbook.live``'s recorder would write. See - ``recipes/_template/tests/test_build_fixtures.py``, which this follows.""" + ``recipes/_template/tests/test_build_fixtures.py``, which this follows. + + ``mode="replay-images"`` returns the variant for a recipe with ``assets/``: the temporary + copy it rebuilds in also holds the committed images (their bytes are part of the replay + key of each request that carries them), and it adds the checks that the images and ``assets-manifest.json`` are reproduced and + that a changed picture is refused. A recipe that has ``assets/`` commits this variant; called + with one argument, the function returns the text variant.""" + if mode == IMAGE_MODE: + return _fill(IMAGE_BUILD_FIXTURES_TEST, recipe) slug = recipe["slug"] module_name = f"recipe{recipe['rank']:02d}_build_fixtures_for_test" return f'''"""build_fixtures.py follows the recipes/_template/build_fixtures.py pattern: inputs and @@ -960,6 +1289,705 @@ def test_the_generator_reproduces_the_committed_fixtures(tmp_path): ''' +# ------------------------------------------------------------------------ the image variant +# +# The three files below (and the image tests after them) are written from plain templates with +# @TOKEN@ placeholders rather than f-strings: they are full of braces and backslashes, and a +# generated file must read exactly as written here. + +IMAGE_HELPERS = r'''"""Python's half of recipe @RANK@: the request Jev sees, the questions and the rules. + +The fixture generator, the notebook and the tests all load this file with +``jev_cookbook.load_helpers``, so the images, the questions and the rules come from one place. + +@TODO@: the example below asks whether an invented sign-in window shows an error banner, and where. +Replace the questions, the request and the rule with your recipe's, and keep the pattern. A path +read from a fixture is data, so an image is loaded with ``image_asset(path, root=ROOT)``; its +reference comes from its bytes; the reference goes into the state once, because every place it +appears adds another copy of the image to the request (docs/vision.md); and a rectangle counts +only after a presence question says the target is there. +""" + +import io +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from PIL import Image + +from jev_cookbook import Box, Noul +from jev_cookbook.evaluation import noul_confidence +from jev_cookbook.vision import ( + box_confidence, + image_asset, + image_reference, + normalized_box, + resolve_asset_path, +) + +# No ``from __future__ import annotations`` here: load_helpers removes this module from +# ``sys.modules``, so typing.get_type_hints cannot resolve postponed annotations on a dataclass. + +# The recipe folder. The image paths in fixtures/inputs.jsonl are relative to it. +ROOT = Path(__file__).resolve().parent + +LOCATED, ABSENT, REVIEW = "located", "absent", "review" # what localization_report counts + + +def build_request(fields: dict[str, Any]) -> tuple[dict[str, str], dict[str, dict[str, str]]]: + """``(state, assets)`` for one example: what Jev sees and the images that go with it. + + ``fields`` names the image by a path relative to the recipe and also carries the capture + identifier. Python keeps the identifier and attaches it to the outcome, so it never needs to + pass through the model. The state holds only the image's reference. + """ + asset = image_asset(fields["image"], root=ROOT) + reference = image_reference(asset) + return {"screenshot": reference}, {reference: asset} + + +def build_questions() -> dict[str, Any]: + """The questions asked about every example: is the target there, and where is it?""" + return { + "present": Noul(instructions="The sign-in window in `screenshot` shows an error banner."), + "location": Box(instructions="The error banner in the sign-in window in `screenshot`."), + } + + +def open_image(path: str) -> Image.Image: + """The image at ``path`` (relative to the recipe) as an RGB Pillow image to draw on.""" + with Image.open(resolve_asset_path(path, root=ROOT)) as picture: + return picture.convert("RGB") + + +def png_bytes(image: Image.Image) -> bytes: + """``image`` as PNG bytes, for ``IPython.display.Image(data=..., format="png")``.""" + buffer = io.BytesIO() + image.save(buffer, format="PNG") + return buffer.getvalue() + + +@dataclass(frozen=True) +class Localized: + """What Python decided: a rectangle, an absence, or a review, with the reason.""" + + capture: str + kind: str + box: dict[str, float] | None + reason: str + + @property + def outcome(self) -> tuple[str, dict[str, float] | None]: + """The pair ``jev_cookbook.vision.localization_report`` takes for this example.""" + return (self.kind, self.box) + + +def locate( + capture: str, + presence: Any, + location: Any, + presence_threshold: float, + box_threshold: float, +) -> Localized: + """Accept a rectangle only when the presence answer is confident and says the target is there. + + ``presence`` is the Noul answer and ``location`` the Box answer. A presence answer whose + confidence is below ``presence_threshold`` goes to review, not to absent. Otherwise a + probability of yes above one half means the target is there, and anything else means it is + not (the Box's rectangle is then ignored). A present target is located only when its + rectangle carries a confidence of at least ``box_threshold``; a rectangle with no confidence + or a lower one goes to review. The rule is code, so it holds whatever the model answers. + """ + for name, value in ( + ("presence_threshold", presence_threshold), + ("box_threshold", box_threshold), + ): + if not 0.0 <= value <= 1.0: + raise ValueError(f"{name} must be between 0 and 1, got {value!r}") + if noul_confidence([presence.noul])[0] < presence_threshold: + return Localized(capture, REVIEW, None, "presence confidence below the threshold") + if presence.noul <= 0.5: + return Localized(capture, ABSENT, None, "no error banner") + confidence = box_confidence(location) + if confidence is None: + return Localized(capture, REVIEW, None, "the rectangle has no confidence") + if confidence < box_threshold: + return Localized(capture, REVIEW, None, "rectangle confidence below the threshold") + return Localized(capture, LOCATED, normalized_box(location), "confident rectangle") + + +# @TODO@: the other rules Python enforces whatever the model answers, with tests. +''' + +IMAGE_BUILD_FIXTURES = r'''"""Write assets/, assets-manifest.json and fixtures/ for recipe @RANK@. + + python build_fixtures.py # from anywhere; it writes next to this file + python build_fixtures.py --force # also overwrite a responses.json holding a recorded answer + python build_fixtures.py --overwrite-assets # accept a picture that changed (replay keys change too) + +The images are drawn here with jev_cookbook.synthetic_images (docs/synthetic-images.md): no +photograph, brand, person or third-party picture, and the gold Box label of each is derived from +what was drawn, not typed in. The responses are synthetic and deliberately imperfect. The replay +keys come from the same ``build_request`` and ``build_questions`` the notebook uses, and cover the +images' bytes. See docs/fixtures.md and docs/vision.md. + +@TODO@: the sign-in windows, the two questions and the answers below are an example. Replace them +with your recipe's images (draw them with ``jev_cookbook.synthetic_images`` primitives, or give +``canvas.provenance("build_fixtures.py:your_function", {...})`` to a scene of your own so the +manifest names a generator that exists here), and keep ROWS' split names and demo example. + +Images are saved under the byte policy of docs/synthetic-images.md. Their bytes are part of the +replay key of each request that carries them, so if a regenerated picture differs from the file that is already there this script +stops and writes nothing more, unless ``--overwrite-assets`` accepts the change. + +Generating inputs and labels is kept separate from generating responses, on purpose: once +responses.json holds even one recorded answer (provenance "recorded", captured from a real Jev +call), running this script again must not silently replace it with a synthetic probability. +inputs.jsonl and labels.jsonl are always rewritten from ROWS, because neither ever holds a +model's answer; responses.json is rewritten only when it does not yet exist, holds only +synthetic answers, or --force is given. +""" + +import argparse +import json +from pathlib import Path + +from jev_cookbook import ( + BoxAnswer, + DecisionResult, + NoulAnswer, + Provenance, + dump_responses, + load_helpers, + replay_key, +) +from jev_cookbook import synthetic_images as si + +HERE = Path(__file__).resolve().parent +helpers = load_helpers(HERE) +QUESTIONS = helpers.build_questions() + +TARGET = "error_banner" # the drawn object whose rectangle is the gold box, when the screen has it + +# (id, split, capture, screen state, drawing seed, probability of yes, the drawn object Jev's +# rectangle is around, that rectangle's pixel offsets from the object (left, top, right, bottom), +# the rectangle's confidence or None for a Box answer that reports none) +ROWS = [ +@ROWS@ +] # @TODO@: your examples, across validation and test, and at least one demo + + +def image_path(ident): + """Where the image of example ``ident`` is saved, relative to the recipe.""" + return f"assets/screen-{ident}.png" + + +def draw(row): + """The canvas of one row. ``login_screen`` is a shared scene, so the manifest entry names it + as the generator and ``si.check_reproducible`` can redraw it from the entry.""" + _ident, _split, _capture, state, seed, *_answer = row + return si.login_screen(state=state, seed=seed) + + +def write_images(rows, overwrite): + """Draw and save every image under the byte policy; return the manifest entries by path. + + Raises ``si.RegenerationMismatch`` (and saves nothing more) for a picture that differs from + the committed file, unless ``overwrite`` is true.""" + entries = [] + for row in rows: + path = image_path(row[0]) + canvas = draw(row) + status = canvas.write(HERE, path, overwrite=overwrite) + print(f"{status}: {path}") + description = f"An invented sign-in window in the {row[3]!r} state (seed {row[4]})." + entries.append(canvas.manifest_entry(HERE, path, description)) + return sorted(entries, key=lambda entry: entry["path"]) + + +def _fields(row): + return {"capture": row[2], "image": image_path(row[0])} + + +def build_inputs_and_labels(rows): + """``(inputs, labels)`` from ``rows``, with the images already saved. Neither ever holds a + model's answer, so both are always safe to regenerate, even after ``responses.json`` has + been recorded. A gold Box label is derived from what was drawn.""" + inputs, labels = [], [] + for row in rows: + ident, split = row[0], row[1] + fields = _fields(row) + state, assets = helpers.build_request(fields) + key = replay_key(state, QUESTIONS, assets=assets) + inputs.append({"id": ident, "split": split, "fields": fields, "replay_keys": [key]}) + if split != "demo": + drawn = draw(row).objects.get(TARGET) + labels.append({"id": ident, "label": drawn.label() if drawn else si.absent_label()}) + return inputs, labels + + +def answers_for(row, provenance: Provenance): + """{question name: answer} for one row: the presence probability and a rectangle around the + object ``row`` names, moved by its offsets (the rectangle a model returns even when the + target is absent).""" + _ident, _split, _capture, _state, _seed, p_yes, shown, offsets, confidence = row + canvas = draw(row) + width, height = canvas.size + left, top, right, bottom = ( + edge + offset for edge, offset in zip(canvas.objects[shown].pixels, offsets, strict=True) + ) + box = { + "top": top / height, + "bottom": bottom / height, + "left": left / width, + "right": right / width, + } + return { + "present": NoulAnswer(p_yes, provenance), + "location": BoxAnswer(box, confidence, provenance), + } + + +def build_responses(rows): + """``{replay_key: stored response}`` from ``rows``, with the images already saved. Always + synthetic: this script never calls Jev, so it can never produce a recorded response.""" + responses = {} + for row in rows: + state, assets = helpers.build_request(_fields(row)) + key = replay_key(state, QUESTIONS, assets=assets) + answers = answers_for(row, Provenance.synthetic()) + responses[key] = DecisionResult(answers, "synthetic").to_dict() + return responses + + +def _is_recorded(path: Path) -> bool: + """True if ``path`` exists and holds at least one response whose model is not + ``"synthetic"`` (a recorded, or otherwise real, answer). A file that fails to parse, or + whose top level is not a JSON object, cannot hold a valid synthetic response either, so it + is treated as not recorded rather than raising; inside an object, an entry that is itself + not an object is treated as if it were recorded, so it blocks an overwrite instead of being + silently skipped.""" + if not path.exists(): + return False + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return False + if not isinstance(data, dict): + return False + return any( + not isinstance(entry, dict) or entry.get("model") != "synthetic" for entry in data.values() + ) + + +def _unresolved_keys(inputs, responses_file: Path) -> list[str]: + """Replay keys the just-rewritten ``inputs`` ask for that ``responses_file`` does not have, + used only to warn when a ROWS edit has desynchronised the two.""" + if not responses_file.exists(): + return [] + try: + data = json.loads(responses_file.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return [] + if not isinstance(data, dict): + return [] + return [key for row in inputs for key in row["replay_keys"] if key not in data] + + +def main() -> None: + parser = argparse.ArgumentParser(description="Write this recipe's assets/ and fixtures/.") + parser.add_argument( + "--force", + action="store_true", + help="overwrite responses.json even if it holds a recorded (non-synthetic) answer", + ) + parser.add_argument( + "--overwrite-assets", + action="store_true", + help="accept an image that differs from the committed file (its replay keys change)", + ) + args = parser.parse_args() + if not ROWS: + raise SystemExit("@TODO@: add examples to ROWS in build_fixtures.py") + try: + entries = write_images(ROWS, args.overwrite_assets) + except si.RegenerationMismatch as error: + raise SystemExit(f"refused: {error}\nnothing else was written") from None + manifest = {"schema": 1, "assets": entries} + (HERE / "assets-manifest.json").write_text( + json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="\n" + ) + folder = HERE / "fixtures" + folder.mkdir(exist_ok=True) + inputs, labels = build_inputs_and_labels(ROWS) + for name, rows in (("inputs.jsonl", inputs), ("labels.jsonl", labels)): + text = "".join(json.dumps(row) + "\n" for row in rows) + (folder / name).write_text(text, encoding="utf-8", newline="\n") + responses_file = folder / "responses.json" + if _is_recorded(responses_file) and not args.force: + message = ( + f"refusing to overwrite {responses_file}: it holds a recorded response " + "(pass --force to overwrite it anyway)" + ) + if _unresolved_keys(inputs, responses_file): + message += ( + "\ninputs.jsonl and labels.jsonl above were rewritten from ROWS; " + "responses.json was not, and at least one of the keys the rewritten inputs " + "ask for is missing from it. The three files are desynchronised until you " + "--force a rewrite or record the missing answers." + ) + raise SystemExit(message) + responses = build_responses(ROWS) + # dump_responses is the serialization the recorder writes with (top-level keys sorted, each + # response keeping the field order DecisionResult.to_dict() emits), so a recording's diff + # stays limited to the values that actually changed. + responses_file.write_text(dump_responses(responses), encoding="utf-8", newline="\n") + print(f"wrote {len(inputs)} examples, {len(labels)} labels, {len(responses)} responses") + + +if __name__ == "__main__": + main() +''' + + +IMAGE_TESTS = r'''"""Tests for recipe @RANK@'s helpers. They load the helpers by file path.""" + +import json +from pathlib import Path + +import pytest + +from jev_cookbook import BoxAnswer, NoulAnswer, Provenance, get_backend, load_helpers, replay_key +from jev_cookbook import synthetic_images as si +from jev_cookbook.evaluation import noul_confidence, select_confidence_threshold +from jev_cookbook.fixtures import load_inputs, load_labels, responses_path +from jev_cookbook.vision import gold_box + +RECIPE = Path(__file__).resolve().parent.parent +helpers = load_helpers(RECIPE) + +BOX = {"top": 0.25, "bottom": 0.5, "left": 0.125, "right": 0.5} + + +def noul(probability): + return NoulAnswer(probability, Provenance.synthetic()) + + +def box(confidence): + return BoxAnswer(BOX, confidence, Provenance.synthetic()) + + +def test_questions_are_built_by_python(): + # @TODO@: assert your questions' instructions and criteria against literal expected values + # (not helpers' own data, which would make the assertion self-referential), then test every + # rule in helpers.py + questions = helpers.build_questions() + assert list(questions) == ["present", "location"] + assert [question.to_dict()["type"] for question in questions.values()] == ["noul", "box"] + + +def test_every_replay_key_in_the_fixtures_matches_the_current_question(): + # The fixture validator cannot see question drift; this test can. One request per example; + # adapt this when an example needs a dependent second request (CONTRIBUTING: a question that + # depends on an earlier answer goes in a later request), where the example lists a key for + # each request in order. The key covers the image's bytes, so a changed picture fails here. + questions = helpers.build_questions() + for example in load_inputs(RECIPE): + state, assets = helpers.build_request(example.fields) + assert example.replay_keys == (replay_key(state, questions, assets=assets),) + + +def test_every_scene_can_be_redrawn_from_its_manifest_entry(): + # A shared scene is rebuilt from the generator and parameters its entry records and compared + # with the committed file; a recipe's own generator is covered by test_build_fixtures.py. + manifest = json.loads((RECIPE / "assets-manifest.json").read_text("utf-8")) + shared = [ + entry + for entry in manifest["assets"] + if entry["generator"].startswith("jev_cookbook.synthetic_images:") + ] + if not shared: + pytest.skip("no entry names a shared scene; test_build_fixtures.py covers the rest") + for entry in shared: + assert si.check_reproducible(RECIPE, entry) in ("unchanged", "kept-committed-bytes") + + +def test_a_confident_yes_with_a_confident_rectangle_is_located(): + found = helpers.locate("C-1", noul(0.9), box(0.9), 0.3, 0.8) + assert (found.kind, found.reason) == ("located", "confident rectangle") + assert found.box == BOX and found.outcome == ("located", BOX) + + +def test_a_confident_no_is_absent_and_ignores_the_rectangle(): + found = helpers.locate("C-1", noul(0.1), box(0.99), 0.3, 0.8) + assert found.outcome == ("absent", None) + + +def test_an_unconfident_presence_answer_is_a_review_never_an_absence(): + # confidence 2 * max(p, 1 - p) - 1: 0.6 for p = 0.8, 0.2 for p = 0.4 + assert helpers.locate("C-1", noul(0.4), box(0.99), 0.3, 0.8).outcome == ("review", None) + assert helpers.locate("C-1", noul(0.8), box(0.99), 0.7, 0.8).outcome == ("review", None) + assert helpers.locate("C-1", noul(0.8), box(0.99), 0.6, 0.8).kind == "located" + + +def test_a_probability_of_exactly_one_half_is_not_a_yes(): + assert helpers.locate("C-1", noul(0.5), box(0.99), 0.0, 0.8).outcome == ("absent", None) + + +def test_a_review_is_never_an_absence_over_a_grid_of_probabilities_and_thresholds(): + # 101 probabilities (0.00 to 1.00) by 5 thresholds: below the threshold the outcome is a + # review; at or above it, an absence is the answer "no" and nothing else. + for step in range(101): + probability = step / 100 + confidence = noul_confidence([probability])[0] + for threshold in (0.0, 0.25, 0.5, 0.75, 1.0): + found = helpers.locate("C-1", noul(probability), box(0.99), threshold, 0.0) + if confidence < threshold: + assert found.kind == "review", (probability, threshold) + elif probability <= 0.5: + assert found.kind == "absent", (probability, threshold) + else: + assert found.kind == "located", (probability, threshold) + + +def test_a_rectangle_without_enough_confidence_is_a_review(): + missing = helpers.locate("C-1", noul(0.9), box(None), 0.3, 0.0) + assert (missing.kind, missing.reason) == ("review", "the rectangle has no confidence") + low = helpers.locate("C-1", noul(0.9), box(0.5), 0.3, 0.8) + assert (low.kind, low.reason) == ("review", "rectangle confidence below the threshold") + assert helpers.locate("C-1", noul(0.9), box(0.8), 0.3, 0.8).kind == "located" + + +def test_a_threshold_outside_zero_to_one_is_an_error(): + with pytest.raises(ValueError, match="presence_threshold"): + helpers.locate("C-1", noul(0.9), box(0.9), 1.1, 0.8) + with pytest.raises(ValueError, match="box_threshold"): + helpers.locate("C-1", noul(0.9), box(0.9), 0.3, -0.1) + + +def test_stored_answers_are_not_all_right(): + # A wrong answer anywhere is a weak guard: it would still pass even if a confidence gate + # caught every mistake. Mirror your notebook's own threshold selection here (the lowest + # confidence at which every validation answer that counts is right) and require a wrong + # `test` answer at or above it: a mistake the gate would still let through, which is what + # evaluating on `test` exists to catch. This checks the presence question; repeat it for the + # rectangle's confidence if that gate matters to your recipe. + backend = get_backend(fixtures=responses_path(RECIPE), images=True) + questions = helpers.build_questions() + labels = load_labels(RECIPE) + examples = load_inputs(RECIPE) + + def answer_of(example): + state, assets = helpers.build_request(example.fields) + return backend.decide(state, questions, assets=assets)["present"] + + def right(example, answer): + return (answer.noul > 0.5) == (gold_box(labels[example.id]) is not None) + + validation = [e for e in examples if e.split == "validation" and e.id in labels] + val_answers = [answer_of(e) for e in validation] + threshold = select_confidence_threshold( + [right(e, a) for e, a in zip(validation, val_answers, strict=True)], + noul_confidence([a.noul for a in val_answers]), + target_accuracy=1.0, + ) + + test = [e for e in examples if e.split == "test" and e.id in labels] + test_answers = [answer_of(e) for e in test] + wrong_and_confident = [ + e.id + for e, a in zip(test, test_answers, strict=True) + if not right(e, a) and noul_confidence([a.noul])[0] >= threshold + ] + assert wrong_and_confident, "expected at least one confidently wrong test answer" +''' + +IMAGE_BUILD_FIXTURES_TEST = r'''"""build_fixtures.py follows the recipes/_template/build_fixtures.py pattern, with images: inputs +and labels generation is separate from responses generation, a recorded responses.json is never +silently overwritten, a picture that differs from the committed file is refused rather than +rewritten, and the committed files match what the script writes: the images and +assets-manifest.json unchanged, and responses.json byte for byte what jev_cookbook.live's writer +produces.""" + +import importlib.util +import json +import shutil +import subprocess +import sys +from pathlib import Path + +from PIL import Image + +from jev_cookbook import dump_responses +from jev_cookbook import synthetic_images as si + +RECIPE = Path(__file__).resolve().parent.parent +SCRIPT = RECIPE / "build_fixtures.py" + + +def load_module_at(path, name): + """Import build_fixtures.py by file path so its functions and module-level values (ROWS, + build_responses) are plain Python objects, not subprocess output or a re-dump of the file + it wrote.""" + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _jsonl(rows): + return "".join(json.dumps(row) + "\n" for row in rows) + + +def _copy_of_the_recipe(folder): + """A folder holding what a rerun needs: the two scripts and the committed images. The images + are copied because their bytes are part of the replay key of each request that carries + them: with the committed files beside it the script compares each picture it draws with the + committed one, and without them it would create the pictures afresh with whatever PNG + encoder this environment has.""" + folder.mkdir() + for name in ("helpers.py", "build_fixtures.py"): + (folder / name).write_bytes((RECIPE / name).read_bytes()) + shutil.copytree(RECIPE / "assets", folder / "assets") + return folder / "build_fixtures.py" + + +def test_the_committed_responses_are_byte_identical_to_the_recorders_writer(): + """jev_cookbook.dump_responses sorts only the top-level keys; json.dumps(..., sort_keys=True) + sorts every nested dict too and so disagrees with it on every response's field order. The + committed file must match dump_responses exactly, not just agree with it on top-level order. + + Comparing against dump_responses(json.loads(raw)) (re-dumping the file's own parsed content) + cannot catch a sort_keys=True regression: json.loads preserves whatever nested order the file + already has, and dump_responses only re-sorts the top level, so that round trip would pass no + matter which writer produced the file. Instead, import build_fixtures.py and compare against + dump_responses of what build_responses computes directly from ROWS, independent of what + main() actually wrote to disk.""" + raw = (RECIPE / "fixtures" / "responses.json").read_text("utf-8") + module = load_module_at(SCRIPT, "recipe@RANK@_build_fixtures_for_test") + assert len(module.ROWS) > 1 + assert raw == dump_responses(module.build_responses(module.ROWS)) + + +def test_build_fixtures_separates_inputs_labels_from_responses(tmp_path): + """Mutation check: regenerating must never silently overwrite a recorded responses.json, + and --force must restore the exact synthetic bytes. inputs.jsonl and labels.jsonl are + rewritten from ROWS on every run, refused or not: only responses.json is guarded, and this + also proves it by corrupting both files before the refused run and checking that they come + back exactly as build_inputs_and_labels(ROWS) computes them, not merely "changed".""" + script = _copy_of_the_recipe(tmp_path / "@SLUG@") + copy = script.parent + first = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert first.returncode == 0, first.stderr + + responses = copy / "fixtures" / "responses.json" + data = json.loads(responses.read_text("utf-8")) + for value in data.values(): + value["model"] = "jev-1.13.0" + responses.write_text( + json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8", newline="\n" + ) + before = responses.read_bytes() + + inputs_path = copy / "fixtures" / "inputs.jsonl" + labels_path = copy / "fixtures" / "labels.jsonl" + inputs_path.write_bytes(b"corrupted") + labels_path.write_bytes(b"corrupted") + + refused = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert refused.returncode != 0 + assert "recorded" in refused.stderr and "--force" in refused.stderr + assert responses.read_bytes() == before + module = load_module_at(script, "recipe@RANK@_build_fixtures_for_test_refused") + expected_inputs, expected_labels = module.build_inputs_and_labels(module.ROWS) + assert inputs_path.read_text("utf-8") == _jsonl(expected_inputs) + assert labels_path.read_text("utf-8") == _jsonl(expected_labels) + + forced = subprocess.run( + [sys.executable, str(script), "--force"], capture_output=True, text=True + ) + assert forced.returncode == 0, forced.stderr + after = json.loads(responses.read_text("utf-8")) + assert all(v["model"] == "synthetic" for v in after.values()) + + +def test_the_generator_reproduces_the_committed_fixtures_and_images(tmp_path): + script = _copy_of_the_recipe(tmp_path / "@SLUG@") + copy = script.parent + done = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert done.returncode == 0, done.stderr + for name in ("inputs.jsonl", "labels.jsonl", "responses.json"): + assert (copy / "fixtures" / name).read_bytes() == (RECIPE / "fixtures" / name).read_bytes() + assert (copy / "assets-manifest.json").read_bytes() == ( + RECIPE / "assets-manifest.json" + ).read_bytes() + committed = sorted(p.relative_to(RECIPE) for p in (RECIPE / "assets").rglob("*") if p.is_file()) + assert committed + assert sorted(p.relative_to(copy) for p in (copy / "assets").rglob("*") if p.is_file()) == ( + committed + ) + for path in committed: + assert (copy / path).read_bytes() == (RECIPE / path).read_bytes(), path + + +def test_a_picture_that_differs_from_the_committed_file_is_refused_not_rewritten(tmp_path): + """The byte policy of docs/synthetic-images.md: a changed picture is a new hash and so new + replay keys, which is a decision. The script stops, names the problem and writes nothing more; + --overwrite-assets accepts the redrawn picture and restores the committed one.""" + script = _copy_of_the_recipe(tmp_path / "@SLUG@") + copy = script.parent + picture = sorted((copy / "assets").glob("*.png"))[0] + committed = picture.read_bytes() + with Image.open(picture) as original: + size = original.size + Image.new("RGB", size, "white").save(picture) + changed = picture.read_bytes() + assert changed != committed + manifest = copy / "assets-manifest.json" + assert not manifest.exists() + + refused = subprocess.run([sys.executable, str(script)], capture_output=True, text=True) + assert refused.returncode != 0 + assert "refused" in refused.stderr and "nothing else was written" in refused.stderr + assert picture.read_bytes() == changed + assert not manifest.exists() and not (copy / "fixtures").exists() + + accepted = subprocess.run( + [sys.executable, str(script), "--overwrite-assets"], capture_output=True, text=True + ) + assert accepted.returncode == 0, accepted.stderr + # The same pixels as the committed picture. Whether the bytes are the committed ones too + # depends on the PNG encoder of the environment running this test (docs/synthetic-images.md). + assert picture.read_bytes() != changed + assert si.png_pixels_equal(picture.read_bytes(), committed) +''' + + +def image_rows_text() -> str: + """``IMAGE_ROWS`` as the lines of ``ROWS`` in the image ``build_fixtures.py``.""" + + def literal(value: object) -> str: + if isinstance(value, tuple): + return "(" + ", ".join(literal(item) for item in value) + ")" + return json.dumps(value) if isinstance(value, str) else repr(value) + + return "\n".join(f" {literal(row)}," for row in IMAGE_ROWS) + + +def _fill(template: str, recipe: dict) -> str: + """``template`` with its ``@TOKEN@`` placeholders replaced.""" + filled = template.replace("@RANK@", f"{recipe['rank']:02d}").replace("@TODO@", TODO_MARK) + return filled.replace("@SLUG@", recipe["slug"]).replace("@ROWS@", image_rows_text()) + + +def image_helpers_text(recipe: dict) -> str: + return _fill(IMAGE_HELPERS, recipe) + + +def image_build_fixtures_text(recipe: dict) -> str: + return _fill(IMAGE_BUILD_FIXTURES, recipe) + + def files_for(catalog: dict, recipe: dict, mode: str = "replay") -> dict[str, str]: """``{relative path: text}`` of everything the scaffold creates.""" files = { @@ -976,16 +2004,50 @@ def files_for(catalog: dict, recipe: dict, mode: str = "replay") -> dict[str, st # logic for these tests to guard (build_scripted_fixtures_text), so the file is emitted # for replay recipes only. files["tests/test_build_fixtures.py"] = build_fixtures_test_text(recipe) + if mode == IMAGE_MODE: + files["tests/test_build_fixtures.py"] = build_fixtures_test_text(recipe, mode) return files +BUILD_TIMEOUT = 300 # seconds the image scaffold waits for the new folder's build_fixtures.py + + +def _draw_images(target: Path) -> None: + """Run ``target``'s own ``build_fixtures.py`` once, which draws ``assets/`` and writes + ``assets-manifest.json`` and ``fixtures/`` (``--mode replay-images``). If it cannot run or + exits with an error, ``target`` is removed and ``ScaffoldError`` carries what it printed, so + the command leaves no half-written recipe.""" + try: + done = subprocess.run( + [sys.executable, "build_fixtures.py"], + cwd=target, + capture_output=True, + text=True, + timeout=BUILD_TIMEOUT, + ) + problem = "" if done.returncode == 0 else "\n".join(done.stderr.strip().splitlines()[-8:]) + if done.returncode != 0 and not problem: + problem = f"exit status {done.returncode}" + except (OSError, subprocess.SubprocessError) as error: + problem = f"{type(error).__name__}: {error}" + if problem: + shutil.rmtree(target, ignore_errors=True) + raise ScaffoldError( + "--mode replay-images draws the example images by running the new folder's " + "build_fixtures.py, which needs the installed package and Pillow " + f'(pip install -e ".[dev]"); it failed, so nothing was created:\n{problem}' + ) + + def scaffold(number: int, catalog_path: Path, recipes_dir: Path, mode: str = "replay") -> Path: """Create ``recipes_dir/NN-slug/`` and return it. Raises ``ScaffoldError`` and writes - nothing when the number is out of range, is not in the catalog, or the folder exists.""" + nothing when the number is out of range, is not in the catalog, or the folder exists. For + ``--mode replay-images`` it then runs the folder's ``build_fixtures.py`` once (see + ``_draw_images``); if that fails the folder is removed again.""" if not FIRST <= number <= LAST: raise ScaffoldError(f"recipe number must be {FIRST} to {LAST}, got {number}") - if mode not in MODES: - raise ScaffoldError(f"mode must be one of {', '.join(MODES)}, got {mode!r}") + if mode not in ALL_MODES: + raise ScaffoldError(f"mode must be one of {', '.join(ALL_MODES)}, got {mode!r}") catalog = _load_catalog(catalog_path) recipe = find_recipe(catalog, number) slug = recipe.get("slug") @@ -1004,6 +2066,8 @@ def scaffold(number: int, catalog_path: Path, recipes_dir: Path, mode: str = "re path = target / name path.parent.mkdir(parents=True, exist_ok=True) path.write_text(text, encoding="utf-8", newline="\n") + if mode == IMAGE_MODE: + _draw_images(target) return target @@ -1020,10 +2084,12 @@ def main(argv: list[str] | None = None) -> int: ) parser.add_argument( "--mode", - choices=MODES, + choices=ALL_MODES, default="replay", help="replay (default): fixtures/responses.json is replayed; scripted: a seeded script " - "in helpers.py answers, for simulator and ScriptedBackend recipes", + "in helpers.py answers, for simulator and ScriptedBackend recipes; replay-images: replay " + "with images, which also draws assets/ and a working example by running the new " + "folder's build_fixtures.py (needs the installed package and Pillow)", ) args = parser.parse_args(argv) if not args.number.isascii() or not args.number.isdigit():