From c85bfc1ce0544f38af9a7754733bfbb5dfe7eed8 Mon Sep 17 00:00:00 2001 From: "David J. Kim" Date: Wed, 22 Jul 2026 14:58:03 -0400 Subject: [PATCH 1/2] feat: release-blocking public-surface reconciliation gate Two external audits reported cache-stale public version numbers that no live fetch returned; there was no mechanical way to prove the public surfaces are reconciled to the shipped version. scripts/check_public_surfaces.py closes that: - Group A (offline, CI-safe): the version agrees across pyproject, __init__, the git tag, the README Action pin + hotato-version input, the CHANGELOG top entry, and llms.txt; any drift exits 1. - Group B (--online): PyPI JSON, hotato.dev/release.json, hotato.dev/llms.txt, and the hotato.dev landing page (Googlebot UA, no-cache) all report the shipped version and positioning, with no retired overclaim. Stdlib-only, no pipe-to-shell of fetched bytes; 16 tests (drift detection proven); wired into docs/RELEASE-CHECKLIST.md as a release-blocking step. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9 --- docs/RELEASE-CHECKLIST.md | 1 + scripts/check_public_surfaces.py | 405 ++++++++++++++++++++++++++++ tests/test_check_public_surfaces.py | 186 +++++++++++++ 3 files changed, 592 insertions(+) create mode 100644 scripts/check_public_surfaces.py create mode 100644 tests/test_check_public_surfaces.py diff --git a/docs/RELEASE-CHECKLIST.md b/docs/RELEASE-CHECKLIST.md index 2e5b19f..75df5c2 100644 --- a/docs/RELEASE-CHECKLIST.md +++ b/docs/RELEASE-CHECKLIST.md @@ -35,6 +35,7 @@ Every gate to clear before a release ships, top to bottom: green means proceed. - [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report. - [ ] Verify `uvx hotato doctor` (self-test path) works from the published package. - [ ] Publish the GitHub release notes for the tag `vX.Y.Z` (created above), pointing at the CHANGELOG entry. +- [ ] **Reconcile the public surfaces (release-blocking).** Run `python3 scripts/check_public_surfaces.py --online` from a clean network. It cross-checks the version across `pyproject.toml`, `hotato.__version__`, the git tag, the README Action pin, the CHANGELOG top entry, and `llms.txt` (offline group A), then asserts PyPI's `/pypi/hotato//json`, `hotato.dev/release.json`, `hotato.dev/llms.txt`, and the `hotato.dev/` landing page (Googlebot UA, no-cache) all report the shipped version and positioning with no retired overclaim (group B). The release is not done until this exits 0. If the site lags, redeploy from the `vX.Y.Z` tag and purge the Cloudflare HTML cache, then re-run. (Group A alone runs in CI with no egress.) ### Publishing path: PyPI Trusted Publishing (OIDC) -- DEFAULT diff --git a/scripts/check_public_surfaces.py b/scripts/check_public_surfaces.py new file mode 100644 index 0000000..9450eb5 --- /dev/null +++ b/scripts/check_public_surfaces.py @@ -0,0 +1,405 @@ +#!/usr/bin/env python3 +"""Release-blocking public-surface reconciliation gate. + +A publisher's success signal ("uploaded to PyPI", "Pages deploy green") does NOT +prove the public surfaces actually SERVE the shipped version: two external audits +both reported cache-stale public numbers (a CDN kept serving an older +``llms.txt`` / landing page / package summary). This gate answers, mechanically +and from a clean network, one question: **are the public surfaces reconciled to +the shipped version?** -- so a release is never declared done on a publisher +success signal alone. + +Two check groups: + + A. LOCAL CROSS-SOURCE CONSISTENCY (no network, always runs). Every in-repo + surface that names the version must agree: pyproject ``version`` == + ``hotato.__version__`` == the git tag ``vX.Y.Z`` (when HEAD is tagged or the + tag exists) == the README action pin (``attenlabs/hotato@vX.Y.Z`` and + ``hotato-version: X.Y.Z``) == the top ``## [X.Y.Z]`` CHANGELOG entry == + ``llms.txt``'s ``Version X.Y.Z``. This is the part CI runs with no egress. + + B. PUBLIC SURFACE RECONCILIATION (network; gated behind ``--online`` so a + no-egress CI job skips it cleanly). PyPI's per-version JSON, the site's + ``release.json``, the CDN-served ``llms.txt`` and landing page are fetched + with a crawler UA + ``Cache-Control: no-cache`` and asserted to carry the + CURRENT version and positioning markers (never the retired overclaim, never + the stale product-identity ````). + +Usage: + python3 scripts/check_public_surfaces.py # group A only + python3 scripts/check_public_surfaces.py --online # A + B + python3 scripts/check_public_surfaces.py --version 1.2.3 # override the version + +Exit 0 = reconciled (or group B cleanly skipped); exit 1 = a mismatch / a public +surface is stale or unreachable (fail closed); exit 2 = usage/env error. +Stdlib-only. Group A does no network; group B uses only ``urllib.request`` and +parses every response JSON in-process (downloaded bytes are never piped anywhere). +""" + +from __future__ import annotations + +import argparse +import json +import re +import subprocess +import sys +import urllib.error +import urllib.request +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent + +EXIT_OK = 0 +EXIT_FAIL = 1 +EXIT_ENV = 2 + +# A crawler UA + no-cache so a CDN cannot serve us a cached (stale) copy: this is +# exactly the surface an external auditor / search crawler sees. +_CRAWLER_UA = ( + "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" +) +_NO_CACHE_HEADERS = { + "User-Agent": _CRAWLER_UA, + "Cache-Control": "no-cache", + "Pragma": "no-cache", +} + +_PYPI_URL = "https://pypi.org/pypi/hotato/{version}/json" +_RELEASE_JSON_URL = "https://hotato.dev/release.json" +_LLMS_URL = "https://hotato.dev/llms.txt" +_SITE_URL = "https://hotato.dev/" + +# Positioning markers the public surfaces MUST carry (the current identity) and +# MUST NOT carry (retired overclaim / stale product identity). Kept in lockstep +# with the 1.15.1 public-copy correction (CHANGELOG) and the operator-set +# positioning "Local-first testing and observability for AI agents". +_RETIRED_OVERCLAIM = "everything you use a hosted platform for" +_REQUIRED_PYPI_SUMMARY = "testing and observability for ai agents" +_REQUIRED_SITE_MARKER = "Local-first testing and observability for AI agents" +_STALE_TITLE_MARKER = "regression testing for voice agents" + + +# --------------------------------------------------------------------------- # +# Version helpers +# --------------------------------------------------------------------------- # +def _normalize(v: str | None) -> str | None: + """Strip a leading ``v`` and surrounding whitespace so ``v1.2.3`` and + ``1.2.3`` compare equal.""" + if v is None: + return None + v = v.strip() + return v[1:] if v.startswith("v") else v + + +# --------------------------------------------------------------------------- # +# Group A source readers. Each takes the repo root and returns the version it +# reports (normalized, no ``v``) or None if the surface does not name one. They +# read files directly so a test can point them at a fixture repo layout (or +# monkeypatch them) without any network or git. +# --------------------------------------------------------------------------- # +def read_pyproject_version(root: Path) -> str | None: + text = (root / "pyproject.toml").read_text(encoding="utf-8") + m = re.search(r'(?m)^version\s*=\s*"([^"]+)"', text) + return _normalize(m.group(1)) if m else None + + +def read_init_version(root: Path) -> str | None: + text = (root / "src" / "hotato" / "__init__.py").read_text(encoding="utf-8") + m = re.search(r'(?m)^__version__\s*=\s*"([^"]+)"', text) + return _normalize(m.group(1)) if m else None + + +def read_readme_action_pin(root: Path) -> str | None: + """The ``attenlabs/hotato@vX.Y.Z`` action pin in the README's CI snippet.""" + text = (root / "README.md").read_text(encoding="utf-8") + m = re.search(r"attenlabs/hotato@v(\d+\.\d+\.\d+)", text) + return _normalize(m.group(1)) if m else None + + +def read_readme_hotato_version(root: Path) -> str | None: + """The ``hotato-version: X.Y.Z`` input in the README's CI snippet.""" + text = (root / "README.md").read_text(encoding="utf-8") + m = re.search(r"(?m)^\s*hotato-version:\s*v?(\d+\.\d+\.\d+)", text) + return _normalize(m.group(1)) if m else None + + +def read_changelog_top(root: Path) -> str | None: + """The first real ``## [X.Y.Z]`` release entry, skipping ``[Unreleased]``.""" + text = (root / "CHANGELOG.md").read_text(encoding="utf-8") + for m in re.finditer(r"(?m)^##\s*\[([^\]]+)\]", text): + tag = m.group(1).strip() + if tag.lower() == "unreleased": + continue + vm = re.match(r"v?(\d+\.\d+\.\d+)", tag) + return _normalize(vm.group(1)) if vm else None + return None + + +def read_llms_version(root: Path) -> str | None: + text = (root / "llms.txt").read_text(encoding="utf-8") + m = re.search(r"(?m)^\s*>?\s*Version\s+v?(\d+\.\d+\.\d+)", text) + return _normalize(m.group(1)) if m else None + + +def read_git_tag(root: Path, version: str) -> str | None: + """The git tag source. + + Returns the tag at HEAD (normalized, no ``v``) when HEAD is exactly tagged, + else ``vX.Y.Z`` when that tag exists in the repo, else None (no relevant tag + -- e.g. a pre-tag release commit, or not a git checkout). None is NOT a + mismatch: a version can be reconciled before it is tagged. + """ + exact = subprocess.run( + ["git", "-C", str(root), "describe", "--tags", "--exact-match", "HEAD"], + capture_output=True, + ) + if exact.returncode == 0: + tag = exact.stdout.decode("utf-8", "replace").strip() + if tag: + return _normalize(tag) + exists = subprocess.run( + ["git", "-C", str(root), "tag", "-l", f"v{version}"], + capture_output=True, + ) + if exists.returncode == 0 and exists.stdout.decode("utf-8", "replace").strip(): + return version + return None + + +# Ordered (label, reader) pairs for the file-backed sources. Git is handled +# separately because it needs the expected version to resolve its tag. +_LOCAL_SOURCES = [ + ("pyproject.toml version", read_pyproject_version), + ("hotato.__version__", read_init_version), + ("README action pin (attenlabs/hotato@vX.Y.Z)", read_readme_action_pin), + ("README hotato-version input", read_readme_hotato_version), + ("CHANGELOG top entry", read_changelog_top), + ("llms.txt Version", read_llms_version), +] + + +def collect_local_sources(root: Path, version: str) -> list[tuple[str, str | None]]: + """Return an ordered list of (source label, reported version) for group A, + including the git-tag source resolved against ``version``.""" + rows: list[tuple[str, str | None]] = [] + for label, reader in _LOCAL_SOURCES: + try: + rows.append((label, reader(root))) + except FileNotFoundError: + rows.append((label, None)) + try: + rows.append(("git tag vX.Y.Z", read_git_tag(root, version))) + except Exception: # noqa: BLE001 - git absence must never crash group A + rows.append(("git tag vX.Y.Z", None)) + return rows + + +def run_group_a(root: Path, version: str) -> tuple[bool, list[str]]: + """Assert every in-repo source that names a version equals ``version``. + + A source that reports None (does not name a version, or the git tag does not + exist yet) is skipped, not failed. Returns (ok, printable table lines). + """ + rows = collect_local_sources(root, version) + lines: list[str] = [] + ok = True + width = max(len(label) for label, _ in rows) + for label, found in rows: + if found is None: + status = "n/a" + elif found == version: + status = "OK" + else: + status = "MISMATCH" + ok = False + shown = found if found is not None else "(not present)" + lines.append(f" {label.ljust(width)} {shown:<12} {status}") + return ok, lines + + +# --------------------------------------------------------------------------- # +# Group B fetch + parse helpers. The parse helpers take already-decoded data so +# they are unit-testable on a fixture without any network. +# --------------------------------------------------------------------------- # +class SurfaceError(Exception): + """A public surface could not be fetched or failed its assertion. Carries a + single-line human message naming the failing URL; never a traceback.""" + + +def _fetch_bytes(url: str, timeout: float = 15.0) -> bytes: + req = urllib.request.Request(url, headers=_NO_CACHE_HEADERS) # noqa: S310 - https only + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 + return resp.read() + except urllib.error.HTTPError as exc: + raise SurfaceError(f"{url} -> HTTP {exc.code} {exc.reason}") from exc + except (urllib.error.URLError, TimeoutError, OSError) as exc: + reason = getattr(exc, "reason", exc) + raise SurfaceError(f"{url} -> unreachable ({reason})") from exc + + +def _fetch_json(url: str) -> dict: + raw = _fetch_bytes(url) + try: + doc = json.loads(raw.decode("utf-8")) + except (ValueError, UnicodeDecodeError) as exc: + raise SurfaceError(f"{url} -> response was not valid JSON ({exc})") from exc + if not isinstance(doc, dict): + raise SurfaceError(f"{url} -> JSON root was not an object") + return doc + + +def _fetch_text(url: str) -> str: + return _fetch_bytes(url).decode("utf-8", "replace") + + +def check_pypi_doc(doc: dict, version: str) -> list[str]: + """Assert PyPI's per-version JSON reconciles to ``version``. Returns a list + of failure strings (empty == OK). Pure; unit-tested on a fixture JSON.""" + problems: list[str] = [] + info = doc.get("info") or {} + got = info.get("version") + if got != version: + problems.append(f"info.version is {got!r}, expected {version!r}") + urls = doc.get("urls") + if not urls: + problems.append("doc.urls is empty (no release files for this version)") + summary = (info.get("summary") or "") + low = summary.lower() + if _RETIRED_OVERCLAIM in low: + problems.append( + f"summary still contains the retired overclaim {_RETIRED_OVERCLAIM!r}" + ) + if _REQUIRED_PYPI_SUMMARY not in low: + problems.append( + f"summary is missing the required phrase {_REQUIRED_PYPI_SUMMARY!r}" + ) + return problems + + +def check_release_json(doc: dict, version: str) -> list[str]: + got = doc.get("version") + if got != version: + return [f"version is {got!r}, expected {version!r}"] + return [] + + +def check_llms_text(text: str, version: str) -> list[str]: + if f"Version {version}" not in text: + return [f"does not contain {'Version ' + version!r}"] + return [] + + +def check_site_html(html: str, version: str) -> list[str]: + """Assert the landing page carries the current positioning marker and does + NOT use the stale product identity as its ``<title>``. Deliberately does + NOT fail on an old version string appearing in the page -- a changelog + legitimately lists past versions; only positioning markers are asserted.""" + problems: list[str] = [] + if _REQUIRED_SITE_MARKER not in html: + problems.append(f"missing the positioning marker {_REQUIRED_SITE_MARKER!r}") + # Only a stale <title> is a true product-identity regression. Match the + # marker specifically inside a <title>..., not anywhere on the page. + title_m = re.search(r"]*>(.*?)", html, re.I | re.S) + if title_m and _STALE_TITLE_MARKER.lower() in title_m.group(1).lower(): + problems.append( + f" still reads the stale product identity {_STALE_TITLE_MARKER!r}" + ) + return problems + + +def run_group_b(version: str) -> tuple[bool, list[str]]: + """Fetch and reconcile every public surface. Returns (ok, table lines). A + network error is a clean failure line naming the URL, never a traceback.""" + checks = [ + ("PyPI /pypi/hotato/{v}/json", _PYPI_URL.format(version=version), + lambda: check_pypi_doc(_fetch_json(_PYPI_URL.format(version=version)), version)), + ("hotato.dev/release.json", _RELEASE_JSON_URL, + lambda: check_release_json(_fetch_json(_RELEASE_JSON_URL), version)), + ("hotato.dev/llms.txt (crawler UA)", _LLMS_URL, + lambda: check_llms_text(_fetch_text(_LLMS_URL), version)), + ("hotato.dev/ (crawler UA)", _SITE_URL, + lambda: check_site_html(_fetch_text(_SITE_URL), version)), + ] + lines: list[str] = [] + ok = True + width = max(len(label) for label, _, _ in checks) + for label, url, run in checks: + try: + problems = run() + except SurfaceError as exc: + ok = False + lines.append(f" {label.ljust(width)} UNREACHABLE {exc}") + continue + if problems: + ok = False + lines.append(f" {label.ljust(width)} STALE") + for p in problems: + lines.append(f" {' ' * width} - {p}") + else: + lines.append(f" {label.ljust(width)} OK {url}") + return ok, lines + + +# --------------------------------------------------------------------------- # +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser( + description="Reconcile hotato's public surfaces to the shipped version.", + ) + ap.add_argument( + "--version", + help="version to reconcile to (default: read from pyproject.toml)", + ) + ap.add_argument( + "--online", + action="store_true", + help="also fetch and reconcile the live public surfaces (network)", + ) + args = ap.parse_args(argv) + + version = _normalize(args.version) + if version is None: + try: + version = read_pyproject_version(REPO_ROOT) + except FileNotFoundError: + print("error: pyproject.toml not found; run inside the repo checkout", + file=sys.stderr) + return EXIT_ENV + if version is None: + print("error: could not read a version from pyproject.toml", + file=sys.stderr) + return EXIT_ENV + + print(f"Reconciling public surfaces to version {version}\n") + + print("A. Local cross-source consistency (no network):") + a_ok, a_lines = run_group_a(REPO_ROOT, version) + for line in a_lines: + print(line) + + print("\nB. Public surface reconciliation (network):") + if args.online: + b_ok, b_lines = run_group_b(version) + for line in b_lines: + print(line) + else: + b_ok = True + print(" SKIPPED (pass --online to fetch the live surfaces)") + + print() + if a_ok and b_ok: + print(f"OK: public surfaces reconciled to {version}.") + return EXIT_OK + reason = [] + if not a_ok: + reason.append("local sources disagree") + if not b_ok: + reason.append("a public surface is stale or unreachable") + print(f"REFUSE: {'; '.join(reason)}. A release is NOT done on a publisher " + "success signal alone -- reconcile the surfaces above and re-check.", + file=sys.stderr) + return EXIT_FAIL + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_check_public_surfaces.py b/tests/test_check_public_surfaces.py new file mode 100644 index 0000000..504fda3 --- /dev/null +++ b/tests/test_check_public_surfaces.py @@ -0,0 +1,186 @@ +"""``scripts/check_public_surfaces.py``: the release-blocking public-surface +reconciliation gate. + +Group A (local cross-source consistency) is exercised end to end against a +TEMP fake-repo layout: every in-repo surface that names the version is written +into a tmp tree, and the gate must (a) pass when they all agree and (b) FAIL +when any single source drifts. Group B is not unit-tested against the live +network here; only its pure parsing helper (``check_pypi_doc``) is asserted on +a fixture JSON, so no test ever performs I/O against pypi.org / hotato.dev. +""" + +from __future__ import annotations + +import importlib.util +import os + +import pytest + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_SCRIPT = os.path.join(_ROOT, "scripts", "check_public_surfaces.py") + + +def _load_module(): + spec = importlib.util.spec_from_file_location("check_public_surfaces", _SCRIPT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +cps = _load_module() + +VERSION = "1.15.1" + + +def _write_fake_repo(root, *, version=VERSION, overrides=None): + """Materialize a minimal repo layout naming ``version`` across every group-A + surface. ``overrides`` maps a surface key to a REPLACEMENT version string so + a single source can be drifted while the rest stay in lockstep.""" + overrides = overrides or {} + + def v(key): + return overrides.get(key, version) + + root = str(root) + (pkg := os.path.join(root, "src", "hotato")) # noqa: F841 + os.makedirs(os.path.join(root, "src", "hotato"), exist_ok=True) + + with open(os.path.join(root, "pyproject.toml"), "w", encoding="utf-8") as fh: + fh.write(f'[project]\nname = "hotato"\nversion = "{v("pyproject")}"\n') + + with open(os.path.join(root, "src", "hotato", "__init__.py"), "w", + encoding="utf-8") as fh: + fh.write(f'__version__ = "{v("init")}"\n') + + with open(os.path.join(root, "README.md"), "w", encoding="utf-8") as fh: + fh.write( + "# hotato\n\n```yaml\n" + f" - uses: attenlabs/hotato@v{v('readme_pin')}\n" + " with:\n" + " contracts: contracts/\n" + f" hotato-version: {v('readme_input')}\n" + "```\n" + ) + + with open(os.path.join(root, "CHANGELOG.md"), "w", encoding="utf-8") as fh: + fh.write( + "# Changelog\n\n## [Unreleased]\n\n" + f"## [{v('changelog')}] - 2026-07-22\n\n### Changed\n- stuff\n\n" + "## [1.6.0] - 2026-01-01\n- older\n" + ) + + with open(os.path.join(root, "llms.txt"), "w", encoding="utf-8") as fh: + fh.write(f"# hotato\n\n> some index text\n>\n> Version {v('llms')}\n") + + return root + + +def test_group_a_passes_when_all_sources_agree(tmp_path): + root = _write_fake_repo(tmp_path) + ok, lines = cps.run_group_a(cps.Path(root), VERSION) + assert ok, "\n".join(lines) + # Every file-backed source resolved to the shipped version (git tag is n/a + # in a non-git tmp tree, which is not a failure). + assert all("MISMATCH" not in line for line in lines) + + +@pytest.mark.parametrize( + "drift_key", + ["pyproject", "init", "readme_pin", "readme_input", "changelog", "llms"], +) +def test_group_a_fails_on_single_source_drift(tmp_path, drift_key): + """A drift in ANY single in-repo surface must fail group A -- this is the + whole point of the gate: a publisher signal cannot hide a stale source.""" + root = _write_fake_repo(tmp_path, overrides={drift_key: "9.9.9"}) + ok, lines = cps.run_group_a(cps.Path(root), VERSION) + assert not ok, f"expected drift in {drift_key} to fail:\n" + "\n".join(lines) + assert any("MISMATCH" in line for line in lines) + + +def test_group_a_git_tag_is_na_without_repo(tmp_path): + """The git-tag source reports n/a (not a mismatch) when the tree is not a + git checkout, so group A is clean pre-tag.""" + root = _write_fake_repo(tmp_path) + rows = cps.collect_local_sources(cps.Path(root), VERSION) + git_rows = [(label, val) for label, val in rows if "git tag" in label] + assert git_rows and git_rows[0][1] is None + + +def test_read_changelog_top_skips_unreleased(tmp_path): + root = _write_fake_repo(tmp_path) + assert cps.read_changelog_top(cps.Path(root)) == VERSION + + +# --------------------------------------------------------------------------- # +# Group B: only the pure parsing helper, on a fixture JSON. No network. +# --------------------------------------------------------------------------- # +def _pypi_fixture(**over): + doc = { + "info": { + "version": VERSION, + "summary": "Local-first testing and observability for AI agents. " + "Trace, evaluate, simulate, and gate agents.", + }, + "urls": [{"filename": "hotato-1.15.1.tar.gz"}], + } + doc["info"].update(over.get("info", {})) + if "urls" in over: + doc["urls"] = over["urls"] + return doc + + +def test_check_pypi_doc_clean_fixture_passes(): + assert cps.check_pypi_doc(_pypi_fixture(), VERSION) == [] + + +def test_check_pypi_doc_flags_version_mismatch(): + doc = _pypi_fixture(info={"version": "1.14.0"}) + problems = cps.check_pypi_doc(doc, VERSION) + assert any("info.version" in p for p in problems) + + +def test_check_pypi_doc_flags_empty_urls(): + doc = _pypi_fixture(urls=[]) + problems = cps.check_pypi_doc(doc, VERSION) + assert any("urls" in p for p in problems) + + +def test_check_pypi_doc_flags_retired_overclaim(): + doc = _pypi_fixture(info={ + "summary": "everything you use a hosted platform for, on your machine. " + "testing and observability for AI agents", + }) + problems = cps.check_pypi_doc(doc, VERSION) + assert any("retired overclaim" in p for p in problems) + + +def test_check_pypi_doc_flags_missing_required_phrase(): + doc = _pypi_fixture(info={"summary": "a local tool for voice stuff"}) + problems = cps.check_pypi_doc(doc, VERSION) + assert any("required phrase" in p for p in problems) + + +def test_check_release_json_and_llms_and_site_helpers(): + assert cps.check_release_json({"version": VERSION}, VERSION) == [] + assert cps.check_release_json({"version": "1.14.0"}, VERSION) != [] + + assert cps.check_llms_text(f"...\n> Version {VERSION}\n", VERSION) == [] + assert cps.check_llms_text("no version line here", VERSION) != [] + + +def test_check_site_html_positioning_markers(): + good = ("<title>hotato" + "

Local-first testing and observability for AI agents

" + "

changelog: 1.6.0, 1.7.0 released earlier

") + assert cps.check_site_html(good, VERSION) == [] + + # Old version strings elsewhere on the page are NOT a failure. + assert "1.6.0" in good and cps.check_site_html(good, VERSION) == [] + + missing_marker = "hotato

voice tools

" + assert cps.check_site_html(missing_marker, VERSION) != [] + + stale_title = ("hotato -- regression testing for voice agents" + "

Local-first testing and observability for AI agents

") + problems = cps.check_site_html(stale_title, VERSION) + assert any("" in p for p in problems) From f656ad39bc1e004c6833eea0f721f14fb0181dc1 Mon Sep 17 00:00:00 2001 From: "David J. Kim" <david@attentionlabs.ai> Date: Wed, 22 Jul 2026 15:20:25 -0400 Subject: [PATCH 2/2] chore: regenerate llms-full.txt (RELEASE-CHECKLIST reconciliation-gate step) The reconciliation-gate checklist edit feeds llms-full; regenerate so the committed copy matches the build (test_llms_full_txt_committed_copy_is_up_to_date). Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9 --- llms-full.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/llms-full.txt b/llms-full.txt index d2f4cf6..05e4eef 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -11324,6 +11324,7 @@ Every gate to clear before a release ships, top to bottom: green means proceed. - [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report. - [ ] Verify `uvx hotato doctor` (self-test path) works from the published package. - [ ] Publish the GitHub release notes for the tag `vX.Y.Z` (created above), pointing at the CHANGELOG entry. +- [ ] **Reconcile the public surfaces (release-blocking).** Run `python3 scripts/check_public_surfaces.py --online` from a clean network. It cross-checks the version across `pyproject.toml`, `hotato.__version__`, the git tag, the README Action pin, the CHANGELOG top entry, and `llms.txt` (offline group A), then asserts PyPI's `/pypi/hotato/<version>/json`, `hotato.dev/release.json`, `hotato.dev/llms.txt`, and the `hotato.dev/` landing page (Googlebot UA, no-cache) all report the shipped version and positioning with no retired overclaim (group B). The release is not done until this exits 0. If the site lags, redeploy from the `vX.Y.Z` tag and purge the Cloudflare HTML cache, then re-run. (Group A alone runs in CI with no egress.) ### Publishing path: PyPI Trusted Publishing (OIDC) -- DEFAULT