Physics-semantics evaluation layer and the 6-scorer architecture

.gitignore

@@ -12,7 +12,10 @@
 
 # Distribution / packaging
 .Python
-build/
+# Anchored to the repo root so the setuptools build/ artifact is ignored WITHOUT
+# also ignoring the src/prkit/semantics/build package (a bare "build/" matches any
+# directory of that name at any depth).
+/build/
 develop-eggs/
 dist/
 downloads/
@@ -251,3 +254,6 @@ uncertainty_*/
 
 
 **/perturbations/
+
+# uv lockfile (incidental; project standardizes on pip + .venv)
+uv.lock

.pre-commit-config.yaml

@@ -1,14 +1,15 @@
 repos:
+  # ruff is the linter only. Formatting is owned by black (the single formatter CI
+  # enforces via `black --check`); ruff-format is intentionally NOT enabled because
+  # the two disagree and would ping-pong files on every run.
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.15.17
     hooks:
       - id: ruff
         files: ^(src/prkit|tests/prkit)/
-      - id: ruff-format
-        files: ^(src/prkit|tests/prkit)/
 
   - repo: https://github.com/psf/black
-    rev: 24.8.0
+    rev: 26.1.0  # keep in sync with the venv/CI black to avoid format ping-pong
     hooks:
       - id: black
         files: ^(src/prkit|tests/prkit)/
@@ -17,7 +18,10 @@ repos:
     rev: v4.6.0
     hooks:
       - id: end-of-file-fixer
+        # Vendored third-party assets/forks are shipped as-is (verbatim upstream).
+        exclude: ^src/prkit/(annotation/tasks/correctness/ui/vendor|evaluation/baselines)/
       - id: trailing-whitespace
+        exclude: ^src/prkit/(annotation/tasks/correctness/ui/vendor|evaluation/baselines)/
 
   # Local hooks mirror the CI checks (.github/workflows/ci.yml) so type and
   # test failures surface at commit time instead of in CI. They invoke the

CHANGELOG.md

@@ -9,6 +9,9 @@ Production releases follow semantic versioning. TestPyPI validation builds use P
 
 ### Added
 
+- **Edit-distance scorer family** in `prkit.scoring`: the faithful PHYBench-EED / CMPhysBench-SEED baselines (`EedScorer`, `SeedScorer`, vendored under `prkit.evaluation.baselines`) and their our-semantics counterparts (`SemanticsEedScorer`, `SemanticsSeedScorer`), plus the model-graded `LLMJudgeScorer` wrapping `prkit.evaluation.llm_judge`. All emit the canonical `Verdict`. A new `[baselines]` optional extra pins `pint` for the SEED unit path; `import prkit.scoring` / `prkit.verify` stay free of `pint`/`openai` (lazy in `score()`).
+- **`Verdict.score == -1.0`** reserved as the not-applicable sentinel (`comparison_mode="not_applicable"`), emitted by the edit-distance scorers for answer kinds/structures with no SEED type. It is an honest "N/A" distinct from `0.0`; numeric aggregators must exclude it (`score >= 0`).
+- **`cmphysbench` loader** — `DatasetHub` gains the CMPhysBench benchmark loader, mapping the dataset-native `answer_type` into `PhysicsAnswer.source_type` (one of the five SEED tokens) for faithful `SeedScorer` dispatch.
 - **`BaseModelClient.parse()`** — dedicated typed structured-output entry point mirroring the SDK `.parse()` idiom (OpenAI `client.responses.parse`, Anthropic `client.messages.parse`). `parse(input, *, response_format=<PydanticModel>, image_paths=None, structured_policy="best_effort", instructions=None, **kwargs)` returns a `StructuredCallResult[T]` (`.parsed`, `.raw_text`, `.validation_error`, `.require_parsed()`). The first parameter is `input` and the schema parameter is `response_format`, unifying naming with `response()`. `response()` remains text-only (passing a Pydantic model still returns the JSON string). Replaces `chat_structured()` (now deprecated).
 - **Batch API support across OpenAI, Anthropic, and Gemini** — `BaseModelClient` gains a synchronous batch job lifecycle (`submit_batch` → `poll_batch` → `retrieve_batch_results`) plus a free-text request builder `build_batch_request(...)` that mirrors `response()` (same `input`/`instructions` handling, no structured output), complementing the existing structured `build_batch_structured_request`. New provider-agnostic types `BatchState`, `BatchStatus`, `BatchItemStatus`, and `BatchResult` (in `prkit.core.model_clients.batch_types`) normalize each provider's status enum and per-request results. Each provider's request-body construction is now shared between `response()` and the batch builders (`_build_responses_body` / `_build_messages_params`) to prevent drift. OpenAI o-family models drop `temperature` at build time. Unsupported providers raise `NotImplementedError`. Batch processing runs asynchronously at ~50% of synchronous cost. Gemini batches are submitted as an uploaded keyed JSONL file (via the File API, `src=<file>`) rather than as inline requests, so results come back as documented keyed JSONL (`{"key": ..., "response": {...}}`) and correlate reliably to each request — inline responses carry no per-request key and cannot be correlated.
 - **`OpenAIModel` custom endpoint support** — new keyword-only constructor params `base_url`, `api_key`, and `api_key_env` allow routing to any proxy or gateway that implements the OpenAI Responses API (`POST /v1/responses`) with an explicit key or key from a named environment variable. Backward-compatible: omitting all three preserves existing `OPENAI_API_KEY` + default endpoint behaviour.
@@ -26,6 +29,8 @@ Production releases follow semantic versioning. TestPyPI validation builds use P
 ### Changed
 
 - **BREAKING: renamed subpackages** to drop the redundant `prkit_` prefix — import from `prkit.core`, `prkit.datasets`, `prkit.evaluation`, `prkit.annotation`, `prkit.semantics` (previously `prkit.prkit_core`, etc.). The `sys.modules` top-level aliasing hack was removed.
+- **BREAKING: renamed domain classes** `Answer` → `PhysicsAnswer` and `PhysicalDataset` → `PhysicsDataset` (the latter also fixes the `physics_dataset.py` file/class stem mismatch). The other domain nouns (`PhysicsProblem`, `PhysicsSolution`, `PhysicsDomain`, `AnswerObjectKind`, `AnswerStructure`, `LicenseSpec`) are unchanged. The contract stays provisional at `API_VERSION "1.0"` (the rename is tracked internally, not signalled by a major bump); no deprecation alias is provided. The answer-ontology module `core/domain/answer_kinds.py` was also renamed to `answer_taxonomy.py` (symbols unchanged).
+- **`PartialCreditScorer` removed.** The graded edit-distance scoring it provided is now covered by `SemanticsEedScorer` / `SemanticsSeedScorer`; `verify(..., partial_credit=True)` routes to `SemanticsSeedScorer`. No deprecation alias (it was never part of the frozen `prkit.api` surface).
 - The model-client factory is now an extensible provider registry (`register_model_client`) instead of an if/elif chain; image/MIME/data-URL helpers are centralized in `prkit.core.model_clients.utils`.
 - Packaging: removed the erroneous `pip` runtime dependency, expanded trove classifiers (Python 3.11/3.12, Education, OS Independent), and aligned `black` / `requires-python` targets.
 - Release publishing now uses automated version selection instead of manual version bumps.

Makefile

@@ -1,4 +1,4 @@
-.PHONY: lint format format-check typecheck test test-prkit build check
+.PHONY: lint format format-check typecheck test test-prkit build check ci
 
 lint:
 	python -m ruff check src/prkit tests/prkit
@@ -22,4 +22,18 @@ test-prkit:
 build:
 	python -m build
 
-check: lint format-check test-prkit
+check: lint format-check typecheck test-prkit
+
+# Faithful local mirror of .github/workflows/ci.yml: runs every gate the same
+# way CI does, in a CI-like environment with NO .env and NO OPENAI_API_KEY, so
+# tests that secretly rely on a local key fail here instead of in CI. The .env
+# is moved aside and restored afterwards even if a gate fails. Run before pushing.
+ci:
+	@bash -c 'set -u; \
+	  if [ -f .env ]; then mv .env .env.cibak; fi; \
+	  trap "[ -f .env.cibak ] && mv .env.cibak .env" EXIT; \
+	  unset OPENAI_API_KEY; \
+	  python -m ruff check src/prkit tests/prkit && \
+	  python -m black --check src/prkit tests/prkit && \
+	  python -m mypy src/prkit && \
+	  python -m pytest tests/prkit'

README.md

@@ -8,8 +8,8 @@ PRKit applies a “unified interface” idea to the full physical-reasoning loop
 
 PRKit centers on **core components** that define the physical reasoning ontology. Three integrated subpackages build on this foundation:
 
-- **Core components**: `PhysicsDomain`, `AnswerCategory`, `PhysicsProblem`, `Answer`, `PhysicalDataset`, `PhysicsSolution`, `BaseModelClient`, `create_model_client`, `PRKitLogger`—the shared abstractions used across the toolkit.
-- **`prkit.datasets`**: A Datasets-like hub that downloads/loads benchmarks into the unified schema (`PhysicsProblem`, `PhysicalDataset`).
+- **Core components**: `PhysicsDomain`, `PhysicsProblem`, `PhysicsAnswer`, `PhysicsDataset`, `PhysicsSolution`, `BaseModelClient`, `create_model_client`, `PRKitLogger`—the shared abstractions used across the toolkit.
+- **`prkit.datasets`**: A Datasets-like hub that downloads/loads benchmarks into the unified schema (`PhysicsProblem`, `PhysicsDataset`).
 - **`prkit.annotation`**: Workflow-oriented tools for structured, lower-level labels (e.g., domain/subdomain, theorem usage).
 - **`prkit.evaluation`**: Evaluate-like components for physics-oriented scoring and comparison (e.g., symbolic/numerical answer matching).
 
@@ -19,7 +19,7 @@ PRKit centers on **core components** that define the physical reasoning ontology
 from prkit.datasets import DatasetHub
 from prkit.core.model_clients import create_model_client
 
-# Load any benchmark into the unified schema (PhysicsProblem, PhysicalDataset)
+# Load any benchmark into the unified schema (PhysicsProblem, PhysicsDataset)
 dataset = DatasetHub.load("physreason", variant="full", split="test")
 
 # Run inference with the unified model client (core component)
@@ -30,12 +30,45 @@ for problem in dataset[:3]:
 
 The same pattern works across different datasets and model providers—swap the dataset name or model identifier.
 
+#### Just verify an answer (`prkit.verify`)
+
+For the standalone "is this physics answer right?" use case, use the light-import
+verifier—a `math-verify`-shaped API that, unlike `math-verify`, is unit- and
+symbolic-aware and imports no model clients, dataset hub, or provider SDKs:
+
+```python
+from prkit.verify import verify
+
+v = verify("9.8 m/s^2", "9.8 m/s²")   # verify(gold, pred) -> Verdict
+v.correct          # True  — the unit suffix normalizes (math-verify strips units)
+v.units_ok         # True
+v.symbolic_equiv   # None  (numeric case); True for e.g. verify("v = a t", "v = t a")
+v.scorer_version   # stamped so a stored score is attributable to its scorer
+```
+
+#### Physics semantics (`prkit.semantics`)
+
+Underneath `verify` is the **physics-semantics** layer. It models a question's contract
+`q` and an answer's typed semantics `a`, and judges equivalence as a question-conditioned
+relation `Eq(a_pred, a_ref ; q)` — deterministically, not by string match. It exposes three
+build actions and two judge entry points, all importable from `prkit.semantics`:
+
+- `extract_prediction_answer_semantics(answer_text)` — deterministically type a prediction;
+- `create_reference_semantics(problem, model_client=None)` — build a reference `(q_ref, a_ref)`
+  (deterministic when `model_client` is omitted, LLM-assisted otherwise);
+- `generate_prediction_semantics(problem, solver_client, ...)` — solve, then type the answer;
+- `compare_protocol_answers(pred, ref, ...)` — reference-based judgement;
+- `compare_predictions(a_i, a_j, ...)` — reference-free (symmetric) judgement.
+
+See **[PHYSICS_SEMANTICS.md](docs/PHYSICS_SEMANTICS.md)** for the full story and doc map.
+
 ### 📖 Documentation
 
 **Quick Links:**
 - 🔧 **[CORE.md](docs/CORE.md)** - Core components: domain model, model client, logger, and definitions
 - 📚 **[DATASETS.md](docs/DATASETS.md)** - Complete guide to supported datasets and benchmarks
-- 📊 **[EVALUATION.md](docs/EVALUATION.md)** - Evaluation metrics and comparison strategies
+- 🧪 **[PHYSICS_SEMANTICS.md](docs/PHYSICS_SEMANTICS.md)** - Physics-semantics layer: `q`/`a`, the five build/judge steps, and the doc map
+- 📊 **[EVALUATION.md](docs/EVALUATION.md)** - The deterministic physics-semantics scorer (`verify` / `SemanticsScorer` → `Verdict`)
 - 🏷️ **[ANNOTATION.md](docs/ANNOTATION.md)** - Human annotation tasks (gold, correctness)
 - 📝 **[CHANGELOG.md](CHANGELOG.md)** - Version history and release notes
 
@@ -178,21 +211,25 @@ The toolkit is organized around **core components** and three subpackages that u
 The essential building blocks of the physical-reasoning-toolkit. All datasets, inference, evaluation, and annotation workflows use these components.
 
 * **PhysicsDomain** — Enumeration of physics subfields (mechanics, thermodynamics, quantum mechanics, optics, etc.) for problem classification. Aligned with UGPhysics, PHYBench, TPBench. Use `PhysicsDomain.from_string()` for flexible parsing.
-* **AnswerCategory** — Enumeration of answer types for normalization and evaluation: `NUMBER`, `PHYSICAL_QUANTITY`, `EQUATION`, `FORMULA`, `TEXT`, `OPTION`. Drives how answers are compared (numerical precision, symbolic equivalence, exact match).
-* **PhysicsProblem** — The canonical representation of a physics problem. Required: `problem_id`, `question`. Optional: `answer` (Answer), `solution`, `domain`, `image_path`, `problem_type` (MC/OE), `options`, `correct_option`. Supports dictionary-like access and `load_images()` for visual problems.
-* **Answer** — Unified answer model. `value` holds the number (NUMBER), numeric part (PHYSICAL_QUANTITY), option string (OPTION), or plain string (EQUATION, FORMULA, TEXT). `unit` is optional and used only for PHYSICAL_QUANTITY. Type checks, unit helpers, LaTeX handling, option indexing.
-* **PhysicalDataset** — Collection of `PhysicsProblem` instances. Indexing, slicing, `get_by_id()`, `filter_by_domain()`, `take()`, `sample()`, `save_to_json()` / `from_json()`. Provides `get_statistics()` for domain and problem-type distribution.
+* **PhysicsProblem** — The canonical representation of a physics problem. Required: `problem_id`, `question`. Optional: `answer` (PhysicsAnswer), `solution`, `domain`, `image_path`, `problem_type` (MC/OE), `options`, `correct_option`. Supports dictionary-like access and `load_images()` for visual problems.
+* **PhysicsAnswer** — Thin observation record: `value` (str, verbatim), optional `unit` (observed unit string), optional `source_type` (dataset-native type tag, verbatim), and `metadata` dict. The canonical answer kind (`AnswerObjectKind`, 9 object kinds) is derived on demand by the `prkit.semantics` layer — it is not stored on `PhysicsAnswer`.
+* **PhysicsDataset** — Collection of `PhysicsProblem` instances. Indexing, slicing, `get_by_id()`, `filter_by_domain()`, `take()`, `sample()`, `save_to_json()` / `from_json()`. Provides `get_statistics()` for domain and problem-type distribution.
 * **PhysicsSolution** — Bundles a `PhysicsProblem`, model `agent_answer`, and optional `intermediate_steps`. Captures the full solution trace for evaluation and analysis.
 * **BaseModelClient** — Abstract base for model clients. Subclasses implement `chat(user_prompt, image_paths=None)`.
 * **PRKitLogger** — Centralized logging with colored output, file logging, and env config (`PRKIT_LOG_LEVEL`, `PRKIT_LOG_FILE`, etc.).
 
 📖 See [CORE.md](docs/CORE.md) for the full domain model, entity relationships, subpackage dependency diagram, and import reference.
 
 
-### prkit.evaluation 📈
-Answer comparators (symbolic, numerical, textual, option-based), accuracy evaluator, and physics-focused assessment protocols.
+### prkit.scoring / prkit.verify 📈
+The deterministic physics-semantics scorer: `prkit.verify.verify` (light-import, one-call)
+and the `prkit.scoring` family — `SemanticsScorer` (binary), the `EedScorer`/`SeedScorer`
+edit-distance baselines, the graded `SemanticsEedScorer`/`SemanticsSeedScorer`, and the
+model-graded `LLMJudgeScorer` — all returning the canonical `Verdict`. Wraps the
+`prkit.semantics.comparison` engine. (The legacy `prkit.evaluation` comparator/evaluator
+stack is deprecated; `prkit.evaluation.llm_judge` stays.)
 
-📖 [EVALUATION.md](docs/EVALUATION.md)
+📖 [EVALUATION.md](docs/EVALUATION.md) · [PHYSICS_SEMANTICS.md](docs/PHYSICS_SEMANTICS.md)
 
 ### prkit.datasets 📊
 Dataset hub with a Datasets-like interface: `DatasetHub.load()` for PHYBench, PhysReason, UGPhysics, SeePhys, PhyX (plus JEEBench, TPBench loaders). Auto-download, variant selection, and reproducible sampling.

cookbooks/enrich_quantity_views.py

@@ -12,7 +12,7 @@
 from typing import Any
 
 from prkit.core import PRKitLogger
-from prkit.semantics.inference import (
+from prkit.semantics.build import (
     load_prediction_semantics_artifact,
     load_reference_semantics_artifact,
     load_semantics_artifact,

cookbooks/generate_reference_semantics.py

@@ -14,7 +14,7 @@
 from prkit.core import PRKitLogger
 from prkit.core.model_clients import create_model_client
 from prkit.datasets import DatasetHub
-from prkit.semantics import infer_reference_semantics, save_semantics_json
+from prkit.semantics import create_reference_semantics, save_semantics_json
 
 logger = PRKitLogger.get_logger(__name__)
 
@@ -83,9 +83,9 @@ def main() -> None:
 
         artifact_path = output_dir / f"{_safe_filename(problem.problem_id)}.json"
         try:
-            artifact = infer_reference_semantics(
+            artifact = create_reference_semantics(
                 problem,
-                client,
+                model_client=client,
                 max_output_tokens=args.max_output_tokens,
             )
             save_semantics_json(artifact, artifact_path)

cookbooks/physics_reasoning_with_semantics.py

@@ -14,7 +14,7 @@
 from prkit.core import PRKitLogger
 from prkit.core.model_clients import create_model_client
 from prkit.datasets import DatasetHub
-from prkit.semantics import infer_prediction_semantics, save_semantics_json
+from prkit.semantics import generate_prediction_semantics, save_semantics_json
 
 logger = PRKitLogger.get_logger(__name__)
 
@@ -83,7 +83,7 @@ def main() -> None:
 
         artifact_path = output_dir / f"{_safe_filename(problem.problem_id)}.json"
         try:
-            artifact = infer_prediction_semantics(
+            artifact = generate_prediction_semantics(
                 problem,
                 client,
                 max_output_tokens=args.max_output_tokens,