Physics-semantics evaluation layer and the 6-scorer architecture - #5
Merged
Conversation
…r-backed - New light-import prkit.verify (parse/verify) wrapping SemanticsScorer; an import-isolation test asserts no clients/hub/datasets/pandas on the path. - Enrich Verdict with correct/units_ok/symbolic_equiv/numeric_within_tol/ extracted_answer (derived losslessly in scoring/_adapt.py); partial_credit and rationale reserved as None. - AccuracyEvaluator gains an injectable scorer= (default SemanticsScorer) and routes evaluate() through it; the legacy comparator= path is preserved. - Broaden SemanticsScorer.score to accept PhysicsAnswerSemantics inputs. - Bump package to 0.2.0; document the facade and Verdict fields in CONTRACT.md and README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pre-commit ran both ruff-format and black, which format differently and ping-ponged files on every `pre-commit run --all-files`; CI enforces only black (`black --check`). Drop the ruff-format hook (keep ruff as the linter), bump the pinned black from 24.8.0 to 26.1.0 to match the version in the venv/CI, and exclude vendored minified katex assets from the end-of-file/whitespace hooks. Apply the resulting stable formatting to the docs and test file that were not yet clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The deterministic verifier is binary, so a near-miss answer scores the same as a completely wrong one. Add a graded scorer based on PHYBench Expression Edit Distance and CMPhysBench Scalable EED, run on PRKit's existing SymPy parser and unit backend (no pint / latex2sympy dependency). - New pure package prkit/semantics/edit_distance/: ExprNode + sympy_to_tree, an extended Zhang-Shasha tree-edit distance (forest matrix initialized to math.inf, fixing the upstream 1000-sentinel), a cost-aware subtree discount and eed_score, a thread-safe simplify timeout, and the SEED dispatch pipeline (eed_compare). - New PartialCreditScorer (Scorer protocol) maps the graded result onto Verdict, populating partial_credit and rationale; PARTIAL_CREDIT / BINARY / TOLERANCE modes. - verify(partial_credit=True) now routes to the scorer instead of raising. - CONTRACT.md and the conformance gate updated to cover the new scorer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Decide equality relations by their homogeneous form H = L - R: two
equalities are equivalent iff their denominator-cleared numerators agree up
to a nonzero constant. This admits cross-"=" rearrangement (F=ma vs a=F/m,
1/f=1/u+1/v vs f=uv/(u+v)) while rejecting equations with extra roots
(x=0 vs x*y=0). The per-operator criteria (_equalities_equivalent /
_inequalities_equivalent) are stated directly rather than as fallbacks after
a stricter check.
Fold functional-form notation into the canonical clause form so r(t)=... is
compared as r=..., and fix two parsing canonicalizations that previously
blocked symbolic matches: an "=" inside a summation limit (\sum_{n=1}^{N})
no longer splits relation clauses, and compact products carrying a subscript
expand (NmV_r -> N*m*V_r).
Add accept and adversarial-reject tests for each behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add EQUIVALENCE.md, a detailed reference for compare_protocol_answers: the pipeline, the per-object-kind criteria, the cross-kind bridges, the policy modes and numeric tolerance, with diagrams and a worked example for every condition. Add METHODOLOGY.md, the precision-preserving design discipline for changing the judgement (canonical forms and principled per-kind criteria, not fallback rescues). Point the answer-semantics enum taxonomy comment at the new reference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Carry per-symbol real-domain assumptions into SymPy parsing so expression/relation equivalence is decided over the intended real-physical domain instead of the generic complex default: - symbol assumptions: realness derived in-engine (precision-safe), positivity/nonnegativity declared via q.symbol_assumptions; threaded to Symbol(token, **assumptions). - de-radicalization: a solved even root (c = sqrt(E/m)) canonicalizes to its squared form when the non-radical side is provably nonnegative, falling into the existing exact equality criterion. - numeric identity testing: domain-honoring multi-point evaluation decides 'is a - b the zero function', rejecting on any disagreement (exact) and guarding assumption-empowered symbolic accepts. Also rename the previously dormant symbol_domains -> symbol_assumptions (enum SymbolAssumption, model PhysicsSymbolAssumptionSemantics, inner field assumption; coercion still accepts the legacy 'domain' key). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tomic)
Structure misclassification is an unrecoverable false negative (the structure-mismatch gate has no bridge), so reliably reach genuine atomic-vs-atomic before the atomic judgement:
- canonicalize_structure: a symmetric, idempotent, precision-safe pre-comparison pass that collapses structural degeneracies to atomic (1-element tuple/set/vector, closed point-interval [a,a], single-case trivially-conditioned piecewise), inserted last in _repair_answer_for_comparison.
- gate non-atomic comparison to proven-sound accepts only: ordered (tuple/interval/vector with atomic cells), piecewise, and label-aligned per-part are enabled; set/unordered accept only an exact multiset match (sound tolerant matching is a deferred roadmap milestone); matrix/tensor, per-part positional fallback, one-sided coordinate frame, and unparsed shaped payloads return a TBD sentinel (comparison_mode=not_implemented), or raise under STRICT_STRUCTURE_COMPARISON.
- contract admits ATOMIC whenever a collapsible structure is admitted, so a collapsed answer is not a spurious contract violation.
Adds STRUCTURE.md, structure decision guidance in the inference prompt role, a gold structure corpus + confusion harness, and per-comparator adversarial-reject batteries (incl. the {1.0,1.0} != {1.0,1.1} set-tolerance regression). Also carries the atomic-equivalence test battery (shared test file).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Spell out cardinality (was abbreviated 'Card.') and add a short explanation of what each signature column means — denotation (meaning, drives equivalence), cardinality (part count, drives length checks and degeneracy collapses), ordering, and surface evidence (textual cues, drives classification) — and why keeping denotation separate from surface evidence is the point. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…red, reference-free judgement Build the question/answer semantics the equivalence judgement consumes with the same precision discipline the engine enforces, and expose the three steps as independent capabilities. Reference creation: a staged builder produces q_ref + a_ref (problem + golden) and q_prob (problem only). The deterministic backbone (normalize_physics_answer + canonicalize_structure) is authoritative for structure/object_kind; advisory LLM calls run best-effort and only clean surfaces, fill question policy, and declare justified symbol assumptions, each cross-checked (round-trip, contract self-consistency, q_ref<->a_ref mutual consistency) with a provenance build report. Symbol assumptions are declared-not-derived and keyed by canonical post-alias tokens; numeric tolerance is relative (never absolute); allowed_* are widen-only. Replaces the single fused reference call (old body kept commented for review). Answer generation: problem-only isolated solve with a leakage guard, producing an LLM-structured record (a_pred_llm) and a deterministically-extracted one (a_pred_ext); it degrades to a_pred_ext alone when native structured output is unavailable. Equivalence judgement: add a symmetric reference-free compare_predictions(q_prob) that builds an explicit contract and requires agreement both directions, so neither prediction's printed precision nor kind/structure decides and no self-contract-violation arises; the reference-based path is unchanged. The verify/scoring facade gains an optional, duck-typed q_ref context so a caller can supply reference-built assumptions without the light facade importing the inference layer (backward compatible). The three steps are independent: the judgement core imports nothing from the build/generation layer at runtime, generation never calls the reference build, and each entry takes plain problem/semantics inputs. Methodology, the three-step ecosystem, and the engine-compatibility rules are documented in the comparison reference docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two directional answers can differ by a global sign because each was expressed under an opposite, unstated axis/sign-convention choice (a reference velocity -20 m/s right-as-positive vs a prediction +20 m/s left-as-positive describe the identical motion). These were false negatives. Add a convention-aware criterion that reconciles two *stated* conventions to a common frame. The convention is concrete per-answer data, not a toggle: a global flip is forgiven iff the question fixes no convention (q.sign_convention/q.coordinate_frame absent), both answers declare conventions whose positive axes are a provable global reversal (antonym directions), and the values are an exact global -1. Covers number / physical_quantity / sign_direction (atomic) and vector (component-wise) answers; reuses _proportional_ratio (already returns -1 for a negation) and numbers_close. New comparison/sign_convention.py holds the pure criteria; engine wiring runs it before the strict same-kind criterion (so it can also override a plain-equality false accept) and refines the _compare_shaped both-set- incompatible frame branch (opposite global-reversal frames reconcile, genuinely-incompatible frames still reject, partial flips reject). Registered as the audited sign_convention TIER2 bridge: on under audited, blocked under strict, with bridge_id/tier/evidence recorded. Precision is held two ways. A flip is forgiven only with both stated opposite conventions, so a bare intrinsic sign (charge -5 C vs +5 C) is never reconciled -- directional intent is declared, not derived. And the precision dual ships alongside the accept: opposite conventions with equal values denote physically opposite quantities and are rejected under every policy, avoiding an asymmetric relaxation. Also preserve answer-level coordinate_frame/sign_convention through _repair_atomic_answer's reparse so the lane can read them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record the implemented lane across the three comparison references: METHODOLOGY.md flips the section 4 recall-gap row to implemented and adds the discipline (convention as concrete per-answer data, the canonical- reframe lever, the admitted-class proof, and the precision dual) plus the section 5 residual. EQUIVALENCE.md documents the sign_direction resolution (7.5), the sign_convention bridge row and same-kind note (8), a policy example (9), the mode catalogue entry (11), and an end-to-end trace (13). STRUCTURE.md documents the refined _compare_shaped frame branch for vectors (both-unset / one-sided / opposite / incompatible) and adds the per-axis-frame residual. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on built data The committed sign-convention lane reconciles a global sign flip on a directional answer only when both answers carry stated, opposite conventions and the question fixes none. It was correct-but-dormant on built data: the build never populated answer-level conventions and routed any inferred convention onto q_ref, which closes the lane's gate. Split q vs a so each record carries its convention where the judge reads it: - a_ref: Call A declares the convention the golden is expressed in (free axis); sign_convention added to _ANSWER_SURFACE_FILL_FIELDS, adopted fill-only. - q_ref: Call B sets a convention only when the problem text itself fixes one. - a_pred: a solver-declared field for a_pred_llm; a conservative, declaration-only parser (_extract_sign_convention_declaration) for a_pred_ext that reads an explicit "<dir> as positive" clause, strips it before value parsing, and never guesses from a bare sign. reference_pair_consistency flags a q-fixed-vs-a_ref-opposite convention. Conventions are recorded in sign_convention at every capture point (coordinate_frame stays for a named frame) so the field-specific vector judge path cannot manufacture a spurious one-sided TBD. Prompt versions bumped v5->v6 and v4->v5; no schema change. Also make the prediction answer-semantics form a consumer choice rather than a provider-capability decision: infer_prediction_semantics gains answer_semantics — "structured" requires native structured output and raises otherwise, "extracted" uses the plain-text deterministic route, and "auto" (default) keeps the prior best-effort behavior. Tests: extended staged-build and isolated-prediction suites; new parser adversarial battery, built-records->verify integration cases (flip accepts; no-convention / precision-dual / q-fixed reject; vector both-sided accept + one-sided TBD), and an opt-in live smoke. Full gate green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
License metadata for the bundled benchmarks was duplicated as free-text strings in each loader's get_info() and each downloader's download_info, with no shared source of truth, no SPDX normalization, and several wrong/missing values (phybench, seephys, and jeebench mislabeled "Research use"; physreason a non-SPDX "CC BY-NC-SA / MIT"; ugphysics and tpbench missing the key entirely). PhysReason's load() built a literal info dict that dropped the license, so its two public read paths disagreed, and auto_download fetched any dataset with no license check. Introduce a typed LicenseSpec (SPDX id + machine-readable usage flags) and a registry (license_registry.get_license) that owns one spec per dataset. Loaders and downloaders now embed get_license(...).to_info_dict() at info["license"] plus an info["license_spdx"] shortcut, so every read path reports the same corrected license. Gate DatasetHub.load(auto_download=True) on the spec's redistributable flag (override with allow_nonredistributable=True) and warn on eval_only. Corrected values: phybench, jeebench, and physreason are MIT; seephys is Apache-2.0 (eval-only); ugphysics is CC-BY-NC-SA-4.0; tpbench stays LicenseRef-unknown pending upstream verification. Tests: new registry/gating/parity suite (every key resolves, unknown degrades to non-redistributable, corrected flags per dataset, loader<->downloader parity, the PhysReason load-vs-get_info regression, and the redistributable gate); the two phyx assertions read the dict/shortcut. Full gate green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the semantics.inference subpackage to semantics.build (the layer creates references and generates predictions, it does not only "infer"); keep semantics.inference as a package-level deprecation shim that re-exports from build and warns on import. Rename the action functions infer_reference_semantics -> create_reference_semantics and infer_prediction_semantics -> generate_prediction_semantics, keeping the old names as deprecated aliases. create_reference_semantics gains a deterministic mode: when model_client is None the three advisory LLM calls are skipped (None-guard in _advisory_inference) and the build records the existing *_call_unavailable flags, so it yields a (q_ref, a_ref) bundle with no LLM call. model_client stays positional-or-keyword so the alias keeps working for positional callers. Surface extract_prediction_answer_semantics on prkit.semantics, and remove the duplicate prkit.verify.parse (callers use the extractor instead). Anchor the ruff and git "build" excludes to the repo root (/build) so the new semantics/build package is linted and tracked rather than silently skipped. Repoint all internal test and cookbook imports to prkit.semantics.build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace docs/EVALUATION.md (which named deleted comparator/metric symbols) with a scorer-focused stub, and add docs/PHYSICS_SEMANTICS.md: a narrative on-ramp covering question/answer semantics, the five build/judge steps and their entry points, the judgement at a glance, and the canonical doc map. Add a physics-semantics section to the README and repoint the package overview at prkit.scoring / prkit.verify. Fix the stale code anchors in EQUIVALENCE.md, extend its comparison_mode catalogue (identical_text, not_implemented, and an asymmetric_match note), repoint METHODOLOGY.md / CONTRACT.md / the semantics README at the build package and the new function names, and drop parse from the contract examples. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move AnswerObjectKind/AnswerStructure (and the _StrEnum base) into prkit.core.domain as the toolkit's single canonical answer taxonomy, re-exported from prkit.semantics.schema for back-compat and promoted onto prkit.api. Add a 9th object kind, descriptive_text, for free-form answers, judged by conservative normalized-text equality (curated controlled-vocabulary phrases still classify as qualitative_label; sign/boolean detection now tolerates trailing punctuation). Retire the legacy AnswerCategory enum: Answer.answer_category becomes answer_kind: AnswerObjectKind (mapping NUMBER->number, EQUATION->relation, PHYSICAL_QUANTITY->physical_quantity, FORMULA->expression, OPTION->choice, TEXT->descriptive_text), with the loaders, conformance suite, physics_problem, and the llm_judge payload migrated and the serialized key renamed to answer_kind. Delete the deprecated evaluation comparator/evaluator/utils/similarities stacks (only llm_judge remains; the semantics layer is the canonical replacement). Breaking change: bump API_VERSION to 2.0 and record the removal in CONTRACT.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reduce Answer to a thin observed-data record (value/unit/source_type/ metadata) and keep the answer ontology (AnswerObjectKind/AnswerStructure) solely in the semantics layer; simplify the dataset loaders that built the heavier record. Mark the integration contract as provisional API_VERSION "1.0" — breaking changes are tracked in the delta doc rather than via a major bump — and drop the removed verify.parse from the facade docs. Split the front-end-free EED/SEED tree-edit core (tree/zss/score/timeout) out of semantics/edit_distance/ into evaluation/edit_distance/, leaving the SEED dispatch glue under semantics/ to keep the package graph acyclic, and narrow the import-isolation guard to prkit.evaluation.llm_judge. Refresh the docs and tests to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce prkit.scoring.LLMJudgeScorer, the one non-deterministic reference scorer. It adapts the OpenAIJudgeRunner (which stays in evaluation/llm_judge) into the canonical Verdict: a "correct" judgement maps to score 1.0 / equivalent True, the model's reasoning lands in rationale, and confidence / expected_answer_type / raw_response / verdict_type / model are recorded in details. partial_credit stays None (the judge is binary) and deterministic False is reported via get_info(). All judge imports are deferred to method bodies (with TYPE_CHECKING-only annotations) so that re-exporting the scorer keeps import prkit.scoring free of openai and prkit.evaluation.llm_judge — verified by the import-isolation test. Reframe the evaluation package docstring as the engine home the thin scoring wrappers adapt. Tests inject a FakeJudgeRunner (no network/client). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The legacy/ directory was removed, so excluding it from ruff is no longer needed. "/build", "dist", and "htmlcov" remain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add faithful, lightly-modified forks of the two upstream edit-distance scorers under evaluation/baselines/, each split into a front-end-free pure core and a LaTeX front-end, with the verbatim upstream LICENSE/NOTICE/PROVENANCE shipped alongside: - phybench_eed/ (github.com/phybench-official/phybench@706feb4, MIT) - cmphysbench_seed/ (github.com/CMPhysBench/CMPhysBench@b2cd857, Apache-2.0; attribution chain to PHYBench EED MIT + the zss BSD license preserved in NOTICE) Local modifications (documented per-file and in each PROVENANCE.md): lift the top-level latex front-end import out of the core so it imports without latex2sympy2_extended; make pint a lazy singleton in the SEED core; and replace the SIGALRM timeout_decorator bounds with the thread-safe prkit.evaluation.edit_distance.timeout.run_with_timeout. Importing either pure core pulls no latex2sympy2_extended/pint/timeout_decorator. Wire the baselines into the Scorer contract with two thin wrappers, EedScorer and SeedScorer, that lazily import the vendored core inside score() so import prkit.scoring stays free of pint and the LaTeX front-end. SeedScorer resolves its answer_type from the kwarg, then reference.source_type (validated against the SEED enum), then default Expression, with an opt-in classifier (default off) recorded in get_info(). Add the [baselines] extra (pint only) plus its all-aggregate entry, ship the vendored licenses as package data, and exclude the vendored tree from ruff/black/mypy and the whitespace pre-commit hooks so it stays auditable against upstream. Create the CMPhysBench loader (HF weidawang/CMPhysBench) mapping the curated answer_type column into Answer.source_type as one of the five SEED tokens, and register it in DatasetHub with an Apache-2.0 license-registry entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add pint to the import-isolation guard's forbidden list. The vendored CMPhysBench SEED baseline pulls pint only on its unit-aware Numeric path, and the EedScorer / SeedScorer wrappers import the vendored core lazily inside score(), so pint must never reach import prkit.scoring or the verify facade. latex2sympy2_extended and its antlr4 runtime are a required core dep already on this path by design and stay deliberately unforbidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire PRKit's own answer normalization to the vendored PHYBench-EED and CMPhysBench-SEED pure cores via a new semantics→pure-core adapter (scoring/_edit_distance_adapt.py), shipping SemanticsEedScorer and SemanticsSeedScorer. The adapter parses each answer with the semantics parser (never latex2sympy2) and the object_kind+structure → SEED-type map, then calls the cores' own tree-edit / numeric-tier algorithm verbatim, so a SemanticsEed/SeedScorer number stays directly comparable to its vendor Eed/SeedScorer baseline. The semantics path never loads pint. Reserve score == -1.0 as the not-applicable sentinel: Verdict accepts it (comparison_mode="not_applicable") for kinds/structures with no SEED type, check_scorer accepts it and skips the equivalence/identity expectations for it, and CONTRACT records that aggregators must exclude it. Remove PartialCreditScorer (and PartialCreditMode), which the two semantics edit-distance scorers supersede; verify(partial_credit=True) now uses SemanticsSeedScorer. The PHYBench/CMPhysBench reimplementations under evaluation/edit_distance and semantics/edit_distance are relabeled as parked reference (eed-reimpl / seed-reimpl), no longer wired to any Scorer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the consumer-repo-locating machinery in core/project_env.py (the PRKIT_TOOLKIT_ROOT env var, the src/prkit marker walk, and the named-sibling fallback) with a single nearest-ancestor-with-pyproject.toml lookup, so the toolkit resolves only its own project .env and never reaches into a sibling or consumer repository. Off-repo callers (e.g. an installed wheel with no pyproject.toml ancestor) get a best-effort empty result. find_toolkit_root is dropped from the public surface; load_project_dotenv and ensure_openai_api_key keep their signatures, so the model-client and llm-judge importers are unaffected. The tests are reworked to key off pyproject.toml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prkit/verify/ held only __init__.py, so flatten it to prkit/verify.py. The import path prkit.verify is unchanged, so no caller, test, or doc edits are needed; the light-import isolation guarantee and its test are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move core/domain/answer_kinds.py to answer_taxonomy.py (a clearer name for the canonical AnswerObjectKind/AnswerStructure taxonomy) and repoint its two importers (core/domain/__init__.py and semantics/schema/enums.py, plus the latter's :mod: docstring reference). Symbol names are unchanged, so every import-via-package site is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the two core domain classes Answer -> PhysicsAnswer and PhysicalDataset -> PhysicsDataset (the latter also resolves the physics_dataset.py file/class stem mismatch and completes the four-noun Physics* symmetry with PhysicsProblem/PhysicsSolution). The other domain nouns and the AnswerObjectKind/AnswerStructure taxonomy are unchanged. Edits target only files that reference the domain symbols as code; string and comment occurrences (mock-answer fixtures, "Answer:" prompt-section text, the docs' unrelated structured-output Answer(BaseModel) example, the PASEC prose, and historical release notes) are deliberately left untouched. The contract stays provisional at API_VERSION "1.0": the breaking rename is tracked in internal/PAPER_V1_TO_V2_DELTA.md rather than signalled by a major bump, and no deprecation alias is provided. CONTRACT.md, CHANGELOG.md, and the API docs (README, CORE, DATASETS, semantics/README) are updated accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two independent "green locally, red in CI" failures on this branch: - test_model_client_satisfies_protocol built a real OpenAI client, which the openai SDK validates a credential for at construction. The key came from the gitignored .env locally; CI has neither .env nor a secret, so construction raised OpenAIError. Inject a dummy OPENAI_API_KEY in the test so it runs offline anywhere. - mypy's Type check failed on Python 3.12 because a fresh install floated numpy to 2.5.0, whose bundled stub uses PEP 695 `type` statements that mypy rejects for a target python_version below 3.12 (we target 3.10). Skip following numpy stubs so dependency stub drift can't break the type gate. Prevent recurrence: pin the gate tools (black/ruff/mypy) to exact versions so a fresh CI install can't silently drift from local, add a `make ci` target that mirrors the workflow with no .env and no key, and fold the previously-missing typecheck into `make check`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First PR for the
feat/roadmapbranch (27 commits). It builds out PRKit's physics-aware evaluation and reshapes the toolkit's public contract around a clean, version-stamped scoring surface. The work falls into five themes:1. Physics-semantics judgement layer (PASEC)
A deterministic, question-conditioned equivalence engine — judges
Eq(a_pred, a_ref ; q)instead of string-matching.prkit.semantics, and reference-free (symmetric) judgement.simplify, a sign-convention reconciliation bridge, and an answer-structure decision layer.2. Scoring architecture — a 6-scorer family (this branch's headline)
All implement the frozen
prkit.api.Scorer→Verdictcontract:SemanticsScorer(deterministic binary),EedScorer/SeedScorer(faithful vendored PHYBench-EED / CMPhysBench-SEED baselines),SemanticsEedScorer/SemanticsSeedScorer(our-semantics front-end over the same pure cores), andLLMJudgeScorer(model-graded).latex2sympy2+heuristics; EED vs SEED).prkit.evaluation.baselines. New optional[baselines]extra (pinspint);import prkit.scoring/prkit.verifystay free ofpint/openai.Verdict.score == -1.0reserved as an honest not-applicable sentinel (excluded from aggregation).PartialCreditScorerremoved (superseded by the two semantics edit-distance scorers).3. Streamlining the domain + contract
PhysicsAnsweris now a thin observation record (value,unit,source_type,metadata) — the canonical answer ontology (AnswerObjectKind/AnswerStructure, 9 kinds) lives only in the semantics layer; legacyAnswerCategoryretired.API_VERSIONheld at provisional"1.0"; breaking changes are tracked, not signalled by a major bump.4. Domain & module renames (breaking, provisional 1.0)
Answer→PhysicsAnswer,PhysicalDataset→PhysicsDataset(four-nounPhysics*symmetry); ontology moduleanswer_kinds.py→answer_taxonomy.py. No deprecation alias.5. Infrastructure & hygiene
prkit.verifyfacade;prkitCLI; dataset license registry + gated auto-download; toolkit-independent.envloading (ownpyproject.tomlonly); pre-commit (ruff/black/mypy/pytest) convergent with CI;cmphysbenchdataset loader.Validation
ruff/black/mypyclean; import-isolation test green (pint/openaioff the scoring/verify path).Notes for review
CONTRACT.md/CHANGELOG.md; the contract stays provisional1.0.score == -1.0.🤖 Generated with Claude Code