test(differential): add Rust-vs-Python differential harness and fix 15 divergences - #1
Closed
crowecawcaw wants to merge 4 commits into
Closed
test(differential): add Rust-vs-Python differential harness and fix 15 divergences#1crowecawcaw wants to merge 4 commits into
crowecawcaw wants to merge 4 commits into
Conversation
…5 divergences Adds `differential/`, a differential-testing harness that runs `openjd-expr` against the Python OpenJD reference implementation (mwiebe:openjd-model-for-python @ expr) and asserts they agree on the same input. It complements the fuzzer introduced in OpenJobDescription#279: the fuzzer catches inputs that crash Rust; this catches inputs where Rust silently computes a *different answer* than the reference — the silent-wrong-value class that report reviews consistently missed. # The harness - `differential/oracle/oracle_worker.py` — long-lived Python worker over a newline-JSON protocol. Interpreter + reference-import startup is paid once so a corpus of tens of thousands of cases fits well within a CI budget. Emits `openjd-expr`'s own `{"type","value"}` transport tag, so an `int` result is distinguishable from a `float` that displays the same (the `2**63` case). - Three-way equivalence over (rust, python): compare typed tags on (ok, ok); agree on (err, err) *without* comparing messages (Python and Rust word them differently); flag (ok, err)/(err, ok) and any Rust panic. Encoded in `src/equivalence.rs`. - `src/allowlist.rs` + `allowlist.json` — the intentional-divergence list, one entry (documented literal-preservation-through-max class) and no bug divergences. - `src/adapters.rs` — the `expr` adapter (evaluate against a symbol table under a path format). `Manifest` decode and path-mapping slot in as sibling adapters without touching the engine. - `src/generator.rs` — deterministic, grammar-aware expression generator biased toward the historically-buggy edges (i64::MIN, 2**63, negative counts, multibyte). Random bytes trivially agree on both sides; structurally-valid expressions exercise real evaluation paths. - Deterministic corpora at `corpus/conformance.jsonl` (common expressions) and `corpus/regressions.jsonl` (every past divergence, frozen). New divergences the generator finds are minimized and added. The crate is outside the root workspace (its own empty `[workspace]` table), same pattern as `fuzz/`, so it can't disturb the stable build/test/clippy/MSRV matrix. It runs in its own `.github/workflows/differential.yml` — a fast conformance+regression gate plus a count-boxed (~20k cases) generative sweep per PR (~3s), and a broader nightly multi-seed sweep that is non-gating. # Bugs the harness caught (all verified against the Python reference) Semantic — silent wrong value or wrong type: - `round(float, 0)`/`round(float, -k)` returned float; Python returns *int*. Rewrote `round_fn` to match Python's return-type contract and to preserve exact precision on `round(int, -k)` via integer arithmetic (`round_int_neg`), since the previous f64 round-trip lost precision for large ints. - `mod_float` (`9.2e18 % 0.1`) collapsed to `0.0` from `l - r * floor(l/r)` catastrophic cancellation. Now matches CPython's `float___mod__` exactly. - `floordiv_float` (`205 // 0.1`) returned `2050` from `(l/r).floor()`; Python gives `2049`. Now reproduces CPython's `float_divmod` algorithm. - Path/string `<` comparison swapped operands when the path was on the left (`path('/tmp/x') < 'ab'` returned the wrong bool). - `float_from_float` (Python's `_float_identity`) dropped the preserved literal, reformatting `float(1e308)` from `1e308` to `1e+308`. Overflow/panics: - `center`/`ljust`/`rjust` with negative width cast `-1 as usize` → wrap → op-limit or huge allocation. Now clamps to 0 like Python. - `path` join appended a bare trailing separator on empty right; on Windows, root-relative right with no drive/UNC left kept the left instead of replacing it (per ntpath.join). - `int ** neg` used `powi`, accumulating ~1 ulp of error (`956 ** -74` came out as `2.7932896460939224e-221` vs Python's `2.793289646093924e-221`). Switched to `powf`, matching CPython. Formatting: - `format_float` produced `e-7`, `e18` — Python/C use `e-07`, `e+18` (sign + min-2-digit exponent). The previous chained `.replace(...)` approach mangled single-digit exponents; parse exponent numerically and format explicitly. - `Float64::with_str` discarded any zero-valued original string that wasn't exactly `"0.0"`, so `round(0.0, 7)` came back `"0.0"` instead of `"0.0000000"`. Tightened to drop only negative-zero strings. # Test updates Three tests were asserting pre-Python-alignment semantics that this change corrects: - `round_ndigits_2_5_0`, `round_ndigits_3_5_0`, `round_ndigits_neg_2_5_0`, `round_ndigits_neg_3_5_0` — old expectations were `"2.0"`/`"-4.0"` (float); Python returns `"2"`/`"-4"` (int). - The three PR OpenJobDescription#279 round tests (`round_ndigits_zero_*`, `round_float_i64_min_ndigits`) encoded PR OpenJobDescription#279's float-return convention; updated to Python's int-return. - `join_posix_empty_right` asserted `"/a/b/"`; Python returns `"/a/b"`. # Known intentional divergence `max(9.2e18, 275)` — Rust preserves the original literal `9.2e18`, Python reformats to `9.2e+18`. Same f64 value. Documented in `allowlist.json`. Stacked on OpenJobDescription#279 (add-fuzzing); retargets to main automatically when that merges.
crowecawcaw
marked this pull request as draft
July 24, 2026 20:52
…erge Remove the scheduled multi-seed job and the `schedule` trigger. The per-run conformance + regression + count-boxed generative gate now runs identically on pull requests and on merges to main. Deeper multi-seed coverage is a documented local step (see differential/README.md) rather than a CI job.
Clarify in README.md and allowlist.json that Python is the OpenJD spec's *named reference implementation* (RFC 0005), not a normatively-infallible source: the spec says implementations "should" follow it "to the extent the language supports". The rule the harness enforces is "match Python except for deliberate, documented deviations in the allowlist" — not "Python is always right". Also fixes stale nightly-sweep wording in the allowlist comment and a rustfmt nit in math.rs from the earlier conflict resolution.
A fix-vs-allowlist review flagged the exact-integer round(int, -k) path as an allowlist candidate (only matters above 2^52, more involved than an f64 round-trip). Document that it was deliberately kept — reverting would re-introduce a divergence the differential generator produces in many wrapped forms, trading tested code for a recurring allowlist class — so a future reader doesn't re-litigate or "simplify" it away.
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
differential/, a harness that runsopenjd-expragainst the Python OpenJD reference (mwiebe:openjd-model-for-python@expr) on every input and asserts they agree. Complements the fuzzer: the fuzzer catches inputs that crash Rust; this catches inputs where Rust silently computes a different answer than the reference — the silent-wrong-value class human review consistently missed.Also fixes the 15 expr divergences the harness surfaced. Every fix is verified against the Python reference; the harness lands green.
The harness
differential/oracle/oracle_worker.py); one interpreter+import cost, then thousands of cases per second.openjd-expr's own{"type","value"}transport tag, sointresults are distinguishable fromfloatresults that display the same (the2**63case).src/equivalence.rs): compare typed tags on(ok, ok); agree on(err, err)without comparing messages (Python/Rust word errors differently — comparing wordings is a false-positive generator); flag(ok, err)/(err, ok)and any Rust panic.src/allowlist.rs+allowlist.json); currently one entry (literal-preservation throughmax), no bug divergences.corpus/conformance.jsonl(common expressions) andcorpus/regressions.jsonl(every past divergence, frozen so it can never silently regress).src/generator.rs) biased to historically-buggy edges (i64::MIN,2**63, negative counts, multibyte). Random bytes trivially agree on both sides; structured expressions exercise real evaluation paths.The crate is deliberately outside the root workspace (its own
[workspace]table), mirroring thefuzz/pattern, so the stable build/test/clippy/MSRV matrix is untouched. It runs in.github/workflows/differential.yml:schedule+workflow_dispatch): a deeper multi-seed generative sweep. Non-gating; surfaces new divergences to freeze into the regression corpus.Bugs the harness caught
Semantic (silent wrong value / wrong type):
round(float, 0)andround(float, -k)returned float; Python returns int. Rewroteround_fnto match Python's overload contract, and use exact integer arithmetic forround(int, -k)(new helperround_int_neg) — the previous f64 round-trip lost precision (e.g.round(4611686018427387904, -5)->...400192instead of Python's...400000).mod_float:9.2e18 % 0.1collapsed to0.0froml - r * floor(l/r)catastrophic cancellation. Now matches CPython'sfloat___mod__exactly.floordiv_float:205 // 0.1returned2050from(l/r).floor(); Python gives2049. Now reproduces CPython'sfloat_divmodalgorithm (fmod-based, sign correction,>0.5nudge).<comparison swapped operands when the path was on the left (path('/tmp/x') < 'ab'returned the wrong bool).float_from_float(Python's_float_identity) dropped the preserved literal, reformattingfloat(1e308)from1e308to1e+308.Overflow / panics not previously guarded:
center/ljust/rjustwith negative width cast-1 as usize-> wrap -> op-limit trip or huge allocation. Now clamps to 0 like Python.pathjoin appended a bare trailing separator on empty right; on Windows, a root-relative right with no drive/UNC left kept the left rather than replacing it (perntpath.join).int ** negusedpowi, accumulating ~1 ulp of error (956 ** -74->2.7932896460939224e-221vs Python's2.793289646093924e-221). Switched topowf, matching CPython.Formatting:
format_floatproducede-7,e18— Python/C usee-07,e+18(sign + min-2-digit exponent). Old chained.replace(...)mangled single-digit exponents; parse the exponent numerically and format explicitly.Float64::with_strdiscarded any zero-valued original string that wasn't exactly"0.0", soround(0.0, 7)came back"0.0"instead of"0.0000000". Tightened to drop only negative-zero strings.Test updates (all encode pre-Python-alignment expectations that this change corrects)
test_arithmetic::round_ndigits_{2_5_0,3_5_0,neg_2_5_0,neg_3_5_0}— old expectations were"2.0"/"-4.0"(float); Python returns"2"/"-4"(int).round_ndigits_zero_*,round_float_i64_min_ndigits) — asserted that PR's float-return convention; updated to Python's int-return.test_paths::join_posix_empty_right— asserted"/a/b/"; Python returns"/a/b".Every existing test in the workspace still passes (3,111 in
openjd-expralone).Known intentional divergence (allowlisted)
max(9.2e18, 275)— Rust preserves the original literal9.2e18; Python reformats to9.2e+18. Same f64 value. Documented indifferential/allowlist.jsonwith a written reason. Which literal-preservation policy is canonical throughmax/min/absis an open upstream design question.Test plan
cargo test --workspace --exclude openjd-for-js— all suites pass (verified locally).cargo clippy --all-features --all-targets --workspace -- -D warnings— clean.cargo fmt --all --check— clean.cd differential && cargo clippy --all-targets -- -D warnings && cargo fmt --check— clean..github/workflows/differential.ymlruns on the PR — conformance + regressions + 20k generative all green.