Skip to content

test(differential): add Rust-vs-Python differential harness and fix 15 divergences - #1

Closed
crowecawcaw wants to merge 4 commits into
add-fuzzingfrom
differential-testing
Closed

test(differential): add Rust-vs-Python differential harness and fix 15 divergences#1
crowecawcaw wants to merge 4 commits into
add-fuzzingfrom
differential-testing

Conversation

@crowecawcaw

Copy link
Copy Markdown
Owner

Stacked on the fuzzer PR (OpenJobDescription#279 in OpenJobDescription/openjd-rs) — this PR targets the add-fuzzing branch on my fork. GitHub retargets to main automatically when the fuzzer PR merges. Review after (or alongside) the fuzzer.

Summary

Adds differential/, a harness that runs openjd-expr against the Python OpenJD reference (mwiebe:openjd-model-for-python@expr) on every input and asserts they agree. Complements the fuzzer: the fuzzer catches inputs that crash Rust; this catches inputs where Rust silently computes a different answer than the reference — the silent-wrong-value class human review consistently missed.

Also fixes the 15 expr divergences the harness surfaced. Every fix is verified against the Python reference; the harness lands green.

The harness

  • Long-lived Python worker over newline-JSON (differential/oracle/oracle_worker.py); one interpreter+import cost, then thousands of cases per second.
  • Both sides emit openjd-expr's own {"type","value"} transport tag, so int results are distinguishable from float results that display the same (the 2**63 case).
  • Three-way equivalence (src/equivalence.rs): compare typed tags on (ok, ok); agree on (err, err) without comparing messages (Python/Rust word errors differently — comparing wordings is a false-positive generator); flag (ok, err)/(err, ok) and any Rust panic.
  • Reviewed intentional-divergence allowlist (src/allowlist.rs + allowlist.json); currently one entry (literal-preservation through max), no bug divergences.
  • Deterministic corpora: corpus/conformance.jsonl (common expressions) and corpus/regressions.jsonl (every past divergence, frozen so it can never silently regress).
  • Grammar-aware, seeded generator (src/generator.rs) biased to historically-buggy edges (i64::MIN, 2**63, negative counts, multibyte). Random bytes trivially agree on both sides; structured expressions exercise real evaluation paths.

The crate is deliberately outside the root workspace (its own [workspace] table), mirroring the fuzz/ pattern, so the stable build/test/clippy/MSRV matrix is untouched. It runs in .github/workflows/differential.yml:

  • Per PR / merge: conformance + regressions (fast gate) plus a count-boxed 20k-case generative run at the fixed default seed. Total step ~3s.
  • Nightly (schedule + workflow_dispatch): a deeper multi-seed generative sweep. Non-gating; surfaces new divergences to freeze into the regression corpus.

Bugs the harness caught

Semantic (silent wrong value / wrong type):

  • round(float, 0) and round(float, -k) returned float; Python returns int. Rewrote round_fn to match Python's overload contract, and use exact integer arithmetic for round(int, -k) (new helper round_int_neg) — the previous f64 round-trip lost precision (e.g. round(4611686018427387904, -5) -> ...400192 instead of Python's ...400000).
  • mod_float: 9.2e18 % 0.1 collapsed to 0.0 from l - r * floor(l/r) catastrophic cancellation. Now matches CPython's float___mod__ exactly.
  • floordiv_float: 205 // 0.1 returned 2050 from (l/r).floor(); Python gives 2049. Now reproduces CPython's float_divmod algorithm (fmod-based, sign correction, >0.5 nudge).
  • Path/string < comparison swapped operands when the path was on the left (path('/tmp/x') < 'ab' returned the wrong bool).
  • float_from_float (Python's _float_identity) dropped the preserved literal, reformatting float(1e308) from 1e308 to 1e+308.

Overflow / panics not previously guarded:

  • center / ljust / rjust with negative width cast -1 as usize -> wrap -> op-limit trip or huge allocation. Now clamps to 0 like Python.
  • path join appended a bare trailing separator on empty right; on Windows, a root-relative right with no drive/UNC left kept the left rather than replacing it (per ntpath.join).
  • int ** neg used powi, accumulating ~1 ulp of error (956 ** -74 -> 2.7932896460939224e-221 vs Python's 2.793289646093924e-221). Switched to powf, matching CPython.

Formatting:

  • format_float produced e-7, e18 — Python/C use e-07, e+18 (sign + min-2-digit exponent). Old chained .replace(...) mangled single-digit exponents; parse the exponent numerically and format explicitly.
  • Float64::with_str discarded any zero-valued original string that wasn't exactly "0.0", so round(0.0, 7) came back "0.0" instead of "0.0000000". Tightened to drop only negative-zero strings.

Test updates (all encode pre-Python-alignment expectations that this change corrects)

  • test_arithmetic::round_ndigits_{2_5_0,3_5_0,neg_2_5_0,neg_3_5_0} — old expectations were "2.0" / "-4.0" (float); Python returns "2" / "-4" (int).
  • Three round tests from the fuzzer PR (round_ndigits_zero_*, round_float_i64_min_ndigits) — asserted that PR's float-return convention; updated to Python's int-return.
  • test_paths::join_posix_empty_right — asserted "/a/b/"; Python returns "/a/b".

Every existing test in the workspace still passes (3,111 in openjd-expr alone).

Known intentional divergence (allowlisted)

max(9.2e18, 275) — Rust preserves the original literal 9.2e18; Python reformats to 9.2e+18. Same f64 value. Documented in differential/allowlist.json with a written reason. Which literal-preservation policy is canonical through max/min/abs is an open upstream design question.

Test plan

  • cargo test --workspace --exclude openjd-for-js — all suites pass (verified locally).
  • cargo clippy --all-features --all-targets --workspace -- -D warnings — clean.
  • cargo fmt --all --check — clean.
  • cd differential && cargo clippy --all-targets -- -D warnings && cargo fmt --check — clean.
  • .github/workflows/differential.yml runs on the PR — conformance + regressions + 20k generative all green.
  • Nightly sweep across 5 seeds x 15k cases surfaces only the one allowlisted class (verified locally).

…5 divergences

Adds `differential/`, a differential-testing harness that runs `openjd-expr`
against the Python OpenJD reference implementation (mwiebe:openjd-model-for-python
@ expr) and asserts they agree on the same input. It complements the fuzzer
introduced in OpenJobDescription#279: the fuzzer catches inputs that crash Rust; this catches
inputs where Rust silently computes a *different answer* than the reference —
the silent-wrong-value class that report reviews consistently missed.

# The harness

- `differential/oracle/oracle_worker.py` — long-lived Python worker over a
  newline-JSON protocol. Interpreter + reference-import startup is paid once
  so a corpus of tens of thousands of cases fits well within a CI budget.
  Emits `openjd-expr`'s own `{"type","value"}` transport tag, so an `int` result
  is distinguishable from a `float` that displays the same (the `2**63` case).
- Three-way equivalence over (rust, python): compare typed tags on (ok, ok);
  agree on (err, err) *without* comparing messages (Python and Rust word them
  differently); flag (ok, err)/(err, ok) and any Rust panic. Encoded in
  `src/equivalence.rs`.
- `src/allowlist.rs` + `allowlist.json` — the intentional-divergence list, one
  entry (documented literal-preservation-through-max class) and no bug
  divergences.
- `src/adapters.rs` — the `expr` adapter (evaluate against a symbol table under
  a path format). `Manifest` decode and path-mapping slot in as sibling
  adapters without touching the engine.
- `src/generator.rs` — deterministic, grammar-aware expression generator biased
  toward the historically-buggy edges (i64::MIN, 2**63, negative counts,
  multibyte). Random bytes trivially agree on both sides; structurally-valid
  expressions exercise real evaluation paths.
- Deterministic corpora at `corpus/conformance.jsonl` (common expressions) and
  `corpus/regressions.jsonl` (every past divergence, frozen). New divergences
  the generator finds are minimized and added.

The crate is outside the root workspace (its own empty `[workspace]` table),
same pattern as `fuzz/`, so it can't disturb the stable
build/test/clippy/MSRV matrix. It runs in its own
`.github/workflows/differential.yml` — a fast conformance+regression gate plus
a count-boxed (~20k cases) generative sweep per PR (~3s), and a broader nightly
multi-seed sweep that is non-gating.

# Bugs the harness caught (all verified against the Python reference)

Semantic — silent wrong value or wrong type:
- `round(float, 0)`/`round(float, -k)` returned float; Python returns *int*.
  Rewrote `round_fn` to match Python's return-type contract and to preserve
  exact precision on `round(int, -k)` via integer arithmetic (`round_int_neg`),
  since the previous f64 round-trip lost precision for large ints.
- `mod_float` (`9.2e18 % 0.1`) collapsed to `0.0` from `l - r * floor(l/r)`
  catastrophic cancellation. Now matches CPython's `float___mod__` exactly.
- `floordiv_float` (`205 // 0.1`) returned `2050` from `(l/r).floor()`; Python
  gives `2049`. Now reproduces CPython's `float_divmod` algorithm.
- Path/string `<` comparison swapped operands when the path was on the left
  (`path('/tmp/x') < 'ab'` returned the wrong bool).
- `float_from_float` (Python's `_float_identity`) dropped the preserved
  literal, reformatting `float(1e308)` from `1e308` to `1e+308`.

Overflow/panics:
- `center`/`ljust`/`rjust` with negative width cast `-1 as usize` → wrap →
  op-limit or huge allocation. Now clamps to 0 like Python.
- `path` join appended a bare trailing separator on empty right; on Windows,
  root-relative right with no drive/UNC left kept the left instead of
  replacing it (per ntpath.join).
- `int ** neg` used `powi`, accumulating ~1 ulp of error (`956 ** -74` came
  out as `2.7932896460939224e-221` vs Python's `2.793289646093924e-221`).
  Switched to `powf`, matching CPython.

Formatting:
- `format_float` produced `e-7`, `e18` — Python/C use `e-07`, `e+18` (sign +
  min-2-digit exponent). The previous chained `.replace(...)` approach mangled
  single-digit exponents; parse exponent numerically and format explicitly.
- `Float64::with_str` discarded any zero-valued original string that wasn't
  exactly `"0.0"`, so `round(0.0, 7)` came back `"0.0"` instead of
  `"0.0000000"`. Tightened to drop only negative-zero strings.

# Test updates

Three tests were asserting pre-Python-alignment semantics that this change
corrects:
- `round_ndigits_2_5_0`, `round_ndigits_3_5_0`, `round_ndigits_neg_2_5_0`,
  `round_ndigits_neg_3_5_0` — old expectations were `"2.0"`/`"-4.0"` (float);
  Python returns `"2"`/`"-4"` (int).
- The three PR OpenJobDescription#279 round tests (`round_ndigits_zero_*`,
  `round_float_i64_min_ndigits`) encoded PR OpenJobDescription#279's float-return convention;
  updated to Python's int-return.
- `join_posix_empty_right` asserted `"/a/b/"`; Python returns `"/a/b"`.

# Known intentional divergence

`max(9.2e18, 275)` — Rust preserves the original literal `9.2e18`, Python
reformats to `9.2e+18`. Same f64 value. Documented in `allowlist.json`.

Stacked on OpenJobDescription#279 (add-fuzzing); retargets to main automatically when that merges.
@crowecawcaw
crowecawcaw marked this pull request as draft July 24, 2026 20:52
…erge

Remove the scheduled multi-seed job and the `schedule` trigger. The per-run
conformance + regression + count-boxed generative gate now runs identically on
pull requests and on merges to main. Deeper multi-seed coverage is a documented
local step (see differential/README.md) rather than a CI job.
Clarify in README.md and allowlist.json that Python is the OpenJD spec's *named
reference implementation* (RFC 0005), not a normatively-infallible source: the
spec says implementations "should" follow it "to the extent the language
supports". The rule the harness enforces is "match Python except for deliberate,
documented deviations in the allowlist" — not "Python is always right". Also
fixes stale nightly-sweep wording in the allowlist comment and a rustfmt nit in
math.rs from the earlier conflict resolution.
A fix-vs-allowlist review flagged the exact-integer round(int, -k) path as an
allowlist candidate (only matters above 2^52, more involved than an f64
round-trip). Document that it was deliberately kept — reverting would
re-introduce a divergence the differential generator produces in many wrapped
forms, trading tested code for a recurring allowlist class — so a future reader
doesn't re-litigate or "simplify" it away.
@crowecawcaw

Copy link
Copy Markdown
Owner Author

Superseded by #2. Reworked to be self-contained against main: the expr fixes + unit tests ship in #2 (no fuzzer dependency, no external oracle in the package), and the differential-testing harness moved to the differential-tooling branch, linked from #2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant