Skip to content

test: compare the ported ConstraintSetParser against the oracle - #358

Merged
Lavmee merged 13 commits into
mainfrom
test/constraintset-differential
Aug 19, 2026
Merged

test: compare the ported ConstraintSetParser against the oracle#358
Lavmee merged 13 commits into
mainfrom
test/constraintset-differential

Conversation

@Lavmee

@Lavmee Lavmee commented Aug 19, 2026

Copy link
Copy Markdown
Owner

A fourth differential harness in :parity, comparing the ported Kotlin ConstraintSetParser against the vendored upstream Java copy over generated ConstraintSet JSON documents.

Before this, ConstraintSetParser — 2040 lines, the largest file in core/state and the hot path for every JSON-driven ConstraintLayout and MotionLayout use — was covered by one test file of 101 lines containing two tests that print to stdout and share a single assertNotNull between them.

Nothing under compose/src/ is touched.

Results

Differential comparison 2000 generated documents × 2 entry points, 0 divergences
Axis liveness 42 of 42, none ignored
:parity:test 126 tests across 20 classes, 0 skipped

How it works

A syntax-free ConstraintSetSpec describes a document; JsonEmitter renders it to DSL text; two subjects parse that identical string with their own package's classes, apply the resulting State to a container, and render the geometry plus every custom attribute. The differential test requires the two renderings to be equal. Same shape as the solver/ and motion/ harnesses.

The subjects drive populateState(CLParser.parse(json), state, LayoutVariables()) rather than the public parseJSON wrapper. The wrapper catches CLParsingException and prints it, leaving the State half-populated with no signal — faithful to upstream, which does the same, but blind as an observation point.

The part worth reading

Nine of the forty-two axes were dead on the first pass. Seven — biases, weights, chain style, centerVertically — were generated and emitted while the observed result did not depend on them at all. The cause turned out to be deeper than a missing measure() call: neither subject told State the document's root size, so the root defaulted to WRAP_CONTENT, which silently disables bias-centering and chain/MATCH_CONSTRAINT resolution. The other two were variables, which the parser resolves through layoutVariables[...] for every transform and bias attribute — the emitter simply never referenced one by name.

Without that check, the headline "zero divergences over 2000 documents" would have been reported over a corpus where a large slice of the generated variety never reached the code. This is the same failure the motion/ harness shipped a month ago with five dead axes, green the whole time.

The first fix landed one level short. setWidth/setHeight went into the new measure entry, but the differential test called only parse — so seven axes were certified live through a path the 2000-seed comparison never used. The test now runs both entries, comparing each against its own counterpart, as SolverDifferentialTest already does. Measured independently: parse and measure produce different geometry for 1862 of 1876 laid-out documents, so the second entry is real coverage rather than a copy.

The harness is shown able to fail before being trusted to pass. Making parseDimensionMode return createWrap() for "spread" turns the differential test red on 405 of 2000 seeds; reverting turns it green. The guards are derived from measurement, not guessed: minimumPopulated catches wholesale rejection, minimumGeometryRows catches wholesale vacuity — a corpus of {} documents would satisfy the first and fail the second.

Upstream defects found, and deliberately not fixed

Three, all present verbatim in the vendored Java, so the port is faithful and correcting any of them would create a divergence:

  • parseDesignElementsJSON indexes its inner loop with the outer loop's variable and labels every element with the literal string "Design" instead of its own id. Three separate agents reported this as a port defect during development; it is not.
  • parseColorString's KDoc promises -1 on unparseable input but throws NumberFormatException for #-prefixed values that are not six hex digits.
  • parseVariables reads numbers through CLNumber.getInt(), which does content().toInt() and throws on any fractional literal, inconsistently with getFloat() in the same class.

The generator deliberately draws inputs that reach the last two at a low rate, because a crash both sides produce identically is legitimate differential coverage of an error path.

What this does not cover

No generated document anchors a widget to a guideline, a barrier or a chain — the generator builds widgets first, so anchor targets can only name widgets. Guidelines and barriers do appear in the corpus and their own positions are compared, but the interaction they exist for is compared nowhere. This is recorded in the generator's KDoc so the coverage claim is not read as broader than it is.

parseMotionSceneJSON and parseTransitions are out of scope; they reach Transition.kt (1247 lines) and need their own scenario model. Natural next increment.

parseDesignElementsJSON's differential test necessarily passes, because the function is broken identically on both sides. Its own KDoc says so plainly: a pass there confirms the port matches upstream, not that the function does anything useful.

Lavmee added 13 commits August 19, 2026 16:00
… documents

minimumPopulated counted the Populated case, not what was in it: a generator
that regressed to emitting {} for every spec would have both sides return
Populated("", "") for all 2000 seeds, clearing the 1800 floor while comparing
nothing. Add a geometryRows accumulator and a 5000-row floor -- under half the
10647 measured against the current corpus -- which a corpus of substance-free
documents can never reach.
AxisLivenessTest mutates one axis of the generated document model at a time and
requires the rendered outcome to differ, so a field that varies without reaching
the parser (the failure mode a sibling harness shipped silently for weeks) shows
up immediately instead of hiding inside "everything is green".

alpha, the rotations, the scales, the translations and the pivots land on
WidgetFrame, a sibling of the solver's box, not on left/top/width/height -
GeometryRow widens to carry them, folded into the existing one-line-per-widget
row so ConstraintSetDifferentialTest's newline-counting geometry-row floor stays
meaningful unchanged.

Two independent findings came out of making every axis prove itself, both left
as @ignore'd tests rather than deleted, per this repo's own precedent
(NestedLayout's @ignore, pinned in NestedContainerTest):

- hBias, vBias, hRtlBias, centerVertically, chainStyle, hWeight and vWeight all
  reach ConstraintReference correctly, but never change the observed box: bias
  and chain-run resolution live in ConstraintWidgetContainer's dependency-graph
  analysis, which only runs via measure(...) with a real Measurer - neither
  subject calls it, confirmed by an isolated widget giving the identical
  position for hBias = 0.1 and hBias = 0.9. This is a harness gap, not a dead
  axis, and fixing it (teaching both subjects to measure()) is a bigger, riskier
  change than this task's scope.
- VariableSpec.Num and VariableSpec.Generator are stored into LayoutVariables by
  parseVariables and then never read again: nothing in this document model lets
  any field reference a variable by name, so both are genuinely dead as
  currently generated.

Full investigation in the task report.
Two rounds of fix. First round's @ignore diagnoses turned out to need a real
fix rather than acceptance, per the coordinator - both landed, so every axis
in AxisLivenessTest now passes for real: 42/42, zero skipped.

hBias/vBias/hRtlBias/centerVertically/chainStyle/hWeight/vWeight: added a
`measure` entry point to ConstraintSetSubject (mirroring solver.OracleSolver's
Measurer pattern) alongside the untouched `parse`. Bisecting outside the
harness (raw ConstraintWidget, then State directly, no JSON) found the actual
defect was never layout() vs measure() as first suspected - it was that
neither entry point ever told State the document's root size, so State
defaulted the root to WRAP_CONTENT. That default is invisible to a
single-anchor widget, which is why parse-backed cases were unaffected, but
it silently disables ConstraintWidget.applyConstraints' bias-centering
equation and the chain/MATCH_CONSTRAINT machinery, both of which special-case
an unresolved parent. Fixed by adding state.setWidth/setHeight to `measure`
only; `parse` is untouched (verified via git diff: zero removed lines in
either subject), so ConstraintSetDifferentialTest's corpus is unaffected.

variableNum/variableGenerator: added FloatValue (Literal | Named) and changed
WidgetSpec.alpha to carry one, so a document can write `alpha: 'v0'` instead
of a number - matching how ConstraintSetParser.applyAttribute actually reads
every transform/bias attribute. Scenarios.generate now draws its variables
before its widgets so a widget can reference one by name, rarely (1 in 20)
and only among variables the same document declares. Re-measured the
differential corpus after reordering the seed's draw sequence: 1876
populated / 10576 geometry rows, versus 1889/10647 before - a ~0.7% shift,
comfortably clear of both floors.

Full investigation, including the wrong first diagnosis and how it was
narrowed to the real one, is in the task report.
AxisLivenessTest certified hBias/vBias/hRtlBias/centerVertically/chainStyle/
hWeight/vWeight live through ConstraintSetSubject.measure, but the 2000-seed
differential test - the only thing that actually compares oracle against
port at scale - only ever called parse. A liveness certificate for a path
nothing checks for correctness said nothing about whether the port agrees
with the oracle on bias or chain resolution, which is exactly the region
Fix 1 just finished unlocking.

Rewrote ConstraintSetDifferentialTest following solver.SolverDifferentialTest's
Entry-enum structure: both parse and measure run for every seed, each
compared only against its own counterpart (never parse against measure, a
different contract), with per-entry populated/geometry-row floors and
per-entry-keyed divergence examples so one entry's failures can't crowd the
other's out of the report.

parse is untouched - this is the only file this fix needed to change.

Measured both entries before setting floors, per seed 1..2000: populated=1876,
geometryRows=10576, divergences=0 for BOTH parse and measure. The identical
counts aren't a mistake - row count depends only on how many
widgets/guidelines/barriers a document materialises, which parse and measure
agree on; they disagree (or, per this run, don't) on the numbers inside each
row. Zero divergences on measure is the first corpus-scale confirmation that
the port's bias, chain-style and chain-weight resolution actually agrees with
the oracle - previously only spot-checked one hand-built document at a time.

Full investigation in the task report.
parseDesignElementsJSON is broken identically on both sides (two
indexing bugs present verbatim in the vendored upstream Java and
reproduced line-for-line in the port), so the differential equality
check here is a real but weak signal: it proves the port matches
upstream, not that the entry point does anything useful. Floors are
calibrated against an instrumented 500-seed run and cover both
failure directions (generator decaying toward "nothing produces" or
"nothing throws"), documented in the class kdoc.
Closes out the review's cleanup wave on the constraintset differential
harness: strengthens the liveness check to also require the mutated
document lay out (so a leak/crash can't masquerade as proof an axis is
live), corrects four kdocs that stated something false about the
parser or the type they described, notes a real coverage gap in the
generator's anchor targeting, pins JsonEmitter's Bounded dimension
shape with a test, extracts the byte-identical observation loop
duplicated across both subjects, removes the one unforced asymmetry
between the two subject files, and puts the previously-dead
ConstraintSetSubject.name to use in an assertion message.

No production code changes; parity module only.
@Lavmee
Lavmee merged commit e8fd5ca into main Aug 19, 2026
2 checks passed
@Lavmee
Lavmee deleted the test/constraintset-differential branch August 19, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant