Skip to content

[ntuple] Reject incompatible late class schemas - #23684

Open
Elvand-Lie wants to merge 1 commit into
root-project:masterfrom
Elvand-Lie:fix/ntuple-union-late-class-schema
Open

Elvand-Lie wants to merge 1 commit into
root-project:masterfrom
Elvand-Lie:fix/ntuple-union-late-class-schema

Conversation

@Elvand-Lie

@Elvand-Lie Elvand-Lie commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

This Pull request:

Changes or fixes:

A Union merge can silently copy member pages to the wrong fields when a class first appears in a later source and its loaded dictionary differs from the file. For example, a file written with members fA,fC and merged with a dictionary containing fA,fB,fC can put fC data into fB.

Reconstruct and validate all new ordinary and projected fields before extending the destination model. Compare type, structure, versions, checksum and ordered child layout recursively. Report mismatches and ROOT field-creation errors through the existing Abort/Skip policy, so a skipped input leaves no partially added fields.

This deliberately requires the loaded dictionary to match the source schema and class version, including appended members, version-only bumps and streamer fields. It does not add schema conversion or transaction rollback for unrelated failures. Matching scalar, nested and projected late fields retain their values. The existing restriction on adding late streamer fields remains; matching common streamer fields are a readback control.

The schema safety issue predates and is separate from #23646.

This branch is rebased onto master after #23646 and the merger Strategy refactor #23576. It keeps the restoration of truncated and quantized field parameters from #23646, and passes the pre-validated late fields through RNTupleMergeStrategy::ExtendDestinationModel. It does not include the still-unmerged #23702. When #23702 lands, this change will need a follow-up adjustment to preserve its late-field unknown-type emulation and update the missing-dictionary rejection scenarios.

Validation

Validation applies to commit 15a039048d3b8a64b7526a59d38037408be226df, rebased onto master fbcf83fe9a58e65f36ba55bdc3821a7983f4799a. The RNTuple and RNTupleUtil test targets built successfully with GCC 13.3 at -j 3. The full evolution suite passed all 20 tests, including all 16 late-class scenarios, the full merger executable passed all 340 tests, and the emulated-field suite passed all 12 tests. With only the merger implementation reverted to master, RNTupleEvolution.UnionMergeLateClassSchema fails on the inserted-member scenario.

The CTest filter ntuple|hadd, excluding largefile|largevector|tutorial, passed all 51 selected tests at -j 2. Runs used ulimit -c 0. Large-file, large-vector and tutorial tests were not run.

[==========] 20 tests from 1 test suite ran. (21293 ms total)
[  PASSED  ] 20 tests.
[==========] 340 tests from 4 test suites ran. (38942 ms total)
[  PASSED  ] 340 tests.
[==========] 12 tests from 1 test suite ran. (1855 ms total)
[  PASSED  ] 12 tests.
100% tests passed, 0 tests failed out of 51
Total Test time (real) = 108.00 sec

Changed-line clang-format 18.1.8 and diff whitespace checks are clean. The original diagnostic registrations passed unchanged.

Compact test evidence

The source writer declares LateMergeClass { int fA = 11; int fC = 33; }. Ordinary cases Union-merge two one-entry files: the first has only e; the second adds before_record=77 and record={fA=11,fC=33}. A separate merging process uses the dictionary variation below. Abort is the default.

Input variation Expected Runtime assertions
Matching fA,fC Successful merge, missing record defaulted in the first entry Success; 2 entries, record values (0,0) then (11,33)
Inserted member: fA,fB,fC Reject incompatible schema Failed RResult with mismatch text; record and before_record absent from sink descriptor
Reordered members: fC,fA Reject incompatible schema Failed RResult with mismatch text; record and before_record absent
Vector of matching class, one item in the second file Preserve collection and values Success; vector sizes 0 then 1; item values (11,33)
Matching late alias projected from record present in both inputs Preserve record and alias Success; 2 entries; both record and alias (11,33) in both entries
Inserted member in nested or projected class Reject nested/projected mismatch Failed RResult with mismatch text; new record or alias, and before_record, absent
Inserted member with Skip, followed by valid third file containing before_record=99 Omit the bad second input and continue Success; warning emitted; 2 entries; no record; before_record values [0,99]
Missing usable dictionary with Abort Return field-creation error Failed RResult containing unknown type: LateMergeClass; record and before_record absent
Missing usable dictionary with Skip and the same valid third file Omit the bad input and continue Success; warning emitted; 2 entries; no record; before_record values [0,99]

This table shows representative scenarios. The 16 scenarios also cover appended/removed members, a changed member type, a class version-only bump, common matching streamer readback, and an evolved late streamer rejection. The runtime-assertions column lists fatal assertions checked by the test; the test does not print separate value dumps or full error reports for each scenario.

AI assistance: Codex (gpt-6.1-sol) assisted with implementation, rebase conflict resolution and validation, and Claude Code (opus) independently reviewed the work.

Checklist:

  • Tested changes locally.
  • Updated the Union-mode API documentation.

This PR fixes #23683

Late Union fields are built from the loaded dictionary, then their
member pages are paired by position. A changed dictionary can therefore
silently place source data in the wrong destination members.

Preflight every new field against its source schema before extending the
model. Route mismatches and ROOT field-creation errors through the
existing Abort/Skip policy. Require matching class metadata and ordered
member layouts rather than attempting schema conversion.

Cover matching, evolved, nested and projected classes, streamer fields,
and missing dictionaries, including continuing after a skipped source.

Assisted-by: codex:gpt-6.1-sol
Assisted-by: claude-code:opus
@Elvand-Lie
Elvand-Lie force-pushed the fix/ntuple-union-late-class-schema branch from 5d174ac to 15a0390 Compare October 18, 2026 10:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ntuple] Union merging can corrupt a late class field when its dictionary differs from the file

3 participants