Skip to content

CLI: check subcommand - cached-value calc oracle and XLSX round-trip fidelity on any corpus - #416

Merged
h-a-n-a-s-h-i merged 1 commit into
storytold:mainfrom
neilcawse:feat/oracle-check
Oct 11, 2026
Merged

h-a-n-a-s-h-i merged 1 commit into
storytold:mainfrom
neilcawse:feat/oracle-check

Conversation

@neilcawse

@neilcawse neilcawse commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

What changed

New gridcraft-cli check <file-or-dir>... [--json] [--tolerance 1e-9] [--max-report 20] (apps/gridcraft-cli/src/check.rs). It measures calculation correctness and file fidelity on any set of .xlsx/.xlsm files and writes nothing next to them.

  • Calc oracle. Excel stores its own result next to every formula. check reads each file and keeps those cached values, recalculates every formula with GridCraft, and compares the two. Numbers must agree within a relative tolerance (absolute when the magnitude is below 1). Errors must be the same kind. Text and logicals must match exactly.
    • Cells with no cached value (including fullCalcOnLoad files from libraries that don't calculate, such as xlsxwriter) are counted as no oracle. Volatile formulas, and everything downstream of them in the dependency graph, are counted as volatile. Neither is compared.
    • Each mismatch lists the cell, formula, expected and actual values, and the functions it calls. Mismatches are also summed by function, where primary means none of the cell's precedents mismatch, so that function is the likely culprit. They are also summed by type signature (e.g. number → #SPILL!), which tends to group one root cause.
  • Round trip. The recalculated workbook is saved with GridCraft's XLSX writer, then read back. The package is validated against the things Excel would "repair": missing content types, dangling relationship targets, duplicate relationship ids. Then sheets, values, formulas, per-cell number formats, merges, names, tables, conditional formats, validations, charts, comments, hyperlinks and pivots are compared with the original.
  • --json gives a machine-readable report for CI or scripts. Usage is documented in docs/cli.md.

Why

The parity docs count which functions exist, not whether they return what Excel returns. This gives a repeatable number from real Excel-saved files without needing Excel installed. It's clean-room: it only uses values Excel already wrote into the files.

Current main (ce0483f) on 40 Excel-saved files (20 calculation-heavy Apache POI test workbooks plus 20 assorted POI/Microsoft sample files):

formula cells: 11387 (11387 checked: 10742 matched, 645 mismatched; 0 no oracle; 0 volatile)
round trip: 2 of 40 files differ; categories (files): tables 2, values 1

Top primary culprits: TEXT 226 (number-format rendering), formulas with no function call 70, CONCATENATE 22, then a long tail. Most come from a few root causes, the biggest being legacy formulas opened without Excel's implicit @. The follow-up PRs listed below each fix part of this. On a local branch with all of them merged (on a slightly older main), the same run gave 3 mismatches, all VBA user-defined functions, and 0 round-trip differences.

Follow-up PRs, each independent of this one (they don't need the subcommand to build), with the mismatches they remove where measurable:

Tests

apps/gridcraft-cli/tests/cli.rs::check_exit_codes_and_json: a GridCraft-written workbook matches itself and round-trips (exit 0), the --json schema and totals, an unopenable file is reported and fails the run (exit 1), and usage errors exit 2.

Commands run:

cargo xtask ci    # all 7 steps passed
gridcraft-cli check <poi-calc-workbooks>/ <sample-files>/   # the 40 files above

Docs

  • docs/cli.md: new check section.

Not done

  • The corpus isn't added to the repo, because the craftrules test-corpora rules decide where corpora live. The POI files are Apache-2.0 and easy to fetch, and I can add a fetch script if you want one.
  • Not wired into cargo xtask ci. It could become a step that fails when the mismatch count goes up.

…fidelity on any corpus

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@echelon

echelon commented Oct 11, 2026

Copy link
Copy Markdown
Member

Merged — thank you for the contribution! If you haven't already, please join our Discord: https://discord.gg/artcraft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants