Skip to content

calc: rebuild the dependency graph from shared syntax trees - #2693

Open
vangel76 wants to merge 3 commits into
storytold:mainfrom
vangel76:perf/calc-expr-arc-2685
Open

vangel76 wants to merge 3 commits into
storytold:mainfrom
vangel76:perf/calc-expr-arc-2685

Conversation

@vangel76

@vangel76 vangel76 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2685

What

Calc::rebuild and Calc::cells_changed used Formula::expr(), which deep-copies the whole syntax tree only to hand &Expr to precedents. They now call a new Formula::with_expr(|e| ...), which lends the cached tree to the closure, or, for a formula that only has its text (for example after loading a file), parses it and drops the tree afterwards. No copy, no Arc allocation and no reference count change. (The first version of this PR used expr_arc(); a reviewer's measurements showed it adds an Arc allocation on the text-only path, so it was replaced.)

Why

Every file open (recalc_all runs rebuild), every undo/redo and every structural edit rebuilds the graph, so each one paid for a full copy and drop of every formula's tree (about a fifth of a rebuild in the profile).

Numbers

Linux, release build, one pinned core (taskset), 100,000 formulas in column C. Short is =A{r}+B{r}, complex is =IF(A{r}>B{r},A{r}*B{r}+ROUND(A{r}/3,2),MAX(A{r},B{r})). perf stat -e instructions:u, difference of an 11-rebuild and a 1-rebuild run divided by 10 (millions of instructions per rebuild). "Text only" means Formula.expr = None. Single edit: parse and store one complex formula plus cells_changed, thousand instructions per edit. Instruction counts are repeatable; wall times on this shared machine were too noisy to quote.

Case PR parent upstream/main main + this PR
Rebuild, short, cached tree 408.5 408.2 348.8 (-15%)
Rebuild, short, text only 931.7 933.8 934.3 (+0.1%)
Rebuild, complex, cached tree 1088.3 1088.9 749.0 (-31%)
Rebuild, complex, text only 3421.7 3427.2 3438.5 (+0.3%)
Single complex edit (k instr) 105.6 97.4 93.4 (-4%)

The text-only rows are dominated by parsing (the parent-to-main drift is of the same size as the difference), so they are unchanged within noise. The earlier head of this PR (expr_arc()) measured 940.3 / 3430.1 on the parent base, that is +0.9% / +0.25% on those rows, from the extra Arc allocation per parsed formula.

How verified

The graph is built from the same tree, so it is unchanged. New test graph_built_from_text_only_formulas_matches_cached builds the same workbook with cached trees and with text-only formulas and checks identical values after rebuild, an input edit, and a cells_changed of a text-only formula. cargo test -p gridcraft-calc (91 pass), cargo fmt --all --check, and cargo clippy -p gridcraft-calc -p gridcraft-model --lib --tests -- -D warnings are clean.

I left the other rebuild cost mentioned in the issue (a HashSet per precedent cell in Graph::cell_deps) alone to keep this change minimal.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved dependency updates during recalculation and cell edits for formulas whose expressions are parsed on demand, helping dependent formulas stay in sync.
    • Added coverage for formula parsing failures and recalculation after edits.

Calc::rebuild and Calc::cells_changed called Formula::expr(), which deep-copies each formula's
syntax tree only to read its references. Use expr_arc() (already used when formulas are
calculated), which hands out the cached Arc without copying.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between c567511 and b6bbdcd.

Files selected for processing: 3
  • crates/calc/src/recalc.rs
  • crates/calc/src/tests.rs
  • crates/model/src/cell.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

Walkthrough

Formula::with_expr provides cached or parsed expressions to callbacks without copying the expression tree. Calc::rebuild and Calc::cells_changed use it to compute precedents. A regression test compares dependency graph results for cached and text-only formulas.

Changes: Formula expression sharing

Layer / File(s) Summary
Provide expressions through callbacks
crates/model/src/cell.rs
Formula::with_expr invokes a callback with a cached expression or parses the formula text first. It returns None if parsing fails.
Build precedents through the callback
crates/calc/src/recalc.rs, crates/calc/src/tests.rs
Calc::rebuild and Calc::cells_changed compute precedents inside with_expr callbacks. The regression test compares results for cached and text-only formulas, including parse failure and updates after edits.

Priority: Low

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Refactor · Severity of issue fixed: Low

Possibly related issues

Suggested reviewers: echelon

Merge Risk: Minimal · up to b6bbd

No merge-blocking risk was identified; normal checks remain appropriate.

Security architecture risk: Minimal · up to b6bbd

The change avoids copying formula trees without expanding formula privileges or changing dependency resolution. Cached expressions, text-only formulas and parse failures retain their previous behavior. No material security risk was identified in the changed contract.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The changed behavior is confined to expression access during dependency tracking for the supplied workbook. References still resolve through existing workbook sheet selection; the change does not introduce a new destination or authority for formula-controlled references.

Trust Boundaries and Controls

  • observed — Formula text remains input to the same parser and precedent analyzer, not a source of callback code. The program supplies the callback, which receives only an immutable expression borrow. Evaluation continues through the unchanged expr_arc path.

Resilience and Maintainability Implications

  • observed — Precedent analysis still completes before node insertion. Rebuild retains full-reset recovery, while cells_changed retains replacement/removal, reverse-index cleanup, spill cleanup and dirty-dependent recalculation. Exclusive mutable access and unchanged transition sequencing provide no new concurrency or interruption window attributable to this PR; parse-failure behavior also predates it.
Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: Calc::rebuild now builds the dependency graph from shared syntax trees via Formula::with_expr instead of cloning trees with expr(). It also matches the PR objective of…
Linked Issues check Passed Issue #2685 requires removal of formula-tree deep copies in Calc::rebuild and Calc::cells_changed. The PR uses Formula::with_expr at both call sites. with_expr borrows cached Expr values and…
Out of Scope Changes check Passed The changes stay within issue #2685. The Formula::with_expr helper enables the required no-copy behavior, and the graph-equivalence test supports that change. No unrelated production behavior or unr…
Full details: Title check

Explanation

The title names the main change: Calc::rebuild now builds the dependency graph from shared syntax trees via Formula::with_expr instead of cloning trees with expr(). It also matches the PR objective of avoiding a tree copy and Arc allocation. The title is short, specific, and in one sentence.

Full details: Linked Issues check

Explanation

Issue #2685 requires removal of formula-tree deep copies in Calc::rebuild and Calc::cells_changed. The PR uses Formula::with_expr at both call sites. with_expr borrows cached Expr values and parses text-only formulas without cloning or allocating an Arc. The new regression test compares cached and text-only formulas through rebuild, edits, and cells_changed. The optional Graph::cell_deps optimization is not required.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue #2685. The Formula::with_expr helper enables the required no-copy behavior, and the graph-equivalence test supports that change. No unrelated production behavior or unrelated tests appear in the reviewed diff.

Full details: Docstring Coverage

Explanation

Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 3 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@vangel76

Copy link
Copy Markdown
Contributor Author

This PR is one of five speed-ups from an audit of the functions and calc crates (issues #2679 and #2681 to #2685). Each one changes a different place; all are based on current main and none needs another to merge first.

PR Fixes Change Measured (one pinned core, perf stat -e instructions:u unless noted)
#2686 #2679 MEDIAN, PERCENTILE, QUARTILE, LARGE, SMALL select instead of fully sorting MEDIAN of 1M 0.34 G to 0.054 G (6x); LARGE(1M,1) 0.34 G to 0.034 G (10x); 300 x LARGE(100k,k) 8.75 G to 1.26 G (6.9x)
#2691 #2681 per-pass memo of (sum, count) for constant ranges; longer ranges extend shorter ones 20k rows of =A/SUM(A$1:A$20000) 20.2 s to 19 ms; 20k running totals 8.9 s to 16 ms; 1000 x SUM over 50k rows 2.0 s to 2.1 ms
#2693 #2685 rebuild and cells_changed share the formula AST instead of cloning it rebuild of 100k IF formulas 1086 M to 747 M (-31 %)
#2694 #2683 spatial index for spill ranges 20k MAX(A:B) rows with 2000 spills 1160 ms to 77 ms; no-spill control unchanged
#2696 #2684 set of #SPILL! cells instead of walking all graph nodes and a 64x64 window clearing one cell with 100k formulas 10 ms to 0; 20k-cell paste 20 ms to 10.5 ms

Results are identical in every case. #2686 compares against the old full-sort code bit for bit (lengths 0 to 4096, ties, -0.0, NaN, inf). #2691 compares against a plain fold, bit for bit, over full recalculations and 30 edit rounds and against the unmodified build on random workbooks with errors, text, spills and RANDBETWEEN. #2694 and #2696 compare byte-identical dumps of all values, spill and spill_ranges against the old build on about 2,000 and 1,500 random workbooks with overlapping and blocked spills, iterative calculation on and off, and edit batches of 1 to 76 cells. The measurements use synthetic workbooks.

Open PRs that touch the same code and will need a small merge: #2319 and #2252 (recalc.rs ranges, #2691), #1954, #2319 and #2483 (spills, #2694), #2653, #2639 and #2637 (rebuild, #2693). #2694 and #2696 only collide in the tests appended to tests.rs. The lookup copies (#2678) wait until #2036, #2037 and #2409 are merged.

🤖 Generated with Claude Code

@FredApps FredApps left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed both call-site changes together with Formula::expr, Formula::expr_arc, precedents and graph update/removal. The cached-tree and deserialized/no-cache branches obtain the same Expr; the graph visitor borrows it and does not mutate it. Invalid formulas still return None and follow the existing removal/error handling. I found no new concrete correctness defect or dependency on the held-back lookup/name optimizations.

CI evidence: inspected the successful gate job log: 2,524 tests passed, one timing-sensitive calc scaling test skipped; all ten gate steps passed, including fmt, Clippy and wasm. I have not represented these as local tests.

The 31% instruction reduction remains an author benchmark for rebuilding 100,000 relatively complex IF formulas. It targets avoided AST allocation/clone/drop work; it does not establish the same benefit for short formulas, deserialized formulas without cached trees, or a full edit/recalc operation. I did not reproduce the timing or instruction counts.

Local Cargo validation was blocked before compilation: the pinned egui source was unavailable offline, and the isolated crate build could not download missing crates because the TLS client credentials were unavailable. No security settings were changed. This is a COMMENT because local integration validation and independent performance reproduction remain incomplete.

@Harrison-Booth

Copy link
Copy Markdown
Contributor

I measured this on one Windows machine with a temporary release-mode gridcraft-calc harness: 100,000 formulas, 9 rebuild/edit samples per case, reporting medians. Rebuild rows time only Calc::rebuild; “no cached AST” sets Formula.expr = None to simulate the deserialized state (I did not round-trip XLSX). The edit row parses/stores one replacement formula and calls cells_changed in the populated workbook.

Case upstream main PR head PR parent → head
Rebuild, short, cached AST 80.2 ms 100.3 ms 117.4 → 100.3 ms (−15%)
Rebuild, short, no cached AST 120.3 ms 197.4 ms 179.7 → 197.4 ms (+10%)
Rebuild, complex, cached AST 837.3 ms 433.9 ms 628.6 → 433.9 ms (−31%)
Rebuild, complex, no cached AST 1,763 ms 1,800 ms 1,479 → 1,800 ms (+22%)
Single complex formula edit 34 µs 59 µs 27 → 59 µs

The direct parent comparison supports a speedup for cached complex formulas: −31% wall time here, versus the reported −18%; short and uncached cases did not improve. The current main is newer than the PR parent, so main→head includes other code changes and is not an isolated estimate of this diff. The single-edit numbers are at the noise floor. I did not measure instruction counts or peak allocations, and did not run Linux Rust builds.

vangel76 and others added 2 commits October 29, 2026 00:48
…nly formulas

Formula::with_expr runs a closure on the cached tree, or on a freshly parsed
one that is dropped afterwards, so rebuild and cells_changed need neither a
tree copy nor an Arc allocation / reference count.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vangel76

Copy link
Copy Markdown
Contributor Author

Thanks for measuring this, it was useful.

I repeated it on Linux with a release harness, 100,000 formulas, pinned to one core, counting instructions (perf stat -e instructions:u) for three builds: the PR parent, upstream/main and the PR head. Instruction counts per rebuild, in millions:

Case parent main old head
short, cached 408.5 408.2 347.5 (-15%)
short, no cached AST 931.7 933.8 940.3 (+0.9%)
complex, cached 1088.3 1088.9 746.6 (-31%)
complex, no cached AST 3421.7 3427.2 3430.1 (+0.25%)
single complex edit (thousand instr) 105.6 97.4 101.6

What I confirmed: the cached-AST gains match yours (-15% short, -31% complex), and the old PR text said -18% for a different formula and with wall time, so I corrected it. The uncached path did get slightly worse: expr_arc() wraps every freshly parsed tree in a new Arc, an extra allocation per formula. On Linux that is small (under 1%, because parsing dominates), so your +10% and +22% wall time is larger than what I see in instructions. I assume the Windows allocator makes that extra allocation more expensive, but I did not test Windows. The single edit did not regress in instructions (101.6k vs 105.6k on the parent); I could not reproduce the 27 to 59 us jump, and at that size it is likely noise.

What changed: I pushed b6bbdcd. rebuild and cells_changed now use a new Formula::with_expr, which lends the cached tree to a closure, or parses a text-only formula and drops it afterwards, with no copy, no Arc and no refcount. I also merged current main. On top of main the result is: short cached 348.8 (-15%), short text-only 934.3 (+0.1%), complex cached 749.0 (-31%), complex text-only 3438.5 (+0.3%), edit 93.4k (-4%). The last two text-only rows are within code-layout noise of main. I added a test comparing results for cached and text-only formulas. The PR description has the full table. If you can rerun your Windows harness on the new head, I would like to see the uncached rows.

@FredApps FredApps left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at b6bbdcd (previously commented at c567511).

Since then: a merge of main and one commit (b6bbdcd) that replaces the expr_arc use with a new Formula::with_expr. I read the whole current diff (3 files).

Checked:

  • with_expr runs the closure on the cached tree when there is one, and otherwise on a tree parsed from the text that is dropped afterwards; it returns None when the text does not parse, so an invalid formula still takes the same removal path in cells_changed and is skipped in rebuild, as it did through expr().
  • Both call sites borrow the workbook immutably inside the closure (precedents(wb, si, e) only reads), so the new borrow shape is fine and nothing mutates the tree. A text-only formula no longer costs an Arc allocation, which is what the author's no-cache rows were about.
  • The new test builds the same workbook with cached and with text-only formulas (including one that does not parse), compares the values before and after an edit, and then adds a text-only formula through cells_changed and checks it joins the graph (F1 becomes 107 when A1 is set to 7).

No findings. I did not run the tests or the measurements; the instruction counts are the author's. Approval still needs a run on the current head and target.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perf: rebuilding the dependency graph deep-copies every formula's syntax tree

3 participants