Perf: improve edit session performance on large meshes (3-step plan)

[pcolt]

Overview

Edit session performance on large meshes has several bottlenecks across session begin, per-pick interactions, and commit. This issue tracks a progressive improvement plan — each step is independent and leaves the existing test suite green.

The performance work can be done directly in paraview_backend.py without first extracting backend/selection.py. The selection methods have too many cross-cutting dependencies on backend state (self.source, self.view, self.simple, self.render(), etc.) to benefit from a preventive extraction. The backend refactor can proceed in parallel starting from more self-contained modules (display, coloring, pipeline); selection follows after those have clean interfaces.

See docs/logic_flows/ for detailed flow diagrams.

Data flow and bottlenecks at a glance

─── SESSION BEGIN ───────────────────────────────────────────────────────────
  ParaView pipeline (server-side)
    source proxy  →  UpdatePipeline()
         │
         │  servermanager.Fetch(source)   ⚠ B1: O(mesh), collapses MPI ranks
         ▼
  vtkUnstructuredGrid (Python)
         │
         │  DeepCopy()                    ⚠ B2: O(mesh), second full copy
         ▼
  edit_session.working_dataset

─── DURING SESSION (every pick) ────────────────────────────────────────────

  ParaView side (proxy)            Edit session side (working_dataset)
  ─────────────────────            ────────────────────────────────────
  renders source in viewport       holds all field mutations
  SelectSurfaceCells() fires       tracks selected cell/surface/point IDs

       cell IDs from pick
              │
              │  Fetch(source)            ⚠ B3: O(mesh), every pick
              │  + rebuild coord index    ⚠ B4: O(n_cells) Python, every pick
              │  + coordinate remap
              ▼
       IDs on working_dataset ──────────► edit_session.add/replace_selection()
                                                   │
                                          overlay rebuild  ⚠ B6: every pick

  (surface mode only)
       GeometryFilter pick
              │
              │  coordinate lookup        ⚠ B5:  O(n_source_points), every pick
              │  + boundary map rebuild   ⚠ B5b: O(n_cells×faces),   every pick
              ▼
       boundary face keys ───────────────► edit_session

─── COMMIT ──────────────────────────────────────────────────────────────────
  working_dataset
       │
       │  vtkXMLUnstructuredGridWriter    ⚠ B7: O(mesh) I/O, full file always
       ▼
  output .vtu  →  load as new pipeline node

Current bottlenecks

Reference mesh for timing: test_data/hyper_sphere-3D-6ref_clipped.vtu (~5.6 M cells). AMD Ryzen 7 8845HS, ~25 GB RAM, Ubuntu 24.04.4 LTS, single MPI rank. See benchmark comment for full before/after tables.

ID	Where	Cost	Frequency	~Time (s)	Status
B1	Fetch(source) at session begin	O(mesh), collapses MPI ranks	once	~0.029 s (localhost; >> on MPI)	open
B2	DeepCopy at session begin	O(mesh), second full in-memory copy	once	~1.4 s	open
B3	Fetch(source) inside remap	O(mesh) transfer	every pick	~0.025 s → eliminated	✅ PR #35
B4	Coordinate index rebuild inside remap	O(n_source_cells) Python loop	every pick	~5 s (surface) / ~35 s (volume) → eliminated	✅ PR #35
B5	Coordinate lookup in _surface_keys_from_selected_dataset	O(n_source_points)	every pick, surface mode only	~27–32 s → ~0.07 s (−99.7%)	✅ PR #37
B5b	_boundary_codim_elements rebuild in _normalize_surface_keys_to_source_boundary	O(n_source_cells × faces_per_cell)	every pick, surface mode only	included in B5 baseline → session-once (per-pick cost eliminated)	✅ PR #37
B6	Overlay rebuild	O(selected_cells)	every pick	TBD	open
B7	Full mesh write + reload at commit	O(mesh) I/O	once	TBD	open
B8	Adjacency graph build in _ensure_volume_adjacency / _ensure_surface_adjacency	O(n_cells × faces) Python→C++ calls	first grow per session (cached after)	TBD	open
B9	_surface_boundary_map_for_top_cells() lazy build + uncached _top_dimension() in EditSession	O(n_cells × faces) Python→C++ calls	first surface pick per session (boundary map); every surface pick (_top_dimension)	~20 s (1st pick) / ~2 s (subsequent)	open
B10	top_dimension scan in _pick_surface_keys_native (backend)	O(n_cells) Python→C++ loop	every surface pick	~2 s	open
B11	Re-entry overhead after commit — full session begin (Fetch + DeepCopy + cache builds) paid again on every subsequent field assignment	O(mesh)	once per field assignment beyond the first	~2.5 s (baseline) / ~31.6 s (after PR #37)	open (#39)

B3 and B4 are the most impactful because they repeat on every user interaction during a session. The eager build of the B4 cache at session begin (PR #35) moves that cost to session start — measurably slower on large meshes. Lazy initialization (build on first pick instead) would restore fast session open.

---

Step 1 — Quick wins, no architectural change

Fixes: B3, B4 + pipeline lock correctness bug; B5 investigated — both PassThrough variants ruled out in PV 6.1

✅ Cache the remap (B3, B4) — done in PR #35

_remap_cell_ids_to_edit_target_dataset called Fetch(source) and rebuilt the coordinate index on every single pick. Both are now computed once at begin() and reused for the entire session — world-space coordinates do not change when the user rotates the camera (view transform, not mesh transform).

Known trade-off: the cache is built eagerly at begin(), which makes session start measurably slower on large meshes (confirmed in issue #34 comments). Lazy initialization (defer to first pick) would recover that cost without changing the per-pick result.

❌ PassThroughCellIds on GeometryFilter (B5)

PR #31 ruled out PassThroughPointIds (vtkOriginalPointIds produces an identity mapping through the proxy/Fetch round-trip in PV 6.1).

❌ Ruled out — same failure mode as PassThroughPointIds. PassThroughCellIds was the natural next candidate: for a 3D mesh with N source cells and M boundary faces, if the array carried the source 3-D cell ID (not the output face index 0..M-1), the surface-key mapping could become O(selected_cells) with no source-point index build.
On cube.vtk (64 hex cells, ~96 boundary quads), vtkOriginalCellIds reported value 68 for a selected face, but n_source_cells = 64 (68 ≥ 64). In PV 6.1 the array carries the face's sequential index in the GeometryFilter's PolyData output (0..M-1), not the source 3-D cell ID. Both PassThrough variants produce the wrong reference space through the proxy/Fetch round-trip. All code changes were reverted.

Remaining options for B5 (lower priority — see Step 1 status below):

• vtkStaticPointLocator: drop-in C++ replacement for the Python loop in _point_coordinate_indexes. Limited benefit: _edit_source_point_indexes is already cached at session begin (B4), so the uncached path only runs outside edit sessions or on the very first pick of a session.
• SelectCellsThrough bypass: skip GeometryFilter entirely; run SelectCellsThrough directly on the source (works on unstructured grids), get cell IDs via vtkOriginalCellIds (already reliable in the click-pick path, §6a), filter to boundary cells using the edit session boundary map. Needs a visibility/depth check to discard interior cells — reuse _surface_element_is_visible from the §6c fallback path. More promising than vtkStaticPointLocator but also made obsolete by Step 2.

✅ Boundary map caching (B5b) — fixed in PR #37

_normalize_surface_keys_to_source_boundary called _boundary_codim_elements(source_dataset) on every pick in surface mode. This rebuilt the full boundary map (O(n_source_cells × faces_per_cell)) each time — the same per-pick rebuild pattern that B4 eliminated for the coordinate index. Fixed in PR #37 (Option 1): set_edit_target_dataset builds _edit_boundary_elements once at session begin and passes it through the surface-pick chain.

✅ Pipeline lock during edit (correctness, not performance) — done in PR #35 / #33

Clicking a different pipeline node while an edit session is active silently broke the pick target. Node-switch events are now blocked (with a warning) when edit_session.active.

Step 1 status — paused in favor of Step 2

The main per-pick wins (B3, B4) are done. Remaining Step 1 work:

Item	Decision
Deferred B4 initialization (first pick)	(Optional) Low effort. Moves the cache build from session open to the first pick — session opens faster, but the first pick blocks for the same duration. No reduction in total cost; just different distribution.
B5b boundary map caching	✅ Fixed in PR #37 alongside B5.
B5 SelectCellsThrough bypass	Deferred — Step 2 Option 2 (tracked in PR #37 comment).
B5 vtkStaticPointLocator	Minimal impact given B4 cache; not worth pursuing.

Step 2 (file-based edit node) eliminates B3, B4, B5, and B5b entirely by making pick IDs land directly on the edit dataset with no remap. It is the next priority, on a new branch.

---

✅ Step 2 — Edit pipeline node (PR #37)

Fixes: B3, B4 (remap eliminated); B5 and B5b (session-scoped geometry caches)

Core idea: create a real pipeline node (kind="edit") as the active source during the session. The edit node reads the same backing data as the original node, so pick cell IDs directly index working_dataset with no coordinate-based remapping. A dedicated reader proxy approach was explored first but caused regressions in coloring, LUT, and representation handling (those code paths assume a regular pipeline node); using a real node fits the existing architecture with zero special-casing.

Session begin — two cases

Root reader (no parent node) — open a second Reader proxy on the same backing file. No write needed.

Filter active — write the filter output to a temp .vtu server-side (XMLUnstructuredGridWriter), then open it. Avoids Fetch(source) MPI-rank collapse for the filter case.

In both cases: Fetch(edit_source) → DeepCopy → working_dataset (B2 remains; working_dataset must be a mutable Python object for field assignment and adjacency growth).

Discard: delete edit node, restore pre-edit node visibility, remove temp file.

Commit: write working_dataset to disk, delete edit node, load output as new pipeline node.

What Step 2 fixes

Volume picks: B3, B4 eliminated. Cell IDs from the edit node directly index working_dataset with no remap or coordinate scan. Confirmed fast by comparison with 6101876 (pre-Step-1).

Surface picks: B5 and B5b fixed via session-scoped caches. set_edit_target_dataset builds _edit_source_point_indexes (coordinate index, O(n_points)) and _edit_boundary_elements (boundary face map, O(n_cells×faces)) once at session begin. Both are passed through the surface-pick chain. B5 and B5b were initially marked ✅ after Step 2 but were not actually fixed — _edit_source_point_indexes was removed alongside the remap caches, causing silent per-pick rebuilds. See PR #37 comment for full investigation.

PassThroughPointIds on GeometryFilter is definitively closed: servermanager.Fetch() returns an identity mapping for vtkOriginalPointIds even on meshes with interior points, producing silently wrong face keys (see MIGRATION_LOG.md).

Option 2 (SelectSurfaceCells on volume mesh) — deferred. Tracked in PR #37 comment. Would eliminate the coordinate scan entirely; medium effort; deferred as a future improvement.

Remaining bottleneck — B8 (adjacency growth): for grow selection, _ensure_volume_adjacency / _ensure_surface_adjacency iterate all cells via individual Python→C++ calls — O(n_cells × faces). Cached after first grow per session. Potential fix: vtkStaticCellLinks (C++ build) or numpy connectivity arrays.

---

Step 3 — Server-side mutations

Fixes: B1 completely

Move field-value mutations server-side (e.g. a ProgrammableFilter driven by a selection mask) to eliminate the last Fetch. This is a significant architectural change and should only be pursued if Step 2 is not sufficient for the target cluster workloads.

[pcolt]

For now the idea is to focus on Step 1 only — caching the per-pick Fetch and coordinate index rebuild, because they are surgical changes with no architectural impact and should be straightforward to validate against existing e2e tests. Step 2 is more exploratory and involved therefore I just want to hear your thoughts about it. @luca-heltai @mfranzon @gbrandino @pogmat

[pcolt]

As far as I understand, here is some more context on why the remap exists and how each step changes the picture.

Why the remap is needed today

During an edit session there are two separate objects in play: the live ParaView source proxy (what is rendered and picked against) and working_dataset (a DeepCopy made at session begin from a Fetch(source)). On every pick, SelectSurfaceCells fires on the source proxy and returns cell IDs from a fresh Fetch(source) — a different Python VTK object fetched at a different point in time from working_dataset. VTK gives no guarantee that cell numbering is preserved across two independent fetches of the same proxy, so the code cannot safely assume the IDs match. The remap verifies the match geometrically: it builds a coordinate index over working_dataset and maps each picked cell back by sorted corner-point coordinates (B3 + B4, repeated on every single pick).

In the no-filter case this is always an identity mapping in practice — but the code does not know that, so it pays O(n_source_cells) on every pick to confirm it anyway.

Step 1 — cache the remap (B3, B4)

The two-object structure stays, but the mapping is computed once at begin() and reused for the whole session. Correct as long as the pipeline does not change during the session — which it will not once the pipeline lock is also in place. Eliminates the per-pick Fetch and index rebuild without touching the architecture.

Step 2 — file-based edit node

The two-object problem disappears entirely. working_dataset is Fetch(edit_reader) and picks happen on the same edit_reader proxy. The cell IDs are identical by construction, not by geometric verification — no remap needed at all for volume/cell mode (B3, B4 gone).

For surface mode the GeometryFilter is still needed (because SelectSurfaceCells requires PolyData input). The coordinate lookup (B5) in _surface_keys_from_selected_dataset still runs to map GeometryFilter output cells back to point IDs, but it is now a self-consistent lookup within working_dataset — no cross-dataset translation on top. B5 is what Step 1's PassThroughCellIds attempt targets: if that works it eliminates the coordinate lookup entirely, independent of Step 2.

The two steps are complementary: Step 2 eliminates B3 and B4 for all pick modes; Step 1 eliminates B5 for surface picks. Together they remove all per-pick O(mesh) costs.

ID assignment in Step 2

working_dataset is a standalone Python VTK object — the server-side edit Reader is never updated when field values are assigned. This is fine because topology (cell IDs) is determined by the file's cell ordering, which does not change when array values are modified. After an assignment the next pick on the edit Reader still returns IDs that map correctly to the same cells in the mutated working_dataset. The viewport during editing already shows the original unmodified mesh today (mutations are invisible until commit — only the gold overlay updates), so Step 2 does not change that behaviour.

[pcolt]

Remap cache (Step 1, B3/B4) landed in PR #35 — see above for details.

As visible in screen recordings: picks during a session are considerably faster than before. However, entering edit mode is noticeably slower on large meshes — the cache build (coordinate indexes, cell/point ID maps) now happens eagerly at session begin instead of being deferred to the first pick.

A quick mitigation would be to build the caches lazily on the first pick: enter-edit-mode stays fast, only the first pick pays the build cost, all subsequent picks are O(1) as now.

That said, Step 2 should make this entirely moot: since picks go through the edit Reader (which is the direct source of working_dataset), returned cell IDs already index into working_dataset with no translation needed — the remap and all its caches disappear completely.

[pcolt]

Small grid — new remap cache vs. baseline:

No significant visible improvement on small grids, as expected (the remap cost is negligible when the mesh is small).

New:

cache_remap_small_sphere.webm

Before:

old_version_small_sphere.webm

---

Large grid — new remap cache vs. baseline:

Entering edit mode is noticeably slower (the cache is built upfront here), but subsequent picks are considerably faster than before.

New:

cache_remap_big_sphere.webm

Before:

old_version_big_sphere.webm

[pcolt]

Main work on Step 1 is done and verified — the key wins from caching (B3, B4) are in place: remap structures are built once at session start so subsequent picks are faster. Some B5 approaches were investigated and ruled out (both PassThrough variants fail in PV 6.1); remaining B5 options have diminishing returns and will be superseded by Step 2 if the migration is successful.

Moving to Step 2 (file-based edit node) on a new branch.

PRs #32 and #35 can be reviewed (merge in order and re-target main).

[pcolt]

Overview

PR #37 (Step 2 — edit pipeline node) eliminates B3/B4 entirely and moves B5/B5b to session begin as one-time caches. Key findings on a 6 times refined hyper_sphere with a Clip filter applied on it (~5.6 million cells):

• Volume picks: ~65× faster (36 s → 0.55 s). The dominant baseline cost was B3 + B4 (Fetch + coordinate remap on every pick); both are gone.
• Surface picks: ~8× faster on subsequent picks (44.7 s → 7.0 s); first pick unchanged at ~25.6 s due to B9 (lazy boundary map build).
• Session begin: ~13× slower (2.5 s → 31.6 s), dominated by B5b cache build (22.9 s).
• B11 — multi-field workflows: each field assignment after the first re-pays the full session begin. After PR Perf: edit pipeline node for zero-remap picks (Step 2, issue #34) #37 that re-entry cost is 31.6 s vs 2.5 s at baseline (12× worse per re-entry). Option C in issue Perf/UX: keep edit session alive between field assignments (B11) #39 (lazy B5b build) would drop it back toward ~9 s.
• Remaining open: B9 (EditSession lazy boundary map, first surface pick ~20 s), B10 (backend top_dimension scan ~2 s per surface pick), B11 (issue Perf/UX: keep edit session alive between field assignments (B11) #39).

PR #37 breaks even at N = 1 surface pick per session and wins decisively from N = 2 onward.

Approximation note: timings are approximate and depend on several factors, including whether the ParaView backend rendering window is visible — Ubuntu significantly slows rendering when the window is hidden. Hardware: AMD Ryzen 7 8845HS, ~25 GB RAM, Ubuntu 24.04.4 LTS; single MPI rank, VTK SMP threading active.

Benchmark:

All timings in seconds. Baseline = perf/session-begin-timing (before PR #35 / #37). After = perf/step2-edit-node.

Session begin

Phase	Baseline	After PR #37
export (Fetch + edit node creation)	0.029 s	2.3 s
edit_begin (DeepCopy)	1.4 s	1.4 s
set_target — point_indexes cache (B5)	—	3.4 s
set_target — boundary_elements cache (B5b)	—	22.9 s
Total	2.5 s	31.6 s

B5 and B5b costs moved from per-pick to session begin. Session begin is 13× slower but picks improve dramatically (see below).

Volume box-pick

Phase	Baseline pick 1	Baseline pick 2	After PR #37 pick 1	After PR #37 pick 2
backend_pick (B3 + B4 remap)	34.7 s	34.7 s	0.21 s	0.26 s
sync_overlay	—	—	~0.30 s	~0.30 s
render	—	—	~0.045 s	~0.045 s
Total	~36 s	~36 s	~0.55 s	~0.62 s

B3 + B4 dominated baseline volume picks. After PR #37 the pick itself is fast; sync_overlay and render now dominate the remainder. edit_session_apply (B9) is negligible in volume mode (~0.3 ms — no boundary map involved).

Surface box-pick

Bottleneck	Hot path	Baseline pick 1	Baseline pick 2	After PR #37 pick 1	After PR #37 pick 2
B3 fetch_source_dataset	servermanager.Fetch(source) per pick	0.025 s	0.025 s	eliminated	eliminated
B4 remap_to_edit_target	O(n_cells) coordinate remap to working_dataset	5.1 s	5.1 s	eliminated	eliminated
B5 map_selection_keys	O(n_source_points) coordinate scan per selected face	26.9 s	32.4 s	0.075 s (cache)	0.064 s (cache)
B9 edit_session_apply	lazy _surface_boundary_map_for_top_cells() + uncached _top_dimension()	19.7 s	2.1 s	19.7 s	2.6 s
B10 backend.top_dimension	O(n_cells) GetCellDimension() loop	2.0 s	2.0 s	2.0 s	2.5 s
Total		57.0 s	44.7 s	25.6 s	7.0 s

Net trade-off — surface picks per session

Session begin + N surface picks, total wall time:

N picks	Baseline	After PR #37	Speedup
1	59.5 s	57.2 s	1.0×
2	104.2 s	64.2 s	1.6×
5	238.3 s	85.2 s	2.8×
10	461.8 s	120.2 s	3.8×

PR #37 breaks even at N = 1 and wins decisively from N = 2 onward. The remaining session-begin cost (B5b = 22.9 s) is the next target (B9/B10 on the pick side, B11 for re-entry after commit).