[Feature Request] vtk-wasm render backend, delivered as a separable rendering module

[wayfarer3130]

What feature or change would you like to see made?

A vtk-wasm render backend for Cornerstone3D — covering planar stacks, volume slice (MPR) and 3D volume/geometry rendering — delivered as a separately packaged rendering module, so that a build can include the vtk.js renderer, the vtk-wasm renderer, or both.

This replaces #2796, which added an experimental vtk.js WebGPU backend for planar and MPR GenericViewports.

Why not #2796

#2796 is being rejected on direction, not on quality: vtk.js's WebGPU work is not going to be completed, and the supported route to WebGPU is vtk-wasm, with vtk.js expected to be deprecated in its favour. Extending the vtk.js WebGPU view API therefore builds on a path that will not be maintained.

This is the rationale from the #2796 review, and it is the premise of this request. I could not find a public upstream deprecation announcement for vtk.js, so anyone acting on this should confirm the current Kitware position before committing the work — that confirmation is the first item under "Verify before starting" below.

What #2796 established is still worth keeping, and this request assumes it:

• the render-backend registry (registerRenderBackend(), already on main in packages/core/src/RenderingEngine/helpers/renderBackendRegistry.ts) as the extension seam;
• resolving render-mode kind and surface through the registry rather than through hardcoded ActorRenderMode literals, so an extension backend participates in view resolution, view references, scrolling and labelmap styling;
• the examples with live backend switching and a ?renderBackend= parameter, and the debug panel reporting effective backend / mounted render mode;
• the performance fixes in that branch (per-stroke brush binding resolution, incremental flat-index labelmap slice extraction, slice buffer reuse) — those are independent of the backend and should land on their own regardless of what happens here.

The part that gets thrown away and rewritten is the backend implementation itself: WebGPUImageMapperRenderPath.ts, WebGPUVolumeSliceRenderPath.ts, webgpuViewportRenderWindow.ts.

---

Part 1 — Split rendering into a separate module

Goal: @cornerstonejs/core stops depending on a specific renderer, so vtk.js and vtk-wasm can be included independently or together in a single build.

Update — these prerequisites now have an earlier consumer. #2796 is no longer rejected: it is being repackaged as @cornerstonejs/rendering-webgpu-vtkjs, an opt-in, @deprecated, not-installed-by-default module (terms). That means the render-path SDK exports and the backend-owned surface / actor-hosting capability below get built and exercised against a real second backend before any vtk-wasm work starts, rather than being designed in the abstract for a consumer that does not exist yet. Treat them as shared prerequisites landing on #2796's timeline, not as vtk-wasm tasks. The vtk-wasm backend is then the second consumer of a seam already proven by one.

Current state, honestly measured

• @kitware/vtk.js is a hard dependency of both @cornerstonejs/core (66 source files import it) and @cornerstonejs/tools (41 files). sideEffects: false does not help — viewport construction reaches vtk.js directly, so it is always in the graph.
• The extension seam exists but is deliberately incomplete. The registerRenderBackend() JSDoc already names the gaps:

  "the definition is intended to grow additional parameters describing the backend-specific changes and behaviours being registered (participation in the 'auto' capability resolution, backend-owned surface/canvas creation, per-backend degradation handling)"

• The registry is planar-only. createRegisteredPlanarRenderPaths() is consumed by PlanarRenderPathResolver alone; createVolume3DRenderPathResolver() takes a hardcoded default list (VtkVolume3DPath, VtkGeometry3DPath). A 3D backend has nowhere to register today.
• RenderSurface is a closed union 'vtk' | 'cpu'. A backend cannot own its own canvas. feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796 had to blit WebGPU frames into the cpu surface canvas after device work completed — a workaround, not a design.

Proposed shape

Package	Contents
@cornerstonejs/core	viewport architecture, view resolution, camera and projection math, voxel manager, cache, loaders, CPU render paths. No renderer imports.
@cornerstonejs/rendering-vtkjs	RenderingEngine/vtkClasses/*, VtkImageMapperRenderPath, VtkVolumeSliceRenderPath, VtkVolume3DRenderPath, VtkGeometry3DRenderPath, and the @kitware/vtk.js dependency.
@cornerstonejs/rendering-vtkwasm	the vtk-wasm equivalents plus runtime loading and asset resolution.

• The CPU path stays in core — it is the route that works everywhere and has no third-party renderer dependency.
• Registration follows the existing add-on precedent, e.g. cornerstoneTools.init({ addons: { polySeg } }) and the "PolySeg add-on not configured" diagnostic in packages/tools/src/config.ts. A missing renderer module should produce that same class of actionable message, not a crash.
• No breaking change for apps that just want vtk.js. The default entry point continues to register the vtk.js backend; the split is internal packaging plus an explicit renderer-free entry point.

The larger half: tools

The 41 vtk.js imports in @cornerstonejs/tools — labelmap actor styling, the labelmap render plans, segmentation vtkClasses — are the harder part of this and probably want their own task. It needs an abstract actor/mapper surface for those files to talk to instead of vtk.js types. This should be scoped and estimated separately; treating it as a footnote of the packaging task is how the packaging task overruns.

Also needed for a real backend

• Registry hook for Volume3D and geometry render paths, matching createRegisteredPlanarRenderPaths().
• Backend-owned surfaces — a backend declares and creates its own canvas rather than picking from 'vtk' | 'cpu'.
• auto resolution participation — a registered backend contributes to capability-based backend selection, instead of auto only ever choosing among core backends.
• Per-backend degradation — a declared fallback when the backend is unavailable at runtime (no navigator.gpu, wasm fetch failure, device loss), rather than each call site guarding.

---

Part 2 — The vtk-wasm backend

What we are building against

@kitware/vtk-wasm (2.1.5 at time of writing, actively published) loads as:

import { loadAsync } from '@kitware/vtk-wasm';
const runtime = await loadAsync({ url: BUNDLE });
const session = runtime.createStandaloneSession();
const vtk = session.vtk; // vtk.vtkConeSource(), ...

This is the desktop VTK C++ API through embind, not vtk.js's API. That matters for scoping: each render path is a rewrite against VTK class names and lifetimes, not a port of the vtk.js path. Anyone estimating this from the size of the vtk.js render paths will be wrong.

Scope

• Planar image — stack rendering, pixel parity with the WebGL path.
• Volume slice / MPR — including oblique.
• 3D volume (DVR) and geometry — the 3D viewport is in scope here, unlike feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796, which was planar-only.
• Coplanar overlays — annotations and labelmap segmentation must composite correctly. feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796 needed two upstream vtk.js fixes for this (fix(WebGPU): apply projection once in ImageMapper and refresh imageData textures Kitware/vtk-js#3547 image mapper projection + texture invalidation, fix(WebGPU): pass depth-equal fragments in the translucent pass Kitware/vtk-js#3548 translucent-pass depth for coplanar overlays). The equivalent behaviour in vtk-wasm has to be established and confirmed, not assumed; if it needs upstream work, that is a schedule dependency to find now rather than late.

New failure modes the JS backends do not have

These are the parts that make a wasm renderer different from a JS one, and each needs an explicit answer:

• Asset delivery — where the .wasm bundle is served from, and resolution relative to the app root rather than the current route. We have already been bitten by exactly this with ONNX (fix(ai): resolve the ONNX wasm directory against the app, not the route #2864); the same trap applies here.
• Bundle cost — download and instantiation time, on the low-end devices that most need the faster renderer. Measure before committing.
• Heap boundary — volume scalars must cross into the wasm heap. feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796 already had to materialize one shared, ref-counted contiguous scalar array per volume because Cornerstone volumes are image-backed and own no contiguous array; a wasm backend pays a copy on top of that. See Part 3 — the multi-resolution voxel manager is where this should be answered.
• Threading — if a pthreads build is used, it brings SharedArrayBuffer and therefore COOP/COEP headers, which is a deployment constraint on every consuming app. Prefer the non-threaded build unless measurement forces otherwise.
• Object lifetime — embind objects need explicit release; leaking them across viewport teardown is not the kind of leak the current code is shaped to catch.
• WebGPU maturity — VTK's WebGPU support is documented as experimental, and the WebGPU volume mapper in particular needs measurement before MPR and 3D are committed to it.

---

Part 3 — Build on the multi-resolution voxel manager

The backend should bind a resolution level, not a fully loaded contiguous volume.

• Depend on the voxel-manager multi-resolution interface, with the brick loader in [WIP] feat(core): brick-delivered volumes as a Cornerstone image source #2860 as its first consumer and the analysis in docs(volume3d): analysis of the three Volume3D render paths and an improvement plan #2853 as the rationale. A multi-resolution manager looks like an ordinary voxel manager to existing consumers; level detail is an additional field on a multi-resolution manager only. A change of resident level must not be a change of volume identity — which is what today's decimatedVolumeLoader gets wrong by producing a separate volume with its own id.
• Capability-bound level binding applies to vtk-wasm exactly as it does to WebGL: bind the finest level that fits maxTextureDimension3D and the memory budget, always, at rest, irrespective of frame rate. A 3072×512×512 series has no image at any frame rate on a 2048-limit device.
• Report fidelity. Whatever level and reduction source the backend actually bound has to be visible through the viewport fidelity state, so the displayed image never silently misrepresents its own resolution.
• The strongest case for the combination is full-resolution off-axis MPR. A plane through a bricked volume intersects only ~N^(2/3) of the bricks, and the plane ∩ brick regions are disjoint in screen space — one quad per intersecting brick, nothing to blend, no ray ordering, no page table. That is the tractable half of bricked rendering, and it is what makes a WebGPU path worth having beyond raw frame rate: today a full-resolution sagittal plane on the Juno CT costs 28.34 MB and 174 requests to display a 2 MB image; from bricks it is ~4.5 MB and 31 requests.

Bricked DVR — which does need single-pass page-table indirection — stays out of scope.

---

Verify before starting

1. Confirm the upstream position with Kitware: is vtk.js deprecation stated, and on what timeline? The whole sequencing argument rests on this.
2. Confirm vtk-wasm's WebGPU volume mapper is usable for MPR and DVR today, and measure it against the existing WebGL path and against the vtk.js WebGPU and mview WebGPU numbers already collected. If the answer is "not yet", this becomes a staged item rather than a now item.
3. Confirm coplanar overlay compositing in vtk-wasm, since that is where feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796 needed upstream fixes.

---

Suggested sequencing

1. Land the feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796 performance and correctness fixes standalone — they are backend-independent and shouldn't be stranded.
2. Complete the render-backend registry: render-path SDK exports, backend-owned surfaces / actor-hosting capability, Volume3D and geometry registration hook, auto participation, declared degradation. ⟵ shared with feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796, lands on its timeline
3. Registry-driven render-mode resolution in core and tools, replacing the hardcoded ActorRenderMode literals. ⟵ also shared with feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796
4. Measurement baseline, so the vtk-wasm decision is made against numbers.
5. Package the vtk.js renderer out of core behind the completed registry, with the default entry point unchanged.
6. Scope the tools-side abstraction as its own task.
7. Implement @cornerstonejs/rendering-vtkwasm: planar → MPR → 3D, in that order, each with overlay compositing proven.
8. Level-granular MPR from bricks on the vtk-wasm path, once the multi-resolution interface is in place.

Steps 2 and 3 are the ones #2796 now pulls forward. By the time step 7 starts, the seam it needs should already have one backend living on it.

---

Done when

• A single build can include the vtk.js renderer, the vtk-wasm renderer, or both, selected at runtime through setRenderBackend() / ?renderBackend=, with live switching and no errors.
• @cornerstonejs/core builds and its CPU render paths run with no renderer package installed; a missing renderer produces an actionable message, not a crash.
• The vtk-wasm backend renders stack, MPR (including oblique) and 3D volume/geometry, with annotations and labelmap segmentation compositing correctly on all three.
• Pixel parity with the WebGL path at the same resolution level, within tolerance — so any visible difference is attributable to the level of data, not to the renderer.
• A published measurement of interaction fps, time to first image and time to full resolution, per GPU class, against the WebGL baseline.
• Bound level and reduction source are reported through the viewport fidelity state.
• Existing applications that only want vtk.js need no code change.

Non-goals

• Removing vtk.js, or forcing a vtk.js → vtk-wasm migration. Both coexist.
• Retiring the CPU path — it stays the route that works everywhere.
• Bricked GPU DVR with page-table indirection.
• Reviving the vtk.js WebGPU backend from feat(webgpu): experimental WebGPU render backend for GenericViewport stacks and MPR #2796.

---

Why should we prioritize this feature?

The WebGPU capability has to come from somewhere, and this is the only supported route. WebGPU was the point of #2796: a measurable interaction-performance ceiling that the WebGL path cannot reach. Rejecting #2796 does not remove that need — it removes the only implementation we had, and vtk-wasm is the replacement with a maintainer behind it. Continuing to improve the WebGL path is the right short-term call precisely because it is supported, but it has a ceiling, and nothing else on the roadmap raises it.

#2796's work is otherwise stranded. 8,000+ lines, upstream fixes landed in vtk.js 36.4.2, three working examples, and — importantly — the registry-driven render-mode resolution that made an extension backend possible at all. Redirecting it to vtk-wasm keeps everything except the two backend files.

The module split pays off independently of vtk-wasm. Today every consumer ships vtk.js even if it only ever displays a stack — 66 files in core and 41 in tools guarantee it. The split is also the seam that any future backend needs, so it is not speculative work: it is the missing half of a registry that already exists and already documents what it is missing.

It is the natural consumer of the multi-resolution work now in flight. The brick store (#2860) and the voxel-manager interface exist to make level-granular access possible; a renderer that binds a level rather than a whole volume is what turns that into a visible improvement. Full-resolution off-axis MPR from bricks — 28.34 MB / 174 requests down to ~4.5 MB / 31 requests — is not reachable from the current renderer at all.

Timing. Doing the registry completion and the packaging split now, while the WebGL performance work is live and touching the same files, is much cheaper than retrofitting it around a finished WebGL path later. The vtk-wasm implementation itself can then be scheduled against whatever the upstream and measurement answers turn out to be.

Interaction with existing features

• Additive. A new registered backend; gpu (vtk.js/WebGL) and cpu stay the defaults and stay unchanged.
• View resolution, view references, scrolling, labelmap styling already resolve render-mode kind and surface through the registry rather than through ActorRenderMode literals, so an added backend participates without touching those call sites.
• GenericViewport only. Legacy viewports keep the vtk.js path; no legacy behaviour changes.
• Segmentation and annotation tools are the main integration risk, via the 41 vtk.js imports in @cornerstonejs/tools. That is called out as its own task above rather than absorbed silently.

[linear]

CS-212

[wayfarer3130]

@daker - we met with the kitware people and they indicated that they will not be continuing working on the vtk-js webgpu side of things as they are deprecating vtk-js in favour of a more flexible and faster vtk-wasm. There are real/hard limits in vtk-js that aren't clear how or if we can get around them. The plan is, as of right now, to merge the vtk-js webgpu work that @sedghi has done, but modified so that it is an external library. That will allow vtk-js and vtk-wasm rendering to co-exist. If there are good reasons to use vtk-js for initial render, eg for planar/simple 3d, then it would certainly be possible to load both libraries depending on what the work being done is - the wasm version for complex work or the js version for simpler. It isn't decided yet for sure, but we will need the wasm version unless kitware completely changes their mind about vtk-js - there are just some types of volumes that are too hard to render using only the vtk-js capabilities.