feat(layers): analytic antialiasing for PathLayer and LineLayer

[chrisgervang]

Background

PathLayer and LineLayer have no antialiasing of their own; their edges are smoothed entirely by the framebuffer's MSAA. This adds an opt-in antialiasing prop that computes edge coverage analytically in the fragment shader, for the situations where MSAA is unavailable.

At a glance

Where deck.gl has multisampling today, and which remedy applies. "Host MSAA" is asking the base map or canvas for antialias: true; "luma.gl#2741" is the proposed color-only MSAA for offscreen framebuffers; "this prop" is antialiasing: true (lineAntialiasing on composite layers).

Situation	MSAA today	Host MSAA	luma.gl#2741	This prop	Recommended
Standalone canvas	yes	on by default	—	optional	nothing needed
Standalone + PostProcessEffect (#10404)	no	no — bypassed	yes	yes	luma.gl#2741; this prop meanwhile
Interleaved MapLibre / Mapbox	no	yes	no	yes	either; this prop is cheaper at 4K
Interleaved Google Maps vector (#7647)	no	no option exposed	no	yes	this prop — only avenue
@deck.gl/arcgis	no	no	no — depth attachment	yes	this prop — only avenue
App-supplied _framebuffer	no	no	if color-only	yes	whichever fits the target
WebGPU, any target	no	no such attribute	no — deferred	yes	this prop — only avenue
PathStyleExtension offset (#8063, #9395)	no — edge is a discard	no	no	partial	this prop; full fix needs an extension change

Three rows have no alternative at all, and one — post-processing — is better served by luma.gl#2741 than by this proposal. The two efforts overlap only there; neither subsumes the other.

If luma.gl#2741 lands

No part of this proposal is descoped by it, and the reasons are structural. Interleaved base maps draw into the host's default framebuffer rather than an offscreen target, and routing them through one would break the depth interaction that interleaving exists for — the same reason post-process effects cannot be used in interleaved mode. WebGPU is a deferred follow-up in that RFC. And the PathStyleExtension offset edge is defined by a discard, which kills every sample of a fragment, so no sample count smooths it. ArcGIS could move once multisampled depth is supported, since its framebuffer is depth-attached.

What does change is post-processing: deck's render buffers pass only colorAttachments, and luma auto-creates a depth attachment only when both attachment lists are empty, so they are genuinely color-only and fall squarely in that RFC's initial scope. The better fix there is to set samples on them, closing #10404 for every layer rather than for these two.

Measured impact

A 2px diagonal path rendered into a 240×180 context, counting pixels by alpha:

context	antialiasing	partial-coverage pixels	distinct alpha levels
no MSAA (base-map-like)	false	0	0
no MSAA (base-map-like)	true	719	104
MSAA canvas	false	1361	3
MSAA canvas + PostProcessEffect	false	0	0

Where the framebuffer provides no multisampling there is no antialiasing from any source — every covered pixel is fully opaque and the edge is a hard staircase. The third row shows the honest comparison where MSAA genuinely is available; the fourth shows how easily that stops applying.

Change List

Layers

• antialiasing prop on PathLayer and LineLayer, defaulting to false. TripsLayer inherits it by subclassing PathLayer.
• lineAntialiasing on PolygonLayer and GeoJsonLayer, following the existing pointAntialiasing precedent. These are composites that forward stroke props explicitly rather than by inheritance, so the prop would not otherwise reach the stroke sub-layer — which is the case that matters most, since they are the layers most often drawn over a base map.
• Analytic edge coverage in the GLSL and WGSL fragment shaders for both leaf layers; no vertex shader changes.

Tests

• Golden image (test/render/test-cases/path-antialiasing.spec.ts) and coverage assertions (test/render/path-antialiasing.spec.ts), both on a device created with antialias: false.
• Uniform plumbing (test/modules/layers/antialiasing.spec.ts) and composite forwarding (test/modules/layers/antialiasing-composite.spec.ts).
• Render harness: runRenderTestSuite accepts webgl context attributes, and TestCase.imageDiffOptions threads includeAA through (and now honours tolerance, previously declared but ignored).

Docs

• Prop reference for all four layers.
• A short antialiasing remark in each integration whose users have no alternative: @deck.gl/google-maps, @deck.gl/arcgis, PostProcessEffect, and @deck.gl/mapbox. Each points inward at the layer prop; core layer docs carry no integration commentary.
• Full RFC at dev-docs/RFCs/v9.4/path-line-antialiasing-rfc.md, covering the design, prior art, alternatives, limitations and testing rationale.

No CHANGELOG entry (maintained at release time).

Notes for reviewers

• Why not just antialias: true on the map? That has been the standing answer since #5742 and is still right where it applies — the RFC positions this prop as complementary, not a replacement. Two things have narrowed it: MapLibre v5 moved the option into canvasContextAttributes, so the top-level form silently does nothing on current versions, and it was never available for Google Maps, ArcGIS, offscreen targets or WebGPU.
• Default is false everywhere, so every existing render output is byte-identical.
• Coverage comes from screen-space derivatives, not a stroke-width varying. The varying approach was implemented first and was wrong under PathStyleExtension's offset (feather collapsed to 0.328×) and under perspective foreshortening.
• A golden image needs includeAA: true — pixelmatch drops antialiased pixels from the mismatch count by default, and this prop changes nothing else, so a diff without it passes with the feature entirely removed.

---

Note

Medium Risk
Shader and uniform changes affect core stroke rendering on both backends; default-off preserves existing output, but enabling the prop changes blending at edges and has documented overlap/flat-cap limitations.

Overview
Adds opt-in analytic antialiasing for stroked geometry when framebuffer MSAA is missing (interleaved maps, offscreen targets, WebGPU, ArcGIS, post-process effects). PathLayer and LineLayer gain antialiasing (default false); GeoJsonLayer and PolygonLayer forward lineAntialiasing to stroke PathLayer sublayers.

Fragment shaders (GLSL + WGSL) feather stroke width using fwidth-based device-pixel edge distance and smoothedge, applied before premultiplication on WebGPU. Path handling distinguishes body vs rounded corner silhouettes; line/path ends stay hard along the stroke direction.

project.wgsl.ts remaps clip-space Z from WebGL convention to WebGPU so depth is not half-clipped.

Render tests: optional webgl: { antialias: false }, includeAA / tolerance in image diffs, golden path-antialiasing, framebuffer coverage tests, WebGPU premultiplied-alpha check; CI runs tests under xvfb-run and extends Chromium flags for headless WebGPU capture. Docs/RFC describe when to use the prop vs host MSAA.

^{Reviewed by Cursor Bugbot for commit 14434fe. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coveralls]

coverage: 83.914% (+0.8%) from 83.077% — claude/deckgl-path-aliasing-4tx9a3 into master

[Pessimistress]

There is a utility shader function smoothedge defined in https://github.com/visgl/deck.gl/blob/master/modules/core/src/shaderlib/misc/geometry.ts I believe scatterplot layer is using it.

[Pessimistress]

You should enable WebGPU render test for line layer. Right now it fails due to lack of antialiasing.

[chrisgervang]

Thanks — both addressed in 2d98227.

smoothedge — now used at all four coverage sites. I'd looked at it early and passed because SMOOTH_EDGE_RADIUS is a fixed 0.5 CSS pixels and not DPR-aware, but that objection belonged to an earlier implementation that carried the stroke half-width down as a varying. That's gone; coverage now comes from screen-space derivatives, so edgePixels is already a signed device-pixel distance and smoothedge(0.0, edgePixels) is exactly smoothstep(-0.5, 0.5, edgePixels). Units line up. Falloff goes from linear to cubic, the existing goldens absorb it, and both regression guards still fire.

LineLayer WebGPU — the case is prepared but the suite entry is still commented out, and I want to be transparent about why rather than quietly not doing it.

Setting antialiasing: true unconditionally dropped the WebGL match to 98.28%, since it layers analytic coverage on top of MSAA that's already there. So it's now set per device — on for WebGPU only. That way one golden serves both backends, and the WebGL case keeps covering the default configuration with its golden untouched.

What I couldn't verify is the WebGPU side, because WebGPU rasterizes nothing under the software renderer this suite runs on. line-lnglat comes back as a completely blank frame — and so does every path-layer case. The device is created, navigator.gpu is present, commands submit with no validation errors, and the output is pure white. The 84.32% match is that blank frame against the golden, and it's identical before and after this change, so antialiasing isn't what that number is measuring.

Since test-ci includes --project render and CI uses the same --use-angle=swiftshader --enable-unsafe-webgpu chromium, enabling it looked likely to red CI for a reason unrelated to this PR. If WebGPU does rasterize in your environment, it's a one-line uncomment and I'd expect it to pass now — happy to flip it if you confirm, or if you'd rather see the CI result I can enable it and we can judge from that.

Same root cause, for what it's worth, explains a dead end I hit earlier trying to assert on WGSL coverage directly: rendering deck layers into an app-supplied _framebuffer also came back empty on WebGPU. I'd attributed that to the incomplete post-processing area of #9504, but blank-everything is the simpler explanation.

---

Generated by Claude Code

[ibgreen-openai]

@chrisgervang Very cool, this could have some potential.

• General antialiasing is rather complicated and different between WebGPU and WebGL. I have several pages about the topic in luma.gl docs, and even a demo that lets users experiment. Trying to consolidate.
• https://luma.gl/next/docs/api-guide/gpu/gpu-antialiasing

For this technique I am thinking it might benefit from being done in premultiplied colors.
WebGPU shaders already return pre-multiplied colors but in most cases as a last step, they don't do blending math in premultiplied space, which could make this less effective than it could be

[chrisgervang]

Correcting myself: my earlier comment said WebGPU "rasterizes nothing under the software renderer." That was wrong, and it was hiding a real deck bug.

Root cause

Viewports build projection matrices for the WebGL clip volume, -w ≤ z ≤ w. WebGPU's is 0 ≤ z ≤ w. A default OrthographicView projects to z/w = -0.998, so every primitive fell in the half of the range WebGPU clips, and nothing rasterized. Fixed in d7996ed by remapping depth where the clip position is assembled — the only place in WGSL where the convention differs. The GLSL path is a separate file and is untouched.

How it was isolated

probe	result
Raw WebGPU — clear + readback, no luma	[255,0,0,255] — rasterizes
luma Model triangle → offscreen FBO → readback	[0,0,255,255] — draws
deck ScatterplotLayer / LineLayer	blank
deck's own model, driven manually in a plain luma pass	blank, with drawResult=true, pipelineErrored=false, validationError=null

A valid pipeline issuing a valid draw and producing nothing. From there: CPU-side uniform bytes are byte-identical between backends; all attributes arrive (verified by encoding them into fragment color); a clip-space triangle bypassing the projection renders fine. Encoding the computed clip position as color gave w = 1, z/w ≈ -0.998 — outside WebGPU's clip volume.

Two things I chased that were dead ends, recorded so nobody repeats them:

• Chromium flags. Adding --ignore-gpu-blocklist and --use-webgpu-adapter=swiftshader to match luma's set changed nothing. Reverted.
• Uniform buffers reading back all-zero. An artifact — UniformStore allocates them UNIFORM | COPY_DST, and readAsync copies through a staging buffer needing COPY_SRC. The copy fails inside an error scope and the staging buffer stays zero. Uniform buffers can't be read back that way.

Result

Reading back an offscreen framebuffer, LineLayer, PathLayer and ScatterplotLayer all now paint on WebGPU within a few pixels of their WebGL counts, under both orthographic and lng/lat viewports:

webgl/line 450    webgpu/line 450
webgl/path 700    webgpu/path 700
webgl/scatterplot 1304    webgpu/scatterplot 1264   (WebGL's MSAA edges)

On enabling the LineLayer WebGPU render test

Still can't, but for a different and narrower reason: the WebGPU canvas does not present under the headless software renderer, so the screenshot this suite diffs comes back blank no matter what deck draws. That reproduces with a plain luma triangle and no deck involved, and readback of the same content from an offscreen texture is correct. The antialiasing: deviceType === 'webgpu' wiring is committed and ready; the row just needs CI with hardware WebGPU. Comment and RFC updated to say this accurately.

Note the depth fix is @deck.gl/core and unrelated to antialiasing — happy to split d7996ed into its own PR if you'd prefer to review it separately.

Full render suite re-run: 168 passing, 4 pre-existing network failures (scenegraph duck ×2, terrain ×2), no goldens moved.

---

Generated by Claude Code

[chrisgervang]

@ibgreen do canvas-based render tests work in CI on luma in your experience?

I'm a bit stuck with blank canvases.

Claude claims to have made a minimal reproduction of the issue in pure luma too, which I'll post for a deeper look

[chrisgervang]

Thanks @ibgreen-openai — the guide is a better home for the taxonomy than this RFC, so the RFC and both layer docs now link out to it instead of restating it.

On premultiplied: coverage already lands before deckgl_premultiplied_alpha in both layers (path-layer.wgsl.ts:278-284, line-layer.wgsl.ts:170-172 → :200), giving (rgb·a·c, a·c) — algebraically the same as scaling the premultiplied vec4 by coverage. Nothing defended that ordering though, so 2f6f0b6 adds a guard: renders on WebGPU into an offscreen framebuffer and asserts red tracks alpha at partial-coverage pixels. Moving the multiply after premultiplication takes the worst overshoot from 2 to 126.

Alpha-to-coverage was the most useful thing in the guide — I hadn't considered it. Per-sample masks would fix the flat-cap and self-overlap limitations and the discard-defined edges together. Written up in the RFC as parked, for three reasons: luma never maps sampleAlphaToCoverageEnabled on WebGL (declared in core/.../parameters.ts, no case in webgl/.../device-parameters.ts — silent no-op); on WebGPU it maps but needs sampleCount > 1, so it waits on luma.gl#2741; and the mask comes from fragment alpha while deck blends SRC_ALPHA, so coverage gets counted twice unless an explicit @builtin(sample_mask) separates it from opacity. Worth revisiting once #2741 lands.

---

Generated by Claude Code

[chrisgervang]

@Pessimistress I've enabled the line render tests so we can see where we're at - they render but don't match yet

Also, I had to jump through some configuration hoops to make canvas-based headless WebGPU renders work.

Rendering to offscreen textures was easy, but getting the entire pipeline to work requires a virtual display.

I recorded findings and options in the linked luma issue

[chrisgervang]

I advocate for adding this prop in 9.4 to fix long standing WebGL issues like #7647, but also want to hear if there are any concerns.

I'd land this PR with WebGPU render tests functional but disabled to keep CI green

[cursor]

WebGPU line test fails CI

High Severity

The 'webgpu' line-layer render case is enabled while the comments document it as expected to fail (~83% vs a 99% threshold). That fails yarn test-ci under the new xvfb-run path, so the suite cannot stay green. The RFC still describes this row as commented out, matching the earlier plan to keep WebGPU goldens disabled until they match.

 

^{Reviewed by Cursor Bugbot for commit 241d28b. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 14434fe. Configure here.}

[cursor]

AA misses outer coverage half

Medium Severity

Coverage is centered with smoothedge(0.0, edgePixels), but stroke geometry is not expanded by half a device pixel and round joints still discard past the silhouette. Pixel centers outside the edge never shade, so the outer half of the claimed one-pixel feather is lost and edges cut off around 50% alpha.

Additional Locations (2)

• modules/layers/src/line-layer/line-layer-fragment.glsl.ts#L20-L27
• modules/layers/src/path-layer/path-layer-fragment.glsl.ts#L43-L52

 

^{Reviewed by Cursor Bugbot for commit 14434fe. Configure here.}