Support reverse-PRIME

[AprilGrimoire]

It finally works!

Now by itself yserver doesn't automatically enable outputs on secondary devices. However, xrandr --auto works.

After some preparations on modelling the relevant objects, now reverse-PRIME works in three modes:

1. Output-owned scanout buffer, DMA-exposed to the renderer
2. Renderer-owned buffer, DMA-exposed to the output device while being scannable
3. copy to scanout buffer on output device, the fallback

Yserver now automatically probes which of these modes work.

Changes by commit:

• refactor(kms): use Rc for single-threaded DRM ownership
• feat(randr): add direct CRTC-safe request handling
• feat(platform): split DRM discovery boundary
• chore(platform): make udev dependency Linux-only
• refactor(platform): move KMS discovery into Linux layer
• feat(prime): add device-qualified RANDR topology
• feat(prime): open all KMS devices and allow headless startup
• feat(prime): finish device-qualified topology and lifecycle
• fix(kms): initialize cursor plane after headless startup
• feat(prime): map DRM devices to Vulkan GPUs
• feat(prime): expose RANDR provider topology
• feat(prime): qualify and probe scanout routes
• fix(prime): attempt scanout routes with unknown support
• feat(prime): enable RANDR output source routing
• feat(prime): add ordered DRM device override
• fix(kms): route DRM events by device fd
• fix(randr): resolve DRM modes by connector handle
• feat(prime): probe copy-free scanout ownership
• feat(prime): add copied reverse-PRIME scanout

[joske]

That's amazing! I'll have a look in the morning

[joske]

Really strong contribution — thank you. The scope is large but well-structured: spec + impl plan in docs/superpowers/, one logical change per commit, careful error unwinding, and the architecture is sound. I reviewed it statically and then HW-tested it on a real dual-GPU box; writing up both, because the HW testing surfaced two blockers that static review can't.

Static gates (local, master…prime)

• cargo build ✅
• cargo clippy --all-targets -- -D warnings (as CI) ✅ zero warnings
• cargo +nightly fmt --check ✅
• cargo test ✅ 312 passed, 0 failed

The high-risk surfaces hold up well on read: the VK_EXT_physical_device_drm join is conservative (exact major:minor match, ambiguity is a hard error, generic fallback only when no device exposes the ext), the sync_file fd ownership rules in sync.rs are correct (TEMPORARY one-shot, consumed only on success), and completions are keyed by monotonic job_id + stable OutputKey rather than fd/index, as the design requires.

HW test — silence (Intel UHD 770 / i915 + AMD RX 580 / amdgpu+RADV, Polaris gfx8)

Transport plumbing works. yserver renders across both GPUs, drives both monitors, MATE enables the second screen, and RANDR topology/modes report correctly. Core-protocol apps composite fine on both screens.

But real GPU/dmabuf app usage fails in both PRIME directions on this pairing — and both are multi-GPU-only (they do not reproduce on master with the same 2-GPU hardware, nor on prime with a single device opened, so they're regressions the multi-device path introduces, not pre-existing):

Bug A — render = RX 580, sink = i915: DRI3 client buffers won't import (apps render empty)

wezterm, gnome-console (kgx), eog, loupe all draw window decoration but empty interiors — and the window backing is empty, i.e. content is lost in client-ingestion upstream of scanout (the transport itself is exonerated). The log:

WARN vk::device] physical device lacks VK_EXT_image_drm_format_modifier — Vulkan-fed scanout will not work
DRI3::PixmapFromBuffer 634x532 stride=3072 depth=32 bpp=32
  -> BadAlloc (DRI3 import_dmabuf: Vk(ERROR_INVALID_EXTERNAL_HANDLE))

dri3_open (crates/yserver/src/kms/render/backend.rs:17110) correctly hands the client the RX 580 render node (renderD128, matching the compositing GPU) — so this is not a wrong-device issue. The failure is in dri3_import_pixmap (:16681): the legacy PixmapFromBuffer path imports as LINEAR (modifier 0), but the RX 580 (Polaris/gfx8) lacks VK_EXT_image_drm_format_modifier, so yserver can't express an explicit-stride LINEAR import and falls back to the driver's native layout. Once a second GPU exists, Mesa allocates PRIME-padded buffers (note stride=3072 for a 634px row — padded to 768px, not tight linear), and the size/layout no longer matches → VK_ERROR_INVALID_EXTERNAL_HANDLE. A 1×1 dummy buffer imports fine (single page); real sizes fail.

Confirmed it's not the client's device pick: forcing MESA_VK_DEVICE_SELECT=1002:67df (RX 580) on the client did not help — same import failure. It's the render GPU's missing modifier extension.

Bug B — render = i915, sink = RX 580: enabling the sink resets amdgpu

Forcing the modifier-capable iGPU as the renderer (YSERVER_DRM_DEVICES=/dev/dri/card2:/dev/dri/card1) gets the primary screen up, but enabling the RX 580 as a sink via Displays crashes:

apply_crtc_config: HDMI-4 … no-op
RANDR::SetCrtcConfig -> status=0
FenceTicket::poll_signaled: get_fence_status: ERROR_DEVICE_LOST   (×hundreds)
kms: renderer device lost (GPU reset) — requesting clean shutdown … respawn

The amdgpu reset dump lists name=imported 14400 KB buffers (2560×1440×4) — the reverse-PRIME cross-GPU scanout buffers. So importing full-screen buffers into the RX 580 for sink scanout hangs amdgpu → DEVICE_LOST → exit-for-respawn.

Scope note

The through-line is that the RX 580 (Polaris, no VK_EXT_image_drm_format_modifier) is the weak link in both roles — it can't import client PRIME buffers as the renderer, and hangs on imported full-screen buffers as the sink. Reverse-PRIME may well be fine on modifier-capable pairs (two Intel, or newer RDNA AMD). I'd suggest either handling the modifier-less-GPU case or making the supported-GPU scope explicit before this lands as "works."

Code-review finding (independent of the HW bugs)

fire_randr_changes omits the scanout-BO reset that every sibling quiesce path performs — crates/yserver/src/kms/render/backend.rs:6271.

Every other quiesce site pairs three calls:

wait_idle_bounded() → scene.drain_all() → platform.reset_scanout_bos_for_suspend()

• VT suspend (backend.rs:6095–6124) ✅
• apply_crtc_config (:11888–:11890) ✅ (with a comment on why all three)
• DPMS sleep (:18391–:18396) ✅
• fire_randr_changes (:6271–:6272) — only the first two ❌

requery_outputs_and_modeset cancels pending completions only for dropped connectors and preserves a surviving output's pool. So a hotplug while a copied frame is in flight on another output leaves that output's platform BO busy while the scene thinks it's idle → the later completion is dropped and the bo_idx is orphaned until pool exhaustion (NoBO). Fix mirrors the siblings — add self.platform.reset_scanout_bos_for_suspend(); after the drain_all at :6272.

Diagnostics / robustness nits

• Per-DRI-device driver= in the discovery log (crates/yserver/src/platform/drm.rs, the candidate {path}: … line). On my box the iGPU came up on simpledrm (not i915) and the log gave no hint — the drm::Device is already open there, so a get_driver() name (amdgpu/i915/simpledrm) would make this obvious. Same on the DRI3 render node ready … line.
• Capability-based render-GPU selection. primary_kms_card_candidate_index picks the first connected device as primary and then uses it as the render GPU; when that device can't render (a dumb simpledrm fb, or a display-only device) yserver aborts instead of falling back to a render-capable GPU. Reverse-PRIME makes display-only devices a normal thing, so the render GPU should be chosen by render capability with a fallback, not by first-connected.
• render_node_via_dev_walk blind fallback (crates/yserver/src/platform/drm_linux.rs:657): when no render node matches the primary's PCI parent it returns render_nodes.into_iter().next() — an arbitrary node from a different GPU. Ok(None) is the correct answer for a device with no render sibling, and it feeds the capability-based selection above. (Only bit me because of the simpledrm state, but it's latent.)

Bottom line

Architecture is sound and the transport is validated; I don't think it should merge as "works" until Bugs A/B are addressed or the supported-GPU scope is stated explicitly. Happy to help test fixes on the same dual-GPU box.

[AprilGrimoire]

Supposedly fixed. Now the exported copied buffer is first copied into linear layout.

Fixed another bug of different monitors plugged into the same port not triggering reprobes of RANDR data.

[joske]

great, I'm not sure when I can test it though. I don't always have access to that machine.

[joske]

yserver-hw-mate.log
yserver-hw-cinnamon.log
Tested. Had huge rendering issues. Lots of flickering and windows half rendered, and even lost.

[AprilGrimoire]

Sorry, I'll get more hardware to test on.

[joske]

No worries, you can't test/own all hardware.

[AprilGrimoire]

Hello. I resurrected an old desktop and bought a CPU with iGPU, GTX1050 + i3-12100, and couldn't reproduce the issue. I merged the changes upstream. The previous test run is at log level warn and doesn't tell a lot. Could you please test again with log level debug? I'll clean up the git history to rebase after the final state looks good for merge.

[joske]

Oh wow, you bought hardware for yserver ❤️

How exactly did you test on your HW? I output via the iGPU and one vie the dGPU?

[AprilGrimoire]

Oh wow, you bought hardware for yserver ❤️

How exactly did you test on your HW? I output via the iGPU and one vie the dGPU?

Haha! It's a second-hand CPU (since it's i3-12100). That's the cheapest CPU I can find that fits my motherboard. I can resell it after testing so the cost is infinitesimal.

I tested both directions, and they all worked.

[joske]

I'll test again on Monday (the PC I have for that is in office)