ExtendedDesktopSize advertised with screen id 0 breaks client-initiated resize (Guacamole/guacd)

[ThilinaTLM]

Summary

neatvnc advertises its ExtendedDesktopSize screen with id = 0. libvncclient (LibVNCServer) explicitly treats a screen with id == 0 as invalid and never stores it into client->screen. As a result, any libvncclient-based client cannot perform a client-initiated desktop resize against neatvnc/wayvnc.

The most visible victim is Apache Guacamole / guacd (which added VNC auto-resize in 1.6.0, GUACAMOLE-1196): the browser desktop never resizes to the viewport. guacd logs Screen data has not been initialized and Failed to send desktop size message on every resize attempt.

Root cause

In src/server.c, send_extended_desktop_size_rect() builds the screen entry without setting id, so it defaults to 0:

struct rfb_screen screen = {
    .width = htons(width),
    .height = htons(height),
};

On the client side, libvncclient's ExtendedDesktopSize handler (libvncclient/rfbproto.c) only records the screen when id != 0:

if (screen.id != 0 && screen.width && screen.height) {
    client->screen = screen;
} else {
    invalidScreen = TRUE;   /* id == 0 is discarded */
}

Because the advertised id is 0, client->screen.width/height stay 0. guacd then refuses to send a SetDesktopSize, guarded by:

/* guacamole-server: src/protocols/vnc/display.c */
if (client->screen.width == 0 || client->screen.height == 0) {
    guac_client_log(gc, GUAC_LOG_DEBUG, "Screen data has not been initialized, yet.");
    return FALSE;
}

So the negotiation looks healthy (the client advertises extended-desktop-size, neatvnc sends the resize rect), but the resize round-trip can never start.

Reproduction

1. Run wayvnc on a headless Sway output.
2. Connect through Guacamole/guacd 1.6.0 (VNC, disable-display-resize unset/false).
3. Resize the browser window → desktop stays at its initial size (letter-/pillar-boxed).
4. guacd trace shows repeated Setting VNC display size. → Failed to send desktop size message. and Screen data has not been initialized.

Confirmed via wayvnc debug logs: Client set encodings: ...,extended-desktop-size,... and Sending extended desktop resize rect: 1920x1080, but no client-initiated resize ever follows.

Proposed fix

Advertise a non-zero screen id. id 1 works well in practice — it is what guacd uses for its own resize message (GUAC_VNC_SCREEN_ID = 1), so the subsequent SetDesktopSize is accepted by check_desktop_layout() for the single-output case:

 	struct rfb_screen screen = {
+		.id = htonl(1),
 		.width = htons(width),
 		.height = htons(height),
 	};

With this one-line change, Guacamole/guacd resizes the wayvnc desktop to match the browser viewport on connect and on every window resize.

(Arguably libvncclient's id != 0 requirement is also overly strict, but advertising a non-zero id is the simplest interoperable fix and is harmless to spec-compliant clients.)

Environment

• neatvnc 0.9.5, wayvnc 0.9.1, Sway 1.12 (wlroots headless backend)
• guacd / guacamole-server 1.6.0 (bundled libvncclient 0.9.15)
• Arch Linux

[any1]

Did you report this to libvncserver?

[ThilinaTLM]

Did you report this to libvncserver?

No, not yet. Should I? Could you direct me to the repo?

[Micket]

@ThilinaTLM It's very easy to find. https://github.com/LibVNC/libvncserver

In particular, note this commit:
LibVNC/libvncserver@041ea57

Not part of any release yet.
In summary: guacd should update their vendored sources to include at least this commit. Also libvnserver should make a release.

(Though, the slow rate of releases, updating libcvncserver, then vendored sources in guacd, and updates to remmina, then distros need to pick it up, it's probably many more years until this is useful.. so i'll probably steal that 1 line patch to pick screen id = 1)

EDIT: The workaround wasn't that simple for Remminas issues. Something about the display layout isn't getting initialized it seems. But just building libvncserver from latest commit, and building remmina on top of that, client side resizing works against Wayvnc. So, we it really only needs a libvncserver release for the sake of guacd and remmina (and all other clients that might exist).