server: Normalize pixel format depth from client - #165
Merged
Conversation
Owner
|
Did you see this? #160 (comment) |
Some VNC clients (e.g. macOS Screen Sharing) report depth=32 in their SetPixelFormat message for standard RGB888 pixel formats. This causes calc_bytes_per_cpixel to return 4 instead of 3, corrupting ZRLE tile data. Normalize the depth field using rfb_pixfmt_depth() when receiving the client's pixel format. This derives the true depth from the RGB max values, ensuring correct CPIXEL encoding downstream.
alistairreay
force-pushed
the
fix/cpixel-calculation
branch
from
February 20, 2026 18:12
74f6621 to
c5de84f
Compare
Contributor
Author
Ahh yes, thanks for pointing that out. I've reworked the fix to normalize depth using rfb_pixfmt_depth() in on_client_set_pixel_format() instead of modifying the encoding logic. Should be a much cleaner approach. |
Owner
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Some VNC clients (e.g. macOS Screen Sharing) report
depth=32in theirSetPixelFormat message for standard RGB888 pixel formats. This causes
calc_bytes_per_cpixelto return 4 instead of 3, corrupting every ZRLEtile -- the encoder writes 4 bytes per pixel but the client expects 3.
This patch normalizes the
depthfield usingrfb_pixfmt_depth()whenreceiving the client's pixel format in
on_client_set_pixel_format().This derives the true depth from the RGB max values (via popcount),
ensuring correct CPIXEL encoding downstream without modifying the
encoding logic itself.
Testing
Tested with macOS Screen Sharing (which sends depth=32) connecting to
wayvnc with ZRLE encoding. Before the fix, the display showed corrupted
flashing tiles. After the fix, the display renders correctly.
Also verified that existing clients sending depth=24 are unaffected --
rfb_pixfmt_depth()returns 24 for standard RGB888, same as what thoseclients already report.
I have read and understood CONTRIBUTING.md