H264: fix botw 1080p cutscene green screen

[lijunyu-cn]

No description provided.

[Exzap]

can you explain the changes

[rcaridade145]

This does conflict with #1894

[lijunyu-cn]

Typically, the length parameter passed to H264DECGetImageSize points to several bytes after the end of the SPS NAL unit. These extra bytes are passed into h264Parser_ParseSPS, which eventually returns false when readTrailingRBSPBits is called within parseNAL_seq_parameter_set_rbsp. This behavior causes H264DECGetImageSize to consistently fail to obtain the correct width and height values.

Considering that h264Parser_ParseSPS, parseNAL_seq_parameter_set_rbsp, and readTrailingRBSPBits are very fundamental decoder functions, much existing code might rely on their current behavior. Therefore, I only made modifications within H264DECGetImageSize. By simply truncating the extra data after the first found SPS NAL unit, h264Parser_ParseSPS can now return the correct width and height parameters.

[Exzap]

Appreciate the PR, I am closing this as it's been superseded by a39e99e.

I had time to look into this a bit and it seems like the implementation changes proposed by this PR are modeled after the h.264 spec and not after how the function actually behaves on Wii U. Or maybe it's just trying to replicate the old behavior but for some reason the code is refactored quite a bit and I don't fully understand for what purpose. But either way since I wanted to make sense of the changes I did my own research and uncovered more problems (that were already present in the existing code in Cemu) which I addressed in commit a39e99e so that essentially ended up superseding this work.

[ppsspp-gamer]

BOTW green screen bug is still there, Exzap your fix fixes nothing. At least test BOTW to verify if your fix work or not.

[Exzap]

I kept tweaking the code after I tested it (bad habit, I know), so I probably broke it then. I'll look into it.

[godelite-lab]

Can you also double check the intro video for Rayman Legends? On Cemu 2.6 it still runs green. Thanks

[Exzap]

That's caused by the resolution graphic pack. You can open an issue over at https://github.com/cemu-project/cemu_graphic_packs