Use RGBA16F for feedback texrenders to fix 8-bit precision loss in self-feedback shaders

[h-e-d-o-n]

Problem

create_or_reset_texrender() hardcodes gs_texrender_create(GS_RGBA, GS_ZS_NONE) (8-bit UNORM) for all four ping-pong feedback texrenders: input_texrender, previous_input_texrender, output_texrender, previous_output_texrender.

Any shader that reads its own previous output back through previous_image/previous_output and applies a small per-frame change — accumulating a value, or decaying it by a fraction each frame — gets stuck at an 8-bit UNORM rounding fixed point instead of continuing to converge. This is a general correctness issue for any self-feedback shader with fine gradients, not tied to any specific effect.

How it was measured

A minimal test shader seeds a state to 1.0 and multiplies it by 0.99 every frame, reading the value back from previous_output. On stock obs-shaderfilter (OBS Studio 32.2.1-6, obs-shaderfilter 2.6.0, Linux) the state decayed 249 → 202 → 159 → 133 → 110 → 87 → 65 → 49 and then got stuck at exactly 50/255 for 2468 consecutive frames (82.3s) and never moved again. That's the 8-bit UNORM round-trip fixed point: 50 * 0.99 = 49.5, which rounds straight back to 50.

The fixed point isn't specific to 0.99 — any self-feedback loop with a multiplicative or additive step small enough to round away at 8 bits will converge to some non-zero rounding floor instead of the mathematically correct limit.

Notably, gs_get_format_from_space() is already called elsewhere in this file (get_input_source(), draw_output(), and the per-source-param texrender at line ~3091) to pick a float format for HDR/extended color spaces — but that path only returns a float format when the source itself is HDR. For an ordinary SRGB source (the common case), gs_get_format_from_space(GS_CS_SRGB) returns plain 8-bit GS_RGBA, so threading that value into create_or_reset_texrender() would not have fixed the measured bug for typical content — the feedback loop needs float precision independent of the source's own color space.

Fix

Switch the one shared gs_texrender_create() call in create_or_reset_texrender() from GS_RGBA to GS_RGBA16F. This is a minimal, single hardcoded-constant change — no new parameters, no behavior change to anything else in the file.

What was and wasn't verified

• Verified: the modified obs-shaderfilter.c compiles cleanly (gcc -std=gnu17 -Wall -Wextra) against the real, installed OBS Studio 32.2.2 SDK headers (libobs.so.30) on the machine this was developed on, with zero new warnings versus the unmodified file.
• Verified: a full CMake configure + build against that same installed libobs (via find_package(libobs)) succeeds and produces a working obs-shaderfilter.so that links correctly against libobs.so.30 and exports obs_module_load/obs_module_get_string/etc. (One unrelated pre-existing warning in load_shader_from_file() — a const-qualifier discard at the strrchr(file_name, '/') call, line ~299, present in master before this change — had to be demoted from the SDK's -Werror to build cleanly on a newer GCC; it is untouched by and unrelated to this PR, called out here rather than silently worked around.)
• Not verified: end-to-end behavior inside a running OBS Studio (i.e., actually loading the built plugin into OBS and confirming the smooth-decay shader now converges to 0 instead of sticking at a fixed point). That would need a live OBS instance with a GPU; not available in the environment this fix was developed in. The 8-bit-vs-16F reasoning and the surrounding format-selection code were verified directly against the compiled OBS SDK headers, and the plugin itself now builds and links, but the actual GPU-side runtime behavior of GS_RGBA16F texrenders in this plugin has not been observed on-screen.

Scope

Single hardcoded constant changed in one function. No other files touched.