fix: Render a synchronous parent with the default effect when capturing filter input directly above it

[hanatyan128]

Summary

Fixes the black output reported in #170. In filter input mode, when Branch Output is the filter closest to its parent source and the parent is a synchronous source that is neither custom-draw nor async (for example an Image source), the branch output now shows the source and the source stays visible in the main preview. effect_setval_inline: invalid param and device_draw (D3D11): No vertex shader specified are no longer logged on every frame.

• FilterVideoCapture::captureFilterInput() renders the filter target with obs_source_default_render() when the target is the parent itself and the parent has neither OBS_SOURCE_CUSTOM_DRAW nor OBS_SOURCE_ASYNC. Otherwise it keeps obs_source_video_render().
• The condition is the same one libobs uses in obs_source_process_filter_begin_with_color_space() and obs_source_skip_video_filter().
• The parent is read with obs_filter_get_parent() on every frame, next to obs_filter_get_target(), so the branch follows the filter's current position in the list.
• The trigger render in renderTexture(), which drives the parent's filter chain while the source is not rendered by the main mix, reaches captureFilterInput() through that chain and is fixed by the same change.

Only src/video/filter-video-capture.cpp is changed.

Root cause

• When Branch Output is the only filter or the first one in the list, libobs sets its target to the parent itself.
• captureFilterInput() rendered the target with obs_source_video_render(). Inside the parent's own filter chain, that call goes to obs_source_main_render(), which passes the current effect (gs_get_effect()) to the parent's video_render. No technique is active at that point, so the parent receives NULL.
• A source that draws with the effect it is given, such as image_source, then draws nothing. The texrender stays transparent, but captureFilterInput() still returned true, so the transparent texture replaced the source in the main mix and was also encoded as the branch output.

Design notes

• The trigger render in renderTexture() keeps obs_source_video_render(parentSource). obs_source_default_render() would draw the parent without its filter chain, so captureFilterInput() would never be called from that path.
• Behavior is unchanged for custom-draw parents (scenes, groups, color sources), async parents (media sources, video capture devices), a Branch Output with another filter between it and the parent, and source output mode.

Known limitations

• An Image source with "Unload image when not showing" enabled still outputs black in filter input mode while it is not rendered by the main mix. Only the proxy source is attached to the branch view in filter input mode, so the parent is never marked as showing and the image stays unloaded. This is unchanged from dev and applies to any source that releases its content while hidden.
• When a filter without video rendering (such as an audio filter) sits between the parent and Branch Output, libobs forwards the render from that filter to the parent, which again receives a NULL effect. Effect filters placed there are affected the same way in libobs itself, so this is not handled here.

Verification

• clang-format 17 check passed.
• RelWithDebInfo build passed (Windows x64).
• One round of design review (OBS reviewer) found no Critical or Major issues.
• One round of code review (OBS reviewer) and comment review found no issues.
• One round of adversarial review (OBS, C++, A/V and comment reviewers) found no actionable issues. Its one Info note is the "Unload image when not showing" limitation above.

Manual testing

Set the video source to "Filter Input (Experimental)" and enable any output (recording is enough). Keep "Unload image when not showing" off on the Image source. Watch the log for effect_setval_inline: invalid param and device_draw (D3D11): No vertex shader specified.

Image source parent:

• Branch Output as the only filter. The recording shows the image, the image stays visible in the main preview, and neither log line appears.
• Branch Output first in the list, followed by another effect filter (for example Color Correction). Same result as above.
• Another effect filter between the Image source and Branch Output. The image is captured as on dev.
• Branch Output as the only filter, with the Image source's scene not in Program. The recording shows the image.

Other parents and modes:

• Scene, Color Source and Media Source parents. The output and the main preview are the same as on dev.
• Source output mode ("Independent Mix (Default)"). The output is the same as on dev.

Shutdown:

• After enabling and disabling the filter, exit OBS. It exits normally and the log reports 0 memory leaks.

Environments:

• OBS 30.1.2 (Windows x64)
• OBS 31.1.2 (Windows x64)
• OBS 32.2.2 (Windows x64)

Fixes #170

🤖 Generated with Claude Code