Skip to content

fix(call): improve virtual background performance - #17450

Open
Antreesy wants to merge 3 commits into
mainfrom
fix/noid/virtual-bg-performance
Open

fix(call): improve virtual background performance#17450
Antreesy wants to merge 3 commits into
mainfrom
fix/noid/virtual-bg-performance

Conversation

@Antreesy

@Antreesy Antreesy commented Mar 20, 2026

Copy link
Copy Markdown
Contributor

☑️ Resolves

Goal is to reduce GPU load while maintaining similar blur/mask quality and responsiveness

fix(virtual-bg): use requestVideoFrameCallback when available

  • render loop was scheduled via TimerWorker that fired with 1000/frameRate interval
  • this allowed delay for up to 1-2 frames
  • requestVideoFrameCallback fires exactly when a new video frame is available
  • cancel on stopping the effect
  • add _lastInferenceTimestamp guard for MediaPipe's VIDEO mode (strictly requires incrementing values)
  • should reduce delay, but negligible

fix(virtual-bg): keep _lastMask open while being reused

  • when inference hasn't finished yet, previous _lastMask is reused for frame output
  • _lastMask references should be kept open until replaced ot not needed anymore
  • most noticeable, if inference is throttled (manually or due to perf issues)

fix(virtual-bg): avoid GPU/canvas reallocation every frame

  • texMaskFiltered, texBlurred1, texBlurred2 were reallocated on every single render() call
  • reallocating GPU texture every frame is a confirmed overhead on some GPUs/drivers
  • needs to be only reallocated when the actual size changes
  • similar with output canvas element
  • impact is negligible, so more a hardening

AI (if applicable)

  • The content of this PR was partly or fully generated using AI

🖌️ UI Checklist

🖼️ Screenshots / Screencasts

Tested on Windows with Intel(R) UHD Graphincs

🏚️ Before 🏡 After
A B

🚧 Tasks

  • Check for possible mask freezing in _runInference with active video frame and _postProcessing
  • Uncaught (in promise) RuntimeError: divide by zero
  • Other errors

🏁 Checklist

  • 🌏 Tested with different browsers / clients:
    • Chromium (Chrome / Edge / Opera / Brave)
    • Firefox
    • Safari
    • Talk Desktop
    • Integrations with Files sidebar and other apps
    • Not risky to browser differences / client
  • 🖌️ Design was reviewed, approved or inspired by the design team
  • ⛑️ Tests are included or not possible
  • 📗 User documentation in https://github.com/nextcloud/documentation/tree/master/user_manual/talk has been updated or is not required

@Antreesy

This comment was marked as resolved.

@nemphys

This comment was marked as resolved.

@Antreesy
Antreesy force-pushed the fix/noid/virtual-bg-performance branch from b75813b to 873612b Compare March 23, 2026 20:52
@Antreesy Antreesy changed the title Fix/noid/virtual bg performance fix(call): improve virtual background performance Mar 23, 2026
@Antreesy
Antreesy force-pushed the fix/noid/virtual-bg-performance branch from 873612b to 3291f0a Compare April 2, 2026 10:58
@Antreesy
Antreesy force-pushed the fix/noid/virtual-bg-performance branch from 3291f0a to 0362ce6 Compare June 10, 2026 15:05
@Antreesy
Antreesy force-pushed the fix/noid/virtual-bg-performance branch from 0362ce6 to c5e9ef3 Compare September 1, 2026 13:14
@nextcloud-command nextcloud-command added the AI assisted This PR contains AI-assisted commits label Sep 1, 2026
- render loop was scheduled via TimerWorker that fired with 1000/frameRate interval
- this allowed delay for up to 1-2 frames
- requestVideoFrameCallback fires exactly when a new video frame is available
- cancel on stopping the effect
- add _lastInferenceTimestamp guard for MediaPipe's VIDEO mode (strictly requires incrementing values)
- should reduce delay, but negligible

Assisted-by: ClaudeCode:claude-sonnet-5
Signed-off-by: Maksim Sukharev <antreesy.web@gmail.com>
- when inference hasn't finished yet, previous _lastMask is reused for frame output
- _lastMask references should be kept open until replaced ot not needed anymore
- most noticeable, if inference is throttled (manually or due to perf issues)

Assisted-by: ClaudeCode:claude-sonnet-5
Signed-off-by: Maksim Sukharev <antreesy.web@gmail.com>
- texMaskFiltered, texBlurred1, texBlurred2 were reallocated on every single render() call
- reallocating GPU texture every frame is a confirmed overhead on some GPUs/drivers
- needs to be only reallocated when the actual size changes
- similar with output canvas element
- impact is negligible, so more a hardening

Assisted-by: ClaudeCode:claude-sonnet-5
Signed-off-by: Maksim Sukharev <antreesy.web@gmail.com>
@Antreesy
Antreesy force-pushed the fix/noid/virtual-bg-performance branch from c5e9ef3 to 941c071 Compare September 1, 2026 13:16
@Antreesy
Antreesy marked this pull request as ready for review September 1, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI assisted This PR contains AI-assisted commits feature: call 📹 Voice and video calls feature: frontend 🖌️ "Web UI" client performance 🚀

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants