Repository navigation
[BUG] Higher audio latency after fixes for audio desync #9
Description
Describe the bug
Audio latency is roughly twice as much after commits for fixing audio desync issues
To Reproduce
gpu-screen-recorder -w screen -f 120 -a "$(pactl get-default-sink).monitor" -cr full -fm cfr -ac opus -k hevc -q very_high -o ~/videos/video.mkv
Expected behavior
Lower audio latency
Screenshots
"nodesync-build" uses the commits that fixed the audio desync. "desync-build" is the build I used to report #7. the lines show when it appears on video, the waveform shows the audio.
Desktop (please complete the following information):
-
X11 or Wayland: Wayland
-
Desktop environment/Window Manager: Sway
-
Distro: Debian
-
GPU: Radeon RX 6700XT
-
Version: Source
-
I played the video with the command
mpv --no-config video.mp4(if applicable) -
I use a laptop with an integrated GPU and a dedicated GPU
Activity
dec05eba commentedon Apr 11, 2024
Just for reference, what is the audio latency like with obs studio when compared in the same way when recording at 120 fps?
Stoppedpuma commentedon Apr 11, 2024
It's nearly in the middle of the two, I do have obs recording multiple audio tracks (all of which are the same on timing) so I don't know if that affects latency for obs or not.
dec05eba commentedon Apr 11, 2024
Ok thanks so it might not be possible to sync audio 100% with video if other screen recorders cant do it either, but I know what I can do to improve it at least. I'll test some things when I have time.
Just a question, what type of audio output device are you recording with? is it a bluetooth device, speakers, analog headset or what?
Stoppedpuma commentedon Apr 11, 2024
My output device is my headphones being plugged into the 3.5mm headphone jack on my motherboard.
I'm using pipewire with a minimum quantum of 16 and the following allowed clock rates:
44100 48000 88200 96000 176400 192000OBS audio settings: 48khz Sample rate, Stereo channels, 3 Audio tracks, FFMpeg Opus 192kbps
dec05eba commentedon Apr 12, 2024
I think one of the reasons for this might be that opus is temporary disabled and with aac a larger audio buffer is used so it might take a tiny bit longer to fill the audio buffer before its saved. But this should be fixable.
dec05eba commentedon Apr 15, 2024
Hmm I tested this and I cant reproduce it myself (with the command you used). I pretty much dont have any audio/video desync with the latest commit, kdenlive displays the video and audio at the exact same time. I tested with this: https://www.youtube.com/watch?v=ucZl6vQ_8Uo. I tested with taking latency into account but there wasn't any noticable difference.
I do have latency if I use my bluetooth device but I get the same latency in all audio recording software with that device and I can fix that by changing latency offset in pavucontrol for the output device.
But I pushed the code with latency compensation to git that you can try.
Stoppedpuma commentedon Apr 16, 2024
I'll test the new commit now.
Were you using a 120 fps preset on kdenlive during your testing?
Stoppedpuma commentedon Apr 16, 2024
New commit doesn't seem to help?On kdenlive I noticed it was inconsistent between 8-10 moves on the waveform on the new commit (I don't recall it being inconsistent before but I might be wrong)(See below the tests), I'm also receiving an error I've not had before on the new commitamdgpu: amdgpu_cs_ctx_create2 failed. (-13)Comparison tests done in osu!:
compensation (latest commit)
desync fix
obs
I've rewatched the videos using osu! as a method of comparison instead of the same video you linked, the latest commit does seem to help a bit? The new commit in comparison to the desync fix build seems to be ahead of the previous commit, I can't tell if it's ahead of the video or just me being used to obs latency. I would verify this manually but I seem to have two different applications telling me different things. In case of kdenlive all of them are late, in case of manually scrolling frames in mpv they are early? I am also noticing that the framerate on the latest commit appears to be lower even though it's supposed to be recording at 120 fps? The video file shows 120 fps but it doesn't look like 120 fps when compared to obs and the desync fix build? This might just be from an issue on my end or my eyes tricking me but I'll do more tests spread across more applications, videos, etc for both.
Majority of people aren't really going to notice the delay in less latency sensitive applications and I don't want to burden you with a bunch of different attempts to fix this. If you're fine with how the latency is now then I think it's in an alright enough state.
Stoppedpuma commentedon Apr 16, 2024
@dec05eba Question, I'm assuming you're not aware of gpu-screen-recorder basically just giving up on trying to output the correct framerate when using stupidly high values such as 240? Should I open another issue for this? I'm able to record just fine at 240 fps on obs.
dec05eba commentedon Apr 16, 2024
@Stoppedpuma it's likely a hardware limitation with your gpu. are you recording with vaapi and hevc on obs at the same resolution? you can switch codec to record at higher fps. But also when recording at such high fps it wont really work to record with constant framerate if the gpu cant keep up. obs studio doesn't even let me select higher than 120fps with vaapi even though the gpu can handle at least 200 fps.
Stoppedpuma commentedon Apr 16, 2024
200fps seems to be where things break if I go above it. On OBS I'm able to record 240fps at 1440p, gpu-screen-recorder can't seem to? Both are using vaapi hevc for encoding. Its fine if you consider this out of the scope since it's extremely high, I was just trying to use this high of a framerate for the latency tests.
dec05eba commentedon Apr 16, 2024
I tested recording with pure ffmpeg (kmsgrab) and it records at around the same fps as gpu screen recorder so if you are recording with vaapi hevc in obs studio then the fps it reports is not accurate. You would have to step frame by frame in the video it outputs to see that it's really 240fps and not just duplicating frames
Stoppedpuma commentedon Apr 16, 2024
When going frame by frame in mpv, obs seems to output 240? Maybe I'm misunderstanding this option in obs?
Just to make sure you're seeing the same behaviour on your end, if you record with gpu-screen-recorder at 240fps it just outputs like 15 fps for you as well correct?
12 remaining items
dec05eba commentedon Apr 16, 2024
i tested and audio buffer didn't seem to change the audio/video sync. And for the obs recording I meant that its more accurate for your than for me (for me it has the exact same latency as gpu screen recorder), and if that is because you have changed some obs settings or because of multiple audio tracks i dunno.
dec05eba commentedon Apr 16, 2024
Oh I think I know what it can be... the latency is affected by the devices you record. Which devices did you record in obs studio? some devices can actually make the audio play before the video (my bluetooth does this), so if you have multiple different devices they cancel the latency out. Thats why you need to compare exact same recording in obs and gpu screen recorder with exact same audio device.
Stoppedpuma commentedon Apr 16, 2024
My multiple tracks contain the following:
Track 1: Desktop and Mic
Track 2: Desktop
Track 3: Mic
The comparisons are with Track 1 and just desktop audio on gpu-screen-recorder. Both are using the same output device.
dec05eba commentedon Apr 16, 2024
i dont know if you can just compare the track alone. Obs might have some track syncing going on. It can also affect how pipewire itself behaves when you record from multiple devices at once.
Stoppedpuma commentedon Apr 16, 2024
Wait I might be stupid and possibly know what it is, gpu-screen-recorder records at 44.1khz, my obs profile records at 48khz. Sample rates affect pipewire latency so maybe this could be it?
dec05eba commentedon Apr 16, 2024
gpu screen recorder records at 48khz
Stoppedpuma commentedon Apr 16, 2024
I must have misread 44.1khz some where then, I'll do single track testing.
I still do have the
amdgpu: amdgpu_cs_ctx_create2 failed. (-13)error from latest commit, should I open this as another issue or is this a quick fix?dec05eba commentedon Apr 16, 2024
https://gitlab.freedesktop.org/wlroots/wlroots/-/issues/3543#note_1661557 it's this warning, it's harmless. But that shouldn't appear if you run gpu screen recorder after you install it with sudo ./install.sh. Are you instead building it and then running it from the same directory?
dec05eba commentedon Apr 16, 2024
it might not just be single track but other obs settings you have changed. I dont know if its worth testing without resetting obs to default. But its ok, you dont have to go through the trouble for that. I believe from the video you posted that it's not an issue in gpu screen recorder as it matches obs for me.
But oh I have one question. When I tested with obs studio and gpu screen recorder I used the audio output capture (pulseaudio) with a pipewire server (gpu screen recorder also uses pulseaudio client code). Did you also do that in obs studio? or did you use a pipewire audio output capture (i dont even have such an option in my obs)?
Stoppedpuma commentedon Apr 16, 2024
Didn't even think of that being a reason, fixed after install.
I didn't add the audio as a source, I chose my output in settings so whichever one that uses?
Maybe these values are why I'm noticing a difference?
dec05eba commentedon Apr 16, 2024
that only seems to make a difference in microseconds so I dont think so. But obs syncs well for me now as well, I dont know what I changed.. But I pushed a change to gpu screen recorder. Can you try it? if it doesn't work well then fuck it
Stoppedpuma commentedon Apr 16, 2024
I haven't the slighest fucking idea what you did either but it's basically on the same frame now!
dec05eba commentedon Apr 16, 2024
I added a hardcoded value that it shifts the audio by. Hey, if it works it works lol
Stoppedpuma commentedon Apr 16, 2024
The yellow is where the audio outputs for all the recordings
Correction: the red one titled desync fix is acutally the commit prior to the latest
dec05eba commentedon Apr 16, 2024
Thanks for testing. I'll go with this until somebody complains that it broke audio sync for them xd