Extremely long first CUDA graph warm-up (~99s) before first TTS request on RTX 5070 Ti (Windows 11)

[cinetube]

Hi,

I'm seeing an unexpectedly long delay during the first TTS request when using CUDA.

System:

• OS: Windows 11
• GPU: NVIDIA RTX 5070 Ti
• Backend: CUDA

The model loads normally:

[2026-07-25 10:25:53] [webui] model MOSS-TTS-Local v1.5 (tts) loaded, elapsed 14.7s

Then I run a very small TTS request (54 characters):

[2026-07-25 10:25:53] [webui] TTS started: model=moss-tts-local, 1 chunks / 54 chars total

Nothing happens for about 99 seconds, then:

[10:27:32] ggml_backend_cuda_graph_compute: CUDA graph warmup complete

The request finishes shortly afterwards:

[2026-07-25 10:27:45] [webui] TTS chunk 1/1 done (54 chars, 112.5s)

Subsequent requests are much faster. For example:

• 1,027 characters
• 2 chunks
• Total time: 29.1s

So this appears to be a one-time CUDA warm-up cost.

My questions are:

1. Is a ~99 second CUDA graph warm-up expected?
2. Is this specific to Windows or the RTX 50-series GPUs?
3. Could this indicate an issue with my CUDA environment or build?
4. Is there a way to perform this warm-up during model loading so the first user request doesn't incur the delay?

I'd be happy to provide additional information such as:

• audio.cpp commit/version
• NVIDIA driver version
• CUDA Toolkit version
• Build configuration
• Full logs

Thanks!

[0xShug0]

@cinetube A 99-second warm-up is definitely long, but there was a Windows-specific loading issue that was fixed just an hour ago in PR #97. Would you like to try the latest commit 99c3725 and test via server or CLI?

There are three possible outcomes:

(1) It fixes the issue in both the CLI and the WebUI.
(2) It fixes the issue in the CLI but not the WebUI.
(3) It does not fix the issue.

I’d like to separate any WebUI-specific issue from the core audio.cpp issue, since the WebUI is maintained by community contributors.

If (2) and (3), please share those additional information.

[cinetube]

@cinetube A 99-second warm-up is definitely long, but there was a Windows-specific loading issue that was fixed just an hour ago in PR #97. Would you like to try the latest commit 99c3725 and test via server or CLI?

There are three possible outcomes:

(1) It fixes the issue in both the CLI and the WebUI. (2) It fixes the issue in the CLI but not the WebUI. (3) It does not fix the issue.

I’d like to separate any WebUI-specific issue from the core audio.cpp issue, since the WebUI is maintained by community contributors.

If (2) and (3), please share those additional information.

Thanks for the quick response.

I'm currently using the prebuilt Windows binaries, so I don't have a way to test commit 99c3725 directly.

If there's a prebuilt binary (or CI artifact) that includes this commit, I'd be happy to test it and report whether:

1. it fixes both the CLI and WebUI,
2. it fixes only the CLI, or
3. it doesn't change the behavior.

If needed, I can also provide my NVIDIA driver version, CUDA version, and complete logs.

[0xShug0]

@cinetube Please try the prebuilt in release v0.4.1.

[cinetube]

@cinetube Please try the prebuilt in release v0.4.1.

Thanks for the suggestion.

I did another test using the prebuilt Windows binaries from release v0.4.1

This time I tested Qwen3-TTS 0.6B, and I'm seeing similar behavior:

• OS: Windows 11
• GPU: NVIDIA RTX 5070 Ti
• Release: v0.4.1 (prebuilt binaries)

Results:

• Model load: 1.7 s

• First TTS request (54 chars): 68.0 s

• ggml_backend_cuda_graph_compute: CUDA graph warmup complete appears about 67 seconds after the request starts.

• Subsequent requests are much faster:

  • 1544 chars (3 chunks): 17.8 s total
  • Chunk times: 6.5 s, 6.6 s, and 4.7 s

Relevant log:

[2026-07-25 11:27:59] [webui] starting audiocpp_server (backend=cuda), loading model Qwen3-TTS 0.6B (tts) …
[11:27:59] ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16302 MiB):
[11:27:59]   Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0
[2026-07-25 11:28:01] [webui] model Qwen3-TTS 0.6B (tts) loaded, elapsed 1.7s
[2026-07-25 11:28:01] [webui] TTS started: model=qwen3-tts, 1 chunks / 54 chars total
[11:29:08] ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[2026-07-25 11:29:09] [webui] TTS chunk 1/1 done (54 chars, 68.0s)

[2026-07-25 11:29:26] [webui] TTS started: model=qwen3-tts, 3 chunks / 1544 chars total
[2026-07-25 11:29:32] [webui] TTS chunk 1/3 done (517 chars, 6.5s)
[2026-07-25 11:29:39] [webui] TTS chunk 2/3 done (625 chars, 6.6s)
[2026-07-25 11:29:44] [webui] TTS chunk 3/3 done (402 chars, 4.7s)
[2026-07-25 11:29:44] [webui] TTS done: total 17.8s

I am going to try again and inform you the results.

[cinetube]

I also tested MOSS-TTS-Local v1.5 using the prebuilt Windows binaries from release v0.4.1

Environment:

• OS: Windows 11
• GPU: NVIDIA RTX 5070 Ti
• Backend: CUDA
• Release: v0.4.1 (prebuilt binaries)

The behavior is consistent with my previous test:

• Model load: 9.6 s

• First TTS request (54 chars): 105.8 s

• ggml_backend_cuda_graph_compute: CUDA graph warmup complete appears about 90 seconds after the request starts.

• The request then finishes about 16 seconds later.

• Subsequent requests are much faster:

  • 1027 chars (2 chunks): 24.4 s total
  • Chunk times: 17.8 s and 6.7 s

Relevant log:

[2026-07-25 11:56:27] [webui] starting audiocpp_server (backend=cuda), loading model MOSS-TTS-Local v1.5 (tts) …
[11:56:27] ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16302 MiB):
[11:56:27]   Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 16302 MiB
[2026-07-25 11:56:36] [webui] model MOSS-TTS-Local v1.5 (tts) loaded, elapsed 9.6s
[2026-07-25 11:56:36] [webui] TTS started: model=moss-tts-local, 1 chunks / 54 chars total
[11:58:06] ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[2026-07-25 11:58:22] [webui] TTS chunk 1/1 done (54 chars, 105.8s)

[2026-07-25 11:58:39] [webui] TTS started: model=moss-tts-local, 2 chunks / 1027 chars total
[2026-07-25 11:58:57] [webui] TTS chunk 1/2 done (699 chars, 17.8s)
[2026-07-25 11:59:03] [webui] TTS chunk 2/2 done (328 chars, 6.7s)
[2026-07-25 11:59:03] [webui] TTS done: total 24.4s

So far, I've observed the same pattern with both Qwen3-TTS 0.6B and MOSS-TTS-Local v1.5 on the v0.4.1 prebuilt binaries: the first CUDA inference incurs a very long one-time warm-up, while subsequent requests are significantly faster.

[0xShug0]

Thanks for the update. Could you use the cli to do one-shot inference?

[cinetube]

Thanks for the update. Could you use the cli to do one-shot inference?

I tested one-shot inference with the CLI using the v0.4.1 prebuilt binaries.

Interestingly, the behavior is different from the WebUI.

• Model: MOSS-TTS-Local v1.5
• OS: Windows 11
• GPU: RTX 5070 Ti
• Backend: CUDA

I ran the CLI twice (each as a separate process).

Both runs completed normally:

• First run: session.wall_ms = 9895 (~9.9 s)
• Second run: session.wall_ms = 13325 (~13.3 s)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete is still printed during the CLI runs, but it does not result in the 68–105 second delay that I consistently observe in the WebUI.

So the long startup latency appears to be reproducible in the WebUI/server, but not in one-shot CLI inference.

log.txt

[0xShug0]

I'll debug it tomorrow morning.

[mirek190]

How much RAM do you have?
I see you tried transformer version but the model is VERY big so can be a problem with caching.
Can you try a Q8 gguf version?

https://huggingface.co/audio-cpp/audio.cpp-gguf/tree/main/MOSS-TTS-Local-v1.5-GGUF

[0xShug0]

@cinetube

It looks like Windows cold disk/OS cache cost.

Windows + Safetensor first run:

[2026-07-25 11:31:02] [webui] starting audiocpp_server (backend=cuda), loading model MOSS-TTS-Local v1.5 (tts) …
[11:31:03] ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32606 MiB):
[11:31:03]   Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
[2026-07-25 11:31:40] [webui] model MOSS-TTS-Local v1.5 (tts) loaded, elapsed 37.8s
[2026-07-25 11:31:56] [webui] TTS started: model=moss-tts-local, 1 chunks / 54 chars total
[11:32:09] ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[2026-07-25 11:32:09] [webui] TTS chunk 1/1 done (54 chars, 13.8s)

model MOSS-TTS-Local v1.5 (tts) loaded, elapsed 37.8s; 54 chars, 13.8s

Windows + Safetensor 2nd run (relaunch server):

[2026-07-25 11:42:31] [webui] starting audiocpp_server (backend=cuda), loading model MOSS-TTS-Local v1.5 (tts) …
[11:42:31] ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32606 MiB):
[11:42:31]   Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
[2026-07-25 11:42:35] [webui] model MOSS-TTS-Local v1.5 (tts) loaded, elapsed 4.6s
[2026-07-25 11:42:48] [webui] TTS started: model=moss-tts-local, 1 chunks / 54 chars total
[11:42:49] ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[2026-07-25 11:42:49] [webui] TTS chunk 1/1 done (54 chars, 1.1s)

model MOSS-TTS-Local v1.5 (tts) loaded, elapsed 4.6s; 54 chars, 1.1s

[0xShug0]

Another test:

Windows + Qwen3 TTS first launch:

[11:52:01] ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32606 MiB):
[11:52:01]   Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
[2026-07-25 11:52:18] [webui] model Qwen3-TTS 1.7B Base (tts) loaded, elapsed 16.3s
[2026-07-25 11:52:18] [webui] TTS started: model=qwen3-tts-1.7b, 1 chunks / 535 chars total
[11:52:20] ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[2026-07-25 11:52:24] [webui] TTS chunk 1/1 done (535 chars, 6.2s)

Relaunch

[2026-07-25 11:54:50] [webui] starting audiocpp_server (backend=cuda), loading model Qwen3-TTS 1.7B Base (tts) …
[11:54:51] ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32606 MiB):
[11:54:51]   Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
[2026-07-25 11:54:53] [webui] model Qwen3-TTS 1.7B Base (tts) loaded, elapsed 2.6s
[2026-07-25 11:54:53] [webui] TTS started: model=qwen3-tts-1.7b, 1 chunks / 535 chars total
[11:54:55] ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[2026-07-25 11:54:59] [webui] TTS chunk 1/1 done (535 chars, 5.9s)
[2026-07-25 11:54:59] [webui] TTS done: F:\audio.cpp\webui\output\audiocpp_tts_1784994899404.wav, total 5.9s

[cinetube]

[2026-07-26 07:44:54] [webui] starting audiocpp_server (backend=cuda), loading model Qwen3-TTS 1.7B Base (tts) …
[07:44:55] ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16302 MiB):
[07:44:55]   Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 16302 MiB
[2026-07-26 07:44:59] [webui] model Qwen3-TTS 1.7B Base (tts) loaded, elapsed 5.2s
[2026-07-26 07:45:00] [webui] input transcoded: audiocpp_up_0p51uxyn.mp3 (.mp3) → 16-bit PCM WAV
[2026-07-26 07:45:00] [webui] TTS started: model=qwen3-tts-1.7b, 1 chunks / 54 chars total
[07:46:08] ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[2026-07-26 07:46:09] [webui] TTS chunk 1/1 done (54 chars, 69.6s)
[2026-07-26 07:46:09] [webui] TTS done: C:\TTS\AudioCPP-2\webui\output\audiocpp_tts_1785030369663.wav, total 69.6s
[2026-07-26 07:46:37] [webui] TTS started: model=qwen3-tts-1.7b, 3 chunks / 1544 chars total
[2026-07-26 07:46:45] [webui] TTS chunk 1/3 done (517 chars, 7.6s)
[2026-07-26 07:46:53] [webui] TTS chunk 2/3 done (625 chars, 7.9s)
[2026-07-26 07:46:59] [webui] TTS chunk 3/3 done (402 chars, 5.9s)
[2026-07-26 07:46:59] [webui] TTS done: C:\TTS\AudioCPP-2\webui\output\audiocpp_tts_1785030419264.wav, total 21.4s

using - prebuilt binaries v0.4.2.

Problem not solved yet.

[0xShug0]

Based on my testing, this doesn't appear to be an audio.cpp or WebUI issue. It looks more like the cost of Windows cold disk/OS caching. The evidence is that simply restarting the server makes the abnormal cold-start time disappear. If this were a code issue, restarting the server should produce consistent behavior. That said, the long response time on the first request is probably unavoidable on Windows.

[mirek190]

cinetube used webui which is extremely heavy for current use ( python and gradio ) if you compare to cli / server.
We should go to a lighter implementation like is using llamacpp (SvelteKit/TypeScript) then we could build in webui straight into server binary.