Skip to content

Optimize CUDA inference - #40

Draft
alvarobartt wants to merge 15 commits into
mainfrom
optimized-for-cuda
Draft

alvarobartt wants to merge 15 commits into
mainfrom
optimized-for-cuda

Conversation

@alvarobartt

@alvarobartt alvarobartt commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Description

This PR improves CUDA serving for Qwen3.5 decision models with fused operations, packed batching and memory-aware token admission. Model heads reuse the shared architecture code, with CPU and Metal implementations kept separate.

Cloudflare/clef-flash BF16 (17f0b0a), one NVIDIA H200; sys1 16965e3, SGLang 0.5.23. Two runs per engine: 748 requests per curve point; 17,952 requests per sustained run.

Concurrency Engine Mean request (ms) p95 (ms) Requests/s
1 sys1 34.64 89.94 28.79
1 SGLang 55.75 98.10 17.89
4 sys1 113.76 289.62 35.11
4 SGLang 137.58 295.19 29.05
16 sys1 430.40 692.41 37.07
16 SGLang 445.08 722.31 35.86
32 sys1 834.51 1,147.65 38.08
32 SGLang 834.58 1,155.36 37.81
64 sys1 1,615.77 2,104.03 38.66
64 SGLang 1,627.23 2,195.67 38.11
128 sys1 3,085.89 4,901.79 38.83
128 SGLang 3,185.50 4,412.97 38.18
128 sustained sys1 3,281.88 5,296.82 38.86
128 sustained SGLang 3,335.62 4,163.64 38.27
512 sustained sys1 12,964.54 15,511.16 38.87
512 sustained SGLang 13,188.64 14,420.82 38.30

@alvarobartt
alvarobartt force-pushed the optimized-for-cuda branch 7 times, most recently from 8affc78 to 7a6881a Compare November 4, 2026 19:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant