Sleeping logic and max_num_batched_tokens - #8

Open

toslali-ibm wants to merge 4 commits into

Commits on May 27, 2025

Fix batched tokens
toslali-ibm
committed
2dedc6e

Commits on Jun 9, 2025

Bring back is sleeping logic to help remove warnings
toslali-ibm
committed
dae750a
Bring back is sleeping logic to help remove warnings
toslali-ibm
committed
6d5d179
Adjust max_num_batched_tokens
toslali-ibm
committed
0d2189f