Reproducing Latency Results for SparseD

[Justinhhhh]

Hi,

I attempted to reproduce the latency results reported in the paper; however, in both cases, the original model appears to be more efficient than SparseD.

Here are the commands I used:

python dream_generation.py --origin --prompt 4k
Running Time: 45.8935
python dream_generation.py --skip 0.2 --select 0.3 --block_size 128 --prompt 4k
Running Time: 79.7773

[INV-WZQ]

Hi @Justinhhhh ,

Thank you for sharing your experimental results. Regarding the latency issue you mentioned, we recommend conducting a warm-up inference first when using our sparse attention, as it is implemented with FlexAttention. After the warm-up, subsequent inferences should demonstrate improved speed. For instance, running python dream_generation.py --skip 0.2 --select 0.3 --block_size 128 --prompt 4k a second time will reflect this improvement. Additionally, the speed enhancement is more noticeable on GPUs like the A800.

If you have any further questions or need additional assistance, please feel free to reach out!

Thank you!