Inference Speed Comparisons

[crlandsc]

This is a fantastic paper, repository, and contribution to the field of audio source separation! Thank you so much for sharing. The reduction in model size while still improving separation ability is impressive!

I saw inference speed comparisons in the paper with different configurations of the DTTNet as well as model parameter size comparisons vs other frameworks. However, I was curious if you had inference speed comparisons with other networks.

• i.e. Expand Table 1 with BSRNN, HDemucs/HTDemucs, & TFC-TDF UNet, etc.

I am also interested if you tested its inference time on CPU in addition to GPU.

Again, thanks so much for this contribution!

[crlandsc]

I also noticed that you were slicing the dim_f to 2048 for "vocals", "drums", and "other", and 864 for "bass". This effectively cuts off frequencies for the first 3 stems at 14,700 Hz and bass at ~6,200 Hz.

I was curious about the reasoning for this compared to just using an n_fft of 4096 and hop_length of 1024 to achieve 2048 frequency bins (with no slicing) at full resolution? Was it determined that frequencies above the cutoffs were not critical for SDR for each different output as a tradeoff for computational efficiency?

Thanks so much!

Vocals:

Bass:

[junyuchen-cjy]

1. Inference speed: The published version includes it, but it can't be uploaded to arXiv due to IEEE restrictions. We generally don't track CPU time, mainly because the inference process can become quite time-consuming, especially with a 3-minute 44100Hz audio file.

Time on gpu:

Time on cpu:

2. Window size: A higher n_ftt yields higher SDR, as per TFC-TFC Unet v2. Setting it to 4096 might not be effective.

3. Cut-off: Since we have a big window size, this trick reduces computation costs. Low-frequency bins contribute more to SDRs, so it is fine to remove some of the high-frequency bins. Also, It prevents nan SDRs in our model during training.

[crlandsc]

Thanks so much for your response! I am interested in testing the differences between different n_fft sizes as well as without the frequency cutoff. Because the tradeoff in speed or slightly lower SDR may be worth it for improved perceptual quality.