TUNA-FISH

Minimal PyTorch Implementation of RQ-Transformer(Fish Speech S2-Pro...) and DAC style RVQ Audio Codec Inference

Fish Speech S2-Pro Paper | GitHub | Hugging Face model

Complete rewrite of Fish Speech S2-pro Inference with around 150 lines of clean code for dualAR/RQ-Transformer, 100 lines for DAC style RVQ audio tokenizer. using flex-decoding to achieve fullgraph in RQ-Transformer model decoding stage(typically slowAR decode + fastAR loop in fishspeech s2-pro).

Featuring:

1. Very clean code, <150 lines of python for dualAR inference, <100 lines for pure RQ-Transformer model forward and prefill/decode. <100 lines for fish-speech RVQ audio tokenizer inference.
2. Achieved fullgraph in dualAR decode stage, and got considerable decoding speed on H20 GPU with 135 token/s. for audio tokenizer got encode 1596.55X ratio and 0.000626 RTF, decode 1171.61X ratio and 0.000854 RTF on single 5090 GPU.
3. Clean dependence, torch is all you need.
4. Support int8/fp8/int4/nvfp4 quantization. Due to the simplicity of the code, you can easily use various quantization methods in torchao
5. ‌Minimized Hardware Requirements, Only 5GB GPU needed to run the online inference web app with both dualAR (4bit) and codec (chunk inference)

Inference

Important

License Notice fishaudio s2-pro model weights are released under FISH AUDIO RESEARCH LICENSE. Please refer to LICENSE for more details.

1. build environment

uv pip install -U --pre torch torchao --index-url https://download.pytorch.org/whl/nightly/cu132
uv pip install tqdm safetensors tokenizers soundfile soxr
# install mslk for torchao NVFP4 Quantized Inference  
# uv pip install -U --pre mslk --index-url https://download.pytorch.org/whl/nightly/cu132
# install torch-tensorrt for codec decode maximum throughput
# uv pip install -U --pre torch-tensorrt --index-url https://download.pytorch.org/whl/nightly/cu130

git clone https://github.com/reesertg/tuna-fish
cd tuna-fish

# get vocab codec and weights
# replace huggingface.co to hf-mirror.com if needed
mkdir ./s2-pro

wget https://huggingface.co/fishaudio/s2-pro/resolve/main/tokenizer.json -P ./s2-pro
wget https://huggingface.co/fishaudio/s2-pro/resolve/main/codec.pth -P ./s2-pro
wget https://huggingface.co/fishaudio/s2-pro/resolve/main/model-00001-of-00002.safetensors -P ./s2-pro
wget https://huggingface.co/fishaudio/s2-pro/resolve/main/model-00002-of-00002.safetensors -P ./s2-pro

local generate

# Generate RVQ codes based on the given prompt_text, prompt_tokens and text in dualAR.py
# you can run fishspeech s2-pro on 8GB GPU
# quantize + compile may take several minutes at first run
# got prefill 5000~6000 token/s, decode 26~27 token/s with batch 1 + RTX 4070 Laptop GPU + fp8 quantization
python dualAR.py

web app

python app.py

Inference estimation

model:                slowAR              fastAR            dualAR
params w/o emb:       3644000256          406346240         3644000256+406346240*9
kv_cache:             2*B*L*C*T           2*B*L*C*T         2*1*36*1024*1024+2*1*4*1024*10*9
batch=1, max_length=1024, C_kv=1024, slowAR n_layers=36, fastAR n_layers=4
decode params load: 
(3644000256+406346240*9)+(2*1*36*1024*1024+2*1*4*1024*10*9)=7377351168, bfloat16 memory: 7377351168*2/1024**3 = 13.74 GB 
params w/o emb + buffer(static kv_cache only and no others, refer to https://github.com/meta-pytorch/gpt-fast)

Inference speedtest

bfloat16:    3090: 48.1it/s|661.0GB/s    4090: 56.3it/s|773.6GB/s    5090: 94.8it/s|1302.7GB/s    H20: 135.6it/s|1863.3GB/s

memory use:  bfloat16: 9~10GB      int8wo: 6~7GB       fp8: 6~7GB       int4wo: 4~5GB       nvfp4: 4~5GB
4070m:       CUDA out of memory    int8wo: 24.5it/s    fp8: 26.8it/s    int4wo: 43.7it/s

# prefill 128, decode 896
4070m: Q:fp8, P:2861.5tok/s 0.045s, D:25.1tok/s 172.2GB/s, Context: 100%|██████████████████| 1024/1024 [00:35<00:00, 24.96it/s]
H20:   Q:bfloat16, P:8986.6tok/s 0.014s, D:135.0tok/s 1855.0GB/s, Context: 100%|██████████████████| 1024/1024 [00:06<00:00, 134.34it/s]
5090:  Q:bfloat16, P:9628.9tok/s 0.013s, D:91.4tok/s 1255.9GB/s, Context: 100%|██████████████████| 1024/1024 [00:09<00:00, 91.18it/s]

# dingzhen prompt, prefill 163, decode: ~280
Q:fp8, P:3416.6tok/s 0.048s, D:26.7tok/s 183.2GB/s, Context:  43%|████████▏          | 440/1024 [00:10<00:22, 26.23it/s]
Q:fp8, P:3354.9tok/s 0.049s, D:26.6tok/s 182.9GB/s, Context:  43%|████████▎          | 445/1024 [00:11<00:23, 24.41it/s]
Q:fp8, P:3399.0tok/s 0.048s, D:26.8tok/s 183.8GB/s, Context:  44%|████████▍          | 452/1024 [00:11<00:22, 25.76it/s]

Audio tokenizer speedtest

gpu	audio length	input shape	encode Ratio	encode RTF	decode Ratio	decode RTF
original fp32 4070m	1 x 7.73 s	[1, 1, 340982]	56.64X	0.017655	29.23X	0.034205
original fp16 4070m	1 x 7.73 s	[1, 1, 340982]	96.19X	0.010396	57.85X	0.017286
4070m	1 x 7.73 s	[1, 1, 340982]	259.38X	0.003855	169.33X	0.005906
5090	1 x 7.73 s	[1, 1, 340982]	1435.84X	0.000696	858.49X	0.001165
5090	1 x 27.70 s	[1, 1, 1221570]	1518.96X	0.000658	925.42X	0.001081
5090	8 x 27.70 s	[8, 1, 1221570]	1585.13X	0.000631	946.42X	0.001057
H800 PCIe	1 x 7.73 s	[1, 1, 340982]	1393.78X	0.000717	919.74X	0.001087
H800 PCIe	1 x 27.70 s	[1, 1, 1221570]	1824.94X	0.000548	1115.22X	0.000897
H800 PCIe	8 x 27.70 s	[8, 1, 1221570]	2067.92X	0.000484	1094.86X	0.000913
tensorRT 4070m	1 x 7.73 s	[1, 1, 340982]	259.02X	0.003861	104.95X	0.009528
tensorRT 5090	1 x 7.73 s	[1, 1, 340982]	1408.37X	0.000710	943.62X	0.001060
tensorRT 5090	1 x 27.70 s	[1, 1, 1221570]	1502.82X	0.000665	1144.86X	0.000873
tensorRT 5090	8 x 27.70 s	[8, 1, 1221570]	1596.55X	0.000626	1171.61X	0.000854
tensorRT H800PCIe	1 x 7.73 s	[1, 1, 340982]	1394.65X	0.000717	397.00X	0.002519
tensorRT H800PCIe	1 x 27.70 s	[1, 1, 1221570]	1906.83X	0.000524	547.08X	0.001828
tensorRT H800PCIe	8 x 27.70 s	[8, 1, 1221570]	2078.78X	0.000481	594.15X	0.001683

training estimation

slowAR tied    emb params: 4032298496
fastAR without emb params: 414210560

10 million hour data, 21hz audio codec, 4~5hz text BPE: 
semantic token: 1e7*3600*21/1e9=756 B, text token: 1e7*3600*5/1e9=180 B

32 x H100, BF16 989TFLOPS, Linear+Attention 0.6 MFU, training cost:
slowAR: toks=756e9+180e9; p=4032298496; L,C,T=32,4096,2048; FLOPs_slow=(6*p+12*L*C*T)*toks
fastAR: toks=756e9*10;    p=414210560;  L,C,T= 4,4096,10;   FLOPs_fast=(6*p+12*L*C*T)*toks
training cost: (FLOPs_slow+FLOPs_fast)/(32*989e12*0.6)/86400=27.1day

Next Step

the bset TTS model you can get in ￥100 and 4hour

Data: 50000 hour audio and text, 21hz 10codebook RVQ audio codec, 4~5hz text BPE tokenizer
Temporal Transformer: params: 360M, training token: 50000*3600*(21+5)/1e9=4.68 B
Depth Transformer:    params: 40M,  training token: 50000*3600*21*10/1e9= 37.8 B
training FLOPS: 6*(360e6*4.68e9+40e6*37.8e9)=1.92e19

audodl, GPU: 1 x 5090, Transformer training speed(torch2.9.1+cu130):
model: L12 C1024 GQA1, param: 150M, fp8+Adamw,  TPS:233721tok/s | FLOPS:249TFLOPs  | MFU:118.6%
model: L16 C2048 GQA2, param: 750M, fp8+Adamw,  TPS:59982tok/s  | FLOPS:320TFLOPs  | MFU:152.8%

conservative estimate—assuming: 249T per gpu and an 8-card efficiency loss of 0.2
400M RQ-Transformer + 50000 hours data requires:
duration=6*(360e6*4.68e9+40e6*37.8e9)/(249e12*8*(1-0.2))/3600=3.34 hour
the price is 3.03/hour on autodl, which mean cost=3.34*8*3.03=81 yuan

Credits

• fish-speech, dualAR.py was writed refer to fish_speech/models/text2semantic, codec.py was writed refer to fish_speech/models/dac.
• gpt-fast, dualAR.py using flex-decoding in gpt-fast to achieve fullgraph in decode stage.

Cite

If you find tuna-fish helpful in your research cite simply as:

@misc{tuna-fish,
  author = {Reese},
  title = {tuna-fish: Minimal PyTorch Implementation of RQ-Transformer Inference},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/reesertg/tuna-fish}
}

License

MIT