Skip to content

sense_asr.md

Latest commit

 

History

History
136 lines (105 loc) · 6.01 KB

sense_asr.md

File metadata and controls

136 lines (105 loc) · 6.01 KB
Raw

SenseVoice-Small Community ASR

Default model-manager downloads use the published GGUF package when available; the original source/conversion instructions below remain valid for manual use.

sense_asr is a native audio.cpp port of SenseVoice-Small, a multilingual speech-to-text model from FunAudioLLM that reads audio through a 50-block SAN-M encoder and a CTC head. It recognizes 23 language tags and emits rich <|event|>/<|emotion|>/<|language|> meta tags, with optional inverse text normalization (ITN). The port adapts the SenseVoice llama.cpp runtime (sensevoice-server.cpp) into the framework layout and uses the bundled silero_vad for audio segmentation instead of the reference FSMN-VAD.

Capabilities

Field Value
Task asr
Modes offline, buffered streaming
Languages auto, zh, en, yue, ja, ko, pt, ru, es, it, fr, de, nl, pl, tr, ar, hi, vi, th, id, ms, fa, nospeech
Input WAV; audio.cpp converts to 16 kHz mono
Output Transcript, detected language, event/emotion tags
Frontend 80-mel filterbank + LFR (140ms context), Kaldi-compatible
Encoder SAN-M, 1 stem + 49 main blocks + 20 timestamp-prediction blocks, 4 heads, 512-wide, kernel 11
Decoding CTC greedy collapse with SentencePiece detok
ITN Query token 14 (withitn) on, 15 (woitn) off; on by default
Package sensevoice_small_q8 Q8 GGUF

The language request option hard-codes a language ID query token when one of the model's native tags (zh, en, yue, ja, ko, nospeech) is given; the rest of the tags fall back to auto. keep_tags keeps the <|event|>/<|emotion|>/<|language|> tags inline in the output text.

Install

Install the default package:

Download the standalone GGUF directly from FunAudioLLM/SenseVoiceSmall-GGUF-audiocpp (such as sensevoice-small-q8-audiocpp-v1.gguf) and point --model at the file.

Run (offline)

audiocpp_cli --task asr --family sense_asr \
  --model models/SenseVoice-Small-GGUF/sensevoice-small-q8-audiocpp-v1.gguf \
  --backend cpu --audio samples/zh.wav

audio_chunk_mode=auto (default) segments long audio with the bundled silero VAD; audio_chunk_mode=fixed splits on audio_chunk_duration_sec; none runs the whole clip as one encoder pass. Chunk transcripts are joined at ASCII word boundaries so CJK output stays space-free.

Run (streaming)

audiocpp_cli --task asr --family sense_asr \
  --model models/SenseVoice-Small-GGUF/sensevoice-small-q8-audiocpp-v1.gguf \
  --backend cpu --mode streaming --audio - \
  --request-option audio_chunk_duration_sec=5 --request-option audio_chunk_mode=none \
  < 16k_s16.pcm

Buffered streaming holds accumulated PCM and transcribes one fixed window per audio_chunk_duration_sec; each window yields a partial_text delta and finalize() returns the full accumulated transcript. The SenseVoice service also runs through audiocpp_server with POST /v1/audio/transcriptions/live for chunked PCM ingest with SSE deltas.

Run as a server (offline & streaming)

Start the WebUI-enabled server with the SenseVoice model:

audiocpp_server --ui --backend cpu \
  --config <(echo '{"models":[{"id":"sense_asr","family":"sense_asr","path":"models/SenseVoice-Small-GGUF/sensevoice-small-q8-audiocpp-v1.gguf","task":"asr","mode":"streaming"}]}')

Then open http://127.0.0.1:8080 and select "SenseVoice-Small (asr, 流式, 社区)" from the ASR tab.

Or load the model dynamically via the WebUI's model manager (click "📥 加载模型" after selecting the model).

For headless API-only mode without WebUI:

audiocpp_server --backend cpu \
  --config <(echo '{"models":[{"id":"sense_asr","family":"sense_asr","path":"models/SenseVoice-Small-GGUF/sensevoice-small-q8-audiocpp-v1.gguf","task":"asr","mode":"streaming"}],"ui":false}')

Then use the OpenAI-compatible endpoint:

# Offline transcription
curl -X POST http://127.0.0.1:8080/v1/audio/transcriptions \
  -F 'model=sense_asr' -F 'file=@assets/resources/3.wav'

# Streaming transcription (SSE) — requires chunked transfer encoding
curl -X POST http://127.0.0.1:8080/v1/audio/transcriptions/live \
  -H 'Accept: text/event-stream' \
  -H 'Transfer-Encoding: chunked' \
  -F 'model=sense_asr' -F 'file=@assets/resources/3.wav'

Request options

Option Values Default Meaning
language auto|zh|en|yue|ja|ko|pt|ru|... auto Recognition language; auto lets the model infer it
enable_itn true|false true Inverse text normalization via the withitn query token
keep_tags true|false false Keep `<
audio_chunk_mode auto|fixed|none auto VAD segmentation, fixed split, or one pass
audio_chunk_duration_sec seconds 30 Max chunk/window duration

Session options: sense_asr.weight_type (native|f32|f16|bf16|q8_0), sense_asr.encoder_graph_arena_mb, and sense_asr.vad_model_path (defaults to the bundled assets/framework/models/silero_vad).

Source and conversion

The port is based on the SenseVoice llama.cpp runtime at /workspace/SenseVoice/runtime/llama.cpp/sensevoice-server/ (sensevoice-server.cpp): 80-mel filterbank + LFR, the SAN-M encoder, CTC collapse, and SentencePiece detok. The reference FSMN-VAD and HTTP server layer are replaced by the framework session plus the bundled silero_vad. The GGUF is produced by that runtime's export_sensevoice_gguf.py script and carries the sv.* metadata keys and cmvn.shift/cmvn.scale tensors (the offline fbank path applies no CMVN, matching the reference offline engine).

Verification notes

A 6-second Chinese sample (sample.wav) transcribes to 我想问我在滨海新区有房。 in the offline path, matching the reference llama-funasr-sensevoice output exactly. Streaming with 3-second windows emits per-window partials (我想问。 then 我在滨海新区有房。) and the joined final transcript with a ~1.75s TTFT on CPU.