Skip to content

feat: Add Audio8-ASR-0.1B community model port (audio8_asr) - #337

Open
gqf2008 wants to merge 2 commits into
0xShug0:mainfrom
gqf2008:feat/audio8-asr
Open

feat: Add Audio8-ASR-0.1B community model port (audio8_asr)#337
gqf2008 wants to merge 2 commits into
0xShug0:mainfrom
gqf2008:feat/audio8-asr

Conversation

@gqf2008

@gqf2008 gqf2008 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #336.

Native audio.cpp port of Audio8/Audio8-ASR-0.1B as community family audio8_asr: a compact multilingual offline ASR (en / zh / yue / ja / ko / fr / de; 324M params total, ~103M LM). Architecture: Whisper 128-mel frontend → Qwen3-ASR audio encoder → 4× pre-norm residual MLP tower → adaptive avg-pool (merge factor 4) → LayerNorm + Linear(1024→512) → 8-layer Qwen2-style decoder (hidden 512, tied embeddings, vocab 151936, RoPE 1e6).

Design points:

  • Encoder + frontend are 100% reused from qwen3_asr via a small renaming TensorSource (audio_encoder.*model.audio_tower.* / model.multi_modal_projector.{linear_1,linear_2}), so the C++ encoder implementation is exercised unchanged; audiocpp_add_model(... DEPENDS qwen3_asr) links it.
  • Decoder reuses the shared QwenCausalDecoder with use_qk_norm=false, per-projection Q/K/V attention biases, tied LM head (lm_head = embed_tokens).
  • Projector (tower + adaptive pool + projection) is the only new graph code; the adaptive average pool replicates torch.nn.functional.adaptive_avg_pool1d windowing exactly.
  • Parity detail: the reference processor emits bfloat16 mel; the audio8 frontend rounds mel to bf16 (RNE) before encoding so fp32 pipelines see the reference values.
  • License: checkpoint is CC-BY-NC-4.0. Following the mms_forced_aligner precedent, package_defaults.download is unsupported (local conversion only, no release GGUF redistribution); the safetensors package points at the upstream HF repo and loads directly in the runtime.

Verification

Build (macOS, Apple M4):

cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug -DENGINE_ENABLE_OPENMP=OFF -DGGML_OPENMP=OFF \
    -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=qwen3_asr,audio8_asr \
    -DENGINE_BUILD_TESTS=ON -DENGINE_BUILD_MODEL_TESTS=ON
cmake --build build/debug --target audiocpp_cli --target audiocpp_gguf -j 8
cmake --build build/debug --target test_audio8_asr_units --target test_audio8_asr_golden_transcription -j 8

Convert + run:

python tools/community_models/convert_audio8_asr.py \
    --checkpoint models/Audio8-ASR-0.1B-hf \
    --converter build/debug/bin/audiocpp_gguf --type q8_0 \
    --output models/Audio8-ASR-0.1B-GGUF/audio8-asr-0.1b-q8_0.gguf

audiocpp_cli --task asr --family audio8_asr \
    --model models/Audio8-ASR-0.1B-GGUF/audio8-asr-0.1b-q8_0.gguf --audio assets/resources/a.wav

Parity evidence (greedy, fp32 HF trust_remote_code reference vs audio.cpp Q8_0 GGUF, exact text match on both backends):

Audio Reference (fp32) audio.cpp (Q8_0, Metal) audio.cpp (Q8_0, CPU)
assets/resources/a.wav (5.95 s) "This little work was finished in the year eighteen o three, and intended for immediate publication." identical
assets/resources/sample_16k.wav (14.07 s) "Some call me nature. Others call me Mother Nature. I've been here for over four point five billion years, twenty-two thousand five hundred times longer than you." identical identical
61 s clip, fixed 30 s windows per-window reference transcripts identical, space-joined

The F16 GGUF also matches exactly, and the safetensors package loads directly (no conversion) with the same output.

Tests:

  • test_audio8_asr_units — token-count formula (matches reference-observed 1407→176 / 595→74 / 3000→375, clamp-to-1 edge, zero-frame rejection) and bf16 RNE rounding. ctest -R audio8 passes.
  • test_audio8_asr_golden_transcription — end-to-end golden transcript via the loader registry; exits 125 (skip) when weights are absent.
  • python3 tools/check_loader_catalog_sync.py --self-test and the repo check both pass; audiocpp_cli --list-loaders --json emits audio8_asr with asr: [offline].

Timing: 14.07 s audio transcribed in ~2.7 s wall including model load and session setup (~175% CPU, Metal backend); 61 s in ~6.5 s.

Reference/validation tooling committed: tools/community_models/audio8_asr_reference.py (fp32 reference + golden dumps) and audio8_asr_stages.py (mel / encoder / projector staged tensors).

Model Used

Role Model
Implementation Claude Code + GLM5.3-Flash 辅助开发 (assisted development)

Known limitations

  • Offline only; no streaming, no word timestamps, no hotword boosting (decode-time logit bias is future work).
  • Audio longer than 30 s is transcribed in fixed 30 s windows with space-joined text (no VAD segmentation).
  • CC-BY-NC-4.0: local conversion only; GGUF must not be redistributed, so no release GGUF package is advertised by default.

Native port of Audio8-ASR-0.1B as community family audio8_asr: a compact
multilingual ASR (en/zh/yue/ja/ko/fr/de) whose Qwen3-ASR audio encoder is
loaded through the existing qwen3_asr implementation via a renaming tensor
source, followed by a new MLP-tower + adaptive-pool projector graph into an
8-layer Qwen2-style decoder (shared QwenCausalDecoder with use_qk_norm=false
and attention biases). The reference mel bfloat16 rounding is replicated for
parity.

The checkpoint is CC-BY-NC-4.0, so the package catalog ships no release GGUF
(users convert locally from the HF snapshot, following the
mms_forced_aligner precedent); the safetensors package loads directly.

Parity: greedy transcripts match the HF trust_remote_code reference exactly
on assets/resources/a.wav and sample_16k.wav with the Q8_0 GGUF on Metal and
CPU backends; a 61 s clip matches per-window reference transcripts.
Covered by test_audio8_asr_units (token-count formula, bf16 rounding) and
test_audio8_asr_golden_transcription (end-to-end, skipped without weights).

Closes 0xShug0#336
Review round 1 (independent reviewer + PR feedback):

- Remove the stray friend declaration for the anonymous-namespace loader:
  MSVC resolves the name to the incomplete namespace-scope class declared
  by the friend and fails the shared_ptr conversion (Windows CI), while
  clang picked the anonymous-namespace definition.
- Size transcription windows from config.max_audio_samples in the input
  sample domain (rate-correct at any input sample rate; sub-16 kHz inputs
  no longer overflow the encoder position table) and fold tails shorter
  than 0.5 s into the previous window.
- Release the resident weights file blob after the session's weight stores
  upload (mirrors qwen3_asr; roughly halves session RSS).
- Split weight storage options: audio_encoder_weight_type is limited to
  native/f32/f16 like the qwen3_asr encoder path; audio8_asr.weight_type
  governs decoder/adapter and accepts the full set. Options are parsed
  with the shared runtime helpers that fail loudly on malformed values,
  and the arena knobs are now declared in the model spec.
- Require tokenizer.json (the legacy vocab/merges branch could never
  load), validate model_type inside parse_config, drop the duplicated
  create_task_session guards, unify the two language lists (no 'Auto' —
  the port passes no language hint), and order the tensor-source reverse
  rename map longest-prefix-first so diagnostics list exact names.
- Converter rejects sharded safetensors checkpoints instead of silently
  converting the last shard, and skips sidecar copies onto themselves.
- Drop the WebUI catalog entry: the GGUF packages are
  download.kind=unsupported (CC-BY-NC-4.0, local conversion only), so the
  entry only offered a dead install button (mms_forced_aligner precedent
  has no catalog entry).
- Golden test now asserts the raw transcript verbatim in addition to the
  normalized comparison; model test targets gate on
  'audio8_asr IN_LIST AUDIOCPP_LINKED_MODELS' per the a76ec04 convention.

Reviewed-by: independent fresh-context reviewer (APPROVE-WITH-NITS);
parity re-verified after the changes: exact transcripts on Metal and CPU
for a.wav and sample_16k.wav (q8_0 GGUF), 61 s clips at 16 kHz and 8 kHz.

Closes 0xShug0#336
@gqf2008

gqf2008 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Independent review (fresh-context reviewer, APPROVE-WITH-NITS) + fork CI round 1 done. Fix commit f7bc36b addresses the findings:

  • Windows CI failure: stray friend class Audio8ASRLoader; made MSVC resolve the name to an incomplete namespace-scope class (clang picked the anonymous-namespace definition) — declaration removed.
  • Window sizing now derives from max_audio_samples in the input sample domain (rate-correct at 8/16/24/44.1/48 kHz; sub-16 kHz no longer overflows the encoder position table) with a 0.5 s minimum-tail fold.
  • Resident weights file blob is released after weight-store upload (mirrors qwen3_asr; ~halves session RSS).
  • Weight storage split: audio_encoder_weight_type limited to native/f32/f16 (qwen3 encoder precedent); malformed option values now fail loudly via the shared runtime parsers; arena knobs declared in the spec.
  • Converter rejects sharded safetensors checkpoints (previously silently converted the last shard); same-file sidecar copies skipped.
  • tokenizer.json now required (dead legacy branch removed), model_type validated in parse_config, duplicated task/mode guards collapsed, language lists unified, reverse rename map ordered longest-prefix-first.
  • WebUI catalog entry removed: GGUF packages are download.kind=unsupported (CC-BY-NC-4.0, local conversion only), so the entry only offered a dead install button (matches the mms_forced_aligner precedent).
  • Golden test asserts the raw transcript verbatim; test targets gate on audio8_asr IN_LIST AUDIOCPP_LINKED_MODELS (a76ec04 convention).

Re-verified after the changes: exact-transcript parity on Metal + CPU (a.wav, sample_16k.wav, q8_0 GGUF), 61 s clips at 16 kHz and 8 kHz, unit + golden tests green, check_loader_catalog_sync.py ok. Round-2 fork CI (ci/audio8-asr @ f7bc36b) running.

@0xShug0 0xShug0 added the new model Request for new model support label Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new model Request for new model support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🔄 [处理中][wt-audio8-asr] feat: Add Audio8-ASR-0.1B community model port (audio8_asr)

2 participants