feat: Add Audio8-ASR-0.1B community model port (audio8_asr) - #337
Open
gqf2008 wants to merge 2 commits into
Open
feat: Add Audio8-ASR-0.1B community model port (audio8_asr)#337gqf2008 wants to merge 2 commits into
gqf2008 wants to merge 2 commits into
Conversation
Native port of Audio8-ASR-0.1B as community family audio8_asr: a compact multilingual ASR (en/zh/yue/ja/ko/fr/de) whose Qwen3-ASR audio encoder is loaded through the existing qwen3_asr implementation via a renaming tensor source, followed by a new MLP-tower + adaptive-pool projector graph into an 8-layer Qwen2-style decoder (shared QwenCausalDecoder with use_qk_norm=false and attention biases). The reference mel bfloat16 rounding is replicated for parity. The checkpoint is CC-BY-NC-4.0, so the package catalog ships no release GGUF (users convert locally from the HF snapshot, following the mms_forced_aligner precedent); the safetensors package loads directly. Parity: greedy transcripts match the HF trust_remote_code reference exactly on assets/resources/a.wav and sample_16k.wav with the Q8_0 GGUF on Metal and CPU backends; a 61 s clip matches per-window reference transcripts. Covered by test_audio8_asr_units (token-count formula, bf16 rounding) and test_audio8_asr_golden_transcription (end-to-end, skipped without weights). Closes 0xShug0#336
8 tasks
Review round 1 (independent reviewer + PR feedback): - Remove the stray friend declaration for the anonymous-namespace loader: MSVC resolves the name to the incomplete namespace-scope class declared by the friend and fails the shared_ptr conversion (Windows CI), while clang picked the anonymous-namespace definition. - Size transcription windows from config.max_audio_samples in the input sample domain (rate-correct at any input sample rate; sub-16 kHz inputs no longer overflow the encoder position table) and fold tails shorter than 0.5 s into the previous window. - Release the resident weights file blob after the session's weight stores upload (mirrors qwen3_asr; roughly halves session RSS). - Split weight storage options: audio_encoder_weight_type is limited to native/f32/f16 like the qwen3_asr encoder path; audio8_asr.weight_type governs decoder/adapter and accepts the full set. Options are parsed with the shared runtime helpers that fail loudly on malformed values, and the arena knobs are now declared in the model spec. - Require tokenizer.json (the legacy vocab/merges branch could never load), validate model_type inside parse_config, drop the duplicated create_task_session guards, unify the two language lists (no 'Auto' — the port passes no language hint), and order the tensor-source reverse rename map longest-prefix-first so diagnostics list exact names. - Converter rejects sharded safetensors checkpoints instead of silently converting the last shard, and skips sidecar copies onto themselves. - Drop the WebUI catalog entry: the GGUF packages are download.kind=unsupported (CC-BY-NC-4.0, local conversion only), so the entry only offered a dead install button (mms_forced_aligner precedent has no catalog entry). - Golden test now asserts the raw transcript verbatim in addition to the normalized comparison; model test targets gate on 'audio8_asr IN_LIST AUDIOCPP_LINKED_MODELS' per the a76ec04 convention. Reviewed-by: independent fresh-context reviewer (APPROVE-WITH-NITS); parity re-verified after the changes: exact transcripts on Metal and CPU for a.wav and sample_16k.wav (q8_0 GGUF), 61 s clips at 16 kHz and 8 kHz. Closes 0xShug0#336
Contributor
Author
|
Independent review (fresh-context reviewer, APPROVE-WITH-NITS) + fork CI round 1 done. Fix commit
Re-verified after the changes: exact-transcript parity on Metal + CPU ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #336.
Native
audio.cppport of Audio8/Audio8-ASR-0.1B as community familyaudio8_asr: a compact multilingual offline ASR (en / zh / yue / ja / ko / fr / de; 324M params total, ~103M LM). Architecture: Whisper 128-mel frontend → Qwen3-ASR audio encoder → 4× pre-norm residual MLP tower → adaptive avg-pool (merge factor 4) → LayerNorm + Linear(1024→512) → 8-layer Qwen2-style decoder (hidden 512, tied embeddings, vocab 151936, RoPE 1e6).Design points:
qwen3_asrvia a small renamingTensorSource(audio_encoder.*⇄model.audio_tower.*/model.multi_modal_projector.{linear_1,linear_2}), so the C++ encoder implementation is exercised unchanged;audiocpp_add_model(... DEPENDS qwen3_asr)links it.QwenCausalDecoderwithuse_qk_norm=false, per-projection Q/K/V attention biases, tied LM head (lm_head = embed_tokens).torch.nn.functional.adaptive_avg_pool1dwindowing exactly.mms_forced_alignerprecedent,package_defaults.downloadisunsupported(local conversion only, no release GGUF redistribution); the safetensors package points at the upstream HF repo and loads directly in the runtime.Verification
Build (macOS, Apple M4):
cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug -DENGINE_ENABLE_OPENMP=OFF -DGGML_OPENMP=OFF \ -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=qwen3_asr,audio8_asr \ -DENGINE_BUILD_TESTS=ON -DENGINE_BUILD_MODEL_TESTS=ON cmake --build build/debug --target audiocpp_cli --target audiocpp_gguf -j 8 cmake --build build/debug --target test_audio8_asr_units --target test_audio8_asr_golden_transcription -j 8Convert + run:
python tools/community_models/convert_audio8_asr.py \ --checkpoint models/Audio8-ASR-0.1B-hf \ --converter build/debug/bin/audiocpp_gguf --type q8_0 \ --output models/Audio8-ASR-0.1B-GGUF/audio8-asr-0.1b-q8_0.gguf audiocpp_cli --task asr --family audio8_asr \ --model models/Audio8-ASR-0.1B-GGUF/audio8-asr-0.1b-q8_0.gguf --audio assets/resources/a.wavParity evidence (greedy, fp32 HF
trust_remote_codereference vs audio.cpp Q8_0 GGUF, exact text match on both backends):assets/resources/a.wav(5.95 s)assets/resources/sample_16k.wav(14.07 s)The F16 GGUF also matches exactly, and the safetensors package loads directly (no conversion) with the same output.
Tests:
test_audio8_asr_units— token-count formula (matches reference-observed 1407→176 / 595→74 / 3000→375, clamp-to-1 edge, zero-frame rejection) and bf16 RNE rounding.ctest -R audio8passes.test_audio8_asr_golden_transcription— end-to-end golden transcript via the loader registry; exits 125 (skip) when weights are absent.python3 tools/check_loader_catalog_sync.py --self-testand the repo check both pass;audiocpp_cli --list-loaders --jsonemitsaudio8_asrwithasr: [offline].Timing: 14.07 s audio transcribed in ~2.7 s wall including model load and session setup (~175% CPU, Metal backend); 61 s in ~6.5 s.
Reference/validation tooling committed:
tools/community_models/audio8_asr_reference.py(fp32 reference + golden dumps) andaudio8_asr_stages.py(mel / encoder / projector staged tensors).Model Used
Known limitations