feat: Phase 42 — Audio Modality (v4.0.0-alpha.2)

[aarambh-darshan]

Summary

Phase 42 adds a fourth input sense — audio — to aarambh-studio,
following the exact frozen-encoder-plus-trainable-projector pattern
v2 §24 established for vision and v3 §35–36 reused for video and
documents. A frozen, pretrained audio spectrogram transformer
converts a log-mel spectrogram into a grid of patch embeddings; a
small trainable projector maps those into the decoder's d_model
space; the result is spliced into the token sequence at the
<audio> special token position. Nothing about the decoder, the
thinking engine, grammar-constrained tool calling, or long-horizon
tool chains changes — audio is just another sense feeding the same
fusion mechanism.

This is the v4.0.0-alpha.2 release tag, the first Phase 42
deliverable defined in ROADMAP_V4.md §Phase 42 and
ARCHITECTURE_V4.md §56.

What's new

New crate: aarambh-studio-audio (Layer 3)

Module	Role
encoder.rs	FrozenAudioEncoder — AST-style transformer over mel-spectrogram patches; load_pretrained via VarBuilder::from_mmaped_safetensors (same loading path as CLIP)
preprocess.rs	MelSpectrogramConfig + AudioPreprocessor — pure-Rust WAV decode (PCM 8/16/24/32-bit + 32/64-bit float), linear resampling, Hann window, radix-2 Cooley-Tukey FFT, triangular mel filterbank, log-mel normalization — zero new dependencies
projector.rs	AudioProjector — two-layer GELU MLP mirroring VisionProjector exactly
fusion.rs	interleave_audio_tokens() — generalizes interleave_image_tokens
instruct_data.rs	AudioQaExample + load_audio_qa_jsonl (caption, QA, LLaVA conversation formats)

Tokenizer

• <audio> / <audio_end> reserved special tokens (IDs 15 / 16)
• AUDIO_SPECIAL_TOKENS table (17 entries — strict superset of the Phase 36 document table)
• BpeTokenizer::validate_audio_special_tokens()
• BpeTokenizer::upgraded_for_audio() (insertion at ID 15, +2 rows, shifts learned IDs ≥ 15)

CLI

• convert --upgrade-audio-vocab — applies the vocabulary migration to a SafeTensors checkpoint + tokenizer together
• finetune audio-dora / audio-qdora — self-contained two-stage DoRA trainer (run_audio_vlm_dora_from_config) reusing DoraAarambhModel, AdamW::from_varmap, CosineScheduleWithWarmup, accumulate_for_optimizer, cross_entropy_loss, save_adapter
• infer --audio <path> — conflicts with --image/--video/--document (one modality per call, same discipline every prior modality flag holds); backed by AudioRuntime, build_audio_prompt_embeddings, project_audio_tokens, run_audio_infer, and an AudioSafetyAdapter implementing SafetyGenerator
• eval --tasks audio-qa / audio-qa-smoke — implements the shared EvalTask trait

Config

• New [vision.audio] block (AudioTrainingConfig: audio_root, encoder config/weights paths, mel: MelSpectrogramConfig, encoder_batch_size, feature_cache_entries) under the shared [vision] multimodal block
• New configs: configs/audio_qa_smoke.toml, configs/audio_qa_smoke_infer.toml

Scripts & docs

• scripts/phase42_make_audio_smoke_fixture.py — synthetic sine-wave WAVs + tiny random-init audio encoder + projector
• scripts/phase42_prepare_audio_data.sh — data-prep orchestrator (free + public policy, synthetic fallback)
• scripts/phase42_smoke.sh — full smoke chain (fixture → convert → finetune → merge → infer → eval)
• docs/phase42_audio.md — full phase documentation

Workspace

• Bumped workspace version 4.0.0-alpha.1 → 4.0.0-alpha.2
• Release audit EXPECTED_PACKAGES 19 → 20
• CI workflow CLI smoke now includes finetune audio-dora --help and finetune audio-qdora --help

Bug fix included

While verifying the audio smoke end-to-end, I found that
BpeTokenizer::encode only recognized SPECIAL_TOKENS (the 15-entry
document table) for special-token splitting, so <audio> /
<audio_end> (IDs 15/16) were silently BPE-split into characters —
breaking audio fusion at the placeholder. Fixed: encode now uses
AUDIO_SPECIAL_TOKENS (the 17-entry superset); the per-token vocab
filter makes this safe for text/vision/video/document tokenizers too.
Three regression tests pin the fix.

Tests

The four Phase 42 proof obligations from ROADMAP_V4.md:

Test	Location	Proves
frozen_audio_encoder_never_receives_gradients	aarambh-studio-audio	detached encoder output blocks gradients from reaching encoder parameters while the projector still trains
projector_pretrain_stage_trains_only_projector_weights	aarambh-studio-audio	the projector-only stage's structural freeze guarantee holds
audio_token_fusion_produces_expected_sequence_length	aarambh-studio-audio	interleave_audio_tokens produces seq - 1 + audio_tokens length
thinking_controller_behaves_identically_after_audio_context	aarambh-studio-audio	fused audio-context embeddings are a well-formed, finite, contiguous sequence the decoder observes identically to text

Plus unit tests for WAV decode (PCM 16-bit + 32-bit float stereo),
FFT frequency recovery, mel frame counting, projector output width,
fusion placeholder validation, JSONL parsing (caption + QA +
conversation), and three tokenizer-table consistency tests + three
regression tests for the encode fix.

CI gates (all green locally)

• cargo fmt --all --check
• cargo check --workspace --all-targets --locked
• cargo clippy --workspace --all-targets --locked -- -D warnings -D clippy::undocumented_unsafe_blocks
• cargo test --workspace --no-fail-fast --locked
• RUSTDOCFLAGS="-D warnings -D missing_docs" cargo doc --workspace --no-deps --locked
• scripts/phase28_release_audit.sh
• find scripts -type f -name '*.sh' -print0 | xargs -0 -n1 bash -n
• cargo build --release --locked -p aarambh-studio
• Full CLI smoke (every --help, incl. the new audio-dora / audio-qdora)

Audio smoke — verified end-to-end

Trained a tiny model → convert --upgrade-audio-vocab (517 → 519 tokens)
→ infer --audio mid_tone.wav succeeded (WAV → mel → frozen encoder →
projector → <audio> fusion → decoder generation,
finish_reason=MaxTokens) → eval --tasks audio-qa-smoke produced a
scorecard. Output is gibberish (expected from a 2-layer / 20-step
model on synthetic text) — the pipeline is what's verified.

Scope and boundaries (consistent with prior modality phases)

• Audio understanding only (no generation) — same boundary v2 §24 drew for vision
• WAV PCM (8/16/24/32-bit + 32/64-bit float) only; MP3/FLAC/Ogg decode is future work (same "visual-only H.264 MP4" boundary v3 §35 set for video containers)
• Mel-spectrogram extraction is pure-Rust from first principles — no rustfft or audio-DSP dependency
• Self-learning remains text/image only (same boundary video and document hold)

Diffstat

39 files changed, 3987 insertions(+), 65 deletions(-)

New crate crates/aarambh-studio-audio/ (6 source files, ~2050 LOC).
Touches: tokenizer, convert, train config, finetune, infer, eval,
main binary Cargo.toml, CI workflow, release audit, README, ROADMAP_V4,
ARCHITECTURE_V4, CHANGELOG, Cargo.toml, Cargo.lock.

Checklist

• Code follows the project's style (cargo fmt --check clean)
• No TODO/FIXME/unimplemented! markers (release-audit enforced)
• All packages share one version (4.0.0-alpha.2) and publish = false
• Cargo.lock is tracked and consistent
• No model artifacts tracked (.safetensors/checkpoints//adapters/ excluded)
• No [ ] unfinished tasks in ROADMAP.md / ROADMAP_V2.md / ROADMAP_V3.md
• Documentation added (docs/phase42_audio.md) and cross-linked from README
• CHANGELOG entry added under v4.0.0-alpha.2
• Tests added and passing
• MSRV (1.89) respected
• Tag v4.0.0-alpha.2 created

Related

• Roadmap: ROADMAP_V4.md §Phase 42
• Architecture: ARCHITECTURE_V4.md §56
• Prior art: v2 §24–25 (vision), v3 §35–36 (video, document)
• Closes the v4.0.0-alpha.2 milestone