feat: add NVIDIA Nemotron 3.5 ASR backend (0.3.0) - #7
Merged
Conversation
The README documented `uv sync --extra stt-server-mlx` / `stt-server-parakeet`, but pyproject names the extras `mlx` / `parakeet` / `client` -- so those documented commands failed (uv resolves extras by their pyproject key). This was leftover monorepo-extraction drift (stt_server/__init__.py said "extras split once extracted"). Fix docs to match the shipped pyproject (non-breaking) rather than rename the extras (which would break consumers already running `--extra parakeet`). Corrected README executable commands + prose and the stray stt-server-* refs in __init__.py, client.py, __main__.py, parakeet.py, mlx_teardown_spike.sh, test_stt_server.py. Launchd labels (pipecat.stt-server.parakeet) left untouched.
New `nemotron` ASR backend via mlx-audio (Blaizzy/mlx-audio#774), mirroring ParakeetBackend. Records two locked decisions: (1) ship the mlx-audio dep as a git-pinned [dependency-groups] dev group so 0.3.0 stays PyPI-publishable until mlx-audio releases #774; (2) extra/group naming uses the corrected convention. Language-default (auto vs en-US vs omit) left as an integration-time design question with trade-offs documented.
All /review-plan findings addressed (folded in by 80b79ad). Verified coverage: _BACKEND_RE allowlist + install_stt_agent DEFAULT_MODEL (mandatory Phase 2/3), full parakeet test-mirror (whitespace/identity/lean-base no-import/split-language), assumption hedges (defensive getattr, Metal parity assumed-by-analogy, full mlx_audio surface as Phase-0 integration gate), wheel METADATA no-leak proof. Writes the /conduct readiness marker; Status -> Reviewed.
Pin mlx-audio to the PR #774 merge SHA in a [dependency-groups] nemotron dev group (PyPI 0.4.3 predates Nemotron STT; keeps published 0.3.0 PyPI-clean). uv.lock updated. Probed the installed package: generate(path, language=...) returns AlignedResult.text; 'auto' is a verified prompt key and the model default -> DEFAULT_NEMOTRON_LANGUAGE='auto'. End-to-end load+decode verified.
NemotronBackend + _NemotronStream mirror ParakeetBackend structurally: lazy mlx_audio import in start()/_get_model(), asyncio+threading decode lock pair, in-flight close() drain, private 0o700 temp-dir decode WAVs, empty-decode contract. Material difference: language is forwarded to generate(path, language=lang) with DEFAULT_NEMOTRON_LANGUAGE='auto' fallback when client sends None. 24 tests pass (full parakeet parity + two-way language split + PII/temp-dir/shutdown invariants); mlx_audio fully stubbed.
Wire nemotron into every backend-enumeration site, mirroring parakeet: - __main__.py: _make_backend nemotron arm (lazy NemotronBackend import), _resolve_model nemotron arm (lazy DEFAULT_NEMOTRON_MODEL), --backend choices gain nemotron. - render_stt_plist.py: _BACKEND_RE widened to include nemotron (hard gate). - install_stt_agent.sh: DEFAULT_MODEL nemotron arm -> Nemotron repo id, not the silent Whisper fallback (hard gate); header enumeration updated. Tests land together: argparse choice include + unknown-reject parity, _resolve_model nemotron default + override, lean-base subprocess no-import proof (blocks mlx_audio, both seams succeed), render-plist allowlist-pass, and the installer default-model regression. 108 passed, 2 skipped (pre-existing).
Bump version 0.2.0 -> 0.3.0. README: NemotronBackend bullet with PII/temp-WAV
note, --backend {echo,mlx,parakeet,nemotron} enumeration, socket-convention
row (pipecat.stt-server.nemotron / nemotron.sock), install/smoke block using
'uv sync --group nemotron' with the dev-group rationale, model-table row.
CHANGELOG: [0.3.0] Added + Notes stating Option 1 landed, why no published
nemotron extra yet, and the PyPI-clean verification.
Packaging gate verified: wheel + sdist Requires-Dist carry no mlx-audio /
direct-URL entry (PEP 735 dev group does not leak); Version: 0.3.0. Full
suite 286 passed, 2 skipped; ruff format + check clean.
- Important: import DEFAULT_NEMOTRON_MODEL in the installer + renderer regression tests instead of re-hardcoding the literal, so a backend model-id rename fails the regression rather than passing against a stale copy (closes the installer second-point-of-truth coupling via its guard). - numpy: move from the nemotron group to the always-synced dev group (it is test infra; mlx-audio pulls it transitively at runtime). - Logic: wrap the source-grep file read in a with-statement (no leaked handle). Full suite 286 passed, 2 skipped; ruff clean; wheel Requires-Dist still carries no mlx-audio/direct-URL (PyPI-clean preserved).
Normalize the deep-review ignore entry to .deep-review/ (matching the .conduct/ convention). Keeps local, run-specific review state out of source control — per the Codex review note, the latest-* file bakes in run ids and base/head commits that go stale immediately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a fourth ASR backend,
nemotron(NVIDIA Nemotron 3.5 ASR streaming 0.6b viamlx-audio), mirroringParakeetBackendstructurally and shipped as 0.3.0 (additive, non-breaking). Built phase-by-phase via/conductagainst the reviewed dev plan.Phase 0 — dependency pin + real API verification (blocking gate)
Re-verified externals and probed the real installed post-#774
mlx-audio(not assumptions):[dependency-groups]nemotrongroup to merge SHA14add66. PyPImlx-audiolatest is 0.4.3 (2026-04-28), which predates #774, so a git-SHA pin is required.generate(audio, *, language=None, …) -> AlignedResult: kwarg islanguage(nottarget_lang), takes a file path (like parakeet), returns the sameAlignedResultdataclass with.text.prompt_dictionaryhas 121 keys incl."auto"(verified) and"en-US";default_language = "auto"; unknown values degrade gracefully. Nostrip_lang_tagsflag exists.DEFAULT_NEMOTRON_LANGUAGE = "auto"decided — accepted token + the model's own default. End-to-endload(...).generate(<temp WAV>, language="auto")ran on-device and returned.text == ""for a sine tone (the empty-decode path).What landed
stt_server/backends/nemotron.py—NemotronBackend+_NemotronStream. Lazymlx_audioimport instart()/_get_model()only (lean-base preserved), asyncio+threading decode lock pair, model load inside the thread lock, in-flightclose()drain (3 s), private0o700temp-dir decode WAVs unlinked after decode, empty-decode contract. One material difference from parakeet:languageis forwarded togenerate()with theDEFAULT_NEMOTRON_LANGUAGEfallback when the client sendsNone. Docstring states the three-way language contract (parakeet ignores / mlx_whisper forwards / nemotron forwards-with-default), streaming-deferred note, and labels the Metal-drain rationale assumed-by-analogy for mlx-audio.__main__.py_make_backend/_resolve_modelarms +--backendchoices;render_stt_plist.py_BACKEND_REwidened;install_stt_agent.shDEFAULT_MODELnemotronarm (avoids the silent Whisper-fallback misconfig).tests/test_nemotron_backend.py(full parakeet parity + two-way language split + PII/temp-dir/shutdown invariants); CLI choice enumeration; lean-base subprocess no-import proof; renderer allowlist; installer default-model regression.uv sync --group nemotroninstall block + dev-group rationale; CHANGELOG[0.3.0].Packaging decision (Option 1) & verification
Nemotron ships behind a git-pinned
[dependency-groups]dev group, not a published extra: a direct-URL dep can't appear in a published wheel'sRequires-Dist(PyPI rejects it), which would block 0.3.0 from PyPI. PEP 735 groups are never emitted into wheel/sdist metadata. Verified: the built wheel and sdistRequires-Distcarry zeromlx-audio/direct-URL entries;Version: 0.3.0. Promote to a versionednemotronextra oncemlx-audioreleases with #774 (tracked follow-up).Validation
ruff format --check+ruff checkclean.echo/mlx/parakeetbackends unaffected (proven by the lean-base no-import subprocess test).Review & follow-up commits
After the initial implementation, a multi-lens deep review + Codex review ran; all actionable findings were fixed:
fix(review)(691bf29): regression tests (test_install_migration.py,test_render_stt_plist.py) now importDEFAULT_NEMOTRON_MODELinstead of re-hardcoding the literal (a backend rename now fails the guard rather than passing stale);numpymoved from thenemotrondependency group to the always-synceddevgroup (it is test infra —mlx-audiopulls it transitively at runtime, so thenemotrongroup is just the git-pinnedmlx-audio); a leaked file handle in a source-grep test wrapped inwith. Re-verified: 286 passed / 2 skipped, ruff clean, wheelRequires-Diststill PyPI-clean.chore(5f61733): gitignore.deep-review//.conduct/local skill state (per the Codex [P3] note — run-specific state shouldn't be source-controlled).