diff --git a/packages/transcription-parakeet/CHANGELOG.md b/packages/transcription-parakeet/CHANGELOG.md index 76af213de7..67ca34436f 100644 --- a/packages/transcription-parakeet/CHANGELOG.md +++ b/packages/transcription-parakeet/CHANGELOG.md @@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- `rnnt` model type: plain RNN-T (Transducer) checkpoints — e.g. the Transducer + branch of NeMo hybrid `EncDecHybridRNNTCTCBPEModel` models such as the Georgian + `stt_ka_fastconformer_hybrid_large_pc` — auto-detected from GGUF metadata and + exposed in the public `ModelType` union. The vendored converter gains rnnt + export (`parakeet.rnnt.*` metadata + `rnnt.predict/joint` tensors, `--type + rnnt` in `convert-nemo.sh`). Requires a `parakeet-cpp` engine that ships the + rnnt head; the registry port pin must be bumped past the engine-side merge + before this is functional. + ## [0.9.1] - 2026-07-08 ### Fixed diff --git a/packages/transcription-parakeet/README.md b/packages/transcription-parakeet/README.md index 66ad141f0d..17b2d68f1a 100644 --- a/packages/transcription-parakeet/README.md +++ b/packages/transcription-parakeet/README.md @@ -1,6 +1,6 @@ # transcription-parakeet -This library simplifies running NVIDIA Parakeet speech-to-text and Sortformer speaker-diarization inference within QVAC runtime applications. It provides an easy interface to load, execute, and manage Parakeet inference instances, supporting CTC, TDT, EOU, and Sortformer checkpoints from a single binding. +This library simplifies running NVIDIA Parakeet speech-to-text and Sortformer speaker-diarization inference within QVAC runtime applications. It provides an easy interface to load, execute, and manage Parakeet inference instances, supporting CTC, TDT, RNNT, EOU, and Sortformer checkpoints from a single binding. ## Table of Contents @@ -179,7 +179,7 @@ The three underlying scripts are also flag-driven if you want to run them separa setup-venv.sh [--python ] [--venv ] [--force] [--help] download-models.sh [--type ctc|tdt|eou|sortformer|sortformer-streaming-v2.1|all] [--output ] [--force] [--help] -convert-nemo.sh [--type ctc|tdt|eou|sortformer|sortformer-streaming-v2.1|all] +convert-nemo.sh [--type ctc|tdt|rnnt|eou|sortformer|sortformer-streaming-v2.1|all] [--quant f16|q8_0|q5_0|q4_0|f32] [--python ] [--nemo-dir ] [--output ] [--force] [--help] @@ -194,6 +194,7 @@ convert-nemo.sh [--type ctc|tdt|eou|sortformer|sortformer-streaming-v2.1|all] | `eou` | [`nvidia/parakeet_realtime_eou_120m-v1`](https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1) | | `sortformer` | [`nvidia/diar_sortformer_4spk-v1`](https://huggingface.co/nvidia/diar_sortformer_4spk-v1) | | `sortformer-streaming-v2.1` | [`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) | +| `rnnt` | [`nvidia/stt_ka_fastconformer_hybrid_large_pc`](https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc) — convert-only; `download-models.sh` has no `rnnt` entry yet, place the `.nemo` manually | NVIDIA Open Model License -- see each repo's model card for terms. @@ -224,7 +225,7 @@ Most users interact with the package through `index.js`. From that entrypoint we | | `backendsDir` | Root directory for dynamically-loaded ggml backend `.so` files (Vulkan, OpenCL, per-arch CPU variants on Android). Defaults to the package's `prebuilds/` folder; the native addon appends `/` before scanning. Pass an explicit path when prebuilds live elsewhere — e.g. Android `ApplicationInfo.nativeLibraryDir` when backend libs ship inside the APK. No-op on Apple (statically linked). | | | `openclCacheDir` | Persistent directory for ggml-opencl's compiled program-binary cache (`$GGML_OPENCL_CACHE_DIR`). Android-only; pass the host app's cache directory (e.g. `Context.getCacheDir()`) to skip cold `clBuildProgram` on every process start. Ignored on other platforms. | -The model type (CTC / TDT / EOU / Sortformer) is **auto-detected from the GGUF metadata**, so callers don't need to pass `modelType`. Other knobs (`captionEnabled`, `timestampsEnabled`, `seed`, `sampleRate`, `channels`) keep sensible defaults. +The model type (CTC / TDT / RNNT / EOU / Sortformer) is **auto-detected from the GGUF metadata**, so callers don't need to pass `modelType`. Other knobs (`captionEnabled`, `timestampsEnabled`, `seed`, `sampleRate`, `channels`) keep sensible defaults. **Sortformer Streaming Diarization (v2.1 + AOSC).** parakeet-cpp ships two streaming-diarization paths picked automatically by the GGUF: @@ -462,7 +463,7 @@ npm run setup-models -- -t tdt -q q8_0 ### 3. Run the bundled examples ```bash -# Single-file transcription (any model type -- CTC / TDT / EOU / Sortformer) +# Single-file transcription (any model type -- CTC / TDT / RNNT / EOU / Sortformer) bare examples/transcribe.js \ --model models/parakeet-tdt-0.6b-v3.q8_0.gguf \ --audio examples/samples/sample-16k.wav @@ -499,6 +500,7 @@ The live-mic examples capture the default input device via `sox -d` (install: `b | **CTC** | English | argmax CTC | ~ 700 MiB | Fast, no PnC. | | **TDT** | ~25 | RNN-T greedy + duration | ~ 715 MiB | Recommended default; PnC + auto-detect. | | **EOU** | English | RNN-T greedy + `` | ~ 132 MiB | Streaming-trained; native end-of-turn token. | +| **RNNT** | checkpoint-dependent (e.g. Georgian for `stt_ka_fastconformer_hybrid_large_pc`) | plain RNN-T greedy | checkpoint-dependent | Duration-less Transducer heads, incl. the RNN-T branch of hybrid transducer+CTC checkpoints. Requires a parakeet-cpp engine with the `rnnt` head. | | **Sortformer v1** | n/a | Diarization head (sliding history) | ~ 141 MiB | 4-speaker. **Default for offline diarization.** | | **Sortformer v2.1 + AOSC** | n/a | Diarization head + speaker cache | ~ 141 MiB | 4-speaker. **Default for streaming diarization.** AOSC anchors speaker slots across silence/re-entry; auto-detected via GGUF metadata tag `parakeet.model_variant`. | diff --git a/packages/transcription-parakeet/addon/src/model-interface/ParakeetTypes.hpp b/packages/transcription-parakeet/addon/src/model-interface/ParakeetTypes.hpp index d932ef51c0..cfa7ad66b9 100644 --- a/packages/transcription-parakeet/addon/src/model-interface/ParakeetTypes.hpp +++ b/packages/transcription-parakeet/addon/src/model-interface/ParakeetTypes.hpp @@ -17,7 +17,7 @@ struct Transcript { float end; size_t id; // True when this segment ends on a recognised end-of-utterance boundary - // (EOU streaming: `` token; CTC/TDT/Sortformer always leave this + // (EOU streaming: `` token; CTC/TDT/RNNT/Sortformer always leave this // false). The text field still carries any speech tokens decoded in the // same chunk; consumers that want a turn-end signal independent of the // transcript should test this flag. @@ -60,6 +60,7 @@ struct Transcript { enum class ModelType : std::uint8_t { CTC, // English-only, fast transcription with punctuation/capitalization TDT, // Multilingual (~25 languages) with auto-detection + RNNT, // Plain RNN-T (Transducer); e.g. the Georgian hybrid transducer head EOU, // Real-time streaming with end-of-utterance detection SORTFORMER // Speaker diarization (up to 4 speakers) }; diff --git a/packages/transcription-parakeet/addon/src/model-interface/parakeet/ParakeetModel.cpp b/packages/transcription-parakeet/addon/src/model-interface/parakeet/ParakeetModel.cpp index 96e721e88c..455696c57f 100644 --- a/packages/transcription-parakeet/addon/src/model-interface/parakeet/ParakeetModel.cpp +++ b/packages/transcription-parakeet/addon/src/model-interface/parakeet/ParakeetModel.cpp @@ -371,7 +371,7 @@ void ParakeetModel::load() { is_loaded_ = true; // Auto-detect modelType from the loaded GGUF's metadata. The engine - // returns "ctc" / "tdt" / "eou" / "sortformer" reflecting the + // returns "ctc" / "tdt" / "rnnt" / "eou" / "sortformer" reflecting the // `parakeet.model.type` GGUF metadata field, so JS callers don't // need to pass `modelType` themselves -- the binding picks the // right dispatch (ASR vs Sortformer) automatically. We only fall @@ -381,6 +381,7 @@ void ParakeetModel::load() { const std::string detected = engine_->model_type(); if (detected == "ctc") cfg_.modelType = ModelType::CTC; else if (detected == "tdt") cfg_.modelType = ModelType::TDT; + else if (detected == "rnnt") cfg_.modelType = ModelType::RNNT; else if (detected == "eou") cfg_.modelType = ModelType::EOU; else if (detected == "sortformer") cfg_.modelType = ModelType::SORTFORMER; diff --git a/packages/transcription-parakeet/index.d.ts b/packages/transcription-parakeet/index.d.ts index ef9cfc9d62..aa81672d28 100644 --- a/packages/transcription-parakeet/index.d.ts +++ b/packages/transcription-parakeet/index.d.ts @@ -7,7 +7,7 @@ import { Readable } from 'stream' * loaded GGUF's `parakeet.model.type` metadata field; this type is * only here for callers that want to surface it in their own UI. */ -declare type ModelType = 'tdt' | 'ctc' | 'eou' | 'sortformer' +declare type ModelType = 'tdt' | 'ctc' | 'rnnt' | 'eou' | 'sortformer' /** * Parakeet-specific configuration options. The model type itself is diff --git a/packages/transcription-parakeet/index.js b/packages/transcription-parakeet/index.js index c5fcd7dd75..9f46bd99ea 100644 --- a/packages/transcription-parakeet/index.js +++ b/packages/transcription-parakeet/index.js @@ -10,7 +10,7 @@ const { END_OF_INPUT, ERR_CODES, QvacErrorAddonParakeet } = require('./lib/error /** * High-level Parakeet speech-to-text client backed by the ggml engine * sourced from qvac-parakeet.cpp. Takes a single `.gguf` checkpoint - * (CTC, TDT, EOU, or Sortformer); the model type is auto-detected + * (CTC, TDT, RNNT, EOU, or Sortformer); the model type is auto-detected * from GGUF metadata, so the same class transcribes or diarizes * depending on the file you load. */ diff --git a/packages/transcription-parakeet/parakeet.js b/packages/transcription-parakeet/parakeet.js index 2bac5f3987..0bc9d1728f 100644 --- a/packages/transcription-parakeet/parakeet.js +++ b/packages/transcription-parakeet/parakeet.js @@ -33,7 +33,7 @@ function createParakeetError (code, message, cause = undefined) { * runtime. Wraps the ggml-backed Parakeet engine sourced from * qvac-parakeet.cpp. The model type is auto-detected from the * loaded GGUF's metadata, so there's no `modelType` field on the - * config -- pass any of CTC / TDT / EOU / Sortformer .gguf files + * config -- pass any of CTC / TDT / RNNT / EOU / Sortformer .gguf files * to `loadWeights()` and the right pipeline is chosen automatically. */ class ParakeetInterface { diff --git a/packages/transcription-parakeet/scripts/convert-nemo-to-gguf.py b/packages/transcription-parakeet/scripts/convert-nemo-to-gguf.py index e18d9f8467..46b3824003 100644 --- a/packages/transcription-parakeet/scripts/convert-nemo-to-gguf.py +++ b/packages/transcription-parakeet/scripts/convert-nemo-to-gguf.py @@ -4,7 +4,10 @@ # (scripts/convert-nemo-to-gguf.py). # qvac-parakeet.cpp owns the GGUF tensor/metadata layout this script # produces; if the layout changes upstream, resync the file rather than -# diverging it locally. Copy is verbatim apart from this header. +# diverging it locally. Copy is verbatim apart from this header, PLUS the +# rnnt head support, which is ahead of the pinned commit: it mirrors the +# engine-side change hunk-for-hunk (see the companion parakeet-cpp PR). +# Re-pin and resync once that merges. # Original copyright/license: MIT, qvac-parakeet.cpp authors -- see # the top-level NOTICE for attribution. """Convert an NVIDIA NeMo .nemo archive to a single GGUF for the @@ -23,10 +26,16 @@ cache-aware streaming, end-of- utterance token detection; parakeet_realtime_eou_120m-v1) + - ``EncDecHybridRNNTCTCBPEModel`` / ``EncDecRNNTBPEModel`` + (no TDT durations, no ```` token) + -> plain RNN-T (Transducer) head; + for hybrids only the Transducer + branch is exported (the CTC aux + head ``ctc_decoder.*`` is ignored) - ``EncDecDiarLabelModel`` -> Sortformer (diar_sortformer_4spk-v1, diar_streaming_sortformer_4spk-v2) -The FastConformer encoder topology is shared across all four flavours; only +The FastConformer encoder topology is shared across all five flavours; only the decoder / head tensors + metadata differ. EOU additionally swaps the conv module's BatchNorm for a LayerNorm and carries cache-aware streaming hyperparameters (att_context_size, subsampling-output cache lookback, and the @@ -43,7 +52,7 @@ Metadata: general.architecture = "parakeet-ctc" (kept for GGUF compat) general.name = "" - parakeet.model.type = "ctc", "tdt", "eou", or "sortformer" + parakeet.model.type = "ctc", "tdt", "rnnt", "eou", or "sortformer" parakeet.encoder.* (hyperparameters, incl. use_bias, xscaling, conv_norm_type, att_context_size, causal_downsampling, conv_context_size) @@ -51,6 +60,9 @@ parakeet.ctc.* (vocab_size, blank_id) [CTC only] parakeet.tdt.* (predictor + joint hyperparameters + durations) [TDT only] + parakeet.rnnt.* (vocab_size, blank_id, pred_hidden, + pred_rnn_layers, joint_hidden, + max_symbols_per_step) [RNNT only] parakeet.eou.* (vocab_size, blank_id, eou_id, eob_id, pred_hidden, pred_rnn_layers, joint_hidden, encoder_chunk_mel_frames, @@ -73,6 +85,10 @@ tdt.predict.lstm.{l}.{w_ih,w_hh,b_ih,b_hh} [TDT only] tdt.joint.{enc,pred}.{weight,bias} [TDT only] tdt.joint.out.{weight,bias} [TDT only] + rnnt.predict.embed.weight [RNNT only] + rnnt.predict.lstm.{l}.{w_ih,w_hh,b_ih,b_hh} [RNNT only] + rnnt.joint.{enc,pred}.{weight,bias} [RNNT only] + rnnt.joint.out.{weight,bias} [RNNT only] eou.predict.embed.weight [EOU only] eou.predict.lstm.0.{w_ih,w_hh,b_ih,b_hh} [EOU only] eou.joint.{enc,pred}.{weight,bias} [EOU only] @@ -124,6 +140,12 @@ def parse_args() -> argparse.Namespace: "stay at f32. f16 default; use q8_0 for ~2x smaller.") p.add_argument("--hf-repo", default="nvidia/parakeet-ctc-0.6b", help="HF model id to download from if --ckpt is missing.") + p.add_argument("--head", choices=["auto", "ctc", "tdt", "rnnt", "eou", "sortformer"], + default="auto", + help="Override the auto-detected head. 'auto' (default) infers from " + "cfg['target']. Use 'rnnt' to force the plain RNN-T (Transducer) " + "head of a hybrid EncDecHybridRNNTCTCBPEModel checkpoint; its CTC " + "aux head (ctc_decoder.*) is ignored.") return p.parse_args() @@ -186,7 +208,9 @@ def load_nemo(ckpt: Path): return cfg, sd, tok_bytes -def detect_model_type(cfg: dict) -> str: +def detect_model_type(cfg: dict, head: str = "auto") -> str: + if head != "auto": + return head target = cfg.get("target", "") if "Sortformer" in target or "sortformer_modules" in cfg: return "sortformer" @@ -200,7 +224,11 @@ def detect_model_type(cfg: dict) -> str: has_eou = any(str(lbl) == "" for lbl in labels) if has_eou: return "eou" - return "tdt" + # RNN-T with neither TDT durations nor an token: a plain + # Transducer head (e.g. the RNN-T branch of a hybrid + # EncDecHybridRNNTCTCBPEModel). Previously mis-tagged "tdt", which + # crashed below on the absent model_defaults.tdt_durations. + return "rnnt" return "ctc" @@ -234,8 +262,9 @@ def detect_sortformer_variant(ckpt: Path) -> str: return "" -def write_gguf(out: Path, ckpt: Path, cfg: dict, sd: dict, tok_bytes: bytes, quant: str): - model_type = detect_model_type(cfg) +def write_gguf(out: Path, ckpt: Path, cfg: dict, sd: dict, tok_bytes: bytes, quant: str, + head: str = "auto"): + model_type = detect_model_type(cfg, head) enc = cfg["encoder"] pre = cfg["preprocessor"] @@ -270,6 +299,7 @@ def write_gguf(out: Path, ckpt: Path, cfg: dict, sd: dict, tok_bytes: bytes, qua model_name = { "ctc": f"parakeet-ctc-{d_model}-{n_layers}l", "tdt": f"parakeet-tdt-{d_model}-{n_layers}l", + "rnnt": f"parakeet-rnnt-{d_model}-{n_layers}l", "eou": f"parakeet-eou-{d_model}-{n_layers}l", "sortformer": f"sortformer-{d_model}-{n_layers}l", }[model_type] @@ -372,6 +402,45 @@ def write_gguf(out: Path, ckpt: Path, cfg: dict, sd: dict, tok_bytes: bytes, qua variant = detect_sortformer_variant(ckpt) if variant: writer.add_string("parakeet.model_variant", variant) + elif model_type == "rnnt": + # Plain RNN-T (Transducer) head. Structurally a TDT head minus the + # duration outputs: joint.out is (vocab+1, joint_hidden) where the +1 + # is blank only (no duration logits). Predictor + joint tensor keys are + # identical to TDT/EOU; the hybrid's CTC aux head (ctc_decoder.*) is + # ignored. Cache-aware streaming params (att_context_size, conv_norm_type) + # ride in the shared parakeet.encoder.* metadata already emitted above. + if "prednet" not in dec or "joint" not in cfg: + raise RuntimeError( + "rnnt head requires a Transducer checkpoint " + "(decoder.prednet / joint config missing -- is this a CTC model?)") + pred_hidden = int(dec["prednet"]["pred_hidden"]) + pred_rnn_layers = int(dec["prednet"]["pred_rnn_layers"]) + joint_hidden = int(cfg["joint"]["jointnet"]["joint_hidden"]) + pred_vocab_size = int(dec["vocab_size"]) # label vocab (no blank) + joint_num_classes = int(cfg["joint"]["num_classes"]) # label vocab, blank excluded (RNNTJoint adds +1) + blank_id = joint_num_classes # blank_as_pad at index vocab_size + if joint_num_classes != pred_vocab_size: + raise RuntimeError( + f"rnnt: joint.num_classes ({joint_num_classes}) != decoder.vocab_size " + f"({pred_vocab_size}); blank_id placement would be wrong") + greedy_cfg = (cfg.get("decoding") or {}).get("greedy") or {} + max_symbols_cfg = greedy_cfg.get("max_symbols", + greedy_cfg.get("max_symbols_per_step")) + if not max_symbols_cfg: + # NeMo treats an unset/None max_symbols as "no cap"; the C++ greedy + # loop needs a finite cap, so bake in NeMo's usual default of 10. + print("[convert] rnnt: decoding.greedy.max_symbols unset (uncapped in " + "NeMo); capping at 10 in the GGUF", file=sys.stderr) + max_symbols = 10 + else: + max_symbols = int(max_symbols_cfg) + + writer.add_uint32("parakeet.rnnt.vocab_size", pred_vocab_size) + writer.add_uint32("parakeet.rnnt.blank_id", blank_id) + writer.add_uint32("parakeet.rnnt.pred_hidden", pred_hidden) + writer.add_uint32("parakeet.rnnt.pred_rnn_layers", pred_rnn_layers) + writer.add_uint32("parakeet.rnnt.joint_hidden", joint_hidden) + writer.add_uint32("parakeet.rnnt.max_symbols_per_step", max_symbols) else: pred_hidden = int(dec["prednet"]["pred_hidden"]) pred_rnn_layers = int(dec["prednet"]["pred_rnn_layers"]) @@ -601,6 +670,36 @@ def try_bias(name: str, key: str): sd["sortformer_modules.single_hidden_to_spks.weight"]) add_f32("sortformer.head.single_hidden_to_spks.bias", sd["sortformer_modules.single_hidden_to_spks.bias"]) + elif model_type == "rnnt": + add_2d ("rnnt.predict.embed.weight", sd["decoder.prediction.embed.weight"]) + + pred_rnn_layers = int(cfg["decoder"]["prednet"]["pred_rnn_layers"]) + for l in range(pred_rnn_layers): + add_2d (f"rnnt.predict.lstm.{l}.w_ih", + sd[f"decoder.prediction.dec_rnn.lstm.weight_ih_l{l}"]) + add_2d (f"rnnt.predict.lstm.{l}.w_hh", + sd[f"decoder.prediction.dec_rnn.lstm.weight_hh_l{l}"]) + add_f32(f"rnnt.predict.lstm.{l}.b_ih", + sd[f"decoder.prediction.dec_rnn.lstm.bias_ih_l{l}"]) + add_f32(f"rnnt.predict.lstm.{l}.b_hh", + sd[f"decoder.prediction.dec_rnn.lstm.bias_hh_l{l}"]) + + add_2d ("rnnt.joint.enc.weight", sd["joint.enc.weight"]) + add_f32("rnnt.joint.enc.bias", sd["joint.enc.bias"]) + add_2d ("rnnt.joint.pred.weight", sd["joint.pred.weight"]) + add_f32("rnnt.joint.pred.bias", sd["joint.pred.bias"]) + # A plain RNN-T joint emits exactly vocab+1 logits (labels + blank). + # A TDT-shaped joint (vocab+1+num_durations rows) reaching this branch + # (e.g. via --type rnnt) must fail here, not decode garbage later. + out_rows = int(sd["joint.joint_net.2.weight"].shape[0]) + vocab_p1 = int(cfg["decoder"]["vocab_size"]) + 1 + if out_rows != vocab_p1: + raise RuntimeError( + f"rnnt: joint output has {out_rows} rows, expected vocab+1 = " + f"{vocab_p1}; duration logits present? (TDT checkpoint -- use " + f"the auto-detected head instead of forcing rnnt)") + add_2d ("rnnt.joint.out.weight", sd["joint.joint_net.2.weight"]) + add_f32("rnnt.joint.out.bias", sd["joint.joint_net.2.bias"]) else: add_2d ("tdt.predict.embed.weight", sd["decoder.prediction.embed.weight"]) @@ -641,6 +740,11 @@ def try_bias(name: str, key: str): f"blank_id={int(cfg['joint']['num_classes'])} eou_id={eou_pos} " f"att_ctx=[{att_ctx_left},{att_ctx_right}] " f"conv_norm={conv_norm_type}") + elif model_type == "rnnt": + vocab_note = (f"rnnt_vocab={int(cfg['decoder']['vocab_size'])} " + f"blank_id={int(cfg['joint']['num_classes'])} " + f"att_ctx=[{att_ctx_left},{att_ctx_right}] " + f"conv_norm={conv_norm_type}") else: vocab_note = f"tdt_vocab={int(cfg['decoder']['vocab_size'])} durations={cfg['model_defaults']['tdt_durations']}" print(f"[convert] wrote {out} ({size_mb:.1f} MiB, type={model_type}, quant={quant}, {vocab_note}, layers={n_layers}, use_bias={use_bias})", file=sys.stderr) @@ -651,7 +755,7 @@ def main(): ckpt = ensure_ckpt(args.ckpt, args.hf_repo) cfg, sd, tok_bytes = load_nemo(ckpt) args.out.parent.mkdir(parents=True, exist_ok=True) - write_gguf(args.out, ckpt, cfg, sd, tok_bytes, args.quant) + write_gguf(args.out, ckpt, cfg, sd, tok_bytes, args.quant, args.head) if __name__ == "__main__": diff --git a/packages/transcription-parakeet/scripts/convert-nemo.sh b/packages/transcription-parakeet/scripts/convert-nemo.sh index 33de47fb53..b7986aac72 100644 --- a/packages/transcription-parakeet/scripts/convert-nemo.sh +++ b/packages/transcription-parakeet/scripts/convert-nemo.sh @@ -17,8 +17,10 @@ # ./scripts/convert-nemo.sh [flags] # # Flags: -# --type, -t -# Which model(s) (default: all) +# --type, -t +# Which model(s) (default: all; +# rnnt = Georgian hybrid transducer, +# not included in `all`) # --quant, -q Quant tier (default: q8_0) # --python Python interpreter (default: # $PYTHON, then ./venv/bin/python, @@ -63,8 +65,8 @@ while [[ $# -gt 0 ]]; do done case "$TYPE" in - ctc|tdt|eou|sortformer|sortformer-streaming-v2.1|all) ;; - *) echo "Error: --type must be ctc|tdt|eou|sortformer|sortformer-streaming-v2.1|all" >&2; exit 2;; + ctc|tdt|rnnt|eou|sortformer|sortformer-streaming-v2.1|all) ;; + *) echo "Error: --type must be ctc|tdt|rnnt|eou|sortformer|sortformer-streaming-v2.1|all" >&2; exit 2;; esac case "$QUANT" in f32|f16|q8_0|q5_0|q4_0) ;; @@ -127,6 +129,10 @@ nemo_filename() { case "$1" in ctc) echo "parakeet-ctc-0.6b.nemo";; tdt) echo "parakeet-tdt-0.6b-v3.nemo";; + # Georgian hybrid RNN-T (brief target, WER ~5.73%); converter auto-detects + # 'rnnt' and ignores the CTC aux head. Confirm streaming vs non-streaming + # variant (target-model task) before wiring downloads. + rnnt) echo "stt_ka_fastconformer_hybrid_large_pc.nemo";; eou) echo "parakeet_realtime_eou_120m-v1.nemo";; sortformer) echo "diar_sortformer_4spk-v1.nemo";; sortformer-streaming-v2.1) echo "diar_streaming_sortformer_4spk-v2.1.nemo";; @@ -137,6 +143,7 @@ gguf_filename() { case "$t" in ctc) echo "parakeet-ctc-0.6b.${q}.gguf";; tdt) echo "parakeet-tdt-0.6b-v3.${q}.gguf";; + rnnt) echo "stt_ka_fastconformer_hybrid_large_pc.${q}.gguf";; eou) echo "parakeet-eou-120m-v1.${q}.gguf";; sortformer) echo "sortformer-4spk-v1.${q}.gguf";; sortformer-streaming-v2.1) echo "diar_streaming_sortformer_4spk-v2.1.${q}.gguf";; @@ -159,7 +166,14 @@ convert_one() { if [[ ! -f "$nemo" ]]; then echo " x ${t}: .nemo missing -- expected ${nemo}" - echo " run \`./scripts/download-models.sh -t ${t}\` first." + if [[ "$t" == "rnnt" ]]; then + # download-models.sh has no rnnt entry yet (see nemo_filename above). + echo " downloads are not wired for rnnt; fetch the checkpoint from" + echo " HF (nvidia/stt_ka_fastconformer_hybrid_large_pc) and place" + echo " it at the path above." + else + echo " run \`./scripts/download-models.sh -t ${t}\` first." + fi return 1 fi if [[ -f "$gguf" ]] && [[ "$FORCE" -eq 0 ]]; then diff --git a/packages/transcription-parakeet/test/benchmark/rtf-benchmark.test.js b/packages/transcription-parakeet/test/benchmark/rtf-benchmark.test.js index 4514868416..77f07326c7 100644 --- a/packages/transcription-parakeet/test/benchmark/rtf-benchmark.test.js +++ b/packages/transcription-parakeet/test/benchmark/rtf-benchmark.test.js @@ -37,7 +37,7 @@ const platform = detectPlatform() const { samplesDir } = getTestPaths() const SAMPLE_RATE = 16000 -const VALID_MODEL_TYPES = ['tdt', 'ctc', 'eou', 'sortformer'] +const VALID_MODEL_TYPES = ['tdt', 'ctc', 'rnnt', 'eou', 'sortformer'] const VALID_QUANTS = ['q8_0', 'q4_0', 'f16'] const RTF_RESULTS_DIR = path.resolve(__dirname, '../../benchmarks/results') const RESULT_MARKER = 'QVAC_RTF_REPORT::'