From 51e90c928693757b8e5882a01a9b4364390ceedf Mon Sep 17 00:00:00 2001 From: dbrkn Date: Tue, 4 Aug 2026 17:03:07 +0300 Subject: [PATCH 1/2] BENCHMARKS: add Qwen3-ASR 1.7B keyword recognition results --- BENCHMARKS.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 339da7a..dd2aa10 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -505,6 +505,13 @@ - **Code Reference:** [openbench/pipeline/transcription/transcription_whisperkitpro.py](https://github.com/argmaxinc/OpenBench/blob/main/src/openbench/pipeline/transcription/transcription_whisperkitpro.py) - **Hardware**: M2 Ultra Mac Studio +### Argmax +- **Latest Run:** `2026-08-04` +- **Model Version:** `qwen3-asr-1.7b` +- **Configuration:** Argmax SDK WhisperKit Pro framework with the Qwen3-ASR 1.7B model and Custom Vocabulary feature enabled. Keywords are applied through Qwen3-ASR's native system-prompt context biasing. +- **Code Reference:** [openbench/pipeline/transcription/transcription_whisperkitpro.py](https://github.com/argmaxinc/OpenBench/blob/main/src/openbench/pipeline/transcription/transcription_whisperkitpro.py) +- **Hardware**: M3 Pro MacBook Pro + ### Apple - **Latest Run:** `2025-12-30` - **Model Version:** `SFSpeechRecognizer` @@ -557,6 +564,7 @@ | Whisper OSS
(large-v3-turbo) | 15.4 | 21.24 | 14.69 | | Argmax
(parakeet-v2) | 14.69 | 12.46 | 12.57 | | Argmax
(parakeet-v3) | 16.89 | 14.57 | 14.73 | +| Argmax
(qwen3-asr-1.7b) | - | 9.19 | 10.18 | | ElevenLabs | 10.53 | 9.13 | 9.08 | | Apple
(SFSpeechRecognizer) | 28.42 | 26.98 | 27.26 | | Apple
(SpeechAnalyzer) | 17 | - | - | @@ -590,6 +598,7 @@ If the model predicts 20 keywords and 15 of them match the ground truth, precisi | Whisper OSS
(large-v3-turbo) | 0.97 | 0.96 | 0.94 | | Argmax
(parakeet-v2) | 0.97 | 0.98 | 0.96 | | Argmax
(parakeet-v3) | 0.98 | 0.98 | 0.95 | +| Argmax
(qwen3-asr-1.7b) | - | 0.98 | 0.93 | | ElevenLabs | 0.97 | 0.99 | 0.96 | | Apple
(SFSpeechRecognizer) | 1 | 0.99 | 0.99 | | Apple
(SpeechAnalyzer) | 0.99 | - | - | @@ -623,6 +632,7 @@ If the ground-truth transcript has 25 keywords and the model correctly finds 15, | Whisper OSS
(large-v3-turbo) | 0.53 | 0.77 | 0.82 | | Argmax
(parakeet-v2) | 0.47 | 0.85 | 0.82 | | Argmax
(parakeet-v3) | 0.45 | 0.82 | 0.8 | +| Argmax
(qwen3-asr-1.7b) | - | 0.87 | 0.81 | | ElevenLabs | 0.75 | 0.96 | 0.94 | | Apple
(SFSpeechRecognizer) | 0.26 | 0.45 | 0.4 | | Apple
(SpeechAnalyzer) | 0.39 | - | - | @@ -658,6 +668,7 @@ F1 = 2 × (0.75 × 0.6) / (0.75 + 0.6) = **66.7%**, reflecting the model's overa | Whisper OSS
(large-v3-turbo) | 0.69 | 0.86 | 0.87 | | Argmax
(parakeet-v2) | 0.63 | 0.91 | 0.88 | | Argmax
(parakeet-v3) | 0.62 | 0.89 | 0.87 | +| Argmax
(qwen3-asr-1.7b) | - | 0.92 | 0.87 | | ElevenLabs | 0.84 | 0.97 | 0.95 | | Apple
(SFSpeechRecognizer) | 0.41 | 0.62 | 0.58 | | Apple
(SpeechAnalyzer) | 0.56 | - | - | From 82d55a83d0d15d6a9a6d11d5a62fcbae2f20ef07 Mon Sep 17 00:00:00 2001 From: dbrkn Date: Wed, 5 Aug 2026 11:49:19 +0300 Subject: [PATCH 2/2] BENCHMARKS: fill Qwen3-ASR no-keywords baseline column --- BENCHMARKS.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index dd2aa10..7f482c0 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -564,7 +564,7 @@ | Whisper OSS
(large-v3-turbo) | 15.4 | 21.24 | 14.69 | | Argmax
(parakeet-v2) | 14.69 | 12.46 | 12.57 | | Argmax
(parakeet-v3) | 16.89 | 14.57 | 14.73 | -| Argmax
(qwen3-asr-1.7b) | - | 9.19 | 10.18 | +| Argmax
(qwen3-asr-1.7b) | 11.86 | 9.19 | 10.18 | | ElevenLabs | 10.53 | 9.13 | 9.08 | | Apple
(SFSpeechRecognizer) | 28.42 | 26.98 | 27.26 | | Apple
(SpeechAnalyzer) | 17 | - | - | @@ -598,7 +598,7 @@ If the model predicts 20 keywords and 15 of them match the ground truth, precisi | Whisper OSS
(large-v3-turbo) | 0.97 | 0.96 | 0.94 | | Argmax
(parakeet-v2) | 0.97 | 0.98 | 0.96 | | Argmax
(parakeet-v3) | 0.98 | 0.98 | 0.95 | -| Argmax
(qwen3-asr-1.7b) | - | 0.98 | 0.93 | +| Argmax
(qwen3-asr-1.7b) | 0.97 | 0.98 | 0.93 | | ElevenLabs | 0.97 | 0.99 | 0.96 | | Apple
(SFSpeechRecognizer) | 1 | 0.99 | 0.99 | | Apple
(SpeechAnalyzer) | 0.99 | - | - | @@ -632,7 +632,7 @@ If the ground-truth transcript has 25 keywords and the model correctly finds 15, | Whisper OSS
(large-v3-turbo) | 0.53 | 0.77 | 0.82 | | Argmax
(parakeet-v2) | 0.47 | 0.85 | 0.82 | | Argmax
(parakeet-v3) | 0.45 | 0.82 | 0.8 | -| Argmax
(qwen3-asr-1.7b) | - | 0.87 | 0.81 | +| Argmax
(qwen3-asr-1.7b) | 0.53 | 0.87 | 0.81 | | ElevenLabs | 0.75 | 0.96 | 0.94 | | Apple
(SFSpeechRecognizer) | 0.26 | 0.45 | 0.4 | | Apple
(SpeechAnalyzer) | 0.39 | - | - | @@ -668,7 +668,7 @@ F1 = 2 × (0.75 × 0.6) / (0.75 + 0.6) = **66.7%**, reflecting the model's overa | Whisper OSS
(large-v3-turbo) | 0.69 | 0.86 | 0.87 | | Argmax
(parakeet-v2) | 0.63 | 0.91 | 0.88 | | Argmax
(parakeet-v3) | 0.62 | 0.89 | 0.87 | -| Argmax
(qwen3-asr-1.7b) | - | 0.92 | 0.87 | +| Argmax
(qwen3-asr-1.7b) | 0.69 | 0.92 | 0.87 | | ElevenLabs | 0.84 | 0.97 | 0.95 | | Apple
(SFSpeechRecognizer) | 0.41 | 0.62 | 0.58 | | Apple
(SpeechAnalyzer) | 0.56 | - | - |