Skip to content

Commit 4e3aea2

Browse files
0xShug00xShug0
authored
Migrate Sortformer and Higgs Audio STT to spec v1 (#179)
* Migrate Sortformer diarization to spec v1 * Migrate Higgs Audio STT to spec v1
1 parent fa2ae57 commit 4e3aea2

14 files changed

Lines changed: 542 additions & 487 deletions

File tree

‎CMakeLists.txt‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -737,9 +737,8 @@ audiocpp_add_model(higgs_audio_stt
737737
src/models/higgs_audio_stt/prompt_asr.cpp
738738
src/models/higgs_audio_stt/postprocess.cpp
739739
src/models/higgs_audio_stt/session.cpp
740-
src/models/higgs_audio_stt/loader.cpp
741740
INCLUDES
742-
engine/models/higgs_audio_stt/loader.h
741+
engine/models/higgs_audio_stt/session.h
743742
LOADERS
744743
engine::models::higgs_audio_stt::make_higgs_audio_stt_loader
745744
)
@@ -904,10 +903,9 @@ audiocpp_add_model(sortformer_diar
904903
src/models/sortformer_diar/graph.cpp
905904
src/models/sortformer_diar/modules.cpp
906905
src/models/sortformer_diar/postprocess.cpp
907-
src/models/sortformer_diar/loader.cpp
908906
src/models/sortformer_diar/session.cpp
909907
INCLUDES
910-
engine/models/sortformer_diar/loader.h
908+
engine/models/sortformer_diar/session.h
911909
LOADERS
912910
engine::models::sortformer_diar::make_sortformer_diar_loader
913911
)

‎docs/asr.md‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,8 +159,29 @@ audiocpp_cli --task asr --family higgs_audio_stt --model models/higgs-audio-v3-s
159159
| `--max-tokens` | integer | model default | Maximum generated transcript tokens. |
160160
| `--request-option enable_thinking=true|false` | bool | `true` | Enable the model thinking prompt. |
161161
| `--audio-chunk-mode` | `auto`, `fixed`, `none` | `auto` | Long-audio chunking mode. `auto` uses fixed chunks. |
162-
| `--audio-chunk-seconds` | float seconds | `4` | Fixed audio chunk duration. |
162+
| `--request-option audio_chunk_duration_sec=<seconds>` / `--audio-chunk-seconds` | float seconds | `4` | Fixed audio chunk duration. |
163163
| `--text-out` | TXT path | not set | Transcript output. The transcript is also printed to stdout. |
164+
| `--session-option higgs_audio_stt.weight_type=<type>` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | Shared text decoder weight storage type. |
165+
| `--session-option higgs_audio_stt.audio_encoder_weight_type=<type>` | `native`, `f32`, `f16` | `native` | Audio encoder convolution weight storage type. |
166+
| `--session-option higgs_audio_stt.text_decoder_weight_type=<type>` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `higgs_audio_stt.weight_type` or `native` | Text decoder matmul weight storage type. |
167+
168+
Compatibility aliases are applied before v1 option validation:
169+
170+
| Legacy request option | v1 request option |
171+
|---|---|
172+
| `audio_chunk_seconds` | `audio_chunk_duration_sec` |
173+
| `audio_chunk_duration_seconds` | `audio_chunk_duration_sec` |
174+
| `audio_chunk_duration` | `audio_chunk_duration_sec` |
175+
176+
| Legacy session option | v1 session option |
177+
|---|---|
178+
| `weight_type` | `higgs_audio_stt.weight_type` |
179+
| `audio_encoder_weight_type` | `higgs_audio_stt.audio_encoder_weight_type` |
180+
| `text_decoder_weight_type` | `higgs_audio_stt.text_decoder_weight_type` |
181+
| `audio_encoder_graph_arena_mb` | `higgs_audio_stt.audio_encoder_graph_arena_mb` |
182+
| `text_decoder_prefill_graph_arena_mb` | `higgs_audio_stt.text_decoder_prefill_graph_arena_mb` |
183+
| `text_decoder_decode_graph_arena_mb` | `higgs_audio_stt.text_decoder_decode_graph_arena_mb` |
184+
| `text_decoder_weight_context_mb` | `higgs_audio_stt.text_decoder_weight_context_mb` |
164185

165186
## Hviske ASR
166187

‎docs/speech_analysis.md‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -103,23 +103,47 @@ Sortformer diarization identifies speaker turns. The packaged model path is the
103103
| Field | Value |
104104
|---|---|
105105
| Family | `sortformer_diar` |
106-
| Model directory | `models/diar_sortformer_4spk-v1` |
106+
| Model directory | `models/Sortformer-Diar-4spk-v1-GGUF` |
107107
| Task | `diar` |
108108
| Modes | `offline` |
109109
| Output | Speaker turn JSON through `--turns-out` |
110110
| Speakers | Up to the speaker count supported by the model package; the default model is 4-speaker |
111111

112112
```bash
113-
audiocpp_cli --task diar --family sortformer_diar --model models/diar_sortformer_4spk-v1 --backend cuda --audio meeting_16k.wav --turns-out turns.json
113+
audiocpp_cli --task diar --family sortformer_diar --model models/Sortformer-Diar-4spk-v1-GGUF/sortformer-diar-4spk-v1-q8_0.gguf --backend cuda --audio meeting_16k.wav --turns-out turns.json
114114
```
115115

116116
| Option | Values | Default | Meaning |
117117
|---|---|---:|---|
118118
| `--audio` | WAV path | required | Meeting or conversation audio. |
119119
| `--turns-out` | JSON path | not set | Write speaker turns. |
120-
| `--session-option speaker_threshold=<float>` | float | `0.5` | Speaker activation threshold. |
121-
| `--session-option speaker_min_frames=<n>` | integer | `0` | Minimum speaker segment frames. |
122-
| `--session-option speaker_pad_frames=<n>` | integer | `0` | Padding around speaker turns. |
123-
| `--session-option session_len_sec=<float>` | seconds | `20.0` | Diarization graph window length. |
120+
| `--request-option speaker_threshold=<float>` | float | session default | Per-request speaker activation threshold. |
121+
| `--request-option speaker_min_frames=<n>` | integer | session default | Per-request minimum speaker segment frames. |
122+
| `--request-option speaker_pad_frames=<n>` | integer | session default | Per-request padding around speaker turns. |
123+
| `--session-option sortformer_diar.speaker_threshold=<float>` | float | `0.5` | Default speaker activation threshold. |
124+
| `--session-option sortformer_diar.speaker_min_frames=<n>` | integer | `0` | Default minimum speaker segment frames. |
125+
| `--session-option sortformer_diar.speaker_pad_frames=<n>` | integer | `0` | Default padding around speaker turns. |
126+
| `--session-option sortformer_diar.session_len_sec=<float>` | seconds | `20.0` | Diarization graph window length. |
127+
| `--session-option sortformer_diar.graph_capacity_mode=<mode>` | `fixed`, `tiered`, `grow`, `double` | backend default | Offline graph capacity policy. |
128+
| `--session-option sortformer_diar.graph_arena_mb=<n>` | MB | `512` | Inference graph arena size. |
129+
| `--session-option sortformer_diar.weight_context_mb=<n>` | MB | `128` | Weight context size. |
130+
| `--session-option sortformer_diar.weight_type=<type>` | storage type | `f32` | Default weight storage type. |
131+
| `--session-option sortformer_diar.matmul_weight_type=<type>` | storage type | `weight_type` | Matmul weight storage override. |
132+
| `--session-option sortformer_diar.conv_weight_type=<type>` | storage type | `weight_type` | Convolution weight storage override. |
133+
134+
Compatibility aliases are applied before v1 option validation:
135+
136+
| Legacy session option | v1 session option |
137+
|---|---|
138+
| `speaker_threshold` | `sortformer_diar.speaker_threshold` |
139+
| `speaker_min_frames` | `sortformer_diar.speaker_min_frames` |
140+
| `speaker_pad_frames` | `sortformer_diar.speaker_pad_frames` |
141+
| `session_len_sec` | `sortformer_diar.session_len_sec` |
142+
| `graph_context_mb`, `sortformer_diar.graph_context_mb` | `sortformer_diar.graph_arena_mb` |
143+
| `graph_capacity_mode`, `offline_graph_capacity_mode` | `sortformer_diar.graph_capacity_mode` |
144+
| `weight_context_mb` | `sortformer_diar.weight_context_mb` |
145+
| `weight_type` | `sortformer_diar.weight_type` |
146+
| `matmul_weight_type` | `sortformer_diar.matmul_weight_type` |
147+
| `conv_weight_type` | `sortformer_diar.conv_weight_type` |
124148

125149
For backend weight-type controls, use `audiocpp_cli --inspect --model <model-dir> --family <family>`.

‎include/engine/models/higgs_audio_stt/loader.h‎

Lines changed: 0 additions & 32 deletions
This file was deleted.

‎include/engine/models/higgs_audio_stt/session.h‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
#pragma once
22

3+
#include "engine/framework/model_spec/metadata.h"
4+
#include "engine/framework/runtime/model.h"
35
#include "engine/framework/assets/tensor_source.h"
46
#include "engine/framework/runtime/session_base.h"
57
#include "engine/models/higgs_audio_stt/assets.h"
@@ -16,12 +18,10 @@
1618
#include <string>
1719
#include <vector>
1820

19-
namespace engine::runtime {
20-
class ILoadedVoiceModel;
21-
}
22-
2321
namespace engine::models::higgs_audio_stt {
2422

23+
std::shared_ptr<runtime::IVoiceModelLoader> make_higgs_audio_stt_loader();
24+
2525
class HiggsAudioSTTSession final
2626
: public runtime::RuntimeSessionBase
2727
, public runtime::IOfflineVoiceTaskSession
@@ -30,7 +30,8 @@ class HiggsAudioSTTSession final
3030
HiggsAudioSTTSession(
3131
runtime::TaskSpec task,
3232
runtime::SessionOptions options,
33-
std::shared_ptr<const HiggsAudioSTTAssets> assets);
33+
std::shared_ptr<const HiggsAudioSTTAssets> assets,
34+
std::shared_ptr<const engine::model_spec::ModelContract> contract);
3435
~HiggsAudioSTTSession() override;
3536

3637
std::string family() const override;
@@ -57,10 +58,11 @@ class HiggsAudioSTTSession final
5758

5859
runtime::TaskSpec task_;
5960
std::shared_ptr<const HiggsAudioSTTAssets> assets_;
60-
size_t audio_encoder_graph_arena_bytes_ = 128ull * 1024ull * 1024ull;
61-
size_t text_decoder_prefill_graph_arena_bytes_ = 256ull * 1024ull * 1024ull;
61+
std::shared_ptr<const engine::model_spec::ModelContract> contract_;
62+
size_t audio_encoder_graph_arena_bytes_ = 512ull * 1024ull * 1024ull;
63+
size_t text_decoder_prefill_graph_arena_bytes_ = 512ull * 1024ull * 1024ull;
6264
size_t text_decoder_decode_graph_arena_bytes_ = 256ull * 1024ull * 1024ull;
63-
size_t text_decoder_weight_context_bytes_ = 64ull * 1024ull * 1024ull;
65+
size_t text_decoder_weight_context_bytes_ = 4096ull * 1024ull * 1024ull;
6466
engine::assets::TensorStorageType audio_encoder_weight_storage_type_ = engine::assets::TensorStorageType::Native;
6567
engine::assets::TensorStorageType text_decoder_weight_storage_type_ = engine::assets::TensorStorageType::Native;
6668
HiggsAudioSTTTextTokenizer tokenizer_;

‎include/engine/models/sortformer_diar/loader.h‎

Lines changed: 0 additions & 35 deletions
This file was deleted.

‎include/engine/models/sortformer_diar/session.h‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
#pragma once
22

3+
#include "engine/framework/model_spec/metadata.h"
4+
#include "engine/framework/runtime/model.h"
35
#include "engine/framework/runtime/session_base.h"
46
#include "engine/models/sortformer_diar/assets.h"
57
#include "engine/models/sortformer_diar/graph.h"
@@ -11,14 +13,17 @@
1113

1214
namespace engine::models::sortformer_diar {
1315

16+
std::shared_ptr<runtime::IVoiceModelLoader> make_sortformer_diar_loader();
17+
1418
class SortformerDiarSession final
1519
: public runtime::RuntimeSessionBase
1620
, public runtime::IOfflineVoiceTaskSession {
1721
public:
1822
SortformerDiarSession(
1923
runtime::TaskSpec task,
2024
runtime::SessionOptions options,
21-
std::shared_ptr<const SortformerAssets> assets);
25+
std::shared_ptr<const SortformerAssets> assets,
26+
std::shared_ptr<const engine::model_spec::ModelContract> contract);
2227
~SortformerDiarSession() override;
2328

2429
std::string family() const override;
@@ -38,9 +43,10 @@ class SortformerDiarSession final
3843

3944
runtime::TaskSpec task_;
4045
std::shared_ptr<const SortformerAssets> assets_;
46+
std::shared_ptr<const engine::model_spec::ModelContract> contract_;
4147
std::shared_ptr<const SortformerDiarWeights> weights_;
4248
SortformerPostprocessConfig default_postprocess_;
43-
size_t graph_context_bytes_ = 512ull * 1024ull * 1024ull;
49+
size_t graph_arena_bytes_ = 512ull * 1024ull * 1024ull;
4450
size_t weight_context_bytes_ = 128ull * 1024ull * 1024ull;
4551
assets::TensorStorageType matmul_weight_storage_type_ = assets::TensorStorageType::Native;
4652
assets::TensorStorageType conv_weight_storage_type_ = assets::TensorStorageType::Native;

‎model_specs/higgs_audio_stt.json‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
{
2+
"schema_version": 1,
23
"family": "higgs_audio_stt",
34
"display_name": "Higgs Audio v3 STT",
45
"description": "Boson AI English speech-to-text model combining a Whisper Large v3 speech encoder with a Qwen decoder for robust ASR.",
@@ -15,6 +16,110 @@
1516
"en"
1617
],
1718
"capabilities": {},
19+
"dependencies": [],
20+
"options": {
21+
"request": [
22+
{
23+
"name": "language",
24+
"type": "string",
25+
"description": "Transcript language code metadata; English is used when omitted.",
26+
"required": false
27+
},
28+
{
29+
"name": "max_tokens",
30+
"type": "int",
31+
"description": "Maximum generated transcript tokens; default 1024.",
32+
"required": false,
33+
"min": 1,
34+
"default": 1024
35+
},
36+
{
37+
"name": "enable_thinking",
38+
"type": "bool",
39+
"description": "Enable the model thinking prompt; default true.",
40+
"required": false,
41+
"default": true
42+
},
43+
{
44+
"name": "audio_chunk_mode",
45+
"type": "enum",
46+
"description": "Audio chunking mode; default auto uses fixed chunks.",
47+
"values": [
48+
"auto",
49+
"fixed",
50+
"none"
51+
],
52+
"required": false,
53+
"default": "auto"
54+
},
55+
{
56+
"name": "audio_chunk_duration_sec",
57+
"type": "float",
58+
"description": "Fixed audio chunk duration in seconds; must be positive when set; default 4.",
59+
"required": false,
60+
"min": 0.0,
61+
"default": 4.0
62+
}
63+
],
64+
"session": [
65+
{
66+
"name": "weight_type",
67+
"type": "enum",
68+
"description": "Shared text decoder weight storage type; default native.",
69+
"preset": "weight_type_full",
70+
"required": false,
71+
"default": "native"
72+
},
73+
{
74+
"name": "audio_encoder_weight_type",
75+
"type": "enum",
76+
"description": "Audio encoder convolution weight storage type; default native.",
77+
"preset": "weight_type_conv",
78+
"required": false,
79+
"default": "native"
80+
},
81+
{
82+
"name": "text_decoder_weight_type",
83+
"type": "enum",
84+
"description": "Text decoder matmul weight storage type; defaults to weight_type when set, otherwise native.",
85+
"preset": "weight_type_full",
86+
"required": false
87+
},
88+
{
89+
"name": "audio_encoder_graph_arena_mb",
90+
"type": "int",
91+
"description": "Audio encoder graph arena size in MiB; default 512.",
92+
"required": false,
93+
"min": 0,
94+
"default": 512
95+
},
96+
{
97+
"name": "text_decoder_prefill_graph_arena_mb",
98+
"type": "int",
99+
"description": "Text decoder prefill graph arena size in MiB; default 512.",
100+
"required": false,
101+
"min": 0,
102+
"default": 512
103+
},
104+
{
105+
"name": "text_decoder_decode_graph_arena_mb",
106+
"type": "int",
107+
"description": "Text decoder cached-step graph arena size in MiB; default 256.",
108+
"required": false,
109+
"min": 0,
110+
"default": 256
111+
},
112+
{
113+
"name": "text_decoder_weight_context_mb",
114+
"type": "int",
115+
"description": "Text decoder weight context arena size in MiB; default 4096.",
116+
"required": false,
117+
"min": 0,
118+
"default": 4096
119+
}
120+
],
121+
"load": []
122+
},
18123
"runtime": {
19124
"tags": [
20125
"gguf",

0 commit comments

Comments
 (0)