Skip to content

feat(embed): add support for audio inputs - #181

Draft
Nina-mvH wants to merge 11 commits into
mainfrom
support-embedding-audio
Draft

feat(embed): add support for audio inputs#181
Nina-mvH wants to merge 11 commits into
mainfrom
support-embedding-audio

Conversation

@Nina-mvH

@Nina-mvH Nina-mvH commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Adds support for embedding audio inputs

This change required adding the helper load_audio; this is a duplicate of the helper created in #176 and should be removed once that helper is available in the shared utils.

Tested with:

  • facebook/wav2vec2-base
  • openai/whisper-tiny
  • laion/clap-htsat-unfused
    All these models produced some error messages/warnings at the beginning of the log (example below from the whisper log), but otherwise worked seemingly fine (produced an embedding vector)
Loading weights: 0it [00:00, ?it/s]
[transformers] �[1mWhisperEncoder LOAD REPORT�[0m from: openai/whisper-tiny
Key                                                           | Status     | 
--------------------------------------------------------------+------------+-
model.encoder.layers.{0, 1, 2, 3}.self_attn.out_proj.bias     | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.fc1.weight                  | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn.k_proj.weight     | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.final_layer_norm.weight     | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn.v_proj.weight     | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn.v_proj.bias       | UNEXPECTED | 
model.encoder.conv1.bias                                      | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn.out_proj.weight   | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.fc2.bias                    | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn_layer_norm.weight | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.final_layer_norm.bias       | UNEXPECTED | 
model.encoder.conv1.weight                                    | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.fc1.bias                    | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.fc2.weight                  | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn.q_proj.bias       | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn.q_proj.weight     | UNEXPECTED | 
model.encoder.conv2.bias                                      | UNEXPECTED | 
model.encoder.conv2.weight                                    | UNEXPECTED | 
model.encoder.layer_norm.weight                               | UNEXPECTED | 
model.encoder.layers.{0, 1, 2, 3}.self_attn_layer_norm.bias   | UNEXPECTED | 
model.encoder.layer_norm.bias                                 | UNEXPECTED | 
model.encoder.embed_positions.weight                          | UNEXPECTED | 
layers.{0, 1, 2, 3}.fc2.bias                                  | MISSING    | 
layers.{0, 1, 2, 3}.fc2.weight                                | MISSING    | 
layers.{0, 1, 2, 3}.self_attn.v_proj.bias                     | MISSING    | 
layers.{0, 1, 2, 3}.final_layer_norm.bias                     | MISSING    | 
layers.{0, 1, 2, 3}.self_attn.k_proj.weight                   | MISSING    | 
layers.{0, 1, 2, 3}.self_attn.v_proj.weight                   | MISSING    | 
layers.{0, 1, 2, 3}.self_attn_layer_norm.weight               | MISSING    | 
layers.{0, 1, 2, 3}.self_attn.q_proj.weight                   | MISSING    | 
layers.{0, 1, 2, 3}.final_layer_norm.weight                   | MISSING    | 
layers.{0, 1, 2, 3}.self_attn.out_proj.weight                 | MISSING    | 
conv1.bias                                                    | MISSING    | 
conv2.bias                                                    | MISSING    | 
layers.{0, 1, 2, 3}.self_attn.out_proj.bias                   | MISSING    | 
embed_positions.weight                                        | MISSING    | 
layers.{0, 1, 2, 3}.fc1.bias                                  | MISSING    | 
layers.{0, 1, 2, 3}.self_attn.q_proj.bias                     | MISSING    | 
layers.{0, 1, 2, 3}.fc1.weight                                | MISSING    | 
conv1.weight                                                  | MISSING    | 
layers.{0, 1, 2, 3}.self_attn_layer_norm.bias                 | MISSING    | 
conv2.weight                                                  | MISSING    | 
layer_norm.weight                                             | MISSING    | 
layer_norm.bias                                               | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
2026-07-08 16:13:16 | INFO     |    Embedding dimension: 384

Closes #244

@Nina-mvH
Nina-mvH marked this pull request as ready for review July 8, 2026 20:46
@Nina-mvH Nina-mvH changed the title Support embedding audio feat(embed): add support for audio inputs Jul 8, 2026
@Nina-mvH Nina-mvH added the feature New feature or request label Jul 10, 2026
@Nina-mvH
Nina-mvH marked this pull request as draft July 28, 2026 18:34
Base automatically changed from support-embedding-images to main August 11, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

embed: support audio inputs

1 participant