Skip to content

multimodal.md

Latest commit

 

History

History
86 lines (63 loc) · 1.88 KB

multimodal.md

File metadata and controls

86 lines (63 loc) · 1.88 KB
Raw

Multimodal prototypes

complexity.multimodal contains research prototypes for vision, audio, video, fusion, and an omni wrapper.

These modules are separate from the text TR-GQA/TR-MHA evidence. Their routing keys are positions, not tokenizer IDs:

Module Route key
vision patch position
audio time-step position
video spatiotemporal position
fusion query position

Do not describe these as lexical TR-MoE without that qualification.

Vision

import torch

from complexity.multimodal import VisionEncoder

encoder = VisionEncoder(
    image_size=224,
    patch_size=16,
    hidden_size=384,
    num_layers=4,
    num_heads=6,
    num_experts=4,
)
pixels = torch.randn(2, 3, 224, 224)
features = encoder(pixels)

num_experts=1 selects the dense MLP fallback. Values above one enable position-routed MLPs.

Audio

from complexity.multimodal import AudioEncoder

encoder = AudioEncoder(
    n_mels=80,
    hidden_size=384,
    num_layers=4,
    num_heads=6,
    num_experts=4,
)
features = encoder(mel_spectrogram)

Video

from complexity.multimodal import VideoEncoder

encoder = VideoEncoder(num_experts=4)
features = encoder(video)

Expected video layout is documented by the module as [batch, channels, frames, height, width].

Fusion

from complexity.multimodal import MultimodalFusion

fusion = MultimodalFusion(num_experts=4)

Available components include cross-attention, gated fusion, concatenation, Perceiver resampling, and a vision-language connector.

Status

The multimodal package is experimental:

  • it does not inherit the text-model training results;
  • it has no production serving contract in this repository;
  • position routing and token-ID routing answer different research questions;
  • users should add modality-specific parity, masking, and benchmark tests before drawing conclusions.