A complete from-scratch PyTorch implementation of the Qwen3.5-VL Vision-Language Model, built for learning and experimentation. Every component is written from first principles with detailed tutorial-style comments explaining what it does and why it's designed that way.
qwen3_5_vlm/
├── scratch/ # From-scratch implementation (pure PyTorch)
│ ├── config.py # Configuration dataclasses
│ ├── norm.py # RMSNorm, RMSNormGated
│ ├── rope.py # Rotary Position Embeddings (RoPE & M-RoPE)
│ ├── mlp.py # SwiGLU feed-forward network
│ ├── attention.py # Full self-attention with GQA + SDPA + KV cache
│ ├── delta_net.py # Gated DeltaNet linear attention (O(n) / O(1) decode)
│ ├── cache.py # HybridCache for KV + recurrent state caching
│ ├── decoder.py # Transformer decoder layer
│ ├── vision.py # Vision encoder (ViT)
│ ├── text_model.py # Text model (decoder-only transformer)
│ ├── vlm.py # Top-level VLM + optimized generation loop
│ ├── modeling.py # Re-export hub (backward-compatible imports)
│ ├── weight_utils.py # SafeTensors weight loading
│ └── pipeline.py # High-level inference pipeline
├── wrapper/ # HuggingFace transformers wrapper (for comparison)
├── compare.py # Compare scratch vs wrapper outputs
├── main.py # Demo entry point
├── download.py # Download model weights
├── utils.py # Image/video loading helpers
└── test_image.jpg # Sample test image
conda create -n qwen35 python=3.11
conda activate qwen35
pip install torch torchvision
pip install transformers safetensors huggingface_hub
pip install pillow requestscd qwen3_5_vlm
python download.pyThis downloads the Qwen3.5-0.8B checkpoint (~873 MB) to qwen3_5_vlm/Qwen3.5-0.8B/.
All commands below should be run from the repo root (the directory containing qwen3_5_vlm/).
python -m qwen3_5_vlm.main --backend scratchpython -m qwen3_5_vlm.main --backend scratch --prompt "Explain attention in transformers."python -m qwen3_5_vlm.main --backend wrapperpython -m qwen3_5_vlm.main --image qwen3_5_vlm/test_image.jpgpython -m qwen3_5_vlm.main --image qwen3_5_vlm/test_image.jpg --prompt "What objects are in this image?"python -m qwen3_5_vlm.main --video path/to/video.mp4Shows every step of inference (config loading, model creation, weight loading, tokenization, generation) with print statements:
python -m qwen3_5_vlm.main --low-level| Flag | Description | Default |
|---|---|---|
--backend |
scratch or wrapper |
scratch |
--image |
Path or URL to an image | None |
--video |
Path or URL to a video | None |
--prompt |
Custom text prompt | Auto |
--max-new-tokens |
Max tokens to generate | 512 |
--dtype |
auto, float16, bfloat16, float32 |
auto |
--device |
cpu, cuda, cuda:0, etc. |
Auto-detect |
--low-level |
Step-by-step educational demo | Off |
Compare the from-scratch implementation against the HF wrapper to verify they produce matching logits:
python -m qwen3_5_vlm.comparepython -m qwen3_5_vlm.compare --with-image qwen3_5_vlm/test_image.jpgpython -m qwen3_5_vlm.compare --dtype float32
python -m qwen3_5_vlm.compare --dtype bfloat16 --device cuda
python -m qwen3_5_vlm.compare --dtype float32 --with-image qwen3_5_vlm/test_image.jpg| Flag | Description | Default |
|---|---|---|
--model-dir |
Path to checkpoint directory | qwen3_5_vlm/Qwen3.5-0.8B |
--dtype |
float16, bfloat16, float32 |
bfloat16 |
--device |
cpu, cuda, etc. |
Auto-detect |
--atol |
Absolute tolerance | 1e-4 |
--rtol |
Relative tolerance | 1e-3 |
--with-image |
Path to image for multimodal test | None |
| Test | dtype | Max Diff | Top-1 Match | Status |
|---|---|---|---|---|
| Text-only | float32 | ~2.8e-05 | 100% | PASS |
| Text-only | bfloat16 | ~0.47 | 100% | Numerically close |
| Image+Text | float32 | ~0.033 | 100% | Numerically close |
Note: bfloat16 differences are expected due to reduced precision in the DeltaNet recurrence. Both implementations produce identical top-1 predictions at all positions.
# === SETUP (one time) ===
cd qwen3_5_vlm && python download.py && cd ..
# === TEXT INFERENCE ===
python -m qwen3_5_vlm.main --backend scratch
python -m qwen3_5_vlm.main --backend wrapper
# === IMAGE INFERENCE ===
python -m qwen3_5_vlm.main --image qwen3_5_vlm/test_image.jpg
# === CORRECTNESS CHECK ===
python -m qwen3_5_vlm.compare --dtype float32
python -m qwen3_5_vlm.compare --dtype float32 --with-image qwen3_5_vlm/test_image.jpg
# === EDUCATIONAL WALKTHROUGH ===
python -m qwen3_5_vlm.main --low-levelfrom qwen3_5_vlm.scratch import Qwen35VLMPipeline
pipe = Qwen35VLMPipeline("qwen3_5_vlm/Qwen3.5-0.8B", device="cuda")
# Text-only
answer = pipe.chat([{"role": "user", "content": "What is attention?"}])
# Image
answer = pipe.describe_image("photo.jpg", "What's in this image?")
# Video
answer = pipe.describe_video("clip.mp4", "Describe this video.")# Import from individual modules for study
from qwen3_5_vlm.scratch.config import Qwen35Config
from qwen3_5_vlm.scratch.attention import Qwen35Attention
from qwen3_5_vlm.scratch.delta_net import Qwen35GatedDeltaNet
from qwen3_5_vlm.scratch.vision import Qwen35VisionModel
from qwen3_5_vlm.scratch.vlm import Qwen35ForConditionalGeneration
# Or import everything via the re-export hub (backward compatible)
from qwen3_5_vlm.scratch.modeling import Qwen35ForConditionalGenerationThe from-scratch implementation includes several key optimizations for fast, memory-efficient generation:
During generation, re-processing the entire sequence for every new token is O(N^2) total. The HybridCache stores intermediate state so each decode step only processes ONE new token:
- Full attention layers: Growing KV cache (append new K,V each step)
- Linear attention layers: Fixed-size conv window + recurrent state matrix (O(1) per step!)
Result: Generation goes from O(N * max_tokens) to O(N + max_tokens).
Uses PyTorch's F.scaled_dot_product_attention which fuses Q@K, softmax, and @V into a single CUDA kernel:
- 2-3x faster than manual attention
- O(sqrt(N)) memory via Flash Attention (no NxN matrix materialized)
- Automatic backend selection (Flash v2, memory-efficient, or math fallback)
DeltaNet uses two different code paths:
- Prefill (full prompt): Chunked O(n) recurrence processes the entire prompt
- Decode (new tokens): Single O(1) recurrent step per token
The decode path:
- Updates a rolling conv1d window (shift + append, no full convolution)
- Does one matrix update:
S = decay * S + k @ delta^T - Reads output:
o = S^T @ q
4 KV heads shared across 16 query heads = 4x less KV cache memory.
If you're learning how the model works, read the source files in this order:
| # | File | What You'll Learn |
|---|---|---|
| 1 | config.py |
Model hyperparameters and their meaning |
| 2 | norm.py |
RMSNorm vs LayerNorm, why RMSNorm is faster |
| 3 | rope.py |
Rotary position embeddings, M-RoPE for 3D positions |
| 4 | mlp.py |
SwiGLU gated feed-forward, dimension flow |
| 5 | attention.py |
Grouped Query Attention, SDPA, KV cache, sigmoid gating |
| 6 | delta_net.py |
O(n) linear attention, O(1) decode, delta rule recurrence |
| 7 | cache.py |
HybridCache: KV cache + conv/recurrent state caching |
| 8 | decoder.py |
Pre-norm residuals, hybrid layer routing |
| 9 | vision.py |
ViT pipeline, Conv3D patches, patch merging |
| 10 | text_model.py |
Decoder stack, M-RoPE position handling, causal masking |
| 11 | vlm.py |
Vision-language bridge (masked_scatter), optimized generation |
[Image/Video pixels]
|
v
+-----------------+
| Vision Encoder | vision.py
| - Conv3D patch | Splits image into 16x16 patches
| - ViT blocks | Self-attention over patches
| - Patch merger | Projects to text model dimension
+-----------------+
|
v (visual embeddings replace <|image_pad|> tokens)
+-----------------+
| Language Model | text_model.py + decoder.py
| - Token embed | Text tokens -> embeddings
| - N layers of: |
| - Attention | Full self-attention OR Gated DeltaNet
| - MLP | SwiGLU feed-forward network
| - LM head | Embeddings -> vocabulary logits
+-----------------+
|
v
[Generated text tokens]