Efficient Vision-Language Model via Multi-head Latent Attention
-
Updated
Dec 30, 2025 - Python
#
multi-head-latent-attention
Here are 13 public repositories matching this topic...
DeepSeek-style MoE + MLA from scratch in PyTorch, with a router-specialization probe (mutual information) and a dense/+MoE/+MLA ablation. Educational, nano-scale, measured.
transformers pytorch moe from-scratch mla mixture-of-experts nanogpt deepseek multi-head-latent-attention llm-from-scratch
-
Updated
Jun 23, 2026 - Python
🔧 Recurrent-Depth Transformer Research Lab — LTI-stable looped inference, switchable MLA/GQA attention, MoE routing & adaptive halting (ACT). Independent research, not affiliated with Anthropic.
open-source opensource deep-learning transformers pytorch codex claude adaptive-computation-time mythos ai-research mixture-of-experts halting latent-reasoning claude-code multi-head-latent-attention recurrent-depth-transformer openmythos
-
Updated
Jul 27, 2026 - Python
PyTorch implementations of attention mechanisms from Self-Attention to DeepSeek MLA, including MHA, MQA, GQA, and KV Cache.
pytorch self-attention multi-head-attention pytorch-implementation kv-cache multi-head-latent-attention
-
Updated
Sep 4, 2026 - Jupyter Notebook
LLM 推理性能决策基线：TTFT/TPOT、KV Cache、吞吐与解码策略对照
numpy pytorch transformer llama attention-mechanism rotary self-attention prefill inference-optimization gqa kv-cache llm rmsnorm llm-inference grouped-query-attention speculative-decoding deepseek swiglu multi-head-latent-attention decode-acceleration
-
Updated
Aug 20, 2026 - Python
This repository shows how to build a DeepSeek language model from scratch using PyTorch. It includes clean, well-structured implementations of advanced attention techniques such as key–value caching for fast decoding, multi-query attention, grouped-query attention, and multi-head latent attention.
transformers pytorch multi-query-attention grouped-query-attention multi-head-latent-attention deepseek-from-scratch
-
Updated
Jan 10, 2026 - Jupyter Notebook
FlashAttention + Multi-Head Latent Attention, from scratch: Triton kernel, checkpointed PyTorch attention, MLA, and a reproduced roofline argument on real PTB-XL multimodal data.
deep-learning pytorch transformer triton attention-mechanism gpu-kernels roofline-model kv-cache ptb-xl flash-attention deepseek multi-head-latent-attention
-
Updated
Aug 29, 2026 - Python
custom Bengali Language Model built from scratch using PyTorch, featuring advanced transformer architectures including Multi-Head Latent Attention, RoPE positional encodings, and Mixture of Experts layers.
-
Updated
Mar 13, 2026 - Jupyter Notebook
Distributed training framework for DeepSeek-V3 (Multi-Head Latent Attention, DeepSeekMoE, auxiliary-loss-free load balancing) with composable DP/FSDP/HSDP/TP/PP/CP/EP parallelism, torchtitan-inspired.
pytorch moe data-parallelism distributed-training pipeline-parallelism kv-cache tensor-parallelism fsdp deepseek context-parallelism multi-head-latent-attention expert-parallelism
-
Updated
Sep 1, 2026 - Python
Hybrid LLM pre-training framework fusing Multi-Head Latent Attention, Gated Delta Net, DeepSeek MoE, and Multi-Token Prediction — 415M active params, 8.31B tokens, single A100 80GB.
nlp deep-learning pytorch transformer mixture-of-experts pretraining single-gpu llm multi-head-latent-attention fusionllm gated-delta-net
-
Updated
Jul 17, 2026 - Python
Multi-head latent attention from DeepSeek-V2: low-rank KV compression, decoupled RoPE, and absorbed inference verified equal to the naive form. 14.22x cache reduction.
machine-learning transformers pytorch attention from-scratch rope inference-optimization kv-cache llm deepseek multi-head-latent-attention
-
Updated
Sep 10, 2026 - Python
Correctness-first PyTorch/CUDA lab for FlashAttention, DeepSeek MLA/MoE, expert parallelism, reproducible benchmarks, and HPC-for-AI notes.
cuda pytorch gpu-kernels mixture-of-experts flash-attention deepseek ai-infrastructure multi-head-latent-attention expert-parallelism
-
Updated
Aug 27, 2026 - Python
Add this topic to your repo
To associate your repository with the multi-head-latent-attention topic, visit your repo's landing page and select "manage topics."