This is a repository with the code for the ACL 2019 paper "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned" and the ACL 2021 paper "Analyzing Source and Target Contributions to NMT Predictions".

PCA analysis of Q/K attention vectors to discover position-correlated components across transformer heads

CLI toolkit that ingests qk-sniffer dumps, measures per-head positional predictability and attention plasticity, and exports CSV stats plus ready-to-share plots.

Configurable character-level transformer training suite with built-in mechanistic interpretability toolkit — scale to 150M+ parameters and beyond, no ceilings, only hardware limits. Inspect attention weights, hidden states, and head specialisation across all layers. Documented circuit findings included.

Mechanistic interpretability tool to detect induction heads in GPT-2 using TransformerLens

TMLR 2026 | Mechanistic interpretability: attention-head binding (EB*) as a marker of concept emergence. 7 models, 5 architectures (Pythia 160M–2.8B, OLMo-1B, CRFM GPT-2, SmolLM3-3B, Qwen2.5-1.5B), 41 terms.

Replication of 'From Reasoning to Answer' (EMNLP 2025) — Reasoning-Focus Heads + Activation Patching on DeepSeek-R1-Distill-Qwen-7B

REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression

Adaptive inference algorithm for transformers inspired by quantum collapse (SR framework)

Public derivative EPI scaffold for attention-head pruning research framing, review gates, and claim boundaries.