diff --git a/_posts/2026-03-27-glassbox.md b/_posts/2026-03-27-glassbox.md new file mode 100644 index 00000000..b90efea0 --- /dev/null +++ b/_posts/2026-03-27-glassbox.md @@ -0,0 +1,263 @@ +--- +layout: post +title: "Glassbox: Grab vLLM's Attention" +author: "Diego Maniloff (dmanilof@redhat.com), Dominik Dahlem (ddahlem@redhat.com), Mac Misiura (mmisiura@redhat.com) — Red Hat AI" +tags: + - ecosystem + - safety +--- + +Transformer internals contain a great deal of information about how a model is behaving. Attention patterns, in particular, are a promising source of signals for hallucination detection, failure mode diagnosis, task drift detection, uncertainty estimation, and ongoing monitoring of model behavior. + +This post introduces [glassbox](https://github.com/dmaniloff/glassbox), a vLLM plugin for extracting structured signals from transformer attention during inference. + +## Why build this? + +Raw activations and full attention matrices are expensive to retain, and modern serving engines like vLLM are specifically optimized to *never materialize* the full L×L attention matrix (FlashAttention, Triton kernels). That's a good thing for performance, but it means the structure we care about is hidden by design. + +Because of that, most tools for inspecting transformer internals live in research harnesses around HuggingFace models, creating a gap between papers and what is practical for inference. + +## What is glassbox? +Glassbox is a vLLM plugin that instruments the attention path during inference to extract compact, structured features from attention-related operators. It is designed to be a practical implementation of ideas from the attention analysis literature in a form that works inside a real inference engine. + +The project is built around three ideas: + +1. Research-informed signals +2. Built for inference +3. vLLM-native + +We will expand on each of these in the following sections. + +## 1. Research-informed signals + +Glassbox implements state-of-the-art methods for extracting and analyzing attention structure, and extends them with new techniques from active research by the safety team at Red Hat AI. + +Today, glassbox extracts features from five attention-derived operators: + +**Spectral features from the pre-softmax scores matrix** S = QKT. The leading singular values of S reveal attention sharpness (is the head focused on one dominant pattern or distributing attention across several?), whether a head is content-adaptive or positional, and how the attention structure evolves over the course of generation. + +**Attention symmetry features from the raw post-softmax matrix** A = softmax(S/√d) ([AttentionTracker, arXiv:2411.00348](https://arxiv.org/abs/2411.00348)). These capture the coupling between symmetric and antisymmetric parts of the attention matrix — structure connected to recent results on mechanistic classification of failure modes such as prompt injection and hallucination. + +**Diagonal self-attention features** from the diagonal of A ([LLM-Check, NeurIPS 2024](https://github.com/GaurangSriramanan/LLM_Check_Hallucination_Detection)). The self-attention weight A[i,i] — how much a token attends to itself — correlates with model confidence and factuality. Glassbox extracts these without materializing A by computing only the diagonal entries. + +**Laplacian eigenvalue features** from the in-degree graph Laplacian L = Din - A ([LapEigvals, EMNLP 2025, arXiv:2502.17598](https://arxiv.org/abs/2502.17598)). Treating attention as a weighted directed graph, the Laplacian diagonal captures how much attention each token receives. For causal attention, eigenvalues are just the diagonal entries — no decomposition needed. + +**Routing features from the degree-normalized post-softmax operator** M = DQ-1/2 A DK-1/2 ([Dahlem et al., upcoming](TODO)). Degree normalization removes heterogeneity from uneven attention distributions and exposes the effective routing structure of attention. Glassbox extracts spectral features from M as well as features from its Hodge decomposition — separating attention asymmetry into potential-driven (gradient-like) and circulatory (irreversible) components. These routing and flow features are new implementations from ongoing research by the safety team at Red Hat AI. + +## 2. Built for inference + +Research tools that analyze attention typically materialize the full L×L matrix — fine for a notebook, but a non-starter at serving time. Glassbox is designed from the ground up with the inference use case in mind. + +**Matrix-free algorithms.** The core idea is that we never need the attention matrix explicitly. We only need to multiply it by vectors: + +``` +S·v = Q·(Kᵀ·v) — two thin matrix-vector products, O(Ld) +Sᵀ·u = K·(Qᵀ·u) — same cost +``` + +Each multiplication costs O(Ld) through the L×d factors Q and K — not O(L²). These "matvec" primitives are used in iterative SVD algorithms within glassbox. + +For the post-softmax operator, the problem is harder. Standard FlashAttention and Triton kernels already tile the softmax computation to avoid materializing the full matrix — but they're fused with the value matrix V from the KV cache. SVD probing requires applying softmax(QKT/√d) to *arbitrary* vectors, not V. Glassbox provides a fused Triton kernel that replaces V with the probe vectors, using the same online-softmax technique to apply the attention operator to multiple test vectors in a single kernel launch. + +**Configurable overhead.** Glassbox provides fine-grained control over the observability-to-latency tradeoff. Each feature group can be independently enabled or disabled. Extraction intervals, monitored layers and heads, SVD rank, and algorithm choice are all configurable: + +```yaml +spectral: + enabled: true + interval: 32 # extract every 32 decode steps + rank: 4 + method: randomized + heads: [0, 1, 2, 3] # monitor only these heads + +routing: + enabled: true + interval: 64 # less frequent for heavier features + rank: 4 + heads: [0] + +tracker: + enabled: false # off by default + +selfattn: + enabled: true + interval: 32 + heads: [0] + +laplacian: + enabled: true + interval: 32 + heads: [0] + +output: + path: glassbox.jsonl + +emit: + otel: true # emit as OpenTelemetry spans +``` + +You choose how much observability you want and how much latency you're willing to pay for it. + +## 3. vLLM-native + +Glassbox works through vLLM's supported extension points: + +**Custom attention backend** Glassbox's `SVDTritonAttentionBackend` provides access to attention internals. + +The implementation class calls the parent `forward()` first (standard Triton attention runs exactly as before), then accumulates Q tokens and periodically extracts K from the paged KV cache to run the feature extraction pipeline. Everything else — metadata builder, KV cache shape, kernel selection — is inherited unchanged. + +We then make the backend available via an entry point in `pyproject.toml`, from which vLLM loads glassbox automatically: + +```toml +[project.entry-points."vllm.general_plugins"] +glassbox = "glassbox.vllm_plugin:register_svd_backend" +``` + +We are also exploring out-of-tree `torch.compile` passes and `register_forward_hook` as additional extraction paths for hidden states and activations — a complementary signal source beyond attention. The latter has coincidentally been proposed as a recent vLLM RFC ([vllm-project/vllm#36998](https://github.com/vllm-project/vllm/issues/36998)), so we are excited to see it land. + +## Running glassbox + +Glassbox supports three run modes depending on the use case: + +**`vllm serve` — production inference.** Start vLLM with the custom backend and glassbox activates automatically via the plugin entry point. With `emit.otel: true` in `glassbox.yaml`, feature snapshots are emitted as OpenTelemetry spans that flow through the same OTel collector vLLM already uses (Jaeger, Tempo, Datadog, etc.) — zero additional infrastructure. + +```bash +vllm serve facebook/opt-125m --attention-backend CUSTOM --enforce-eager +``` + +**`glassbox-run` — single-prompt testing.** Quick way to try it out during development: + +```bash +glassbox-run \ + --model facebook/opt-125m \ + --signal spectral \ + --interval 16 --rank 4 --heads 0 \ + --otel \ + --prompt "The future of artificial intelligence is" +``` + +**`glassbox-extract` — offline feature extraction.** Runs two-phase extraction on labeled datasets and produces JSONL and Parquet files for downstream training: + +```bash +glassbox-extract \ + --model Qwen/Qwen2-7B-Instruct \ + --dataset halueval_hallucination \ + --signal spectral,routing,tracker,selfattn,laplacian \ + --parquet +``` + +### Signal emission + +Extracted signals are packaged into "snapshot" objects and flow through a pluggable handler system. Multiple handlers can be active simultaneously: + +- **JsonlHandler** writes every snapshot as a JSON line — for archival, bulk analysis, and training data. +- **OtelHandler** emits each snapshot as an OpenTelemetry span with `glassbox.*` attributes — for real-time detection pipelines. +- **Custom handlers** implement the `SnapshotHandler` protocol to forward snapshots to any sink: a feature store, Kafka, Redis, a trained classifier, or whatever your detection pipeline needs. + +OTel is just one delivery mechanism. The handler system is designed so that downstream consumers — whether a hallucination classifier, a monitoring dashboard, or a feature store — can plug in directly. + +Each snapshot is a structured record like this: + +```json +{ + "signal": "spectral", + "request_id": 0, + "layer": "model.layers.3.self_attn.attn", + "layer_idx": 3, + "head": 0, + "step": 32, + "L": 38, + "singular_values": [1582.6, 173.6, 102.4, 89.1], + "features": { + "sv1": 1582.6, + "sv_ratio": 9.12, + "sv_entropy": 0.63 + } +} +``` + +## A first look at what comes out + +Here's what it looks like in practice. We run `glassbox-run` on OPT-125m with spectral features enabled: + +```bash +$ glassbox-run --model facebook/opt-125m --signal spectral \ + --interval 16 --rank 2 --heads 0 \ + --prompt "The future of artificial intelligence is" +``` + +As the model generates tokens, glassbox streams feature snapshots for every layer at each configured interval: + +``` +INFO Creating vLLM engine with CUSTOM attention backend +INFO Model: facebook/opt-125m +INFO Signals: spectral=enabled routing=disabled tracker=disabled selfattn=disabled laplacian=disabled +INFO Spectral: interval=16 rank=2 method=randomized heads=[0] + +INFO Starting generation... + +[spectral] layers.0.self_attn head=0 step=32 L=38 sv1=274.6 sv_ratio=1.81 sv_entropy=1.32 +[spectral] layers.1.self_attn head=0 step=32 L=38 sv1=1067.9 sv_ratio=14.4 sv_entropy=0.54 +[spectral] layers.2.self_attn head=0 step=32 L=38 sv1=508.0 sv_ratio=1.94 sv_entropy=1.22 +[spectral] layers.3.self_attn head=0 step=32 L=38 sv1=1582.6 sv_ratio=9.12 sv_entropy=0.63 + ... +[spectral] layers.11.self_attn head=0 step=32 L=38 sv1=816.3 sv_ratio=3.03 sv_entropy=1.15 + +[spectral] layers.0.self_attn head=0 step=64 L=70 sv1=570.3 sv_ratio=2.30 sv_entropy=1.29 +[spectral] layers.1.self_attn head=0 step=64 L=70 sv1=2004.5 sv_ratio=12.5 sv_entropy=0.59 + ... + +INFO Generated: clouding the world. A report from Canalys found that artificial + intelligence is projected to grow significantly by 20% by 2023 ... +``` + +Each line is a feature snapshot for one (layer, head, step) tuple. The same data can be written to JSONL files, emitted as OpenTelemetry spans, or forwarded to any custom handler. + +### What the features tell you + +The σ₁/σ₂ ratio — how sharply attention is concentrated on a single dominant pattern — tells a different story at each layer: + +| Layer | step 16 | step 32 | step 48 | step 64 | Behavior | +|-------|---------|---------|---------|---------|----------| +| 1 | 14.1 | 14.4 | 14.1 | 12.5 | Constant — content-independent (positional) | +| 3 | 12.2 | 9.12 | 6.3 | 5.33 | Decaying — attention spreads as context diversifies | +| 7 | 1.22 | 1.22 | 1.59 | 1.32 | Near-isotropic — no single direction dominates | + +**Layer 1** has a rock-stable ratio of ~14 regardless of what's being generated — the signature of a fixed structural pattern, almost certainly positional attention (e.g. always attend to the BOS token). + +**Layer 3** starts very sharp (12×) but steadily decays to 5× as the generated text introduces diverse entities. The dominant Q-K direction can't capture everything — a second direction gains weight. + +**Layer 7** stays nearly isotropic (~1.2–1.3×) throughout — multiple attention patterns are active simultaneously, with no single direction dominating. + +This works across architectures. We've verified it on GPT-2 (124M, standard multi-head attention) and Qwen2-7B-Instruct (7B, grouped-query attention with 28 Q heads / 4 KV heads) with no code changes. + +This is a small model on a toy prompt. But the point is not the specific numbers — it's that different layers have qualitatively different spectral behaviors during generation, and those behaviors are interpretable. That's the kind of structure downstream systems can learn from. + +## The vision + +Our goal is to make a class of model-internal signals operational inside vLLM — signals that help distinguish normal behavior from anomalous behavior, localize where a model starts to drift, and that downstream systems can learn from, compare, aggregate, and act on. + +Extracting these signals is half the problem. The other half is: once you have them, how do you act on them? + +Tools like [vLLM Semantic Router](https://github.com/vllm-project/vllm/tree/main/examples/online_serving/semantic_router) route requests based on external signals — the query text, domain, urgency. Glassbox provides a complementary set of *internal* signals — what the model is actually doing during generation. The question we're exploring is: what is the right mechanism to close the loop? + +We're experimenting with several directions: + +- **Inline detection.** A `ClassifierHandler` that runs a trained probe on attention features during inference and flags suspicious behavior in real time — via OTel, logging, or a custom handler. +- **vLLM-native feedback.** The Observation Plugin RFC ([vllm-project/vllm#36998](https://github.com/vllm-project/vllm/issues/36998)) proposes ABORT/CONTINUE actions based on model internals. We're prototyping an integration that bridges glassbox verdicts into this mechanism, allowing the engine to halt a request when hallucination is detected from attention features. +- **External serving.** A REST endpoint that exposes trained classifiers to external routing systems — semantic router, llm-d, or custom orchestrators. +- **Shadow deployment.** Running glassbox via [llm-d](https://llm-d.ai/) in shadow mode alongside the serving path, separating the observability workload from inference entirely for cases where zero inline overhead is required. + +We don't yet know which pattern fits best, and the answer may be different for different use cases. We're figuring it out. + +## Where we are going next + +In future posts, we plan to go deeper into: + +- **Feature evaluation.** How do the derived features actually correlate with failure modes? We plan to run across labeled datasets like [HaluEval](https://github.com/RUCAIBox/HaluEval) and [TruthfulQA](https://github.com/sylinrl/TruthfulQA) and see what signal is there — even if the answer is mixed. +- **Overhead and benchmarks.** How much does this cost? What does the observability-to-latency tradeoff look like in practice? +- **Closing the loop.** From signal extraction to action — training detection models on glassbox features, running them inline, and integrating with vLLM's request lifecycle. +- **Design details.** The matrix-free extraction path, the fused Triton kernel, and the engineering of the vLLM integration in more depth. + +--- + +*Code: [github.com/dmaniloff/glassbox](https://github.com/dmaniloff/glassbox)*