From 8508b2ac87a9f99e3543c7961a7d0a1130697071 Mon Sep 17 00:00:00 2001 From: Diego Maniloff Date: Fri, 27 Mar 2026 22:49:10 +0000 Subject: [PATCH 1/6] =?UTF-8?q?Add=20blog=20post:=20Glassbox=20=E2=80=94?= =?UTF-8?q?=20Grab=20vLLM's=20Attention?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces glassbox, a vLLM plugin for extracting structured signals from transformer attention during inference. Covers research-informed signals, matrix-free SVD for inference efficiency, and vLLM-native integration via custom attention backends. Co-Authored-By: Claude Opus 4.6 Signed-off-by: Diego Maniloff --- _posts/2026-03-27-glassbox.md | 161 ++++++++++++++++++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 _posts/2026-03-27-glassbox.md diff --git a/_posts/2026-03-27-glassbox.md b/_posts/2026-03-27-glassbox.md new file mode 100644 index 00000000..adc86056 --- /dev/null +++ b/_posts/2026-03-27-glassbox.md @@ -0,0 +1,161 @@ +--- +layout: post +title: "Glassbox: Grab vLLM's Attention" +author: "Diego Maniloff (dmanilof@redhat.com), Dominik Dahlem (ddahlem@redhat.com), Mac Misiura (mmisiura@redhat.com) — Red Hat AI" +tags: + - ecosystem + - safety +--- + +This post introduces [glassbox](https://github.com/dmaniloff/glassbox), a vLLM plugin for extracting structured signals from transformer attention during inference. + +## Always start with the why + +Transformer internals contain a great deal of information about how a model is behaving. Attention patterns, in particular, can reveal when behavior is becoming concentrated, unstable, or asymmetric — often before those shifts are visible in the output text. + +That makes attention a promising source of signals for: + +- Hallucination detection +- Failure mode diagnosis +- Task drift detection +- Uncertainty estimation +- Ongoing monitoring of model behavior in production + +But there is a practical problem. Raw activations and full attention matrices are expensive to retain, and modern serving engines like vLLM are specifically optimized to *never materialize* the full L×L attention matrix (FlashAttention, Triton kernels). That's a good thing for performance, but it means the structure we care about is hidden by design. + +Because of that, most tools for inspecting transformer internals live in research harnesses around HuggingFace models, creating a gap between papers and what is practical for inference. + +## What is glassbox? +Glassbox is a vLLM plugin that instruments the attention path during inference to extract compact, structured features from attention-related operators. It is designed to be a practical implementation of ideas from the attention analysis literature in a form that works inside a real inference engine. + +The project is built around three ideas: + +1. Research-informed signals +2. Built for inference +3. vLLM-native + +We will expand on each of these in the following sections. + +## 1. Research-informed signals + +Glassbox implements state-of-the-art methods for extracting and analyzing attention structure, and extends them with new techniques from active research by the safety team at Red Hat AI. + +Today, glassbox extracts features from four attention-derived operators: + +**Spectral features from the pre-softmax scores matrix** S = QKT. The leading singular values of S reveal attention sharpness (is the head focused on one dominant pattern or distributing attention across several?), whether a head is content-adaptive or positional, and how the attention structure evolves over the course of generation. + +**Attention symmetry features from the raw post-softmax matrix** A = softmax(S/√d) ([AttentionTracker, arXiv:2411.00348](https://arxiv.org/abs/2411.00348)). These capture the coupling between symmetric and antisymmetric parts of the attention matrix — structure connected to recent results on mechanistic classification of failure modes such as prompt injection and hallucination. + +**Diagonal self-attention features** from the diagonal of A ([LLM-Check, NeurIPS 2024](https://github.com/GaurangSriramanan/LLM_Check_Hallucination_Detection)). The self-attention weight A[i,i] — how much a token attends to itself — correlates with model confidence and factuality. Glassbox extracts these without materializing A by computing only the diagonal entries. + +**Routing features from the degree-normalized post-softmax operator** M = DQ-1/2 A DK-1/2 ([Dahlem et al., upcoming](TODO)). Degree normalization removes heterogeneity from uneven attention distributions and exposes the effective routing structure of attention. Glassbox extracts spectral features from M as well as features from its Hodge decomposition — separating attention asymmetry into potential-driven (gradient-like) and circulatory (irreversible) components. These routing and flow features are new implementations from ongoing research by the safety team at Red Hat AI. + + +Glassbox is designed as a practical implementation of ideas from the attention analysis literature in a form that works inside a real inference engine. + +## 2. Built for inference + +Research tools that analyze attention typically materialize the full L×L matrix — fine for a notebook, but a non-starter at serving time. Glassbox is designed from the ground up with the inference use case in mind. + +**Matrix-free algorithms.** The core idea is that we never need the attention matrix explicitly. We only need to multiply it by vectors: + +``` +S·v = Q·(Kᵀ·v) — two thin matrix-vector products, O(Ld) +Sᵀ·u = K·(Qᵀ·u) — same cost +``` + +Each multiplication costs O(Ld) through the L×d factors Q and K — not O(L²). These "matvec" primitives are used in iterative SVD algorithms within glassbox. + +For the post-softmax operator, the problem is harder: applying A = softmax(QKT/√d) to arbitrary vectors requires computing softmax without materializing the full matrix. Glassbox handles this with blocked row-streaming (computing softmax in tiles and accumulating results) and a fused Triton kernel that uses online softmax to apply the attention matrix to multiple probe vectors in a single kernel launch — avoiding both O(L²) memory and redundant softmax recomputation. + +**Configurable overhead.** Glassbox provides fine-grained control over the observability-to-latency tradeoff. Each feature group can be independently enabled or disabled. Extraction intervals, monitored layers and heads, SVD rank, and algorithm choice are all configurable: + +```yaml +scores_matrix: + enabled: true + interval: 32 # extract every 32 decode steps + rank: 4 + method: randomized + heads: [0, 1, 2, 3] # monitor only these heads + +degree_normalized_matrix: + enabled: true + interval: 64 # less frequent for heavier features + rank: 4 + heads: [0] + +attention_tracker: + enabled: false # off by default + +attention_diagonal: + enabled: true + interval: 32 + heads: [0] + +matvec_strategy: auto # triton > batched > loop +output: glassbox.jsonl +``` + +You choose how much observability you want and how much latency you're willing to pay for it. + +## 3. vLLM-native + +Glassbox works through vLLM's supported extension points: + +**Custom attention backend** Glassbox's `SVDTritonAttentionBackend` provides access to attention internals. + +The implementation class calls the parent `forward()` first (standard Triton attention runs exactly as before), then accumulates Q tokens and periodically extracts K from the paged KV cache to run the feature extraction pipeline. Everything else — metadata builder, KV cache shape, kernel selection — is inherited unchanged. + +We then make the backend available via an entry point in `pyproject.toml`, from which vLLM loads glassbox automatically: + +```toml +[project.entry-points."vllm.general_plugins"] +glassbox = "glassbox.vllm_plugin:register_svd_backend" +``` + +We are also exploring out-of-tree `torch.compile` passes and `register_forward_hook` as additional extraction paths for hidden states and activations — a complementary signal source beyond attention. The latter has coincidentally been proposed as a recent vLLM RFC ([vllm-project/vllm#36998](https://github.com/vllm-project/vllm/issues/36998)), so we are excited to see it land. + +## A first look at what comes out + +Here's a concrete example. We run glassbox on OPT-125m with the prompt "The future of artificial intelligence is", generating 64 tokens. SVD snapshots are taken every 16 steps, extracting the top 2 singular values of the pre-softmax scores matrix for head 0 across all 12 layers. + +The σ₁/σ₂ ratio — how sharply attention is concentrated on a single dominant pattern — tells a different story at each layer: + +| Layer | step 16 | step 32 | step 48 | step 64 | Behavior | +|-------|---------|---------|---------|---------|----------| +| 1 | 14.1 | 13.0 | 14.1 | 14.1 | Constant — content-independent (positional) | +| 3 | 12.2 | 7.6 | 6.3 | 5.1 | Decaying — attention spreads as context diversifies | +| 7 | 1.22 | 1.47 | 1.59 | 2.07 | Increasing — one direction consolidates over time | + +**Layer 1** has a rock-stable ratio of ~14 regardless of what's being generated — the signature of a fixed structural pattern, almost certainly positional attention (e.g. always attend to the BOS token). + +**Layer 3** starts very sharp (12×) but steadily decays to 5× as the generated text introduces diverse entities. The dominant Q-K direction can't capture everything — a second direction gains weight. + +**Layer 7** starts nearly isotropic (1.2×) but slowly sharpens to 2× as the generation settles into a repetitive theme and one attention direction begins to dominate. + +This works across architectures. We've verified it on GPT-2 (124M, standard multi-head attention) and Qwen2-7B-Instruct (7B, grouped-query attention with 28 Q heads / 4 KV heads) with no code changes. + +This is a small model on a toy prompt. But the point here is not the specific numbers — it's that different layers have qualitatively different spectral behaviors during generation, and those behaviors are interpretable. That's the kind of structure downstream systems can learn from. + +## The vision +Our goal is to make a class of model-internal signals operational inside vLLM. We want to provide: + +- Signals that help distinguish normal behavior from anomalous behavior +- Signals that help localize where a model starts to drift +- Signals that downstream systems — classifiers, monitors, debugging tools — can learn from, compare, aggregate, and act on + +The value is not in any single metric, but in making this family of metrics available in a practical serving pipeline, informed by the best current research on what those metrics should be. + +## Where we are going next + +In future posts, we plan to go deeper into: + +- **Feature evaluation.** How do the derived features actually correlate with failure modes? We plan to run across labeled datasets like [HaluEval](https://github.com/RUCAIBox/HaluEval) and [TruthfulQA](https://github.com/sylinrl/TruthfulQA) and see what signal is there — even if the answer is mixed. +- **Overhead and benchmarks.** How much does this cost? What does the observability-to-latency tradeoff look like in practice? And for cases where even minimal inline overhead is unacceptable, we are exploring deployment via [llm-d](https://llm-d.ai/) — running glassbox in shadow mode alongside the serving path, separating the observability workload from inference entirely. +- **Which features matter most.** Across the different operators and decompositions, which attention-derived features appear most predictive for different failure modes? +- **Layer and head dynamics.** How these signals evolve across layers, heads, and decoding steps — and what that tells us about model internals. +- **Design details.** The matrix-free extraction path, the fused Triton kernel, and the engineering of the vLLM integration in more depth. + +--- + +*Code: [github.com/dmaniloff/glassbox](https://github.com/dmaniloff/glassbox)* From 6659469e02b33eee5ac0aefb86f5bf794a53535f Mon Sep 17 00:00:00 2001 From: Diego Maniloff Date: Tue, 14 Apr 2026 17:31:39 +0000 Subject: [PATCH 2/6] Update blog post: add LapEigvals signal, run modes, handler system, vision - Add LapEigvals (EMNLP 2025) as 5th signal group - Update YAML config to current signal names (spectral, routing, tracker, selfattn, laplacian) - Add "Running glassbox" section with three run modes (vllm serve, glassbox-run, glassbox-extract) - Add signal emission subsection (JsonlHandler, OtelHandler, custom handlers) - Rewrite vision section around closing the loop: inline detection, RFC #36998 ABORT/CONTINUE, external serving, llm-d shadow mode - Fix post-softmax description: clarify difference from FlashAttention tiling Co-Authored-By: Claude Opus 4.6 (1M context) Signed-off-by: Diego Maniloff --- _posts/2026-03-27-glassbox.md | 103 ++++++++++++++++++++++++++-------- 1 file changed, 79 insertions(+), 24 deletions(-) diff --git a/_posts/2026-03-27-glassbox.md b/_posts/2026-03-27-glassbox.md index adc86056..0058627f 100644 --- a/_posts/2026-03-27-glassbox.md +++ b/_posts/2026-03-27-glassbox.md @@ -40,7 +40,7 @@ We will expand on each of these in the following sections. Glassbox implements state-of-the-art methods for extracting and analyzing attention structure, and extends them with new techniques from active research by the safety team at Red Hat AI. -Today, glassbox extracts features from four attention-derived operators: +Today, glassbox extracts features from five attention-derived operators: **Spectral features from the pre-softmax scores matrix** S = QKT. The leading singular values of S reveal attention sharpness (is the head focused on one dominant pattern or distributing attention across several?), whether a head is content-adaptive or positional, and how the attention structure evolves over the course of generation. @@ -48,10 +48,9 @@ Today, glassbox extracts features from four attention-derived operators: **Diagonal self-attention features** from the diagonal of A ([LLM-Check, NeurIPS 2024](https://github.com/GaurangSriramanan/LLM_Check_Hallucination_Detection)). The self-attention weight A[i,i] — how much a token attends to itself — correlates with model confidence and factuality. Glassbox extracts these without materializing A by computing only the diagonal entries. -**Routing features from the degree-normalized post-softmax operator** M = DQ-1/2 A DK-1/2 ([Dahlem et al., upcoming](TODO)). Degree normalization removes heterogeneity from uneven attention distributions and exposes the effective routing structure of attention. Glassbox extracts spectral features from M as well as features from its Hodge decomposition — separating attention asymmetry into potential-driven (gradient-like) and circulatory (irreversible) components. These routing and flow features are new implementations from ongoing research by the safety team at Red Hat AI. - +**Laplacian eigenvalue features** from the in-degree graph Laplacian L = Din - A ([LapEigvals, EMNLP 2025, arXiv:2502.17598](https://arxiv.org/abs/2502.17598)). Treating attention as a weighted directed graph, the Laplacian diagonal captures how much attention each token receives. For causal attention, eigenvalues are just the diagonal entries — no decomposition needed. -Glassbox is designed as a practical implementation of ideas from the attention analysis literature in a form that works inside a real inference engine. +**Routing features from the degree-normalized post-softmax operator** M = DQ-1/2 A DK-1/2 ([Dahlem et al., upcoming](TODO)). Degree normalization removes heterogeneity from uneven attention distributions and exposes the effective routing structure of attention. Glassbox extracts spectral features from M as well as features from its Hodge decomposition — separating attention asymmetry into potential-driven (gradient-like) and circulatory (irreversible) components. These routing and flow features are new implementations from ongoing research by the safety team at Red Hat AI. ## 2. Built for inference @@ -66,34 +65,42 @@ Sᵀ·u = K·(Qᵀ·u) — same cost Each multiplication costs O(Ld) through the L×d factors Q and K — not O(L²). These "matvec" primitives are used in iterative SVD algorithms within glassbox. -For the post-softmax operator, the problem is harder: applying A = softmax(QKT/√d) to arbitrary vectors requires computing softmax without materializing the full matrix. Glassbox handles this with blocked row-streaming (computing softmax in tiles and accumulating results) and a fused Triton kernel that uses online softmax to apply the attention matrix to multiple probe vectors in a single kernel launch — avoiding both O(L²) memory and redundant softmax recomputation. +For the post-softmax operator, the problem is harder. Standard FlashAttention and Triton kernels already tile the softmax computation to avoid materializing the full matrix — but they're fused with the value matrix V from the KV cache. SVD probing requires applying softmax(QKT/√d) to *arbitrary* vectors, not V. Glassbox provides a fused Triton kernel that replaces V with the probe vectors, using the same online-softmax technique to apply the attention operator to multiple test vectors in a single kernel launch. **Configurable overhead.** Glassbox provides fine-grained control over the observability-to-latency tradeoff. Each feature group can be independently enabled or disabled. Extraction intervals, monitored layers and heads, SVD rank, and algorithm choice are all configurable: ```yaml -scores_matrix: +spectral: enabled: true - interval: 32 # extract every 32 decode steps + interval: 32 # extract every 32 decode steps rank: 4 method: randomized - heads: [0, 1, 2, 3] # monitor only these heads + heads: [0, 1, 2, 3] # monitor only these heads -degree_normalized_matrix: +routing: enabled: true - interval: 64 # less frequent for heavier features + interval: 64 # less frequent for heavier features rank: 4 heads: [0] -attention_tracker: - enabled: false # off by default +tracker: + enabled: false # off by default + +selfattn: + enabled: true + interval: 32 + heads: [0] -attention_diagonal: +laplacian: enabled: true interval: 32 heads: [0] -matvec_strategy: auto # triton > batched > loop -output: glassbox.jsonl +output: + path: glassbox.jsonl + +emit: + otel: true # emit as OpenTelemetry spans ``` You choose how much observability you want and how much latency you're willing to pay for it. @@ -115,6 +122,47 @@ glassbox = "glassbox.vllm_plugin:register_svd_backend" We are also exploring out-of-tree `torch.compile` passes and `register_forward_hook` as additional extraction paths for hidden states and activations — a complementary signal source beyond attention. The latter has coincidentally been proposed as a recent vLLM RFC ([vllm-project/vllm#36998](https://github.com/vllm-project/vllm/issues/36998)), so we are excited to see it land. +## Running glassbox + +Glassbox supports three run modes depending on the use case: + +**`vllm serve` — production inference.** Start vLLM with the custom backend and glassbox activates automatically via the plugin entry point. With `emit.otel: true` in `glassbox.yaml`, feature snapshots are emitted as OpenTelemetry spans that flow through the same OTel collector vLLM already uses (Jaeger, Tempo, Datadog, etc.) — zero additional infrastructure. + +```bash +vllm serve facebook/opt-125m --attention-backend CUSTOM --enforce-eager +``` + +**`glassbox-run` — single-prompt testing.** Quick way to try it out during development: + +```bash +glassbox-run \ + --model facebook/opt-125m \ + --signal spectral \ + --interval 16 --rank 4 --heads 0 \ + --otel \ + --prompt "The future of artificial intelligence is" +``` + +**`glassbox-extract` — offline feature extraction.** Runs two-phase extraction on labeled datasets and produces JSONL and Parquet files for downstream training: + +```bash +glassbox-extract \ + --model Qwen/Qwen2-7B-Instruct \ + --dataset halueval_hallucination \ + --signal spectral,routing,tracker,selfattn,laplacian \ + --parquet +``` + +### Signal emission + +Extracted signals flow through a pluggable handler system. Multiple handlers can be active simultaneously: + +- **JsonlHandler** writes every snapshot as a JSON line — for archival, bulk analysis, and training data. +- **OtelHandler** emits each snapshot as an OpenTelemetry span with `glassbox.*` attributes — for real-time detection pipelines. +- **Custom handlers** implement the `SnapshotHandler` protocol to forward snapshots to any sink: a feature store, Kafka, Redis, a trained classifier, or whatever your detection pipeline needs. + +OTel is just one delivery mechanism. The handler system is designed so that downstream consumers — whether a hallucination classifier, a monitoring dashboard, or a feature store — can plug in directly. + ## A first look at what comes out Here's a concrete example. We run glassbox on OPT-125m with the prompt "The future of artificial intelligence is", generating 64 tokens. SVD snapshots are taken every 16 steps, extracting the top 2 singular values of the pre-softmax scores matrix for head 0 across all 12 layers. @@ -138,22 +186,29 @@ This works across architectures. We've verified it on GPT-2 (124M, standard mult This is a small model on a toy prompt. But the point here is not the specific numbers — it's that different layers have qualitatively different spectral behaviors during generation, and those behaviors are interpretable. That's the kind of structure downstream systems can learn from. ## The vision -Our goal is to make a class of model-internal signals operational inside vLLM. We want to provide: -- Signals that help distinguish normal behavior from anomalous behavior -- Signals that help localize where a model starts to drift -- Signals that downstream systems — classifiers, monitors, debugging tools — can learn from, compare, aggregate, and act on +Our goal is to make a class of model-internal signals operational inside vLLM — signals that help distinguish normal behavior from anomalous behavior, localize where a model starts to drift, and that downstream systems can learn from, compare, aggregate, and act on. + +Extracting these signals is half the problem. The other half is: once you have them, how do you act on them? + +Tools like [vLLM Semantic Router](https://github.com/vllm-project/vllm/tree/main/examples/online_serving/semantic_router) route requests based on external signals — the query text, domain, urgency. Glassbox provides a complementary set of *internal* signals — what the model is actually doing during generation. The question we're exploring is: what is the right mechanism to close the loop? + +We're experimenting with several directions: + +- **Inline detection.** A `ClassifierHandler` that runs a trained probe on attention features during inference and flags suspicious behavior in real time — via OTel, logging, or a custom handler. +- **vLLM-native feedback.** The Observation Plugin RFC ([vllm-project/vllm#36998](https://github.com/vllm-project/vllm/issues/36998)) proposes ABORT/CONTINUE actions based on model internals. We're prototyping an integration that bridges glassbox verdicts into this mechanism, allowing the engine to halt a request when hallucination is detected from attention features. +- **External serving.** A REST endpoint that exposes trained classifiers to external routing systems — semantic router, llm-d, or custom orchestrators. +- **Shadow deployment.** Running glassbox via [llm-d](https://llm-d.ai/) in shadow mode alongside the serving path, separating the observability workload from inference entirely for cases where zero inline overhead is required. -The value is not in any single metric, but in making this family of metrics available in a practical serving pipeline, informed by the best current research on what those metrics should be. +We don't yet know which pattern fits best, and the answer may be different for different use cases. We're figuring it out. ## Where we are going next In future posts, we plan to go deeper into: -- **Feature evaluation.** How do the derived features actually correlate with failure modes? We plan to run across labeled datasets like [HaluEval](https://github.com/RUCAIBox/HaluEval) and [TruthfulQA](https://github.com/sylinrl/TruthfulQA) and see what signal is there — even if the answer is mixed. -- **Overhead and benchmarks.** How much does this cost? What does the observability-to-latency tradeoff look like in practice? And for cases where even minimal inline overhead is unacceptable, we are exploring deployment via [llm-d](https://llm-d.ai/) — running glassbox in shadow mode alongside the serving path, separating the observability workload from inference entirely. -- **Which features matter most.** Across the different operators and decompositions, which attention-derived features appear most predictive for different failure modes? -- **Layer and head dynamics.** How these signals evolve across layers, heads, and decoding steps — and what that tells us about model internals. +- **Feature evaluation.** How do the derived features actually correlate with failure modes? We plan to run across labeled datasets like [HaluEval](https://github.com/RUCAIBox/HaluEval) and [TruthfulQA](https://github.com/sylinrl/TruthfulQA) and see what signal is there — even if the answer is mixed. +- **Overhead and benchmarks.** How much does this cost? What does the observability-to-latency tradeoff look like in practice? +- **Closing the loop.** From signal extraction to action — training detection models on glassbox features, running them inline, and integrating with vLLM's request lifecycle. - **Design details.** The matrix-free extraction path, the fused Triton kernel, and the engineering of the vLLM integration in more depth. --- From 1e31748c7d35a43b86eacfddf2615efeba9a67da Mon Sep 17 00:00:00 2001 From: Diego Maniloff Date: Tue, 14 Apr 2026 20:04:36 +0000 Subject: [PATCH 3/6] Update first-look section with terminal-style demo output Show actual glassbox-run command and cleaned-up log output from a real OPT-125m run. Add "What the features tell you" subheading for the ratio trajectory table. Update table values to match real run. Co-Authored-By: Claude Opus 4.6 (1M context) Signed-off-by: Diego Maniloff --- _posts/2026-03-27-glassbox.md | 47 ++++++++++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 6 deletions(-) diff --git a/_posts/2026-03-27-glassbox.md b/_posts/2026-03-27-glassbox.md index 0058627f..ca03c814 100644 --- a/_posts/2026-03-27-glassbox.md +++ b/_posts/2026-03-27-glassbox.md @@ -165,25 +165,60 @@ OTel is just one delivery mechanism. The handler system is designed so that down ## A first look at what comes out -Here's a concrete example. We run glassbox on OPT-125m with the prompt "The future of artificial intelligence is", generating 64 tokens. SVD snapshots are taken every 16 steps, extracting the top 2 singular values of the pre-softmax scores matrix for head 0 across all 12 layers. +Here's what it looks like in practice. We run `glassbox-run` on OPT-125m with spectral features enabled: + +```bash +$ glassbox-run --model facebook/opt-125m --signal spectral \ + --interval 16 --rank 2 --heads 0 \ + --prompt "The future of artificial intelligence is" +``` + +As the model generates tokens, glassbox streams feature snapshots for every layer at each configured interval: + +``` +INFO Creating vLLM engine with CUSTOM attention backend +INFO Model: facebook/opt-125m +INFO Signals: spectral=enabled routing=disabled tracker=disabled selfattn=disabled laplacian=disabled +INFO Spectral: interval=16 rank=2 method=randomized heads=[0] + +INFO Starting generation... + +[spectral] layers.0.self_attn head=0 step=32 L=38 sv1=274.6 sv_ratio=1.81 sv_entropy=1.32 +[spectral] layers.1.self_attn head=0 step=32 L=38 sv1=1067.9 sv_ratio=14.4 sv_entropy=0.54 +[spectral] layers.2.self_attn head=0 step=32 L=38 sv1=508.0 sv_ratio=1.94 sv_entropy=1.22 +[spectral] layers.3.self_attn head=0 step=32 L=38 sv1=1582.6 sv_ratio=9.12 sv_entropy=0.63 + ... +[spectral] layers.11.self_attn head=0 step=32 L=38 sv1=816.3 sv_ratio=3.03 sv_entropy=1.15 + +[spectral] layers.0.self_attn head=0 step=64 L=70 sv1=570.3 sv_ratio=2.30 sv_entropy=1.29 +[spectral] layers.1.self_attn head=0 step=64 L=70 sv1=2004.5 sv_ratio=12.5 sv_entropy=0.59 + ... + +INFO Generated: clouding the world. A report from Canalys found that artificial + intelligence is projected to grow significantly by 20% by 2023 ... +``` + +Each line is a feature snapshot for one (layer, head, step) tuple. The same data can be written to JSONL files, emitted as OpenTelemetry spans, or forwarded to any custom handler. + +### What the features tell you The σ₁/σ₂ ratio — how sharply attention is concentrated on a single dominant pattern — tells a different story at each layer: | Layer | step 16 | step 32 | step 48 | step 64 | Behavior | |-------|---------|---------|---------|---------|----------| -| 1 | 14.1 | 13.0 | 14.1 | 14.1 | Constant — content-independent (positional) | -| 3 | 12.2 | 7.6 | 6.3 | 5.1 | Decaying — attention spreads as context diversifies | -| 7 | 1.22 | 1.47 | 1.59 | 2.07 | Increasing — one direction consolidates over time | +| 1 | 14.1 | 14.4 | 14.1 | 12.5 | Constant — content-independent (positional) | +| 3 | 12.2 | 9.12 | 6.3 | 5.33 | Decaying — attention spreads as context diversifies | +| 7 | 1.22 | 1.22 | 1.59 | 1.32 | Near-isotropic — no single direction dominates | **Layer 1** has a rock-stable ratio of ~14 regardless of what's being generated — the signature of a fixed structural pattern, almost certainly positional attention (e.g. always attend to the BOS token). **Layer 3** starts very sharp (12×) but steadily decays to 5× as the generated text introduces diverse entities. The dominant Q-K direction can't capture everything — a second direction gains weight. -**Layer 7** starts nearly isotropic (1.2×) but slowly sharpens to 2× as the generation settles into a repetitive theme and one attention direction begins to dominate. +**Layer 7** stays nearly isotropic (~1.2–1.3×) throughout — multiple attention patterns are active simultaneously, with no single direction dominating. This works across architectures. We've verified it on GPT-2 (124M, standard multi-head attention) and Qwen2-7B-Instruct (7B, grouped-query attention with 28 Q heads / 4 KV heads) with no code changes. -This is a small model on a toy prompt. But the point here is not the specific numbers — it's that different layers have qualitatively different spectral behaviors during generation, and those behaviors are interpretable. That's the kind of structure downstream systems can learn from. +This is a small model on a toy prompt. But the point is not the specific numbers — it's that different layers have qualitatively different spectral behaviors during generation, and those behaviors are interpretable. That's the kind of structure downstream systems can learn from. ## The vision From 0de336334dcd23773d1c8eb358fb166d88ded9d6 Mon Sep 17 00:00:00 2001 From: Diego Maniloff Date: Wed, 15 Apr 2026 16:20:30 +0000 Subject: [PATCH 4/6] snapshot example. Signed-off-by: Diego Maniloff --- _posts/2026-03-27-glassbox.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/_posts/2026-03-27-glassbox.md b/_posts/2026-03-27-glassbox.md index ca03c814..fa26f1c8 100644 --- a/_posts/2026-03-27-glassbox.md +++ b/_posts/2026-03-27-glassbox.md @@ -155,7 +155,7 @@ glassbox-extract \ ### Signal emission -Extracted signals flow through a pluggable handler system. Multiple handlers can be active simultaneously: +Extracted signals are packaged into "snapshot" objects and flow through a pluggable handler system. Multiple handlers can be active simultaneously: - **JsonlHandler** writes every snapshot as a JSON line — for archival, bulk analysis, and training data. - **OtelHandler** emits each snapshot as an OpenTelemetry span with `glassbox.*` attributes — for real-time detection pipelines. @@ -163,6 +163,26 @@ Extracted signals flow through a pluggable handler system. Multiple handlers can OTel is just one delivery mechanism. The handler system is designed so that downstream consumers — whether a hallucination classifier, a monitoring dashboard, or a feature store — can plug in directly. +Each snapshot is a structured record like this: + +```json +{ + "signal": "spectral", + "request_id": 0, + "layer": "model.layers.3.self_attn.attn", + "layer_idx": 3, + "head": 0, + "step": 32, + "L": 38, + "singular_values": [1582.6, 173.6, 102.4, 89.1], + "features": { + "sv1": 1582.6, + "sv_ratio": 9.12, + "sv_entropy": 0.63 + } +} +``` + ## A first look at what comes out Here's what it looks like in practice. We run `glassbox-run` on OPT-125m with spectral features enabled: From 67cef1f2f9a685c80d37037fd2f008182ca0eaff Mon Sep 17 00:00:00 2001 From: Diego Maniloff Date: Wed, 15 Apr 2026 16:20:52 +0000 Subject: [PATCH 5/6] whitespace. Signed-off-by: Diego Maniloff --- _posts/2026-03-27-glassbox.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2026-03-27-glassbox.md b/_posts/2026-03-27-glassbox.md index fa26f1c8..7f231e1d 100644 --- a/_posts/2026-03-27-glassbox.md +++ b/_posts/2026-03-27-glassbox.md @@ -23,7 +23,7 @@ That makes attention a promising source of signals for: But there is a practical problem. Raw activations and full attention matrices are expensive to retain, and modern serving engines like vLLM are specifically optimized to *never materialize* the full L×L attention matrix (FlashAttention, Triton kernels). That's a good thing for performance, but it means the structure we care about is hidden by design. -Because of that, most tools for inspecting transformer internals live in research harnesses around HuggingFace models, creating a gap between papers and what is practical for inference. +Because of that, most tools for inspecting transformer internals live in research harnesses around HuggingFace models, creating a gap between papers and what is practical for inference. ## What is glassbox? Glassbox is a vLLM plugin that instruments the attention path during inference to extract compact, structured features from attention-related operators. It is designed to be a practical implementation of ideas from the attention analysis literature in a form that works inside a real inference engine. From 13824cc6f7dbd09c7878d5d314054bee5a63fdf2 Mon Sep 17 00:00:00 2001 From: Diego Maniloff Date: Wed, 15 Apr 2026 16:29:23 +0000 Subject: [PATCH 6/6] Tighten opening: lead with value prop, remove bullet list Co-Authored-By: Claude Opus 4.6 (1M context) Signed-off-by: Diego Maniloff --- _posts/2026-03-27-glassbox.md | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/_posts/2026-03-27-glassbox.md b/_posts/2026-03-27-glassbox.md index 7f231e1d..b90efea0 100644 --- a/_posts/2026-03-27-glassbox.md +++ b/_posts/2026-03-27-glassbox.md @@ -7,21 +7,13 @@ tags: - safety --- -This post introduces [glassbox](https://github.com/dmaniloff/glassbox), a vLLM plugin for extracting structured signals from transformer attention during inference. - -## Always start with the why +Transformer internals contain a great deal of information about how a model is behaving. Attention patterns, in particular, are a promising source of signals for hallucination detection, failure mode diagnosis, task drift detection, uncertainty estimation, and ongoing monitoring of model behavior. -Transformer internals contain a great deal of information about how a model is behaving. Attention patterns, in particular, can reveal when behavior is becoming concentrated, unstable, or asymmetric — often before those shifts are visible in the output text. - -That makes attention a promising source of signals for: +This post introduces [glassbox](https://github.com/dmaniloff/glassbox), a vLLM plugin for extracting structured signals from transformer attention during inference. -- Hallucination detection -- Failure mode diagnosis -- Task drift detection -- Uncertainty estimation -- Ongoing monitoring of model behavior in production +## Why build this? -But there is a practical problem. Raw activations and full attention matrices are expensive to retain, and modern serving engines like vLLM are specifically optimized to *never materialize* the full L×L attention matrix (FlashAttention, Triton kernels). That's a good thing for performance, but it means the structure we care about is hidden by design. +Raw activations and full attention matrices are expensive to retain, and modern serving engines like vLLM are specifically optimized to *never materialize* the full L×L attention matrix (FlashAttention, Triton kernels). That's a good thing for performance, but it means the structure we care about is hidden by design. Because of that, most tools for inspecting transformer internals live in research harnesses around HuggingFace models, creating a gap between papers and what is practical for inference.