From 9c594e73ddda2baffcdfb9f065701f478633fdcf Mon Sep 17 00:00:00 2001 From: OlalekanAlagbe Date: Sun, 19 Jul 2026 15:54:03 +0100 Subject: [PATCH 1/2] Rewrite abstract, introduction, and add Background section to manuscript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructures the framing around research questions and contributions, adds §2 Background covering prior analogy work, SAE/feature terminology, and attribution graphs, and reworks the causal-validation description back to the 159-experiment/7-prompt framing. --- docs/content.md | 684 ++++++++++++++++++++++++++---------------------- 1 file changed, 370 insertions(+), 314 deletions(-) diff --git a/docs/content.md b/docs/content.md index 8f3807f..ddc9bae 100644 --- a/docs/content.md +++ b/docs/content.md @@ -78,9 +78,7 @@ supplementary: ## Abstract -We present a mechanistic analysis of analogical reasoning in Gemma-2-2B using Neuronpedia attribution graphs and Sparse Autoencoder (SAE) features. By generating and comparing five attribution graphs across structurally distinct analogical prompts — covering geographic analogies (*Paris - France → Berlin - ?*, *Rome - ?*, *Tokyo - ?*) and semantic role analogies (*Doctor - hospital → teacher - ?*, *Fish - water → bird - ?*) — we identify a shared **analogical reasoning circuit** comprising 180 features active across all five prompts and 510 features active across at least three. Each feature is identified by a stable *(layer, feature index)* pair, identifying circuits as lists of recurring internal model feature activation patterns that retain similar structure across analogical prompts. - -We discover dedicated analogy-encoding features at layers 5, 8, 9, and 13, including a feature at layer 5 labeled literally as **"analogies"** and a layer 8 feature encoding **"analogies or comparisons"** appearing across all graphs with high influence. Early layers (0–4) contain circuit templates tracking the "X is to Y as Z is to" pattern, while mid-to-late layers (5–13) house increasingly semantic representations of the relational structure. The circuit spans all 26 transformer layers and exhibits cross-domain generalization, with the same core features activating for both geographic and semantic role analogies. Causal validation via 218 feature-steering experiments shows that ablating the 180-feature circuit — but not a size- and strength-matched random control — collapses the model to the bare analogy template across all five prompts; that this effect is not reducible to deletion of input-token features and reproduces on held-out analogies; and that Phase 2 features collectively implement the relational-transfer operation at the computational core of analogical reasoning. +Analogical reasoning, the ability to recognize a relationship between one pair of concepts and apply it to another, is a hallmark of flexible cognition, and large language models (LLMs) now complete classic analogy tasks with high accuracy. How they do so internally remains poorly understood: behavioral success alone cannot distinguish genuine relational abstraction from surface pattern matching. We present a feature-level, causally validated account of analogical reasoning in Gemma-2-2B. Using Neuronpedia attribution graphs built on the `gemmascope-transcoder-16k` sparse autoencoder (SAE) suite, we analyzed five analogy prompts spanning geographic capitals (*Paris - France → Berlin - ?*) and semantic roles (*Doctor - hospital → teacher - ?*). We identify a shared circuit of 180 SAE features active in all five graphs. Within this circuit, a small set of individually interpretable features sorts into three functional categories: structural features that parse the analogy format (concentrated in layers 0-4), relational features that recognize the relational concept itself (layers 5-9), and integration features that combine the relation with domain knowledge (layers 10-13). A scaling experiment over 50 prompts phrased in four syntactically distinct surface forms contracts the circuit to a stable 67-feature core, ruling out a surface-token explanation. Causal validation through feature steering experiments confirms that the attribution graphs capture real causal structure, and that ten interpretable features drawn from the circuit, spanning all three groups, are causally necessary. Suppressing the features on a prompt's main causal paths destroys the completion for nearly every prompt tested. Suppressing only the four relational features causes every capital analogy to output "France", the answer to the source pair, rather than the correct country: the model stops completing the analogy and reverts to the source-pair answer. No single relational feature is individually necessary, but the four together are, indicating that the relational features collectively carry the transfer operation at the computational core of analogical reasoning. --- @@ -90,278 +88,231 @@ We discover dedicated analogy-encoding features at layers 5, 8, 9, and 13, inclu -Analogical reasoning — the ability to recognize and complete structural relationships between concepts — is a foundational cognitive ability underlying scientific discovery, language understanding, and abstract problem solving. The classic analogy task, *"Paris is to France as Berlin is to \_\_\_\_,"* tests whether a model can identify the capital-city relationship abstractly and apply it to a new country. Large language models (LLMs) exhibit striking competence on such tasks [1], yet the internal computational mechanisms remain poorly understood. +Analogical reasoning is the ability to recognize a relationship between one pair of concepts and apply it to a new pair. It has long been regarded as a core ingredient of flexible cognition: people use analogies to map prior knowledge onto novel domains, and this capacity underlies scientific discovery, language understanding, and abstract problem solving [16, 17]. The classic four-term analogy task, *"Paris is to France as Berlin is to ____,"* tests whether a reasoner can identify the capital-city relationship from the first pair and apply it to a new argument. Relational similarity of this kind has been studied computationally for decades, from early benchmarks of relational similarity between word pairs [13] to analyses of why word embeddings appear to solve analogy problems [14]. -Mechanistic interpretability research has made significant progress in understanding factual recall circuits [2], indirect object identification [3], and syntactic processing [4]. Sparse autoencoders (SAEs) have emerged as a central tool in this effort, learning sparse, interpretable decompositions of model activations [5, 6] that can be applied at scale across all layers and sublayers of large models [7]. The Neuronpedia platform [8] operationalizes this infrastructure, providing public APIs for attribution graph generation and feature steering that democratize circuit-level analysis beyond institutions with direct model access. +Large language models (LLMs) now exhibit striking behavioral competence on such tasks, in some settings matching or exceeding human performance [1]. Behavioral success alone, however, does not reveal mechanism. A model could complete *"Berlin is to ____"* by genuinely inferring the capital-of relation and transferring it to a new pair, or by shallow pattern matching over memorized co-occurrences. The distinction matters. Work at the representation level has shown that some relational concepts in LLMs are encoded by internal vectors that remain stable when the wording of the task changes, while others are not, and that models lacking such invariant representations fail to generalize to new domains [11]. Invariance to surface variation is therefore a natural criterion for genuine relational abstraction, and testing it requires examining the model's internal computation rather than only its outputs. -However, analogical reasoning presents a distinct challenge beyond prior circuit analyses: it requires not merely retrieving a stored fact, but recognizing a **relational structure** and applying it compositionally to novel inputs. The relation type is never named in the prompt — the model must infer *capital-of* from the example alone, hold it as a variable, and transfer it to a new argument pair. Prior work has documented that LLMs exhibit apparently emergent analogical reasoning [1] and identified internal attention-head mechanisms supporting abstract reasoning [9], yet a feature-level, causally-validated circuit account has been absent. +Mechanistic interpretability provides the tools for this examination. The field aims to reverse-engineer neural networks into human-understandable components, and one of its central objects of study is the circuit: a subnetwork of internal components that together implement a specific behavior [18]. Circuit-level analyses have produced detailed accounts of factual recall [2], indirect object identification [3], and automated circuit discovery [4]. Sparse autoencoders (SAEs) have become a central tool in this effort: they decompose model activations into sparse, more interpretable features [5, 6] and can be trained at scale across all layers of a large model [7]. Attribution graphs [12] build on these features to trace how information flows from a prompt to a prediction, and sparse feature circuits [15] show that circuits can be discovered and edited at the feature level. The Neuronpedia platform [8] makes this infrastructure publicly available through a web interface and APIs for attribution graph generation, feature explanation retrieval, and feature steering, without requiring direct model access. -We address this gap using attribution graphs generated from the `gemmascope-transcoder-16k` SAE suite [7], which provides cross-layer transcoder features for every layer of Gemma-2-2B. Our analysis identifies a three-phase circuit with explicitly labeled analogy-concept features, provides causal validation through 218 steering experiments using a matched-null contrast design, and constitutes — to our knowledge — an SAE-level mechanistic account of analogical reasoning in a large language model. +Analogical reasoning is a distinctive target for this toolkit. It requires not merely retrieving a stored fact, but recognizing a relational structure and applying it compositionally to novel inputs. The relation type is never named in the prompt: the model must infer *capital-of* from the example pair alone, hold it as a variable, and transfer it to a new argument pair. Prior work has documented apparently emergent analogical reasoning at the behavioral level [1], identified attention-head mechanisms supporting abstract reasoning [9], probed the internal representations underlying analogy performance [10], and located concept vectors for some verbal relations [11]. A feature-level, causally validated circuit account of analogical reasoning has, however, been absent. - +This paper provides such an account for Gemma-2-2B. We generated attribution graphs from the `gemmascope-transcoder-16k` SAE suite [7], which provides transcoder features for every layer of the model, and analyzed them across analogy prompts from two semantic domains. We study two research questions, listed below. In the course of answering them we made an unplanned observation: the recurring features sort into three functional groups by layer depth. We did not set out to study how the computation is distributed across layers; the structure emerged from the feature labels and we then followed it up, as reported in Section 4.3. ### 1.1 Research Questions -1. Does Gemma-2-2B employ a **shared circuit** for analogical reasoning, or does it use different mechanisms for different analogy types? -2. Which SAE features — identified by stable *(layer, feature index)* pairs — are most **consistently activated** across diverse analogical prompts? -3. Are there interpretable, semantically meaningful features that encode the **abstract relational structure** of analogies, and how are they discovered? -4. How is the analogical computation **distributed across transformer layers**, and can phase boundaries be causally validated? - ---- - -## 2. Methodology +1. Does Gemma-2-2B employ a single shared circuit for analogical reasoning rather than different mechanisms for different analogy types, and if so, which SAE features, identified by stable *(layer, feature index)* pairs, make up that circuit across diverse analogical prompts? +2. Among those features, do interpretable ones exist that encode the abstract relational structure of analogies, and can their causal role be confirmed by steering? - +The contributions of this paper are: -### 2.1 Prompt Selection +1. The identification of a shared analogical reasoning circuit in Gemma-2-2B, comprising 180 SAE features active across all five initial prompts including features whose automated labels explicitly reference analogy. +2. The sorting of a small set of individually interpretable features within the circuit into three functional categories, structural (layers 0-4), relational (layers 5-9), and integration (layers 10-13), grouped by feature labels and confirmed by causal dissociation. This categorization covers the individually inspected features, not the full circuit. +3. A 50-prompt scaling experiment across four syntactically distinct surface forms, showing convergence to a stable 67-feature core and thereby ruling out a surface-token explanation of the circuit. +4. Causal validation through 159 feature steering experiments across 7 prompts, establishing that the late-layer features on each prompt's main causal paths (its backbone) are collectively necessary for prediction, and that ten interpretable features drawn from the circuit, spanning all three groups, are causally necessary, with the relational features collectively implementing the relational transfer operation. -We selected five prompts spanning two structural analogy types to ensure cross-domain coverage: +The remainder of the paper is organized as follows. Section 2 introduces the technical background and defines the terminology used throughout. Section 3 describes the methods: prompt design, attribution graph generation, cross-graph analysis, the scaling experiment, and the steering procedure. Section 4 presents the results. Sections 5, 6, and 7 discuss the findings, their limitations, and our conclusions. -| ID | Prompt | Expected | Type | -|---|---|---|---| -| `analog_berlin` | *"Paris is to France as Berlin is to"* | Germany | Capital | -| `analog_rome` | *"Paris is to France as Rome is to"* | Italy | Capital | -| `analog_tokyo` | *"Paris is to France as Tokyo is to"* | Japan | Capital | -| `analog_teacher` | *"Doctor is to hospital as teacher is to"* | school | Semantic role | -| `analog_bird` | *"Fish is to water as bird is to"* | air / sky | Semantic role | - - - -### 2.2 Attribution Graph Generation +--- -Attribution graphs were generated using the Neuronpedia API [8] (`/api/graph/generate`) with Gemma-2-2B and the `gemmascope-transcoder-16k` SAE [7] — a 26-layer cross-layer transcoder with 16,384 features per layer. Each graph request returns a JSON object containing nodes (SAE feature activations with layer, index, influence score, and activation magnitude) and directed edges (attribution scores). Graphs were downloaded and loaded into NetworkX `DiGraph` objects for analysis. +## 2. Background -| Parameter | Value | -|---|---| -| Model | `gemma-2-2b` | -| SAE | `gemmascope-transcoder-16k` | -| Max feature nodes | 3,000 | -| Desired logit probability | 0.95 | -| Node threshold | 0.80 | -| Edge threshold | 0.85 | +This section defines the terminology used in the rest of the paper and situates the work against prior research. Readers fluent in sparse autoencoders and attribution graphs may skip to Section 3. -**Key technical finding:** The correct API endpoint for `gemmascope-transcoder-16k` requires a **layer-prefixed SAE identifier** (e.g., `4-gemmascope-transcoder-16k` for layer 4) rather than the global SAE name. +### 2.1 Prior Work on Analogical Reasoning in Language Models -### 2.3 Feature Identification and Cross-Graph Analysis +Behavioral studies show that LLMs solve four-term verbal analogies, matrix reasoning problems, and story analogies at levels comparable to human participants [1]. Two lines of work look beneath this behavior. Webb et al. [9] used causal mediation analysis to identify an emergent symbolic architecture in which early attention heads abstract input tokens into relational variables, intermediate heads perform induction over those variables, and later heads retrieve the answer. Opiełka et al. [11] extracted concept vectors from attention head outputs and found that verbal relational concepts such as *antonym* are encoded invariantly across changes in task wording while more abstract concepts are not, and that this lack of invariant representation predicts generalization failures; Lee et al. [10] similarly linked analogy errors to missing relational information in intermediate layers. Together these results frame the question of genuine analogical reasoning as a question about a model's internal representations, and specifically about whether those representations survive changes in surface wording. Both prior mechanistic accounts operate at the level of attention heads or distilled activation vectors. The present work asks the same question at the level of individual SAE features and their causal roles. -#### 2.3.1 Feature Identity via Stable (Layer, Feature Index) Pairs +### 2.2 Model Internals, Sparse Autoencoders, and Features -Each feature in the attribution graphs is identified by a stable *(layer, feature index)* pair — for example, *(5, 5793)* uniquely and persistently identifies a feature within the `gemmascope-transcoder-16k` SAE [7]. These identifiers are fixed properties of the trained SAE and do not vary across prompts, sessions, or API calls. +Gemma-2-2B is a decoder-only transformer with an embedding layer (E) followed by 26 layers, numbered L0 through L25. Each layer reads from and writes to the residual stream, the running vector representation of each token that accumulates every component's contribution; the numerical values a component produces on a given input are its activations; and the final residual stream is projected to logits, the scores from which the next token is predicted. Individual neurons are polysemantic, responding to many unrelated concepts at once, which makes neuron-level analysis hard to interpret [6]. A sparse autoencoder (SAE) is an auxiliary network trained to re-express activations as a sparse combination of a much larger set of learned directions, only a few of which are active on any input [5, 6]. Each learned direction is an SAE feature; its direction is the vector along which it reads and writes, and its activation magnitude is how strongly it fires (zero meaning inactive). Because the SAE is trained once and then fixed, every feature is permanently identified by a *(layer, feature index)* pair: *(5, 5793)* denotes the same direction across every prompt and session. This work uses the `gemmascope-transcoder-16k` suite [7], which provides 16,384 features per layer for all 26 layers; transcoders are a variant of SAE trained to reconstruct a component's output from its input, which suits them to tracing computation between layers. We write features as, for example, L5 #5793. Each feature also carries a feature label, an automated natural-language description of what it activates on, generated by Neuronpedia [8] from the feature's strongest activating examples; labels are informative but not ground truth (Section 6). -Cross-graph feature overlap was computed by finding which *(layer, feature index)* pairs appear as nodes across multiple independently generated graphs. Formally, let $G_i$ denote the set of feature IDs active in graph $i$. The shared circuit at threshold $k$ is: +### 2.3 Attribution Graphs, Influence Scores, and Circuits -$$\mathcal{C}_k = \left\{ f \;\middle|\; \sum_{i=1}^{5} \mathbf{1}[f \in G_i] \geq k \right\}$$ +An attribution graph [12] is a directed graph that traces how the model computes its prediction for a single prompt. Its nodes are the SAE features active on that prompt, plus embedding nodes for the input tokens and a logit node for the predicted token; its edges carry attribution scores estimating how much each source node contributed to each target node. Each node also has an influence score, summarizing how much it contributes, directly and indirectly, to the output. A circuit is a subnetwork, here a set of SAE features and the attribution edges between them, that together implement a specific behavior [3, 4, 15]; circuit discovery isolates the small part of the network responsible for a behavior from the much larger incidental part. We study circuits at the SAE feature level rather than the attention head level of earlier work [3, 9]. Throughout, we reserve the term circuit for the single shared analogical reasoning circuit that is the object of this study; the per-prompt causal structures used in the steering experiments are called backbones (Section 3.7), and the unit each experiment runs on is a prompt. -Three thresholds were analyzed: $k \in \{3, 4, 5\}$. The 180-feature core circuit ($k=5$) is therefore a concrete, enumerable list of *(layer, feature index)* identifiers that recur across all five independently generated graphs regardless of whether the prompt is geographic or semantic in nature. Feature labels were retrieved using the Neuronpedia feature explanation API [8]. +### 2.4 Feature Steering and Causal Validation -### 2.4 Three-Phase Architecture: How Phase Boundaries Were Identified +Attribution graphs are correlational: a feature can appear in a graph without being causally responsible for the prediction. Feature steering provides the causal test. Steering adds a scaled copy of a feature's direction to the activations during a forward pass: boosting (positive strength) raises its activation, suppressing (negative strength) drives it down or below zero. Steering features and observing whether the prediction changes lets us test three properties. A feature or feature set is causally necessary if suppressing it changes the prediction, and this necessity can hold individually (one feature breaks the behavior) or only collectively (the behavior breaks only when several are suppressed together). A feature is causally sufficient if boosting it induces the behavior where it would not otherwise occur. A protocol has specificity if suppressing features outside the hypothesized circuit leaves the prediction unchanged, confirming that steering effects are not an artifact of perturbing the model arbitrarily. -The three-phase architecture was identified through two converging lines of evidence, neither of which required the authors to impose phase boundaries a priori. - -**Semantic label analysis.** After retrieving Neuronpedia automated labels for the top recurring features, a consistent gradient emerged across layer depth. Features in layers 0–4 carry purely syntactic labels: "the word 'to'", "'to' followed by a verb", "the phrase 'it is to'". Features in layers 5–9 carry explicitly relational-semantic labels: "analogies", "analogies or comparisons", "comparison between two things". Features in layers 10–13 carry integrative labels: "comparisons between disciplines and relationships between concepts". The phase boundaries emerge from the content of the labels rather than an arbitrary partition of layers. This three-stage organization mirrors the emergent symbolic architecture documented by Webb et al. [9] for abstract reasoning more broadly, where early layers abstract tokens into relational variables, intermediate layers perform induction over those variables, and later layers retrieve the answer. +--- -**Activation magnitude progression.** Average activation magnitudes of core features increase monotonically through the phases: +## 3. Methods -| Phase | Layer Range | Role | Activation Magnitude | -|---|---|---|---| -| Circuit template | L0 | Token and syntax parsing | 1.5 – 6.4 | -| Analogy recognition hub | L5 | Analogy concept activation | 7.4 – 11.1 | -| Comparison detectors | L8–L9 | Relational detection | ~13.4 | -| Relational integration | L10–L13 | Domain + relation integration | 9.1 – 16.3 | +This section describes everything we did, in the order the analyses were performed: prompt design (Section 3.2), attribution graph generation (Section 3.3), cross-graph overlap analysis (Section 3.4), the feature group identification procedure (Section 3.5), the scaling experiment (Section 3.6), backbone extraction (Section 3.7), and the feature steering protocol (Section 3.8). All results appear in Section 4. -**Causal validation.** Phase boundaries were then tested causally via collective suppression experiments (§3.4), which confirmed that each phase is collectively necessary and that earlier phases are prerequisites for later ones. +### 3.1 Model and SAE Suite -### 2.5 Discovery of Analogy-Concept Features +All experiments used Gemma-2-2B (`gemma-2-2b`) with the `gemmascope-transcoder-16k` SAE suite [7], a 26-layer cross-layer transcoder with 16,384 features per layer (Section 2.2). All model access was through the Neuronpedia APIs [8]. Neuronpedia is a public platform for SAE research that hosts trained SAE suites, the automated feature labels of Section 2.2, and APIs for generating attribution graphs and performing feature steering on hosted models; we used it without any direct access to the model weights. -The key features — L5 SAE#5793 ("analogies") and L8 SAE#13766 ("analogies or comparisons") — were not specifically sought. They emerged from the cross-graph overlap analysis described in §2.3. Once the intersection feature set was computed, each feature's automated Neuronpedia explanation [8] was retrieved. L5 SAE#5793 returned the label "analogies"; L8 SAE#13766 returned "analogies or comparisons." +### 3.2 Prompt Design -The significance of these labels is their **domain-agnosticism**. Both features appear in attribution graphs for Berlin, Rome, and Tokyo (geographic capital analogies) and for teacher and bird (semantic role analogies). This is consistent with the broader finding in the analogical reasoning literature that LLMs encode relational information in a domain-general manner [10, 11], and extends that behavioral finding to a specific, causally-validated internal feature. L8 SAE#13766 was additionally notable for having 21 appearances across the five graphs and an influence score of 0.533, placing it among the highest-influence recurring features. +We selected five prompts spanning two structural analogy types to ensure cross-domain coverage: three geographic capital analogies and two semantic role analogies. Table 1 lists the full prompt set. Each prompt has an internal ID string that serves as the slug name in Neuronpedia API calls and as a row label throughout the result tables. -### 2.6 Phase 2 Definition +**Table 1:** The five analogy prompts used in the initial analysis. Each row is one prompt; the columns give the internal ID (used as the Neuronpedia graph slug and as a row label in later tables), the analogy type, the full prompt text, and the completion the model is expected to produce. -Phase 2 is defined by two jointly applied criteria: **layer position (5–9)** and **feature label content**. Features in this layer range whose Neuronpedia labels explicitly reference analogies, comparisons, or relational structure constitute Phase 2. The four members are: +| ID | Type | Prompt | Expected | +| --- | --- | --- | --- | +| `analog_berlin` | Capital | "Paris is to France as Berlin is to" | Germany | +| `analog_rome` | Capital | "Paris is to France as Rome is to" | Italy | +| `analog_tokyo` | Capital | "Paris is to France as Tokyo is to" | Japan | +| `analog_teacher` | Semantic role | "Doctor is to hospital as teacher is to" | school | +| `analog_bird` | Semantic role | "Fish is to water as bird is to" | air / sky | -| Feature | Label | -|---|---| -| L5 SAE#5793 | "analogies" | -| L5 SAE#2141 | "comparisons of people or figures using well-known public figures" | -| L8 SAE#13766 | "analogies or comparisons" | -| L9 SAE#13344 | "phrases suggesting uncertainty or comparison between two things" | +### 3.3 Attribution Graph Generation -This grouping is validated causally: suppressing all four Phase 2 features simultaneously collapses all five circuits, with capital analogies producing "France" — the source-pair answer — rather than the target country. An arbitrary phase definition would not produce such a consistent and semantically meaningful failure mode. +Attribution graphs were generated using the Neuronpedia API [8] (`/api/graph/generate`) with Gemma-2-2B and the `gemmascope-transcoder-16k` SAE [7]. Each graph request returns a JSON object containing nodes (SAE feature activations with layer, feature index, influence score, and activation magnitude) and directed edges (attribution scores). Graphs were generated with Neuronpedia's default generation parameters. A technical finding during implementation was that the correct API endpoint for `gemmascope-transcoder-16k` requires a layer-prefixed SAE identifier (e.g., `4-gemmascope-transcoder-16k` for layer 4) rather than the global SAE name; initial calls using the global name returned HTTP 404 errors. -### 2.7 Causal Validation via Feature Steering +### 3.4 Cross-Graph Feature Overlap Analysis -The cross-graph analysis (§2.3) identifies the 180-feature shared circuit as a *correlational* object: these features recur across all five analogy graphs. Recurrence does not establish that the circuit *causes* the model's answers — a recurring feature could be a passenger that co-activates with the computation without driving it. We test causation by intervening on the circuit with the Neuronpedia `/api/steer` endpoint [8] (`modelId: "gemma-2-2b"`, `strength_multiplier: 4`, `temperature: 0`, `seed: 42`). +Because every feature is identified by a stable *(layer, feature index)* pair (Section 2.2), features can be matched exactly across independently generated graphs. Cross-graph feature overlap was computed by finding which identifier pairs appear as nodes in multiple graphs. Formally, let $G_i$ denote the set of feature IDs active in graph $i$. The shared circuit at threshold $k$ is: -One methodological fact governs the whole validation. The per-feature `strength` we set (−20 for ablation) is scaled by the global `strength_multiplier` of 4, so the effective intervention is **−80** per feature — a large perturbation. At that magnitude, removing *any* sizable feature set derails the model, so the bare observation "the output changed" (necessity) is necessary but **not sufficient** evidence that a particular circuit is responsible. The discriminating evidence is of two kinds: (i) a **matched contrast** — does ablating the circuit break the task differently from ablating a size- and strength-matched *random* set? — and (ii) a **predicted failure mode** — does the model fail in the specific way the circuit's claimed role implies? Our design is built on these rather than on necessity alone: +$$\mathcal{C}_k = \left\{ f \;\middle|\; \sum_{i=1}^{5} \mathbf{1}[f \in G_i] \geq k \right\}$$ -1. **Circuit necessity vs. a matched null (primary).** For each of the five analogy prompts, run two ablations of identical size (180 features) and strength (−20): (a) all 180 circuit features, and (b) 180 random *non*-circuit features drawn from the same prompt's graph. The evidence is the contrast between them. -2. **Internal structure (individual necessity).** Ablate each of the 180 features one at a time to locate the individually necessary members. -3. **Robustness.** Layer-stratified ablation (is the effect just deletion of L0 token features?), a strength titration (is it an artifact of the large −80 magnitude?), and held-out analogy prompts the circuit was never derived from (does it generalize?). -4. **Architecture.** Collective suppression of the Phase-1 and Phase-2 feature groups (§2.4) to test the three-phase organization. -5. **Single-feature side-tests.** A sufficiency probe (boost the highest-influence hub) and an individual non-circuit specificity scan, reported as supplementary single-feature controls. +Three thresholds were analyzed: $k \in \{3, 4, 5\}$. We refer to the features in the strictest set, those appearing in every attribution graph of a given prompt set, as core features. The set $\mathcal{C}_5$ over the five initial prompts is therefore a concrete, enumerable list of *(layer, feature index)* identifiers that recur across all five independently generated graphs regardless of whether the prompt is geographic or semantic in nature. After the intersection sets were computed, the automated label of each recurring feature was retrieved using the Neuronpedia feature explanation API [8]. -Across these paradigms the validation comprises **218 individual steering API calls**. +### 3.5 Feature Group Identification Procedure ---- +To characterize how the shared circuit is organized in function, we used one descriptive procedure that did not require imposing feature groupings a priori: we retrieved the Neuronpedia labels for the top recurring features and examined whether the label content clusters into functional categories, and whether those categories are associated with layer depth. This procedure suggests a candidate grouping; whether the groups are genuinely functionally distinct is then tested causally, by suppressing each group and comparing the resulting failure modes (Section 3.8). The outcomes of both lines of evidence, the descriptive label clustering and the causal dissociation, and the three-group organization they converge on, are reported in Sections 4.3 and 4.7. -## 3. Results +### 3.6 Scaling Experiment Procedure - +To test whether the shared circuit is an artifact of using only five similarly phrased prompts, we ran a scaling experiment: attribution graphs were generated for 50 analogy prompts in total, and the core feature set was recomputed as prompts accumulated. A prompt's surface form is the particular wording used to express its relational content. From the second batch onward, new prompts were not merely additional examples of the original template; they were distributed across three syntactically distinct surface forms alongside the original, as shown in Table 2. -### 3.1 Graph Structure +**Table 2:** The four surface forms used in the 50-prompt scaling experiment. Each row is one syntactic template; the example column shows a representative prompt. The four forms share no common connective string, so any feature that survives across all of them cannot be tracking surface tokens. -All five attribution graphs exhibited a consistent structural pattern, with features activated across all 26 transformer layers (0–25) plus the embedding layer (E): +| Surface form | Example | +| --- | --- | +| Standard X-to-Y | `Paris is to France as Berlin is to` | +| Diverse-A (Just as…) | `Just as Paris is the capital of France, Berlin is the capital of` | +| Diverse-B (Found in…) | `Doctors are found in hospitals. Teachers are found in` | +| Diverse-C (The way…) | `The way a fish lives in water, a bird lives in` | -| Graph | Prompt | Nodes | Edges | Max Influence | -|---|---|---|---|---| -| `analog_berlin` | Paris - France → Berlin - ? | 930 | 25,915 | 0.8001 | -| `analog_rome` | Paris - France → Rome - ? | 963 | 27,608 | 0.8002 | -| `analog_tokyo` | Paris - France → Tokyo - ? | 905 | 22,414 | 0.8001 | -| `analog_teacher` | Doctor - hospital → teacher - ? | 1,040 | 35,481 | 0.8001 | -| `analog_bird` | Fish - water → bird - ? | 1,071 | 38,741 | 0.8000 | +At each milestone ($N$ = 5, 10, 20, 30, 40, 50), the strictest possible threshold was applied: a feature was counted only if it appeared in every single attribution graph generated up to that point (i.e., $k = N$). The resulting counts are reported in Section 4.6. -The semantic role analogies (*teacher*, *bird*) have notably larger graphs (1,040–1,071 nodes, 35k–38k edges) compared to the capital analogies (905–963 nodes, 22k–27k edges). We interpret this as reflecting greater ambiguity in the expected completion domain: the *capital-of* relation maps to a discrete, well-encoded fact [2], whereas professional and ecological roles require broader world-knowledge access. +### 3.7 Backbone Extraction - +For the causal experiments, the causal structure of each prompt was operationalized through what we call its backbone: the set of features that sit on the main causal paths through that prompt's attribution graph. We refer to these as backbone features. The extraction procedure is identical for all prompts: trace the top-5 causal paths backward from the logit node (a greedy walk following the highest-weight incoming edges) and the top-5 paths forward from the highest-influence embedding nodes. Any transcoder feature appearing on at least one of these 10 paths is a backbone member. This procedure is analogous to the automated circuit discovery approach of Conmy et al. [4], applied here at the SAE feature level rather than the attention head level. Because the paths are traced toward the output token, backbone features naturally cluster in the final layers of the model (primarily L16 to L25); they are the last-mile computation before a word is written, and they are distinct from the cross-graph core features of Section 3.4. Each prompt has its own backbone: the Berlin backbone and the Rome backbone are largely different feature sets, because the features needed to output "Germany" differ from those needed to output "Italy." -### 3.2 The Core Analogical Reasoning Circuit +### 3.8 Feature Steering Procedure - +Causal validation was performed using the Neuronpedia `/api/steer` endpoint [8] with `modelId: "gemma-2-2b"` and `strength_multiplier: 4`. Four experimental paradigms were applied: (1) necessity by individual suppression, in which a single backbone feature is suppressed at strength −20; (2) necessity by full backbone suppression, in which all backbone features of a prompt are suppressed simultaneously; (3) sufficiency via hub boost, in which a backbone hub feature is boosted at strength +20 while the model processes an altered prompt; and (4) specificity testing via non-backbone suppression, in which high-activation features that lie on no traced causal path are suppressed. The same paradigms of individual and collective suppression were then applied to the cross-graph feature groups (Sections 4.7.3 and 4.7.4). Two additional prompts from the expanded 30-prompt analysis (Cairo→Kenya and Puppy→cat) were included for cross-validation, giving 7 prompts in total for the causal experiments alongside the 5 of Table 1. Across all steering paradigms this amounts to 159 individual steering API calls. -Cross-graph feature overlap analysis over the stable *(layer, feature index)* identifier space revealed a substantial shared circuit: +--- -| Threshold | Features Found | -|---|---| -| Active in ≥3/5 graphs | **510 features** | -| Active in ≥4/5 graphs | **277 features** | -| Active in all 5 graphs | **180 features** | +## 4. Results -Core features by layer group (5/5 graphs): +### 4.1 Graph Structure -| Layer Group | Core Features | -|---|---| -| L0 | 12 | -| L1–L4 | 19 | -| L5–L6 | 12 | -| L8–L13 | 7 | +All five attribution graphs exhibited a consistent structural pattern, with features activated across all 26 transformer layers (0-25) plus the embedding layer (E). Table 3 reports the size of each graph. -early layers (L0–L3) account for the plurality of core features, consistent with circuit template processing occurring first. The mid-range layers (L5–L6) show elevated feature counts relative to neighbors — these are the **analogy recognition hub** layers. Isolated high-influence features appear at L8, L9, L11, and L13. +**Table 3:** Size of the five initial attribution graphs. Each row is one prompt; Nodes and Edges count the feature, embedding, and logit nodes and the directed attribution edges in that prompt's graph, and Max influence is the highest influence score among its nodes. All five graphs were generated with the same settings (Section 3.3). - +| Prompt | Type | Nodes | Edges | Max influence | +| --- | --- | --- | --- | --- | +| `analog_berlin` | Capital | 930 | 25,915 | 0.8001 | +| `analog_rome` | Capital | 963 | 27,608 | 0.8002 | +| `analog_tokyo` | Capital | 905 | 22,414 | 0.8001 | +| `analog_teacher` | Semantic role | 1,040 | 35,481 | 0.8001 | +| `analog_bird` | Semantic role | 1,071 | 38,741 | 0.8000 | -### 3.3 The Three-Phase Analogical Reasoning Circuit +As Table 3 shows, the semantic role analogies (*teacher*, *bird*) have notably larger graphs (1,040-1,071 nodes, 35k-38k edges) than the capital analogies (905-963 nodes, 22k-27k edges). We interpret this as reflecting greater ambiguity in the expected completion domain: the *capital-of* relation maps to a discrete, well-encoded fact [2], whereas professional and ecological roles require broader world-knowledge access. - +### 4.2 The 180-Feature Shared Circuit -We provide evidence that Gemma-2-2B performs **genuine multi-step analogical reasoning internally**. The attribution graph reveals a three-phase computational process that activates for both geographic and semantic role analogies — evidence of a domain-agnostic relational reasoning mechanism. This three-stage organization parallels the symbolic architecture identified by Webb et al. [9] through causal mediation analysis and the internal representation findings of Lee et al. [10]. +Cross-graph feature overlap analysis over the stable *(layer, feature index)* identifier space (Section 3.4) revealed a substantial shared circuit. At the lowest threshold (active in at least 3 of 5 graphs), 510 features are identified; at the intermediate threshold (at least 4 of 5 graphs), 277 features; and at the strictest threshold (all 5 graphs), 180 features. This 180-feature set of core features is the focus of our analysis. ---- +To identify the most meaningful contributions, we ranked all 180 core features by influence score. The highest-influence features with the greatest semantic specificity concentrate in the middle-to-late layers and explicitly encode analogical and relational concepts, carrying Neuronpedia labels such as "analogies or comparisons" and "comparisons between disciplines and relationships between concepts." Section 4.3 characterizes these features and the three functional groups they fall into; Section 4.4 reports their recurrence and influence statistics in detail. -**Phase 1 · layers 0–4 · Circuit Template Parsing** +### 4.3 Three Functionally Distinct Feature Groups -| Feature | Label | -|---|---| -| L0 SAE#11651 | *"the word 'to'"* | -| L1 SAE#11356 | *"the word 'to' followed by a verb"* | -| L2 SAE#11475 | *"the word 'refers' and related words"* | -| L4 SAE#10752 | *"uses of the verb 'to be' preceded by 'to'"* | -| L5 SAE#9672 | *"the phrase 'it is to'"* | +The core features do not all do the same job. While retrieving the Neuronpedia labels for the recurring features (Section 3.4), we noticed that the labels sorted by layer depth, and we followed that observation up with the procedure of Section 3.5. This descriptive evidence suggests that a subset of the shared circuit's individually interpretable features sorts into three functional feature groups; the causal experiments of Section 4.7 then test the grouping by showing that suppressing each group produces a qualitatively different failure mode. This categorization applies to the features we inspected individually (Section 4.4), not to every feature in the circuit. We refer to the three groups as the structural features, the relational features, and the integration features, summarized in Table 4. Each group concentrates in a different band of layers, roughly L0-L4, L5-L9, and L10-L13, but the grouping is defined by function, not by layer boundaries. -These features encode the syntactic skeleton of the analogy prompt. Their progression from individual tokens to multi-word patterns reflects hierarchical parsing of the relational connective. These are *structural* features — they fire on any text with this grammatical form, not specifically on analogical content. +**The descriptive evidence: feature labels cluster by layer depth.** After the Neuronpedia labels for the top core features were retrieved, a consistent gradient emerged across layer depth. The gradient was revealed by the labels themselves rather than imposed by the authors: ---- +- Layers 0-5 carry structural labels ("the word 'to'", "'to' followed by a verb", "the phrase 'it is to'"). These features are not merely grammatical; they parse the skeleton of the analogy prompt itself. For a prompt like *"Paris is to France as Rome is to"*, these features track the relational connectives that signal a comparison is being drawn and that a completion is expected. At this stage the model registers the shape of the problem before it registers what the problem is about. -**Phase 2 · layers 5–9 · Analogy Recognition Hub** +- Layers 5-9 carry explicitly relational-semantic labels ("analogies", "analogies or comparisons", "comparison between two things"). Here the model transitions from recognizing the format to recognizing the concept: it now represents that the task is an analogy, regardless of whether the content is geographic (Berlin, Rome, Tokyo) or semantic (teacher, bird). The clearest case is L5 #5793, labeled "analogies" by Neuronpedia's automated explanation system [8]: it is not a geographic feature, and fires equally for "Paris - France → Berlin - ?" and "Doctor - hospital → teacher - ?". This is direct evidence of the kind of abstract relational representation that prior behavioral work [1, 11] hypothesized but did not directly observe inside a model. -| Feature | Label | -|---|---| -| L5 SAE#5793 | *"analogies"* ← dedicated analogy concept feature | -| L5 SAE#2141 | *"comparisons of people or figures using well-known public figures"* | -| L8 SAE#13766 | *"analogies or comparisons"* (21 activations across 5 graphs, influence 0.533) | -| L9 SAE#13344 | *"phrases suggesting uncertainty or comparison between two things"* | +- Layers 10-13 carry integrative labels ("comparisons between disciplines and relationships between concepts"). Here the model shifts from representing the relational structure in the abstract to representing it in domain-specific terms: for a capital analogy it begins operating within the space of countries and capitals, and for a semantic role analogy within the space of professions and their environments. The model is not yet predicting the answer, but it is narrowing down what kind of thing the answer will be. Layers 14-25 then handle domain-specific knowledge retrieval and output token formatting, analogous to the factual recall circuits identified by Meng et al. [2]. -This is where circuit template processing gives way to semantic recognition of the *relational concept itself*. The presence of L5 SAE#5793, labeled "analogies" by Neuronpedia's automated SAE feature explanation system [8], is particularly significant: it activates consistently for both capital-city and semantic role analogies. It is not a geographic feature — it fires equally for "Doctor - hospital → teacher - ?". This is direct evidence of the kind of abstract relational representation that prior behavioral work [1, 11] has hypothesized but not directly observed inside a model. +The grouping therefore emerges from the content of the labels rather than from an arbitrary partition of layers, and the associated layer ranges are an observation about where each group happens to concentrate. The interactive panel below shows one representative feature per group, with the activation patterns that characterize each group: ---- + -**Phase 3 · layers 10–13 · Relational Integration** +**Table 4:** The three functional feature groups of the shared circuit. Each row is one group; the columns give the layer range where the group concentrates and the functional role of the group. Group membership is defined by feature-label content (Section 4.3), not by layer boundaries. -| Feature | Label | -|---|---| -| L11 SAE#15947 | *"references to historical or social change"* | -| L13 SAE#10969 | *"comparisons between disciplines and relationships between concepts"* | +| Group | Layer range | Role | +| --- | --- | --- | +| Structural | L0-L4 | Parsing the shape and format of the analogy | +| Relational | L5-L9 | Recognizing the relational concept itself | +| Integration | L10-L13 | Combining relation with domain knowledge | -L13 SAE#10969 serves an integrative role, combining the recognized relational structure from Phase 2 with domain-specific knowledge to produce the final completion. layers 14–25 then handle domain-specific knowledge retrieval and output token formatting, analogous to the factual recall circuits identified by Meng et al. [2]. +This three-part organization mirrors the emergent symbolic architecture documented by Webb et al. [9] for abstract reasoning more broadly, where early layers abstract tokens into relational variables, intermediate layers perform induction over those variables, and later layers retrieve the answer. It also parallels the internal representation findings of Lee et al. [10]. Table 5 lists the canonical features of each group. ---- +**Table 5:** Canonical features of the three functional groups. Each row is one individually inspected SAE feature, identified by layer (L) and feature index (#), with its group and its Neuronpedia label. These are the individually inspected members of each group, not the full circuit. -> **Note:** This diagram simplifies the true mechanisms considerably. The attribution graph for any single prompt contains hundreds of features; the circuit shown represents the semantically interpretable core. +| Feature | Group | Neuronpedia label | +| --- | --- | --- | +| L0 #11651 | Structural | "the word 'to'" | +| L1 #11356 | Structural | "the word 'to' followed by a verb" | +| L2 #11475 | Structural | "the word 'refers' and related words" | +| L4 #10752 | Structural | "uses of the verb 'to be' preceded by 'to'" | +| L5 #9672 | Structural | "the phrase 'it is to'" | +| L5 #5793 | Relational | "analogies" | +| L5 #2141 | Relational | "comparisons of people or figures using well-known public figures" | +| L8 #13766 | Relational | "analogies or comparisons" | +| L9 #13344 | Relational | "phrases suggesting uncertainty or comparison between two things" | +| L11 #15947 | Integration | "references to historical or social change" | +| L13 #10969 | Integration | "comparisons between disciplines and relationships between concepts" | - +The three interactive panels below show representative features from the structural, relational, and integration groups respectively, with the activation patterns that characterize each. -### 3.4 Top Recurring Features + -**Directly analogical features** (Neuronpedia labels explicitly reference analogical reasoning or comparison): + -| Feature | Appearances | Avg Influence | Label | -|---|---|---|---| -| L5 #5793 | 11/5 | 0.590 | "analogies" | -| L8 #13766 | 21/5 | 0.533 | "analogies or comparisons" | -| L9 #13344 | 14/5 | 0.681 | "comparison between two things" | -| L5 #2141 | 12/5 | 0.647 | "comparisons of public figures" | -| L13 #10969 | 11/5 | 0.676 | "comparisons between disciplines" | + -**Circuit templates** (encode the "X is to Y as Z is to" scaffold): +> **Note:** This description simplifies the true mechanisms considerably. The attribution graph for any single prompt contains hundreds of features; the interpretable core described here is only a subset. -| Feature | Appearances | Avg Influence | Label | -|---|---|---|---| -| L0 #11651 | 10/5 | 0.633 | "the word 'to'" | -| L1 #11356 | 10/5 | 0.609 | "'to' followed by a verb" | -| L2 #11475 | 10/5 | 0.638 | "the word 'refers'" | -| L4 #10752 | 10/5 | 0.626 | "'to be' preceded by 'to'" | -| L5 #9672 | 12/5 | 0.579 | "the phrase 'it is to'" | +### 4.4 The Model Recognizes the Task as an Analogy -**High-recurrence formal text features** (labels unrelated to analogical reasoning): +The most striking finding of the cross-graph analysis was not planned; it emerged from the data. The key features, L5 #5793 ("analogies"), L8 #13766 ("analogies or comparisons"), and L9 #13344 ("phrases suggesting uncertainty or comparison between two things"), were not specifically sought. They emerged from the cross-graph overlap analysis of Section 3.4, when the automated Neuronpedia explanation [8] of each feature in the intersection set was retrieved. -| Feature | Appearances | Avg Influence | Label | -|---|---|---|---| -| L4 #14857 | 22/5 | 0.681 | "code snippets and license agreements" | -| L6 #2267 | 20/5 | 0.724 | "words in programming code, legal jargon, or scientific texts" | -| L3 #3205 | 20/5 | 0.670 | "code snippets and documentation references" | +Several of these features carry labels that explicitly name analogy. Table 6 lists them. Their significance is what they show about the model's internal state: while processing the prompt, the model activates features whose learned meaning is "this is an analogy," before it produces any answer. The model is not merely completing a pattern; it represents that the task in front of it is an analogy. -These formal-text features have higher raw appearance counts than the explicitly analogical features. Causal steering (§3.7.6) confirms they are inert for all high-confidence circuits, consistent with their role as detectors of syntactic formality rather than relational semantics. The polysemanticity of neurons in large models [6] is precisely why SAE-based feature decomposition [5, 6, 7] is necessary to distinguish these classes of activation. +**Table 6:** The analogy-concept features in the shared circuit. Each row is one feature, identified by layer (L) and feature index (#), together with its Neuronpedia label. All five carry labels that explicitly reference analogical or comparative reasoning. - +| Feature | Neuronpedia label | +| --- | --- | +| L5 #5793 | "analogies" | +| L8 #13766 | "analogies or comparisons" | +| L9 #13344 | "comparison between two things" | +| L5 #2141 | "comparisons of public figures" | +| L13 #10969 | "comparisons between disciplines" | -### 3.5 Cross-Domain Generalization +Figures 1 and 2 show two of these features, L9 #13344 and L8 #13766, active inside the attribution graph interface for the *teacher* prompt, one representative of the five. The complete set across all five prompts is provided in Appendix A. - +![L9 #13344 in the attribution graph for the teacher prompt](UI-GRAPHS/13344_doctor-is-to-hospital-as-teacher-is-to.png) -The consistent activation of L5 SAE#5793 ("analogies") and L8 SAE#13766 ("analogies or comparisons") across both capital-city and semantic role analogy types provides the most direct evidence for a **domain-general analogical reasoning mechanism**. The 180 features active in all five graphs form the stable intersection of the two analogy type families, and this intersection includes the core analogy-concept features at L5 and L8. +*Figure 1: Feature L9 #13344 ("phrases suggesting uncertainty or comparison between two things") shown active inside the Neuronpedia attribution graph interface for the `analog_teacher` prompt ("Doctor is to hospital as teacher is to"). Appendix A (Figure A1) shows the same feature for all five prompts.* -The slightly larger graphs for semantic role analogies (teacher, bird: 1,040–1,071 nodes) relative to capital analogies (Berlin, Rome, Tokyo: 905–963 nodes) may reflect that semantic role completions require broader world-knowledge access — knowing that teachers work in schools, or that birds inhabit air — rather than purely relational computation over a discrete, well-encoded geographic fact [2]. +![L8 #13766 in the attribution graph for the teacher prompt](UI-GRAPHS/13766_doctor-is-to-hospital-as-teacher-is-to.png) -### 3.6 Circuit Stability Across Scaled and Syntactically Diverse Prompts +*Figure 2: Feature L8 #13766 ("analogies or comparisons") shown active inside the Neuronpedia attribution graph interface for the `analog_teacher` prompt. Appendix A (Figure A2) shows the same feature for all five prompts.* -To validate that the shared circuit identified in §3.1 is not an artifact of using only five similar prompts, an extensive scaling experiment was performed. The central question: if we keep adding new analogical prompts — including versions phrased very differently from the original format — do the same features keep showing up? +### 4.5 Cross-Domain Generalization -The experiment generated attribution graphs for 50 prompts in total. Crucially, from the second batch onward, the prompts were not just new examples of the same template — they were rephrased into three syntactically distinct surface forms alongside the original: +The consistent activation of L5 #5793 ("analogies") and L8 #13766 ("analogies or comparisons") across both capital-city and semantic role analogy types provides the most direct evidence for a domain-general analogical reasoning mechanism. The 180 core features form the stable intersection of the two analogy type families, and this intersection includes the analogy-concept features at L5 and L8. The slightly larger graphs for semantic role analogies (teacher, bird: 1,040-1,071 nodes) relative to capital analogies (Berlin, Rome, Tokyo: 905-963 nodes) may reflect that semantic role completions require broader world-knowledge access, such as knowing that teachers work in schools or that birds inhabit air, rather than purely relational computation over a discrete, well-encoded geographic fact [2]. -| Surface Form | Example | -|---|---| -| Standard X-to-Y | `Paris is to France as Berlin is to` | -| Diverse-A (Just as…) | `Just as Paris is the capital of France, Berlin is the capital of` | -| Diverse-B (Found in…) | `Doctors are found in hospitals. Teachers are found in` | -| Diverse-C (The way…) | `The way a fish lives in water, a bird lives in` | +### 4.6 Circuit Stability Across Scaled and Syntactically Diverse Prompts -These four forms look very different on the surface — different word order, different connectives, no shared "is to … as" string. If the circuit from §3.1 were merely tracking surface tokens, it would fall apart when these diverse forms were introduced. +The shared circuit of Section 4.2 was identified from five prompts that all share the "is to … as" template. The scaling experiment (Section 3.6) tests whether the circuit survives when analogical prompts are added that do not share this template: if the circuit were merely tracking surface tokens, it would fall apart once the syntactically diverse forms of Table 2 were introduced. Table 7 reports the number of core features at each milestone. -At each milestone (N = 5, 10, 20, 30, 40, 50), the strictest possible threshold was applied: a feature must appear in **every single** attribution graph at that point. +**Table 7:** Core feature counts as the prompt set grows. Each row is one milestone of the scaling experiment; N is the number of attribution graphs analyzed up to that point, the second column counts the features present in every one of the N graphs (threshold k = N), and the third column gives the change from the previous milestone. From N = 10 onward the prompt set mixes all four surface forms of Table 2. -| N | Recurring features (k = ALL) | Drop from previous | -|---|---|---| +| N | Core features (k = N) | Drop from previous | +| --- | --- | --- | | 5 | **180** | — | | 10 | **116** | −64 (−35.6 %) | | 20 | **86** | −30 (−25.9 %) | @@ -369,199 +320,275 @@ At each milestone (N = 5, 10, 20, 30, 40, 50), the strictest possible threshold | 40 | **70** | −7 (−9.1 %) | | 50 | **67** | −3 (−4.3 %) | -![Figure S1: Scaling curve showing the number of features that recur across ALL N attribution graphs as N grows from 5 to 50. The curve drops steeply at first — removing features that only appeared by coincidence in the small prompt set — then nearly plateaus, converging to a stable floor of 67 features.](scaling_curve.png) +![Scaling curve of core feature count](scaling_curve.png) + +*Figure 3: Scaling curve of the shared circuit. The curve shows the number of features that recur in all N attribution graphs as N grows from 5 to 50 (the values of Table 7). It drops steeply at first, removing features that appeared in the small prompt set only by coincidence or through the shared surface template, then nearly plateaus, converging to a stable floor of 67 features.* -The curve has a clear two-phase shape. First, a **rapid contraction** from N = 5 to N = 20: adding the first batch of syntactically diverse prompts eliminates roughly half the initial 180 features — features that appeared in every one of the original five prompts by coincidence, or because all five shared the "is to … as" surface string, do not survive once fundamentally different phrasings are added. Second, a **near-plateau** from N = 20 onward: only 19 further features are lost across 30 additional prompts, and the final step (N = 40 → 50) removes just 3. By N = 50, the curve has essentially stopped moving. +The curve in Figure 3 has a clear two-stage shape. First, a rapid contraction from N = 5 to N = 20: adding the first batch of syntactically diverse prompts eliminates roughly half the initial 180 features. This is expected. Features that appeared in every one of the original five prompts by coincidence, or because all five shared the "is to … as" surface string, do not survive once fundamentally different phrasings are added. Second, a near-plateau from N = 20 onward: only 19 further features are lost across 30 additional prompts, and the final step (N = 40 → 50) removes just 3. By N = 50, the curve has essentially stopped moving. -This convergence behaviour is the key result. The 67-feature core at N = 50 survived 50 prompts spanning two semantic domains, four surface forms, and a strict all-or-nothing threshold. That is not noise — it is a stable circuit. +This convergence behavior is the key result. The 67-feature core at N = 50 survived 50 prompts spanning two semantic domains, four surface forms, and a strict all-or-nothing threshold. Such a set is not noise; it is a stable circuit. -**The five directly analogical features all survive.** Within the 67-feature core, five features carry Neuronpedia labels that explicitly describe analogical or comparative reasoning. Every one of them appears in all 50 attribution graphs: +All five analogy-concept features survive. Within the 67-feature core, five features carry Neuronpedia labels that explicitly describe analogical or comparative reasoning, and every one of them appears in all 50 attribution graphs. Table 8 lists them. -| Feature | Appearances | Avg. Influence | Label | -|---|---|---|---| +**Table 8:** The five analogy-concept features in the 67-feature core at N = 50. Each row is one feature, identified by layer (L) and feature index (#). Appearances counts total node occurrences across the 50 attribution graphs (a feature can appear at more than one token position within a single graph); average influence is the mean influence score over those occurrences; the label is Neuronpedia's automated description. + +| Feature | Appearances | Avg. influence | Label | +| --- | --- | --- | --- | | L13 #10969 | 62 | 0.713 | "comparisons between disciplines and relationships between concepts" | | L9 #13344 | 116 | 0.683 | "phrases suggesting uncertainty or comparison between two things" | | L9 #14231 | 53 | 0.683 | "words representing comparisons and relationships" | | L7 #749 | 80 | 0.652 | "analogies and comparisons" | | L5 #2141 | 62 | 0.639 | "comparisons of people or figures using well-known public figures" | -Three of these — L13 #10969, L9 #13344, and L5 #2141 — were already identified in the original five-prompt analysis (§3.4). The scaling experiment adds two new ones: L9 #14231 ("words representing comparisons and relationships") and L7 #749 ("analogies and comparisons"), which only become visible once the prompt set is large and diverse enough to filter out coincidental co-activations. All five span the analogy recognition and relational integration phases (§3.4–3.5). +Three of these, L13 #10969, L9 #13344, and L5 #2141, were already identified in the original five-prompt analysis (Section 4.4). The scaling experiment adds two new ones: L9 #14231 ("words representing comparisons and relationships") and L7 #749 ("analogies and comparisons"), which only become visible once the prompt set is large and diverse enough to filter out coincidental co-activations. All five belong to the relational and integration groups (Section 4.3). -Despite the 50 prompts being phrased four different ways, the model consistently activated the same five semantic features. This confirms that the circuit is not reading a surface token pattern — it is recognising the underlying relational structure of an analogy, regardless of how that structure is expressed in words. +Despite the 50 prompts being phrased four different ways, the model consistently activated the same five semantic features. The circuit is therefore not reading a surface token pattern; it is recognizing the underlying relational structure of an analogy, regardless of how that structure is expressed in words. -### 3.7 Causal Validation via Feature Steering +### 4.7 Causal Validation via Feature Steering -The preceding results are correlational. The attribution graphs indicate which features are active and influential when the model produces an answer, but not whether those features cause it. In this section we intervene on the features directly, suppressing and boosting them, to test whether the paper's structural findings hold causally. +The attribution graph analysis identifies recurring features and causal path structures, but it does not by itself establish whether those features are causally necessary for the model's predictions (Section 2.4). To answer that question, we performed the systematic steering experiments described in Section 3.8. The experiments test two distinct feature sets that serve different purposes, and understanding which set is being tested at each point is essential for reading the results correctly. -Suppression is performed with the Neuronpedia steering API, which intervenes on a feature by adding a multiple of its direction to the residual stream at that feature's layer; a negative multiple pushes the feature's contribution down and thereby ablates it. The size of the push is set by a per-feature `strength`, which the API then scales by a global `strength_multiplier`. We used a strength of −20 and the default multiplier of 4, so the coefficient applied to each feature was −80. This is a large intervention: the core circuit features have typical activation magnitudes between roughly 1.5 and 16 (§2.4), so a −80 coefficient pushes them well outside their normal operating range and effectively removes them from the computation. +**Feature Set A: backbone features (Sections 4.7.1-4.7.2).** These are the per-prompt backbone features defined in Section 3.7: the features on the main causal paths of a single prompt's attribution graph, clustering in the final layers (primarily L16-L25). They are not the cross-graph core features of Sections 4.2-4.5, and each prompt has its own backbone. Testing them first establishes that the attribution graphs capture real causal structure rather than mere correlations, which is a prerequisite for trusting any downstream steering result on the shared circuit features. -This magnitude governs how every result below should be read (§2.7). Because the intervention is strong, ablating any sufficiently large set of features disrupts the model, so the fact that the output changed does not by itself identify the responsible circuit. We therefore rely on two forms of discriminating evidence. The first is a matched contrast, in which ablation of the circuit is compared against ablation of a random feature set of the same size and strength. The second is a predicted failure mode, in which we ask whether the model fails in the specific way the circuit's hypothesized role implies rather than in an arbitrary way. +**Feature Set B: group features (Sections 4.7.3-4.7.4).** These are the cross-graph core features identified in Sections 4.2-4.5, organized into the three functional groups of Section 4.3: the structural features (L0-L4), the relational features (L5-L9), and the integration features (L10-L13). These are the features the paper's main claim is about, and testing them is the primary causal validation of the shared circuit. If the grouping of Section 4.3 is real, suppressing each group should produce a distinct failure mode; if it is an arbitrary partition, the failure modes should not differ. -#### 3.7.1 Is the Recurring Circuit Causally Load-Bearing? +#### 4.7.1 Late-Layer Backbone Necessity (Individual Suppression) -We tested whether the 180-feature shared circuit (§2.3) drives the answer or merely co-activates with it. For each prompt we ran two ablations of identical size (180 features) and strength (−20), differing only in which features were removed: (A) all 180 circuit features, and (B) 180 random non-circuit features drawn from the same prompt's graph (seed 7). Because the circuit is defined by its recurrence across all five prompts, we ran both ablations on all five. +*Feature Set A.* The backbone of each prompt is every feature appearing on at least one of the 10 path traces described in Section 3.7. These are prompt-specific late-layer features, not the cross-graph group features. The experiments below ask: are these final-stage output features individually necessary for the model's prediction, and does attribution weight predict which ones are? + +For `analog_berlin` ("Paris is to France as Berlin is to" → Germany, p=0.973), nine backbone features were tested; Table 9 reports the results. The science hub at L21/4827 (strongest path entry, edge +198.0), the relay at L22/15670, output driver A at L25/4717 (final amplifier, shared across backbones), the location encoder at L16/6491, the relays at L17/14546 and L19/5773, the integrator at L21/7482 (integration hub, paths 2-4), and the relation applier at L19/855 all returned "Germany" when suppressed individually; none is necessary. Only output driver B at L25/2725 (secondary output driver, edge −2.09) is individually necessary, producing "the" when suppressed. In total, 1 of 9 features is individually necessary. The highest-weight feature (L21/4827, edge +198.0) is not individually necessary, demonstrating that attribution weight alone does not predict causal necessity, a methodological lesson consistent with prior circuit analysis work [3, 4]. -We call condition (B) the *matched null*. It is a control group of features, built to match the circuit in every respect that could trivially disrupt the model — the same number of features (180), the same steering strength, and features drawn from the same attribution graph — but composed of randomly chosen features instead of the recurring circuit. Its purpose is to answer one question: is the circuit *specifically* responsible for the answer, or would removing any 180 features do the same? The matched null is the negative control, or placebo, against which condition (A) is measured. It stands for the null hypothesis that the circuit is not special, so the circuit earns a causal interpretation only if ablating it produces an effect that ablating the null does not. +**Table 9:** Individual suppression of the nine backbone features of the `analog_berlin` backbone. Each row is one backbone feature, identified by layer and feature index; Role is our informal description of its position on the traced paths; Steered token is the model's first output token when that feature alone is suppressed at strength −20; Necessary is "yes" when suppression changes the prediction away from "Germany". -If the circuit implements the analogical computation, removing it should leave the model unable to complete "X is to Y as Z is to ___." The model would then fall back on the template structure it can still represent and repeat the connective "to" rather than produce an answer. A random ablation has no reason to fail in this way. +| Feature | Layer | Index | Role | Steered token | Necessary? | +| --- | --- | --- | --- | --- | --- | +| Science hub | 21 | 4827 | Strongest path entry (edge +198.0) | Germany | no | +| Relay | 22 | 15670 | Path 1 relay | Germany | no | +| Output driver A | 25 | 4717 | Final amplifier (shared across backbones) | Germany | no | +| Location encoder | 16 | 6491 | Location/direction feature, path 2 entry | Germany | no | +| Relay | 17 | 14546 | Mid-cascade relay | Germany | no | +| Relay | 19 | 5773 | Late relay | Germany | no | +| Integrator | 21 | 7482 | Integration hub (paths 2-4) | Germany | no | +| Output driver B | 25 | 2725 | Secondary output driver (edge −2.09) | the | **YES** | +| Relation applier | 19 | 855 | Relation application node | Germany | no | -| Prompt | Baseline: model's answer | (A) Ablate the circuit (180 feat.) | (B) Ablate a matched random set (180 feat.) | -|---|---|---|---| -| Paris - France → Berlin - ? | Germany (96%) | **to (97%)** | Kyrie (6%, a name) | -| Paris - France → Rome - ? | Italy (96%) | **to (97%)** | autorytatywna (44%, Polish) | -| Paris - France → Tokyo - ? | Japan (98%) | **to (97%)** | to (14%) | -| Doctor - hospital → teacher - ? | school (57%) | **to (97%)** | initComponents (45%, code) | -| Fish - water → bird - ? | air (8%) | **to (97%)** | espère (73%, French) | +For `analog_rome` ("Paris is to France as Rome is to" → Italy, p=0.974), ten features were tested; Table 10 reports the results. The relay at L20/15360, the relays at L22/12202 and L22/14727, the relay at L23/5917, and the secondary gate at L24/13277 all returned "Italy." Four features are individually necessary: the late gate at L24/16122 ("the"), the output driver at L25/286 ("the"), the final amplifier at L25/4717 ("the"), and output driver C at L25/10521 ("the"). In total, 4 of 10 features are individually necessary. The Rome backbone has more single points of failure than Berlin despite near-identical confidence (p=0.974 vs 0.973), indicating that path redundancy varies even among structurally similar geographic analogies. -*How the two ablations fail. Both remove the model's correct answer, but ablating the circuit collapses every prompt to the same connective token "to" at ≈97% confidence, whereas ablating a matched random set produces a different idiosyncratic token each time (a name, a Polish word, a code identifier, a French word).* +**Table 10:** Individual suppression of the ten backbone features of the `analog_rome` backbone. Columns as in Table 9; Necessary is "yes" when suppression changes the prediction away from "Italy". + +| Feature | Layer | Index | Role | Steered token | Necessary? | +| --- | --- | --- | --- | --- | --- | +| Relay | 20 | 15360 | Backward path from logit | Italy | no | +| Late gate | 24 | 16122 | Backward path, L24 suppression gate | the | **YES** | +| Output driver | 25 | 286 | Backward path, output driver | the | **YES** | +| Final amplifier | 25 | 4717 | Shared final amplifier (act=265.2) | the | **YES** | +| Output driver C | 25 | 10521 | Tertiary output driver | the | **YES** | +| Relay | 17 | 14546 | Mid-cascade relay (shared with Berlin) | Italy | no | +| Relay A | 22 | 12202 | Late relay | Italy | no | +| Relay B | 22 | 14727 | Late relay | Italy | no | +| Relay | 23 | 5917 | Late relay | Italy | no | +| Secondary gate | 24 | 13277 | Late gate | Italy | no | -Two observations follow. First, both ablations remove the correct answer on all five prompts, confirming that at this strength the removal of 180 features of any kind disrupts the model and that necessity alone does not distinguish the circuit from a random set. Second, the two ablations fail in different ways. Circuit ablation produces a near-deterministic collapse to the connective "to" at the same confidence on every prompt (97%), for both geographic and semantic-role analogies. The matched null, applied to the same prompts at the same strength, produces a different idiosyncratic out-of-distribution token on each prompt, with no token shared across prompts and lower average confidence (26%); only one null run (Tokyo) reaches "to," and only at 14% confidence versus the circuit's 97%. Because the removed feature set is the only variable differing between (A) and (B), the structured collapse is attributable to the circuit rather than to the size of the intervention. +For `analog_tokyo` ("Paris is to France as Tokyo is to" → Japan, p=0.990), ten features were tested and three were found necessary: the output driver at L25/286 ("the"), late relay A at L23/850 ("the"), and late relay B at L23/13914 ("the"). Table 11 reports the results; 3 of 10 features are individually necessary. L23/13914 is necessary in both the Tokyo and Cairo backbones, a shared bottleneck consistent with a late-layer "geographic entity selector" role. L25/286 recurs as necessary in Rome, Tokyo, and Cairo, making it the single most critical output driver across geographic analogies. -One caveat should be stated directly. All five prompts end in the word "to" (for example, "…as Berlin is to"), and ablating the circuit makes the model output "to" — which is also the final word of the prompt. Two different explanations therefore predict the same token, and the output alone cannot separate them: the model may have lost the analogical computation and fallen back to the template connective "to" (the reading we intend), or it may simply be echoing the last word it was shown, as a broken model often does. What breaks the tie is the matched null, which faces the same "to"-final prompts but does not echo "to," emitting unrelated tokens instead. Ending in "to" is therefore not enough on its own to produce the collapse, so the effect is better explained by a loss of relational completion than by trivial token repetition. +**Table 11:** Individual suppression of the ten backbone features of the `analog_tokyo` backbone. Columns as in Table 9; Necessary is "yes" when suppression changes the prediction away from "Japan". -We also examined the internal distribution of the effect by ablating each of the 180 features individually on the Berlin prompt. Only 24 of 180 features (13.3%) shift the prediction on their own, and these concentrate at the embedding layer (17 of 24 at L0; 42.5% of the L0 features tested, compared with 5.0% at L1 and above). The circuit is therefore internally redundant: necessity is a property of the set rather than of most individual members, and the individually necessary features are early template features such as L0/11651 ("the word 'to'"). +| Feature | Layer | Index | Role | Steered token | Necessary? | +| --- | --- | --- | --- | --- | --- | +| Relay | 20 | 15360 | Backward path from logit | Japan | no | +| Output driver | 25 | 286 | Backward path, output driver | the | **YES** | +| Output driver B | 25 | 12223 | Backward path, secondary output | Japan | no | +| Relay | 17 | 14546 | Mid-cascade relay (shared) | Japan | no | +| Late relay A | 23 | 850 | Late relay | the | **YES** | +| Late relay B | 23 | 13914 | Late relay (also necessary in Cairo circuit) | the | **YES** | +| Gate | 24 | 13277 | Late gate (shared with Rome) | Japan | no | +| Output driver C | 25 | 10152 | Tertiary output | Japan | no | +| Hub | 20 | 6648 | L20 convergence hub | Japan | no | +| Integration | 21 | 7764 | Late integration | Japan | no | -#### 3.7.2 Robustness: Layer, Magnitude, and Generalization +For `analog_teacher` ("Doctor is to hospital as teacher is to" → school, p=0.486), ten features were tested and three were found necessary: the embedding-level feature at L0/17 ("the"), the output driver at L25/4975 ("..."), and the final amplifier at L25/4717 ("a"). Table 12 reports the results; 3 of 10 features are individually necessary. The teacher backbone is the only one in which an L0 embedding-level feature (L0/17) is individually necessary, suggesting that the semantic role analogy relies on an early feature not redundantly encoded by later layers, unlike the capital analogies. -We first considered whether the effect is only the deletion of L0 token features. Forty of the 180 features lie at L0, the embedding layer, so suppressing them approximates deleting the entity tokens. If this accounted for the full effect, ablating only the L0 features would remove the answer. It does not. Ablating the circuit one layer band at a time gives: +**Table 12:** Individual suppression of the ten backbone features of the `analog_teacher` backbone. Columns as in Table 9; Necessary is "yes" when suppression changes the prediction away from "school". -| Prompt | L0 only (40) | L1–L4 (67) | L5–L9 (50) | L10+ (23) | All non-L0 (140) | -|---|---|---|---|---|---| -| Berlin | ` Germany` ✓ | broke | broke | broke | broke | -| Rome | ` Italy` ✓ | broke | broke | broke | broke | -| Tokyo | ` Japan` ✓ | broke | broke | broke | broke | -| Bird | ` air` ✓ | broke (→ water) | broke | broke | broke | -| Teacher | broke | broke | broke | broke | broke | +| Feature | Layer | Index | Role | Steered token | Necessary? | +| --- | --- | --- | --- | --- | --- | +| Embedding | 0 | 17 | Backward path, embedding-level | the | **YES** | +| Gateway | 18 | 6532 | Backward path, mid-late gateway | school | no | +| Hub | 20 | 6179 | Backward path, convergence hub | school | no | +| Output driver | 25 | 4975 | Backward path, output driver | ... | **YES** | +| Final amplifier | 25 | 4717 | Shared final amplifier (act=135.6) | a | **YES** | +| Relay | 22 | 15670 | Late relay (shared) | school | no | +| Relay B | 18 | 11952 | Mid-late relay | school | no | +| Legal docs | 18 | 13586 | Legal docs feature | school | no | +| Convergence | 21 | 2655 | Late convergence hub | school | no | +| Gate | 24 | 15259 | Late suppression gate | school | no | -*Layer-band ablation of the circuit across the five prompts. Ablating only the 40 L0 embedding features leaves the answer intact on 4/5; the exception, teacher, has a weak baseline (p = 0.49).* Necessity is carried by the L1-and-above features, so the token-deletion explanation is insufficient: the causal weight lies in the relational mid- and late-layer features rather than the input-token representations. The bird prompt illustrates this further, in that L0 alone leaves "air" intact but the L1–L4 band reverts the answer to "water," the source-domain completion. +For `analog_bird` ("Fish is to water as bird is to" → air, p=0.117), ten features were tested and eight were found necessary. Table 13 reports the results; 8 of 10 features are individually necessary, making this the most fragile backbone in the dataset. Three L22 relay features (15670, 14727, 13619) are all independently necessary despite occupying the same layer, indicating that they carry non-redundant information through parallel channels. This fragility is consistent with the prompt's very low prediction confidence (p=0.117). -We next considered whether the effect is an artifact of the large −80 magnitude. A strength titration on Berlin and Tokyo sweeps the suppression strength across −2, −5, −10, −20, and −40, bracketing the −20 used above; the table below shows the two extremes: +**Table 13:** Individual suppression of the ten backbone features of the `analog_bird` backbone. Columns as in Table 9; Necessary is "yes" when suppression changes the prediction away from "air". -| strength | Berlin circuit | Berlin null | Tokyo circuit | Tokyo null | -|---|---|---|---|---| -| −2 | to (97%) | similar (54%) | to (97%) | onPostExecute (33%) | -| −40 | to (97%) | similar (50%) | to (97%) | onPostExecute (30%) | +| Feature | Layer | Index | Role | Steered token | Necessary? | +| --- | --- | --- | --- | --- | --- | +| Backward A | 22 | 4252 | Backward path from logit | (space) | **YES** | +| Backward B | 24 | 8106 | Backward path, late gate | \_\_\_\_ | **YES** | +| Final amplifier | 25 | 4717 | Shared final amplifier (act=122.3) | the | **YES** | +| Output driver | 25 | 11801 | Output driver | ? | **YES** | +| Relay A | 22 | 15670 | Late relay (shared) | \_\_\_\_\_\_\_\_ | **YES** | +| Relay B | 22 | 14727 | Late relay | (space) | **YES** | +| Relay C | 22 | 13619 | Late relay | (space) | **YES** | +| Gate A | 24 | 4383 | Suppression gate | air | no | +| Gate B | 24 | 12559 | Suppression gate | the | **YES** | +| Hub | 20 | 3094 | Integration hub | air | no | -The collapse to "to" is already complete at the weakest setting tested (−2), and the circuit-versus-null distinction holds across the full range. The effect is therefore not a product of the −80 intervention. +Across all five prompts, individual necessity correlates inversely with prediction confidence: Berlin (p=0.973) yields 1/9; Rome (p=0.974) yields 4/10; Tokyo (p=0.990) yields 3/10; Teacher (p=0.486) yields 3/10; Bird (p=0.117) yields 8/10. Three features recur as necessary across multiple backbones: L25/#286 (Rome, Tokyo, Cairo), L25/#4717 (Rome, Teacher, Bird), and L23/#13914 (Tokyo, Cairo). -Finally, we tested whether the circuit generalizes beyond its defining prompts. The 180 features were derived from five specific prompts. We applied the fixed circuit to four held-out analogies from which it was not derived: +#### 4.7.2 Full Backbone Suppression -| Held-out prompt | Baseline | Circuit ablation | -|---|---|---| -| Lisbon…Vienna is to | Austria | to (97%) | -| Athens…Oslo is to | Norway | to (97%) | -| Pen…knife is to | cutting | to (97%) | -| Bee…ant is to | colony | to (97%) | +Suppressing all late-layer backbone features simultaneously disrupted 6 of 7 prompts, as shown in Table 14. The failure modes are qualitatively informative: capital analogies degenerate to repetitive or archaic text ("of of of of of"; "pleaſure pleaſure plea"; "country country count"), indicating that the backbone is required for entity selection while the prompt structure alone partially activates a "country" category. Teacher collapses to "1111"; bird falls through to generic continuation. Puppy→cat is the sole exception, apparently carried by direct embedding-to-logit connections outside the multi-hop backbone. -The same collapse to "to" (97%) appears on all four prompts, indicating that the circuit signature is not specific to the defining set. +**Table 14:** Simultaneous suppression of all backbone features for each of the seven prompts. Each row is one prompt; p is the model's unsteered probability of the expected completion; N feat. is the number of backbone features suppressed simultaneously at strength −20; Default output is the unsteered continuation; Steered output is the continuation under full backbone suppression; Disrupted indicates whether the correct completion was lost. + +| Prompt | p | N feat. | Default output | Steered output | Disrupted? | +| --- | --- | --- | --- | --- | --- | +| `analog_berlin` | 0.973 | 9 | Germany. It is the | of of of of of | YES | +| `analog_rome` | 0.974 | 10 | Italy. It is the | pleaſure pleaſure plea | YES | +| `analog_tokyo` | 0.990 | 10 | Japan. It is the | country country count | YES | +| `analog_teacher` | 0.486 | 10 | school. The doc | 1111 | YES | +| `analog_bird` | 0.117 | 10 | air. The fish | (newline) The the the | YES | +| Cairo→Kenya | 0.963 | 9 | Kenya. It is the | (whitespace) | YES | +| Puppy→cat | 0.756 | 4 | cat. I' | cat. I think | no | + +#### 4.7.3 Structural and Relational Feature Necessity + +*Feature Set B: primary validation of the shared circuit.* The features tested here are the cross-graph core features identified in Sections 4.3 and 4.4: the five structural features (L0-L4), the four relational features (L5-L9), and the integration feature L13 #10969. These ten features are a subset of the circuit, not the whole of it. They were selected because they are the interpretable, high-influence members of each group; the remaining core features were not steered, so the causal claims below concern these ten features and not all 180 or all 67 (Section 6). Unlike the backbone features of Sections 4.7.1-4.7.2, which were selected by path tracing within individual graphs, these features were selected because they recur across all five independently generated attribution graphs. The question here is whether they are also causally necessary. + +Individual suppression of 9 key group features across all five prompts (45 tests in total) reveals a clear asymmetry between the structural and relational groups. The results are shown in Table 15; each cell shows the steered first token when the feature is suppressed at strength −20, with a dash indicating an unchanged prediction. + +**Table 15:** Individual suppression of nine shared-circuit group features across the five prompts of Table 1 (45 tests). Each row is one feature, with its group (1 = structural, 2 = relational, 3 = integration) and abbreviated Neuronpedia label; each prompt column shows the model's first output token when that feature alone is suppressed at strength −20; a dash (—) indicates the prediction was unchanged. + +| Feature | Group | Label | Berlin | Rome | Tokyo | Teacher | Bird | +| --- | --- | --- | --- | --- | --- | --- | --- | +| L0/11651 | 1 | "the word 'to'" | Berlin | Rome | Tokyo | school | water | +| L1/11356 | 1 | "'to' followed by a verb" | — | — | — | — | — | +| L4/10752 | 1 | "'to be' preceded by 'to'" | — | — | — | classroom | sky | +| L5/9672 | 1 | "the phrase 'it is to'" | — | — | — | — | sky | +| **L5/5793** | **2** | **"analogies"** | — | — | — | — | — | +| L5/2141 | 2 | "comparisons of public figures" | — | — | — | — | — | +| L8/13766 | 2 | "analogies or comparisons" | — | — | — | — | fish | +| L9/13344 | 2 | "comparison between two things" | — | — | — | — | sky | +| L13/10969 | 3 | "comparisons between disciplines" | — | — | — | — | — | + +Regarding the structural group, L0/11651 is necessary in 4 of 5 prompts. Suppressing it causes the capital analogies to predict the city name itself (Berlin, Rome, Tokyo), indicating that the model reverts to the most recently mentioned entity rather than completing the analogy. For the bird prompt, suppression produces "water" (the source-pair element). Regarding the relational group, L5/5793 ("analogies") is never individually necessary in any circuit; it is individually redundant for high-confidence prompts. For the fragile bird prompt (p=0.117), however, relational features become individually necessary: suppressing L8/13766 changes "air" to "fish" (the source-domain animal), and suppressing L9/13344 changes "air" to "sky." Regarding the integration group, L13/10969 is not individually necessary for any prompt. + +#### 4.7.4 Collective Group Suppression + +Collective suppression results are shown in Table 16, with all 20 of 20 cells disrupted. + +**Table 16:** Collective group suppression. Each row suppresses one set of shared-circuit group features simultaneously at strength −20; the second column lists the suppressed features; each prompt column shows the model's first output token under that suppression. All 20 cells are disrupted. + +| Experiment | Features suppressed | Berlin | Rome | Tokyo | Teacher | Bird | +| --- | --- | --- | --- | --- | --- | --- | +| All relational (4 feat.) | L5/5793, L5/2141, L8/13766, L9/13344 | **France** | **France** | **France** | be | fish | +| All structural (5 feat.) | L0/11651, L1/11356, L4/10752, L5/9672, L2/11475 | (empty) | (empty) | (empty) | to | to | +| Structural + relational (9 feat.) | All structural + relational | : | : | : | : | : | +| All three groups (10 feat.) | All structural + relational + L13/10969 | : | : | : | be | : | + +Collective suppression of the relational group is the most informative experiment in this paper. All three capital analogies output "France", and the bird prompt outputs "fish". In each case the model stops completing the analogy and instead emits a token drawn from the source pair of the prompt itself. This is direct causal evidence that the relational features collectively implement the relational transfer operation: with them suppressed, the model no longer carries the relation across to the new argument pair. We note that "France" is present in the text of all three capital prompts (Table 1), so these outputs are consistent both with the model retaining the source-pair association and failing to re-apply it, and with a simpler fallback to copying a salient prompt token. Distinguishing the two would require varying the source pair while holding the target fixed, which we did not test. The failure mode is precisely what one would predict from the internal representation findings of Lee et al. [10], where reasoning failures reflect missing relational information in mid-upper layers. Collective suppression of the structural group produces empty outputs for the capital analogies and "to" for the semantic role analogies, a qualitatively different and more severe failure, consistent with structural parsing being a prerequisite for relational recognition. Combined structural-plus-relational suppression produces ":" for 4 of 5 prompts, consistent with the model defaulting to list-formatting punctuation when both template parsing and analogy recognition are disabled. Each group therefore fails in its own characteristic way, which is the causal confirmation that the grouping of Section 4.3 reflects genuinely distinct functions. The results also establish a dependency ordering, structural → relational → integration and late layers, in which each group is collectively necessary and earlier groups are prerequisites for later ones. + +#### 4.7.5 Sufficiency (Hub Boost on Altered Prompts) + +Sufficiency was tested by boosting one backbone hub feature per prompt at strength +20 while the model processed an altered prompt (Section 3.8). Table 16 reports the outcomes. + +**Table 17:** Sufficiency tests via hub boosting. Each row boosts one backbone hub feature at strength +20 while the model processes an altered prompt; Induced indicates whether boosting caused the model to produce the original prompt's completion on the altered prompt. -#### 3.7.3 The Three-Phase Architecture +| Source prompt | Hub boosted | Altered prompt | Induced? | +| --- | --- | --- | --- | +| `analog_berlin` | L21/4827 | "Cairo is to Egypt as Nairobi is to" | no | +| `analog_berlin` | L21/4827 | "Madrid is to Spain as Berlin is to" | **YES → Germany** | +| `analog_rome` | L20/15360 | "Paris is to France as Tokyo is to" | no | +| `analog_rome` | L20/15360 | "Madrid is to Spain as Rome is to" | **YES → Italy** | +| `analog_tokyo` | L20/15360 | "Paris is to France as Rome is to" | no | +| `analog_tokyo` | L20/15360 | "Beijing is to China as Tokyo is to" | **YES → Japan** | +| `analog_teacher` | L0/17 | "Nurse is to hospital as teacher is to" | no | +| `analog_teacher` | L0/17 | "Doctor is to hospital as chef is to" | no | +| `analog_bird` | L22/4252 | "Cat is to land as bird is to" | no | +| `analog_bird` | L22/4252 | "Fish is to water as eagle is to" | **YES → air** | +| Cairo→Kenya | L15/15954 | "Lagos is to Nigeria as Nairobi is to" | **YES → Kenya** | -Sections 3.7.1 and 3.7.2 establish that the recurring circuit causally drives the model's answer, rather than merely co-occurring with it. A separate claim (§2.4) is that the circuit is organized into three phases. We tested this with phase-level suppression of the representative features from each phase, across all five prompts. +5 of 11 tests succeed. Sufficiency holds when the altered prompt retains the target entity or a semantically close substitute, and fails when it crosses domain boundaries. The capital hubs encode domain-specific geographic associations rather than general-purpose "answer slot" activators. -Suppressing the nine key phase features one at a time (45 tests) shows that the Phase-2 "analogy" features are individually redundant on high-confidence prompts. A Phase-1 template feature, L0/11651 ("the word 'to'"), is individually necessary in four of five circuits: suppressing it makes the capital analogies emit the city name itself rather than complete the analogy. +#### 4.7.6 Specificity (Non-Backbone Feature Suppression) -Collective phase suppression provides the main architecture test: +Specificity was tested by suppressing high-activation features that lie on no traced causal path (Section 3.8). Table 17 reports the outcomes. -| Experiment | Features | Berlin | Rome | Tokyo | Teacher | Bird | -|---|---|---|---|---|---|---| -| All Phase 2 (4 feat.) | L5/5793, L5/2141, L8/13766, L9/13344 | **France** | **France** | **France** | be | fish | -| All Phase 1 (5 feat.) | L0/11651, L1/11356, L4/10752, L5/9672, L2/11475 | (empty) | (empty) | (empty) | to | to | -| Phase 1+2 (9 feat.) | All Phase 1 + Phase 2 | : | : | : | : | : | +**Table 18:** Specificity tests. Each row suppresses one high-activation feature that lies outside that prompt's backbone, at strength −20; Steered token is the resulting first output token; Disrupted indicates whether the prediction changed. An unchanged prediction is a pass, confirming that steering effects are specific to the hypothesized circuit. -Suppressing all four Phase-2 features makes the three capital analogies output "France," the source country. The model retains the factual association "Paris is to France" but loses the relational transfer "as Berlin is to ___." This is the failure mode predicted if Phase 2 implements relational transfer, and it is the strongest single piece of architecture evidence. Phase-1 suppression produces a more severe failure, an empty output for the capital prompts, consistent with Phase 1 being a prerequisite for Phase 2. Together these results indicate an ordered hierarchy: Phase 1 (template) precedes Phase 2 (relational transfer), which precedes answer retrieval in the later layers. +| Prompt | Feature | Label | Steered token | Disrupted? | +| --- | --- | --- | --- | --- | +| `analog_berlin` | L6/3335 | "difficulty/challenges" | Germany | no | +| `analog_berlin` | L13/4435 | "opera-related terms" | Germany | no | +| `analog_rome` | L6/2267 | "formal text/code" | Italy | no | +| `analog_rome` | L4/14857 | "code snippets" | Italy | no | +| `analog_tokyo` | L6/2267 | "formal text/code" | Japan | no | +| `analog_tokyo` | L3/10018 | early structural feature | Japan | no | +| `analog_teacher` | L4/14857 | "code snippets" | school | no | +| `analog_teacher` | L8/13766 | "analogies or comparisons" | school | no | +| `analog_bird` | L6/2267 | "formal text/code" | **sky** | YES | +| `analog_bird` | L5/5793 | "analogies" | air | no | +| Cairo→Kenya | L5/5500 | "profanity and comparisons" | Kenya | no | +| Puppy→cat | L9/2909 | "formulas/ratios" | cat | no | -#### 3.7.4 The Causal Validation Ledger +12 of 13 tests pass specificity. The sole exception, L6/2267 tipping bird from "air" to "sky", occurs at the margin of unresolved token competition (p=0.117), and the feature is confirmed inert for all high-confidence prompts. L5/5793 ("analogies") passes specificity for the bird prompt, consistent with it being individually dispensable but collectively necessary. -The interventions above bear on the paper's structural claims with different degrees of strength. Four claims meet the matched-contrast or predicted-failure-mode standard and we regard as demonstrated: that the 180 recurring features drive the analogy rather than ride along with it (§3.7.1); that the effect is not merely deletion of the L0 entity tokens (§3.7.2); that it is not an artifact of the −80 steering magnitude (§3.7.2); and that the circuit is organized into the ordered three-phase hierarchy of §3.3, with the four Phase-2 features collectively implementing relational transfer (§3.7.3). The dedicated analogy features at layers 5, 8, 9, and 13 participate causally as a group, although individually they are redundant. +#### 4.7.7 Summary of Causal Validation -Two further claims are supported but lack a full matched control. The circuit's collapse signature reproduces on four held-out analogies (§3.7.2) and on the two semantic-role prompts, extending the account across domains; the teacher and bird baselines are weak, however (the model is only 49% and 12% confident at baseline), so cross-domain generalization rests on fewer confident cases than the geographic analogies. Finally, the high-recurrence formal-text features (§3.4) were not steered as a group and appear causally inert, so their role remains descriptive rather than demonstrated. +Table 18 summarizes the causal validation results across all seven prompts. -Three bounds should be stated explicitly: +**Table 19:** Summary of the causal validation results. Each row is one prompt; p is the unsteered probability of the expected completion; Individual necessity counts the backbone features found individually necessary (Tables 9-13); Full suppress reports the outcome of full backbone suppression (Table 14); Relational collective gives the steered output under collective suppression of the relational group (Table 16); Sufficiency counts successful hub-boost inductions (Table 17); Specificity summarizes the control tests (Table 18). Dashes indicate tests not run for that prompt. -1. **Sufficiency.** We show that the circuit is necessary and fails in the predicted way, but not that it is sufficient to produce the answer in isolation. The single-feature sufficiency probe, which boosts the highest-influence hub, is largely negative, inducing the target only when the target entity is already present in the prompt. -2. **Surgical necessity.** Because −80 is a strong intervention, single-feature necessity is layer-dependent, so the strongest claims rest on the matched contrast rather than on any individual ablation. -3. **Mediation.** The phase experiments show that each phase is collectively necessary, but establishing that information flows along the Phase 1 → Phase 2 edge requires path patching on the model weights, which the steering API does not expose. This remains future work. +| Prompt | p | Type | Individual necessity | Full suppress | Relational collective | Sufficiency | Specificity | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `analog_berlin` | 0.973 | Capital | 1/9 | DISRUPTED | → France | 1/2 | PASS | +| `analog_rome` | 0.974 | Capital | 4/10 | DISRUPTED | → France | 1/2 | PASS | +| `analog_tokyo` | 0.990 | Capital | 3/10 | DISRUPTED | → France | 1/2 | PASS | +| `analog_teacher` | 0.486 | Sem. role | 3/10 | DISRUPTED | → be | 0/2 | PASS | +| `analog_bird` | 0.117 | Sem. role | 8/10 | DISRUPTED | → fish | 1/2 | 1/2 | +| Cairo→Kenya | 0.963 | Capital | 2/9 | DISRUPTED | — | 1/2 | PASS | +| Puppy→cat | 0.756 | Sem. role | 0/4 | intact | — | — | PASS | -Within these bounds, the steering evidence supports the paper's central structural claims: a recurring circuit that causally drives the answer across five prompts and four held-out analogies, not reducible to token deletion or to the intervention magnitude, organized into the three phases of §3.3, and shared between geographic and semantic-role analogies. +Across the 159 steering experiments, the two validation tracks converge on five principal findings. ---- +From Feature Set A (backbone, Sections 4.7.1-4.7.2): First, the late-layer backbone is collectively necessary; full suppression disrupts 6 of 7 prompts, establishing that the attribution graphs track genuine causal structure. Second, attribution weight does not predict individual necessity: the highest-weight feature in the Berlin graph (L21/4827, edge +198.0) is not individually necessary, while a lower-weight feature (L25/2725) is. Third, three late-layer features recur as necessary across multiple distinct backbones, L25/#286 (Rome, Tokyo, Cairo), L25/#4717 (Rome, Teacher, Bird), and L23/#13914 (Tokyo, Cairo), revealing a shared output mechanism not visible in the cross-graph overlap analysis. Fourth, individual necessity scales inversely with prediction confidence: the bird backbone (p=0.117) has 8/10 necessary backbone features while the Berlin backbone (p=0.973) has 1/9. - +From Feature Set B (group features, Sections 4.7.3-4.7.4): Fifth, the ten interpretable core features tested from the shared circuit are causally necessary, and the three feature groups they represent are causally dissociable. Structural features are individually necessary; L0/#11651 alone disrupts 4 of 5 prompts, causing capital prompts to revert to the city name. Relational features are collectively necessary but individually redundant; simultaneous suppression collapses every prompt, with the model falling back to a token from the source pair ("France" for the capitals, "fish" for bird), while no single relational feature is individually indispensable. This is direct causal evidence that the relational features collectively implement the relational transfer operation. The integration feature and the formal-text features (L4/#14857, L6/#2267) are not individually necessary, confirming that high recurrence in the attribution graphs does not imply causal necessity. -## 4. Discussion - -### 4.1 The Analogical Reasoning Circuit in Gemma-2-2B - -**Overall synthesis.** The results establish that Gemma-2-2B performs analogical reasoning through a stable, three-phase distributed circuit rather than any single mechanism or layer. The convergence of structural, semantic, and causal evidence — across 218 steering experiments, a 50-prompt scaling study, cross-domain generalization testing, and held-out analogies — provides a mechanistic account at a level of specificity and causal resolution that prior behavioral work on LLM analogical reasoning could not reach. The core argument of this paper is not merely that recurring features exist, but that the recurring features identified through graph overlap are causally load-bearing, and that different phases of the circuit play functionally distinct and experimentally separable roles. - -**The three-phase architecture in context.** The three-phase organization — structural template parsing (L0–L4), analogy recognition (L5–L9), and relational integration (L10–L13) — mirrors the abstract reasoning architecture documented by Webb et al. [9] through causal mediation analysis, where early layers abstract tokens into relational variables, intermediate layers perform induction over those variables, and later layers retrieve answers. The present results extend that framework in two important ways: by identifying specific SAE features at each phase rather than working at the attention head level, and by providing direct causal evidence through feature steering that each phase is collectively necessary for the circuit to function. Crucially, the phase boundaries were not imposed a priori — they emerged from the content of Neuronpedia automated labels naturally clustering by layer depth, with a convergent gradient in activation magnitudes — rising from 1.5–6.4 in Phase 1 to 9.1–16.3 in Phase 3 (§2.4) — confirming the same partition through a second independent line of evidence. - -This is qualitatively distinct from multi-hop factual reasoning. Analogical reasoning requires extracting an unnamed relation type, holding it as a variable, and applying it to a new argument pair. The Phase 2 collective suppression experiment demonstrates that this extraction and transfer are implemented by identifiable internal components whose removal causes the model to echo the source-pair answer rather than transfer the relation — consistent with the "missing relational information" failure mode documented by Lee et al. [10] at the behavioral level. Our work provides a feature-level causal account of this phenomenon. - -Prior behavioral evidence [1] established that LLMs can match human performance on analogical tasks; Webb et al. [9] identified emergent symbolic mechanisms supporting abstract reasoning through causal mediation of attention heads. The present work extends these findings to the SAE feature level: the relational reasoning primitive is not just a pattern of attention head behavior but a specifically labeled feature that plays a causal role in the SAE's learned decomposition of residual stream activations. - -### 4.2 Circuit Stability Across Scaling and Surface Forms - -**Circuit stability across surface forms.** Perhaps the most theoretically significant finding outside the steering results is the convergence of the feature set to a stable 67-feature core across 50 prompts phrased in four syntactically distinct surface forms (§3.6). The initial 180-feature circuit, identified from five prompts sharing the "X is to Y as Z is to" template, contracts rapidly when surface-diverse prompts are introduced — losing roughly half its features by N = 20 — but then plateaus, with only 19 further features lost across the subsequent 30 prompts, and just 3 in the final step. This two-phase scaling behaviour has a clear interpretation: the first contraction eliminates features that were coincidental artifacts of the shared surface template, while the plateau identifies features that activate because of the underlying relational structure, regardless of how that structure is expressed in words. All five directly analogical features survive the full 50-prompt filter. This is strong evidence against a surface-token explanation of the circuit and in favour of a genuine, abstract relational representation inside the model — independent evidence from the same direction as the held-out-prompt generalization result in §3.7.2. - -**Cross-domain generalization.** The cross-domain generalization finding reinforces this interpretation. The shared circuit — and the stable 67-feature core — includes features that activate for both geographic capital analogies and semantic role analogies. The analogy-concept features at L5 and L8 fire equally for "Paris is to France as Berlin is to" and for "Doctor is to hospital as teacher is to", despite these prompts sharing no surface tokens related to analogy. This is consistent with the behavioral finding of Wijesiriwardene et al. [11] that LLMs encode relational information in a domain-general manner, and constitutes the first identification of specific internal features implementing that domain-generality at the feature level. The slightly larger attribution graphs for semantic role analogies (1,040–1,071 nodes) relative to capital analogies (905–963 nodes) may reflect that semantic roles require broader world-knowledge access rather than retrieval of a discrete, well-encoded fact — an interpretation consistent with the ROME findings of Meng et al. [2] on the compactness of factual storage for geographic entities. - -### 4.3 The Role of Formal Text Features - -The high-recurrence "code and legal text" features present an interpretive puzzle best understood through the lens of polysemanticity and superposition [6]. Two complementary explanations: - -**Functional hypothesis:** These features detect formal, template-driven text patterns generally. The analogy syntax "X is to Y as Z is to" is highly structured, resembling legal definitions, code comments, and mathematical notation. The model reuses a general "formal syntax" detector. - -**Training data hypothesis:** The analogy format appears frequently in SAT preparation and educational materials — which also contain code examples and legal definitions — creating a statistical association between formal-text features and analogy-completion contexts. - -Both are compatible with the causal steering data. The formal features process the syntactic surface of the template while the analogy features process the relational semantics; only the latter are collectively necessary for relational transfer. The SAE-based decomposition [5, 6] is what makes this functional distinction visible — raw neuron activations would not cleanly separate these roles. - -### 4.4 Comparison with the Capital City Recall Circuit + -Comparison with the capital city *factual recall* circuit (prompt: "The capital of X is") reveals: +## 5. Discussion -- **Overlap:** Formal-text features (L4/#14857, L6/#2267) appear with high frequency in both circuits, activated by the formal definitional structure of both prompt types. This is analogous to the shared MLP modules Meng et al. [2] identified across different factual recall tasks. -- **Divergence:** The L5 "analogies" feature and L8 "analogies or comparisons" feature appear to be specific to the analogical task — they were not among the top recurring features in the factual recall circuit — supporting the interpretation that these features are selectively activated by relational structure recognition. +**Overall synthesis.** The results establish that Gemma-2-2B performs analogical reasoning through a stable distributed circuit rather than any single mechanism or layer, and that a small set of individually interpretable features within that circuit sorts into three functional categories. The convergence of structural, semantic, and causal evidence, across 159 steering experiments, a 50-prompt scaling study, cross-domain generalization testing, and 7 distinct prompts, provides a mechanistic account at a level of specificity and causal resolution that prior behavioral work on LLM analogical reasoning could not reach. The core argument of this paper is not merely that recurring features exist, but that the interpretable recurring features identified through graph overlap are causally necessary, and that the circuit's feature groups play functionally distinct and experimentally separable roles. -### 4.5 Relation to Anthropic's Attribution Graph Methodology +**The three feature groups in context.** The three-group organization, structural (L0-L4), relational (L5-L9), and integration (L10-L13), mirrors the abstract reasoning architecture documented by Webb et al. [9] through causal mediation analysis, where early layers abstract tokens into relational variables, intermediate layers perform induction over those variables, and later layers retrieve answers. The present results extend that framework in two important ways: by identifying specific SAE features in each group rather than working at the attention head level, and by providing direct causal evidence through feature steering that each group is collectively necessary and fails in its own characteristic way. Crucially, the grouping was neither imposed a priori nor part of our original research questions. It emerged in the course of the analysis, when the Neuronpedia automated labels were seen to cluster by layer depth, and it was then confirmed causally by the distinct failure signatures under group-wise suppression (Section 4.7.4). We emphasize that the groups are defined by function; the associated layer ranges describe where each group concentrates, not sharp architectural boundaries, and the descriptive evidence alone (the LLM-generated feature labels) would not suffice without the causal dissociation. -The present work is in direct methodological continuity with Anthropic's *On the Biology of a Large Language Model* [12], which applied attribution graphs to Claude 3.5 Haiku using cross-layer transcoders. Both papers find that models implement multi-step, staged computation rather than direct input-to-output pattern matching, and both validate circuit hypotheses through feature steering. Anthropic's paper groups related features into manually curated "supernodes" to present a cleaner narrative; the present work uses automated cross-graph intersection, which is more scalable and less susceptible to confirmation bias but produces a less narratively refined picture of any single circuit. The two approaches are complementary. +**Circuit stability across surface forms.** Perhaps the most theoretically significant finding is the convergence of the feature set to a stable 67-feature core across 50 prompts phrased in four syntactically distinct surface forms (Section 4.6). The initial 180-feature circuit, identified from five prompts sharing the "X is to Y as Z is to" template, contracts rapidly when surface-diverse prompts are introduced, losing roughly half its features by N = 20, but then plateaus, with only 19 further features lost across the subsequent 30 prompts and just 3 in the final step. This two-stage scaling behavior has a clear interpretation: the first contraction eliminates features that were coincidental artifacts of the shared surface template, while the plateau identifies features that activate because of the underlying relational structure, regardless of how that structure is expressed in words. All five analogy-concept features survive the full 50-prompt filter (Table 7). This is strong evidence against a surface-token explanation of the circuit and in favor of a genuine, abstract relational representation inside the model. It is also exactly the property that representation-level work identifies as the signature of genuine conceptual abstraction: an internal representation that remains invariant under changes to the low-level form of the input [11]. -### 4.6 Redundancy as a Property of Well-Learned Computation +**Cross-domain generalization.** The cross-domain generalization finding reinforces this interpretation. The shared circuit, and the stable 67-feature core, includes features that activate for both geographic capital analogies and semantic role analogies. The analogy-concept features at L5 and L8 fire equally for "Paris is to France as Berlin is to" and for "Doctor is to hospital as teacher is to", despite these prompts sharing no surface tokens related to analogy. This is consistent with the behavioral finding of Opiełka et al. [11] that LLMs encode some relational information in a domain-general manner, and constitutes the first identification of specific internal features implementing that domain-generality at the feature level. The slightly larger attribution graphs for semantic role analogies (1,040-1,071 nodes) relative to capital analogies (905-963 nodes) may reflect that semantic roles require broader world-knowledge access rather than retrieval of a discrete, well-encoded fact, an interpretation consistent with the findings of Meng et al. [2] on the compactness of factual storage for geographic entities. -Single-feature ablation of the 180-feature circuit leaves the Berlin prediction intact in 156/180 cases (§3.7.1): necessity is a property of the *set*, not of most individual members, and the few individually necessary features concentrate at the embedding layer. Yet the circuit is collectively decisive — removing it collapses every prompt to the analogy template, while a size- and strength-matched random ablation does not. This combination — individually redundant, collectively necessary — suggests well-learned associations are protected by redundant parallel encodings rather than single critical features. It aligns with the circuit redundancy findings in [12] and may reflect a general property of how transformers distribute robust computations across many partially-redundant components. +**Causal validation.** To confirm that the identified groups actually drive the model's predictions, each feature group was suppressed and the output observed (Sections 4.7.3 and 4.7.4). Structural features are individually critical: removing just the feature tracking "the word 'to'" causes the model to stop completing the analogy and instead repeat the last entity it read, outputting "Berlin" instead of "Germany." Relational features behave differently: removing any single one has no effect because the remaining three compensate, but removing all four simultaneously causes every capital prompt to output "France" and the bird prompt to output "fish." The model stops completing the analogy and falls back to a token from the source pair, direct evidence that the relational features collectively carry the relational transfer step. Prediction confidence and fragility are inversely related: high-confidence prompts have backup paths such that few features are individually critical, while low-confidence prompts have no redundancy and break when almost any single feature is removed. Finally, features whose labels describe formal text rather than analogy appear frequently in the graphs but are causally inert; suppressing them changes nothing, confirming that appearing in the graph does not equal driving the prediction. --- - - -## 5. Limitations +## 6. Limitations -1. **SAE-feature-level intervention only.** Steering operates at the SAE feature level, not the attention head or residual stream level. The causal role of non-SAE circuit components is not assessed. -2. **SAE coverage.** The `gemmascope-transcoder-16k` SAE [7] covers only cross-layer transcoder features. Attention head contributions and residual stream features are not captured. -3. **Threshold sensitivity.** Results are sensitive to node and edge thresholds (0.80/0.85). Lower thresholds would reveal more features; higher thresholds would produce sparser, more focused circuits. -4. **Label quality.** Neuronpedia [8] automated feature explanations are LLM-generated and may not perfectly capture feature semantics. -5. **Prompt set size.** Five defining prompts (plus four held-out) are sufficient for circuit identification and a generalization check but too few to claim statistical robustness. A larger prompt set covering arithmetic, cross-lingual, and abstract relational analogies [13] would strengthen conclusions. -6. **Sufficiency not established.** The steering evidence shows the circuit is necessary and fails in the predicted way, but not that it is *sufficient* to produce the answer in isolation. The single-feature sufficiency probe is largely negative — the highest-influence hub induces the target only when the target entity is already present in the prompt. -7. **Large-magnitude intervention.** Steering at effective −80 per feature is a strong perturbation; the strongest claims rest on the circuit-vs-matched-null contrast and on the strength titration (the effect is already present at effective −8), not on any single surgical ablation. Establishing edge-level mediation (Phase 1 → Phase 2) requires activation/path patching on the model weights, which the steering API does not expose. - -**Future work:** activation patching at the attention head level, replication with benchmark prompt sets, and cross-model comparison. +All steering experiments operate at the SAE feature level only; attention heads and residual stream components are not assessed, so the three feature groups describe the SAE-visible portion of the circuit, not necessarily the complete one. The steering experiments tested ten interpretable core features, not the full circuit: 57 of the 67 core features were never steered, and the ten that were had been selected for interpretability and high influence, which biases them toward mattering. The causal claims therefore establish that these ten features are necessary, not that every core feature is. The prompts were constructed by the authors rather than sampled from a standard benchmark, and no formal statistical tests were applied, meaning the findings are descriptive rather than statistically confirmed. The three capital prompts share the source pair "Paris is to France", so the "France" output under relational suppression (Section 4.7.4) cannot be attributed to retention of the source-pair relation as opposed to a fallback to copying a salient prompt token; varying the source pair while holding the target fixed would resolve this. Feature labels are LLM-generated by Neuronpedia and may not perfectly capture feature semantics. All results are from a single model (Gemma-2-2B) with a single SAE suite, so generalization to other model sizes or architectures is unknown. Future work should pursue activation patching at the attention head level, replication with benchmark prompt sets, and cross-model comparison. --- - - -## 6. Conclusions +## 7. Conclusion -We have identified a shared analogical reasoning circuit in Gemma-2-2B comprising 180 features active across five initial prompts, converging to a stable 67-feature core across 50 prompts phrased in four syntactically distinct surface forms. The circuit is organized into three phases: Phase 1 (L0–L4) parses the structural format of the analogy prompt; Phase 2 (L5–L9) recognises the relational concept itself through features explicitly labeled "analogies" and "analogies or comparisons"; and Phase 3 (L10–L13) integrates the relation with domain-specific knowledge. The same core features activate for both geographic capital analogies and semantic role analogies, confirming a domain-agnostic relational reasoning mechanism rather than separate topic-specific circuits. +We have identified a shared analogical reasoning circuit in Gemma-2-2B comprising 180 features active across five initial prompts, converging to a stable 67-feature core across 50 prompts phrased in four syntactically distinct surface forms. Within this circuit, a small set of individually interpretable features sorts into three functional categories: structural features (concentrated in L0-L4) parse the format of the analogy prompt; relational features (L5-L9) recognize the relational concept itself through features explicitly labeled "analogies" and "analogies or comparisons"; and integration features (L10-L13) combine the relation with domain-specific knowledge. The same core features activate for both geographic capital analogies and semantic role analogies, confirming a domain-agnostic relational reasoning mechanism rather than separate topic-specific circuits. -1. **A stable shared circuit exists, identified by common feature IDs.** 180 features — identified by stable *(layer, feature index)* pairs — appear in all five independently generated attribution graphs. -2. **Dedicated analogy features exist at layers 5, 8, 9, and 13.** These features have Neuronpedia explanations explicitly referencing analogies, comparisons, and relational concepts — providing direct SAE-level evidence of interpretable analogy-concept features in a large language model. -3. **The circuit exhibits a three-phase architecture, identified by label semantics and validated causally.** Circuit template parsing (L0–L4), analogy recognition (L5–L9), and relational integration (L10–L13), with activation magnitude increasing through the sequence. -4. **Cross-domain generalization is confirmed.** The same core features, including L5 SAE#5793 ("analogies"), activate for both geographic and semantic role analogies — a domain-agnostic relational reasoning primitive consistent with behavioral findings [1, 10, 11]. -5. **Phase 2 implements relational transfer, collectively but not individually.** Simultaneous suppression collapses every circuit; capital analogies revert to the source-pair answer. -6. **The validation is robust to the obvious confounds.** Ablating the circuit — but not a matched random control — collapses all five prompts to the analogy template; the effect survives excluding the embedding layer (so it is not mere token deletion), holds from the weakest steering strength tested, and reproduces on held-out analogies the circuit was never derived from. Sufficiency and edge-level mediation remain open (§5). +Causal steering experiments across 159 tests confirm that ten interpretable features drawn from these groups are causally necessary, and that the groups are functionally distinct, each failing in its own characteristic way when suppressed. Removing the structural features causes the model to lose the analogy structure entirely. Removing all four relational features simultaneously causes every capital prompt to output "France" and the bird prompt to output "fish": the model stops completing the analogy and falls back to a token from the source pair, which is direct evidence that the relational features carry the relational transfer step. Prompts with higher prediction confidence tolerate individual feature removal better, suggesting that well-learned predictions are encoded through redundant backup paths while uncertain ones are not. These results provide the first causally validated, feature-level account of analogical reasoning in a large language model. --- @@ -601,7 +628,7 @@ The five Neuronpedia attribution graphs generated for this study are publicly av [10] Lee, T., et al. (2025). The curious case of analogies: Investigating analogical reasoning in large language models. https://arxiv.org/abs/2511.20344 -[11] Wijesiriwardene, T., et al. (2025). Analogical reasoning inside large language models: Concept vectors and the limits of abstraction. https://arxiv.org/abs/2503.03666 +[11] Opiełka, G., Rosenbusch, H., & Stevenson, C.E. (2025). Analogical reasoning inside large language models: Concept vectors and the limits of abstraction. https://arxiv.org/abs/2503.03666 [12] Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N.L., et al. (2025). On the biology of a large language model. *Transformer Circuits Thread*. https://transformer-circuits.pub/2025/attribution-graphs/biology.html @@ -611,6 +638,12 @@ The five Neuronpedia attribution graphs generated for this study are publicly av [15] Marks, S., Rager, C., Michaud, E.J., Belinkov, Y., Bau, D., & Mueller, A. (2024). Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. https://arxiv.org/abs/2403.19647 +[16] Hofstadter, D.R. (1979). *Gödel, Escher, Bach: An Eternal Golden Braid*. New York: Basic Books. + +[17] Mitchell, M. (2020). *Artificial Intelligence: A Guide for Thinking Humans*. New York: Picador. + +[18] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020). Zoom In: An introduction to circuits. *Distill*. https://distill.pub/2020/circuits/zoom-in + --- ## Supplementary Materials @@ -640,3 +673,26 @@ https://kkrampis.github.io/autocircuit/presentation.html note = {Neuronpedia API \texttt{gemmascope-transcoder-16k} SAE analysis} } ``` +## Appendix A: Analogy-Concept Features Across All Five Prompts + +This appendix complements Figures 1 and 2 (Section 4.4) by showing the two most prominent analogy-concept features, L9 #13344 and L8 #13766, as they appear inside the Neuronpedia attribution graph UI for each of the five initial prompts (Table 1). + +### A.1 L9 #13344: "phrases suggesting uncertainty or comparison between two things" + +![13344 - paris is to france as berlin is to](UI-GRAPHS/13344_paris-is-to-france-as-berlin-is-to.png) +![13344 - paris is to france as rome is to](UI-GRAPHS/13344_paris-is-to-france-as-rome-is-to.png) +![13344 - paris is to france as tokyo is to](UI-GRAPHS/13344_paris-is-to-france-as-tokyo-is-to.png) +![13344 - doctor is to hospital as teacher is to](UI-GRAPHS/13344_doctor-is-to-hospital-as-teacher-is-to.png) +![13344 - fish is to water as bird is to](UI-GRAPHS/13344_fish-is-to-water-as-bird-is-to.png) + +*Figure A1: Feature L9 #13344 ("phrases suggesting uncertainty or comparison between two things") shown inside the Neuronpedia attribution graph interface for each of the five initial prompts (top to bottom: `analog_berlin`, `analog_rome`, `analog_tokyo`, `analog_teacher`, `analog_bird`). The feature is active in all five graphs, across both capital and semantic role analogies.* + +### A.2 L8 #13766: "analogies or comparisons" + +![13766 - paris is to france as berlin is to](UI-GRAPHS/13766_paris-is-to-france-as-berlin-is-to.png) +![13766 - paris is to france as rome is to](UI-GRAPHS/13766_paris-is-to-france-as-rome-is-to.png) +![13766 - paris is to france as tokyo is to](UI-GRAPHS/13766_paris-is-to-france-as-tokyo-is-to.png) +![13766 - doctor is to hospital as teacher is to](UI-GRAPHS/13766_doctor-is-to-hospital-as-teacher-is-to.png) +![13766 - fish is to water as bird is to](UI-GRAPHS/13766_fish-is-to-water-as-bird-is-to.png) + +*Figure A2: Feature L8 #13766 ("analogies or comparisons") shown inside the Neuronpedia attribution graph interface for each of the five initial prompts (top to bottom: `analog_berlin`, `analog_rome`, `analog_tokyo`, `analog_teacher`, `analog_bird`). The feature is active in all five graphs, across both capital and semantic role analogies.* From a0c188f8ff6abe8c872d6db4d97a4fbd09ee65c6 Mon Sep 17 00:00:00 2001 From: OlalekanAlagbe Date: Sun, 19 Jul 2026 16:02:10 +0100 Subject: [PATCH 2/2] Remove L11 #15947 row and duplicate L13 #10969 mention in feature tables --- docs/content.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/content.md b/docs/content.md index ddc9bae..337a3b4 100644 --- a/docs/content.md +++ b/docs/content.md @@ -262,7 +262,6 @@ This three-part organization mirrors the emergent symbolic architecture document | L5 #2141 | Relational | "comparisons of people or figures using well-known public figures" | | L8 #13766 | Relational | "analogies or comparisons" | | L9 #13344 | Relational | "phrases suggesting uncertainty or comparison between two things" | -| L11 #15947 | Integration | "references to historical or social change" | | L13 #10969 | Integration | "comparisons between disciplines and relationships between concepts" | The three interactive panels below show representative features from the structural, relational, and integration groups respectively, with the activation patterns that characterize each. @@ -289,7 +288,6 @@ Several of these features carry labels that explicitly name analogy. Table 6 lis | L8 #13766 | "analogies or comparisons" | | L9 #13344 | "comparison between two things" | | L5 #2141 | "comparisons of public figures" | -| L13 #10969 | "comparisons between disciplines" | Figures 1 and 2 show two of these features, L9 #13344 and L8 #13766, active inside the attribution graph interface for the *teacher* prompt, one representative of the five. The complete set across all five prompts is provided in Appendix A.