Cleanup/presentation

.gitattributes

@@ -0,0 +1,8 @@
+# Normalise line endings so results and checksums are identical on every OS.
+* text=auto eol=lf
+# Windows batch files need CRLF (LF-only .bat files can break goto/labels in cmd.exe).
+*.bat text eol=crlf
+# Never touch binary files.
+*.png binary
+*.zip binary
+*.pdf binary

CITATION.cff

@@ -1,5 +1,8 @@
 cff-version: 1.2.0
 message: "If you use this project, please cite it as a software project and include the exact Git commit and model/data configuration used."
 title: "Transformer Internals & Mechanistic Interpretability Lab"
+authors:
+  - alias: "Idnotfound-L"
+repository-code: "https://github.com/Idnotfound-L/transformer-interpretability-lab"
 type: software
 license: MIT

README.md

@@ -1,12 +1,30 @@
 # Transformer Internals & Mechanistic Interpretability Lab
 
-A research-engineering project for inspecting and causally probing decoder-only Transformer language models. The repository combines a PyTorch Transformer implementation, Hugging Face model instrumentation, a controlled sequence-copying study (Protocol V2), ablation and activation-patching experiments, statistical summaries, and an offline Streamlit dashboard.
+[![tests](https://github.com/Idnotfound-L/transformer-interpretability-lab/actions/workflows/tests.yml/badge.svg)](https://github.com/Idnotfound-L/transformer-interpretability-lab/actions/workflows/tests.yml)
+![python](https://img.shields.io/badge/python-3.10%2B-blue)
+![license](https://img.shields.io/badge/license-MIT-green)
+
+**In short:** when a language model continues a repeated sequence (`… A B … A → B`), which of its internal components is it relying on? This project finds candidate attention heads and MLP neurons, then *causally* tests them by switching them off and by patching in activations from a clean run, with matched random controls, held-out prompts, and statistics, on GPT-2 and SmolLM3-3B.
+
+The repository combines a PyTorch Transformer implementation, Hugging Face model instrumentation, a controlled sequence-copying study (Protocol V2), ablation and activation-patching experiments, statistical summaries, and an offline Streamlit dashboard.
 
 **Models studied:** GPT-2 (`gpt2`) and SmolLM3-3B-Base (`HuggingFaceTB/SmolLM3-3B-Base`).
 
-**Primary research question:** Which attention heads and MLP coordinates causally contribute to a controlled repeated-token sequence-copying behaviour, and how robust are these effects across independent seeds and model families?
+**Research question:** Which attention heads and MLP coordinates causally contribute to a controlled repeated-token sequence-copying behaviour, and how robust are these effects across independent seeds and model families?
+
+## Key results
+
+![SmolLM3 Protocol V2 summary: head-ablation effects and activation patching across three seeds](docs/figures/smollm3_v2_summary.png)
+
+*Left: extra copy-margin drop from ablating each layer's discovery-selected head, relative to a matched random head in the same layer (SmolLM3-3B-Base, 3 seeds, 95% bootstrap CIs). Right: fraction of the copy-margin gap recovered by patching clean block states at the earlier successor token (solid) versus a random position (dotted). Layer-35 recovery is zero and is not treated as mechanistic evidence. Regenerate with `python scripts/make_figures.py`.*
 
-> **Research integrity.** The results folder contains full raw V2 output bundles (CSV, JSON, report) for GPT-2 and SmolLM3-3B-Base, seeds 7, 13 and 29, and full raw V1 exploratory bundles kept separately. Headline figures were cross-checked against the raw CSVs. Inference is within-seed; no pooled test across seeds has been run. The SmolLM3 Layer 15 Head 11 result is described as a candidate that causally contributes to the synthetic task, not as a proven complete induction circuit. See [`results/README.md`](results/README.md) and [`reports/FINAL_RESEARCH_REPORT.md`](reports/FINAL_RESEARCH_REPORT.md).
+| Model | Finding | Evidence (3 seeds, BH-adjusted within each run) |
+|---|---|---|
+| SmolLM3-3B-Base | **Layer 15, Head 11** is selected by discovery in every seed. Ablating it hurts copying more than a matched random head and has little effect on the control task. | Extra margin drop 0.29-0.31; copy-minus-control specificity +0.26 to +0.32; q < 0.001 in each seed |
+| GPT-2 | **Layer 3, MLP neuron 1718** is selected in every seed and shows a positive selected-vs-random effect. | Extra margin drop 0.20-0.35; specificity +0.23 to +0.36; q < 0.001 in each seed |
+| SmolLM3-3B-Base | Clean block-state patching at the earlier successor token recovers most of the copy-margin gap in early/middle layers; random positions recover ~0. | Component-specific head/neuron patches recover little, which is consistent with distributed computation but does not distinguish it from redundancy |
+
+These are evidence of a causal contribution to a *synthetic* task, not a proven induction circuit. Baseline copying is partial (held-out accuracy about 0.44-0.46 for SmolLM3 and 0.44-0.54 for GPT-2), and inference is within-seed; no pooled test across seeds has been run. Full tables, caveats and negative results are in [`reports/FINAL_RESEARCH_REPORT.md`](reports/FINAL_RESEARCH_REPORT.md) and [`results/`](results/README.md).
 
 ## What the project implements
 
@@ -15,46 +33,45 @@ A research-engineering project for inspecting and causally probing decoder-only
 - **Causal ablation:** selected attention heads and MLP coordinates are compared against same-layer random heads and activity-matched random neurons on held-out copy prompts.
 - **Task-specificity control:** the same components are evaluated on a repetition-free random-transition control task.
 - **Activation patching:** clean block-state patches at the earlier successor position, random-position controls, and component-specific head/neuron patches. The trivial whole-block final-query patch is omitted.
-- **Statistics:** per-prompt metrics, bootstrap confidence intervals, paired sign-flip tests and Benjamini–Hochberg adjusted q-values.
+- **Statistics:** per-prompt metrics, bootstrap confidence intervals, paired sign-flip tests and Benjamini-Hochberg adjusted q-values.
 - **Dashboard:** saved Protocol V2 runs can be inspected without reloading model weights.
 - **Reproducibility:** seed, model revision, hardware, precision, selected layers, prompt hashes and experiment settings are exported in JSON metadata.
 
 ## Repository layout
 
-```text
+```
 transformer-interpretability-lab/
 ├── src/interpretability_lab/     # model, hooks, experiment logic, Protocol V2
-├── scripts/                     # training, research runners, V2 run indexer
-├── dashboard/                   # TinyGPT, Hugging Face and Protocol V2 dashboards
-├── tests/                       # automated tests
-├── data/                        # small prompt examples/documentation
-├── docs/                        # architecture, method, reproducibility notes
-├── results/                     # available run reports and provenance notes
-├── reports/                     # final research report
-└── artifacts/                   # local experiment outputs (git-ignored by default)
+├── scripts/                      # training, research runners, V2 run indexer, figure script
+│   └── windows/                  # optional .bat wrappers for Windows users
+├── dashboard/                    # TinyGPT, Hugging Face and Protocol V2 dashboards
+├── tests/                        # automated tests
+├── data/                         # small prompt examples/documentation
+├── docs/                         # architecture, protocol, reproducibility notes, figures, provenance
+├── results/                      # raw Protocol V1 (exploratory) and V2 (corrected) run bundles
+├── reports/                      # final research report
+└── artifacts/                    # local experiment outputs (git-ignored by default)
 ```
 
 ## Setup
 
-Python 3.11 is recommended. On Windows:
-
-1. Extract this ZIP to a short path, e.g. `C:\mi\` (helps avoid Windows long-path installation issues).
-2. Run `setup_windows.bat`.
-3. Run `setup_huggingface_windows.bat` for pretrained model experiments.
-4. Verify the project with `\.venv\Scripts\python.exe -m pytest -q`.
-
-On Linux/macOS:
+Python 3.10+ (3.11 recommended).
 
-```bash
+```
+git clone https://github.com/Idnotfound-L/transformer-interpretability-lab.git
+cd transformer-interpretability-lab
 python -m venv .venv
-source .venv/bin/activate
+source .venv/bin/activate        # Windows: .venv\Scripts\activate
 python -m pip install --upgrade pip
 pip install -r requirements.txt
-pip install -r requirements-huggingface.txt
+pip install -r requirements-huggingface.txt   # for pretrained-model experiments
 pip install -e .
+pytest -q
 ```
 
-The test suite currently passes **28 tests** in the packaging environment. Tests exercise software components and synthetic model adapters; passing tests is not a substitute for empirical validation on pretrained checkpoints.
+Windows users can instead run `scripts\windows\setup_windows.bat` and `scripts\windows\setup_huggingface_windows.bat` (see [`scripts/windows/README.md`](scripts/windows/README.md)). Clone to a short path (e.g. `C:\src\`) to avoid Windows long-path installation issues.
+
+The test suite has 28 tests, run in CI on every push. They exercise software components and synthetic model adapters; passing tests are not a substitute for empirical validation on pretrained checkpoints.
 
 ## Run Protocol V2
 
@@ -63,7 +80,7 @@ The test suite currently passes **28 tests** in the packaging environment. Tests
 Windows quick runner (defaults to seed 7):
 
 ```powershell
-.\run_protocol_v2_gpt2_windows.bat 7
+scripts\windows\run_protocol_v2_gpt2_windows.bat 7
 ```
 
 Run individual seeds directly on any OS:
@@ -79,7 +96,7 @@ Repeat with `--seed 13` and `--seed 29`. Keep each timestamped output folder. V2
 
 ### SmolLM3-3B-Base on a cloud GPU
 
-Use a CUDA GPU with sufficient memory (the user's completed runs used an NVIDIA T4 in BF16). Install Transformers 4.53+ and required Hugging Face packages without reinstalling a preconfigured CUDA PyTorch build:
+Use a CUDA GPU with sufficient memory (the reported runs used an NVIDIA T4 in BF16). Install Transformers 4.53+ and required Hugging Face packages without reinstalling a preconfigured CUDA PyTorch build:
 
 ```bash
 pip install "transformers>=4.53.0,<5" accelerate safetensors sentencepiece pandas tabulate
@@ -125,7 +142,7 @@ After running experiments, launch the Protocol V2 offline dashboard:
 streamlit run dashboard/protocol_v2_dashboard.py
 ```
 
-On Windows, double-click `run_protocol_v2_dashboard_windows.bat`. Set the dashboard run directory to the folder containing the timestamped run folders if it is not the default `artifacts/sequence_copy_v2`.
+On Windows, double-click `scripts\windows\run_protocol_v2_dashboard_windows.bat`. Set the dashboard run directory to the folder containing the timestamped run folders if it is not the default `artifacts/sequence_copy_v2`.
 
 The dashboard reads saved CSV/JSON outputs only. It does not download model weights or rerun experiments.
 
@@ -139,24 +156,30 @@ python scripts/summarize_v2_runs.py artifacts/sequence_copy_v2
 
 For SmolLM3, point the utility at the persistent parent directory containing all three seed folders. It writes `run_index.csv`, per-seed tables and a descriptive component-consistency table. The utility does not pool p-values or replace a prompt-level meta-analysis.
 
-## Research findings and limitations
-
-See:
-- [`reports/FINAL_RESEARCH_REPORT.md`](reports/FINAL_RESEARCH_REPORT.md)
-- [`results/README.md`](results/README.md)
-- [`results/smollm3_protocol_v2/THREE_SEED_FINDINGS_AUTHOR_SUMMARY.md`](results/smollm3_protocol_v2/THREE_SEED_FINDINGS_AUTHOR_SUMMARY.md)
-
-Key limitations to communicate in interviews:
+## Limitations
 
 1. The probe is a synthetic sequence-copying task, not a natural-language benchmark.
 2. SmolLM3's baseline performance is partial; negative copy margins and low target probability constrain mechanistic claims.
 3. Attention weights are candidate-discovery signals, not functional proof.
 4. Random-transition accuracy is at 0%, so the control may have a floor effect.
 5. Only eight of 36 layers and one top candidate per layer were screened in the reported protocol.
-6. Component-specific patching was weak, and the block-state patching set was small.
-7. Ablation can be off-manifold; avoid claiming that one component is a complete circuit.
+6. Component-specific patching was weak, and the block-state patching set was small (12 prompts per seed).
+7. Ablation can be off-manifold; no single component should be read as a complete circuit.
 8. Cross-model layer/head indices are not directly comparable.
+9. GPT-2 V2 runs used CPU/float32 and did not record a resolved model revision, so exact GPT-2 weights are not pinned.
+
+## Further reading
+
+- [`reports/FINAL_RESEARCH_REPORT.md`](reports/FINAL_RESEARCH_REPORT.md): full write-up, including negative results
+- [`results/README.md`](results/README.md): what each results folder contains (V1 exploratory vs V2 corrected; never pool them)
+- [`results/protocol_v2/smollm3/THREE_SEED_FINDINGS_AUTHOR_SUMMARY.md`](results/protocol_v2/smollm3/THREE_SEED_FINDINGS_AUTHOR_SUMMARY.md): consolidated SmolLM3 three-seed summary
+- [`docs/`](docs/): architecture, experiment protocol, protocol changelog, reproducibility checklist
+- [`docs/provenance/`](docs/provenance/): archive manifest, results inventory and SHA-256 checksums
+
+## Citation
+
+See [`CITATION.cff`](CITATION.cff); please include the exact commit and model/data configuration used.
 
-## CV-ready wording
+## License
 
-Use the evidence-backed entry in [`CV_ADDITION.txt`](CV_ADDITION.txt). The strongest result is worded as a reproducible causal contribution on a synthetic task, not as discovery of a complete induction circuit.
+MIT, see [`LICENSE`](LICENSE).

docs/HUGGINGFACE_RESEARCH.md

@@ -90,4 +90,4 @@ A 3B checkpoint in BF16/FP16 requires approximately 6 GB for weights alone; runt
 - Report prompt-level observations, confidence intervals and null effects.
 - Attention is descriptive; combine it with intervention and patching evidence.
 - Ablation can push activations off the model's normal distribution. Triangulate with patching, matched controls and alternative prompts.
-- Do not write a CV claim about a discovered circuit or a measured effect until you have inspected and understood the saved output files.
+- Do not claim a discovered circuit or a measured effect until you have inspected and understood the saved output files.

docs/PROJECT_SCOPE.md

@@ -0,0 +1,27 @@
+# Project scope
+
+## What the repository implements
+
+- A decoder-only GPT-style Transformer in PyTorch (tied embeddings, causal self-attention, residual blocks, MLPs) with a small training pipeline.
+- Forward-hook instrumentation that captures attention patterns, residual-stream states and MLP activations.
+- Interactive Streamlit views for next-token predictions, attention maps, residual representations, neuron activity and causal interventions.
+- Attention-head and MLP-neuron ablation, multi-prompt component sweeps, KL / prediction-change metrics and prompt-level bootstrap intervals.
+- Position-aligned residual activation patching with clean / corrupted baselines and a corrupted-self-patch control.
+- An optional GPT-2 Small + TransformerLens CLI pipeline for full head sweeps, activity-selected neuron ablations, repeated-token probes and patching.
+- An architecture-aware Hugging Face adapter for decoder-only causal LMs (GPT-2-style and Llama-like stacks, default SmolLM3-3B-Base) with GQA-aware query-head ablation through the attention output projection, gated MLP-coordinate ablation through the down projection, layer-wise block-output patching and cross-model aggregate comparisons. See [`HUGGINGFACE_RESEARCH.md`](HUGGINGFACE_RESEARCH.md).
+- Per-run metadata capture: seed, model revision, hardware, precision, selected layers and prompt hashes.
+
+## Primary study: controlled sequence copying
+
+The main research track is the Protocol V2 sequence-copying study ([`SEQUENCE_COPY_STUDY.md`](SEQUENCE_COPY_STUDY.md), [`PROTOCOL_V2_CHANGELOG.md`](PROTOCOL_V2_CHANGELOG.md)), implemented in `src/interpretability_lab/sequence_copy_v2.py` and `scripts/run_sequence_copy_study_v2.py`.
+
+It builds fixed-length repeated-token prompts, gives discovery and held-out evaluation cases disjoint token pools, ranks attention heads and MLP coordinates on discovery data, and ablates them against same-layer random controls on held-out prompts. Paired candidate-versus-control effects are reported with bootstrap confidence intervals, sign-flip permutation tests and Benjamini–Hochberg q-values. Activation patching corrupts the earlier successor token and restores clean block states at that position.
+
+The study is designed to be falsifiable: if baseline copy accuracy is poor, or discovery-ranked components do not beat random controls, the reports say so. Synthetic copying is a narrow behavioural probe and cannot establish a general semantic role for a head. Results for GPT-2 and SmolLM3-3B-Base are interpreted separately; layer/head/neuron indices are not comparable across architectures.
+
+## Out of scope / future work
+
+- Natural-language induction benchmarks with a difficulty-matched control task.
+- Screening more than one candidate per layer, and more than 8 of the model's layers.
+- Combined ablations and path patching from candidate heads (e.g. SmolLM3 L15 H11) to downstream components.
+- More patching prompts, and a pre-specified prompt-level meta-analysis across seeds.