attention — Single-Head Scaled Dot-Product Attention (Pure C)

Part 6 of: Transformer from Scratch in Pure C

Goal: Implement the full "Attention is All You Need" paper in pure C, building up from primitives to a complete machine translation pipeline.

attention implements the core innovation of the Transformer: scaled dot-product attention. It computes context vectors by letting each token dynamically decide which other tokens are important, without using recurrence or convolutions. Built on top of mini-tensor.

---

Highlights

• Pure C (C99), depends only on mini-tensor
• Faithful to the paper — uses continuous, learned linear projections (Parameter), completely independent from the neural network module's Linear layer
• Clean mathematical separation — internal static helpers handle queries, keys, values, scoring, scaling, and softmax without exposing internal state
• Forward propagation only — cleanly separates the forward algorithm from the immense complexity of attention backpropagation
• Zero parameter bloat — no output projection (W_O) yet, teaching the pure attention mechanics before multi-head concatenation
• 6-test suite with deep identity validation (Scores approaching Identity Matrix) and analytical scaling verification
• Built on mini-tensor ← | Foundation for multi-head-attention →

---

Architecture Context

Where this project fits in the complete Transformer pipeline:

Token
  │
Embedding
  │
Positional Encoding
  │
Attention              ← This project (Part 06)
  │
Multi-Head Attention
  │
LayerNorm
  │
Feed Forward
  │
Encoder Block

---

The problem this solves

Previous modules transformed discrete tokens into continuous embeddings (Part 4) and injected position awareness (Part 5). However, these representations are static—the embedding for "bank" is identical whether the context is "river bank" or "bank account".

Attention solves this context problem. It provides a mechanism for tokens to exchange information:

1. Each token formulates a Query ("What information do I need?").
2. Each token advertises a Key ("What information do I contain?").
3. Each token holds a Value ("What information will I pass along?").

By computing the dot product of every Query against every Key, the network dynamically assigns a Weight to every relationship, producing a contextualized output representation.

---

What attention provides

A single public header (include/attention.h) exposes:

• Attention — holds the three learned projection matrices (W_q, W_k, W_v)
• attention_create() — allocates the struct and initializes the projections via Xavier
• attention_forward() — computes the full scaled dot-product attention
• attention_free() — releases all memory

---

How this completes the Transformer core mechanism

Position-aware inputs (from Part 5)
    │
    ├─► X × W_Q ─► Queries (Q) ─┐
    │                           │
    ├─► X × W_K ─► Keys (K) ────┴─► Scores (Q·Kᵀ)
    │                                 │
    │                                 ▼
    │                           Scale (1/√d_k)
    │                                 │
    │                                 ▼
    │                           Weights (Softmax)
    │                                 │
    ├─► X × W_V ─► Values (V) ────────┘
                                      │
                                      ▼
                                Context Output

---

How this unblocks the Transformer roadmap

Future Part	What this project provides
Part 7 — Multi-Head Attention	Multiple copies of this exact logic executed in parallel, concatenated, and projected through W_O
Part 10 — Encoder Layer	Multi-Head Attention wrapped with residual connections and LayerNorm
Part 12 — Decoder Layer	Adds Masked Attention (preventing looking into the future) and Cross Attention

---

The Formula

The paper defines scaled dot-product attention as:

Attention(Q, K, V) = softmax( (Q Kᵀ) / √d_k ) V

Where:

• Q = X W_Q
• K = X W_K
• V = X W_V

---

Tensor Shapes

Matrix	Shape
Input X	seq_len × d_model
WQ, WK, WV	d_model × d_k
Q, K, V	seq_len × d_k
Scores, Weights	seq_len × seq_len
Output	seq_len × d_k

---

Directory structure

attention/
├── include/
│   └── attention.h          # Public API (single include)
├── src/
│   ├── attention.c          # attention_create, forward, free, static helpers
│   └── demo.c               # demo printing step-by-step intermediate matrices
├── tests/
│   └── test_attention.c     # 6-test suite — make test
├── Makefile
├── README.md
└── Analysis.md

---

Build & Run

Requirements

• GCC (C99 compatible)
• GNU Make
• libm (math library)
• mini_tensor source at ../mini-tensor/

Build

make

Run tests

make test

Run demo

make demo

Memory checks (valgrind)

make valgrind

Clean

make clean

---

Design decisions

Use raw Parameter* instead of Linear layer

While a multi-layer perceptron's Linear layer (with bias vectors and broadcasting) works mathematically, Queries, Keys, and Values are purely learned linear projections. The original paper does not define them as neural networks. Utilizing raw Parameter* matrices aligns strictly with the conceptual model of the paper and eliminates unnecessary bias addition overhead.

Exclude output projection (W_O)

The output projection naturally belongs to Multi-Head Attention (Part 07). Incorporating it here teaches the wrong abstraction. Single-head attention's final state is simply the context vector (Weights × V).

Hide internal mathematical stages

Computing queries, computing scores, and scaling are isolated into distinct helper functions to enhance mathematical clarity and debugging. However, they remain static to maintain an intentionally tiny public API.

---

API reference

typedef struct {
    Parameter *W_q;
    Parameter *W_k;
    Parameter *W_v;
    size_t d_model;
    size_t d_k;
} Attention;

Attention* attention_create(size_t d_model, size_t d_k);
Matrix*    attention_forward(Attention *attn, const Matrix *input);
void       attention_free(Attention *attn);

---

Example usage

#include "attention.h"

int main(void) {
    /* Create Attention Layer */
    size_t d_model = 512;
    size_t d_k = 64;
    Attention *attn = attention_create(d_model, d_k);

    /* Construct input matrix (seq_len=10, d_model=512) */
    Matrix *input = /* ... position-aware embeddings ... */ NULL;
    
    /* Forward pass through attention mechanism */
    Matrix *output = attention_forward(attn, input);   /* (10 × 64) */

    matrix_free(output);
    attention_free(attn);
    return 0;
}

---

Test suite

=== Attention Test Suite ===
Running test_shape...
  Pass.
Running test_attention_score_shape...
  Pass.
Running test_softmax_rows_sum_to_one...
  Pass.
Running test_analytical_scaling...
  Pass.
Running test_identity_case...
  Pass.
Running test_attention_weights_identity...
  Pass.
All tests passed!

Test	What it validates
test_shape	(3 × 8) goes in, (3 × 8) comes out
test_attention_score_shape	Scores matrix is precisely (3 × 3) given seq_len=3
test_softmax_rows_sum_to_one	Ensures probability distribution normalization across seq_len
test_analytical_scaling	Computes scaling on a known minimal matrix vs analytical derivation
test_identity_case	Force-sets W_q = W_k to generate symmetric Scores == Scores^T
test_attention_weights_identity	Evaluates deterministic Softmax forcing an identity matrix, proving Output ≈ V

---

What's explicitly NOT in this project

Missing feature	Why	Where it belongs
Multi-Head implementation	Introduces concatenation complexity; better kept separate	Part 7 — Multi-Head Attention
Backpropagation	Highly complex derivatives via Jacobian matrices	Future Part
Masking	Only needed for the Decoder	Part 12 — Decoder Layer

---

Engineering notes

• The project compiles cleanly under strict GCC flags (-Wall -Wextra -Werror -pedantic).
• Only mini-tensor sources are compiled. It deliberately decouples from neural-net to prove attention is a primitive mathematical operation.

---

Engineering Design © 2026 Shahid Ul Islam.
Built with passion for Mathematical Rigour and Technical Excellence.