Transformer from Scratch (in pure C)

A 17-Part Masterclass in Systems Engineering & Deep Learning

No PyTorch. No NumPy. No BLAS. No Python.
Just pointers, memory allocators, and raw mathematics.

---

The Vision

Building a Neural Machine Translation (NMT) Transformer in Python takes 200 lines of code. Building it in pure C99 requires inventing a tensor library, writing an autograd topological sorter, managing raw heap memory across millions of parameters, and manually deriving the chain rule for matrix calculus.

This repository is the definitive documentation hub for the Transformer from Scratch ecosystem.

Rather than a single, monolithic script, this system was engineered as 17 strictly-scoped, distinct repositories, mapping identically to modern layered software architecture. It is designed to prove that you can build a production-grade AI training pipeline relying on absolutely nothing but the standard C library.

---

Key Engineering Triumphs

• Dynamic Autograd Engine: Implemented a PyTorch-style Directed Acyclic Graph (DAG) that topologically sorts and recursively executes chain-rule closures.
• Absolute Memory Safety: Gated by rigorous Valgrind CI. Features brutal ownership contracts, pointer poisoning on free, and explicit calloc zero-page requests to prevent silent float corruption.
• Mathematical Rigor: Every backward pass is mathematically proven via central finite-difference gradient checkers ($\epsilon = 1e-4$) before being merged.
• Cache-Aware Linear Algebra: Manual $O(N^3)$ loops rewritten with register-local accumulators and contiguous row-major strides to prevent L1 cache thrashing.
• Multiplication-Free PRNG: Built a custom xorshift32 generator to execute Xavier/Glorot initialization without linking external cryptography libraries.

---

The 17-Layer Architecture

This project strictly adheres to the Single Responsibility Principle. Layer 17 (machine-translation) knows nothing about backpropagation; it merely triggers the pipeline. Layer 1 (mini-tensor) knows nothing about Neural Networks; it just multiplies floats.

graph TD
    classDef math fill:#2ecc71,stroke:#27ae60,color:#fff,stroke-width:2px;
    classDef nlp fill:#3498db,stroke:#2980b9,color:#fff,stroke-width:2px;
    classDef attn fill:#9b59b6,stroke:#8e44ad,color:#fff,stroke-width:2px;
    classDef arch fill:#f1c40f,stroke:#f39c12,color:#000,stroke-width:2px;
    classDef train fill:#e67e22,stroke:#d35400,color:#fff,stroke-width:2px;
    classDef app fill:#e74c3c,stroke:#c0392b,color:#fff,stroke-width:2px;

    subgraph "Phase 6: Application"
        P17[17. Machine Translation]:::app
    end

    subgraph "Phase 5: Dynamic Training Engine"
        P16[16. Training Pipeline]:::train
        P15[15. Autograd Tensor]:::train
        P14[14. Autograd Engine DAG]:::train
    end

    subgraph "Phase 4: Transformer Composition"
        P13[13. Transformer]:::arch
        P12[12. Decoder Layer]:::arch
        P11[11. Encoder Stack]:::arch
        P10[10. Encoder Layer]:::arch
    end

    subgraph "Phase 3: The Attention Mechanism"
        P09[09. LayerNorm]:::attn
        P08[08. Feed Forward]:::attn
        P07[07. Multi-Head Attention]:::attn
        P06[06. Core Attention]:::attn
    end

    subgraph "Phase 2: NLP Encodings"
        P05[05. Positional Encoding]:::nlp
        P04[04. Word Embeddings]:::nlp
    end

    subgraph "Phase 1: Mathematical Foundation"
        P03[03. Softmax Classifier]:::math
        P02[02. Neural Network Base]:::math
        P01[01. Mini Tensor Library]:::math
    end

    P17 --> P16
    P16 --> P15
    P15 --> P14
    P14 --> P13
    P13 -.-> P12 & P11
    P12 & P11 -.-> P10 & P09 & P08 & P07
    P07 -.-> P06
    P06 -.-> P05 & P04
    P05 & P04 -.-> P03 & P02
    P03 & P02 -.-> P01

Loading

---

Deep-Dive Documentation Hub

The true value of this ecosystem lies in the brutal engineering realities exposed during construction. I have extracted the architectural decision records, mathematical proofs, and C-code optimizations from all 17 repositories into this central hub.

Explore the Ecosystem

Document	Description
The 17 Projects Portfolio	Start here. Read the overwhelming technical summaries for all 17 implementations.
System Architecture	A breakdown of how the 6 macro-layers interface via opaque pointers and callbacks.
Engineering Decisions (ADRs)	Why we use flat row-major arrays, max-subtraction Softmax, and Lazy Gradients.
Lessons Learned	The brutal realities of Cache Thrashing, Kahan Summation, and DAG memory management.
Timeline & Roadmap	The chronological dependency graph that governed the build order.

The 17 Projects

Project	Documentation	Repository
01. Mini Tensor	01-mini-tensor.md	mini-tensor
02. Neural Network	02-neural-network.md	neural-net
03. Softmax Classifier	03-softmax-classifier.md	softmax-classifier
04. Word Embeddings	04-word-embeddings.md	word-embeddings
05. Positional Encoding	05-positional-encoding.md	positional-encoding
06. Attention	06-attention.md	attention
07. Multi-Head Attention	07-multi-head-attention.md	multi-head-attention
08. Feed Forward	08-feed-forward.md	feed-forward
09. LayerNorm	09-layernorm.md	layernorm
10. Encoder Layer	10-encoder-layer.md	encoder-layer
11. Encoder Stack	11-encoder-stack.md	encoder-stack
12. Decoder Layer	12-decoder-layer.md	decoder-layer
13. Transformer	13-transformer.md	transformer
14. Autograd Engine	14-autograd-engine.md	autograd-engine
15. Autograd Tensor	15-autograd-tensor.md	autograd-tensor
16. Training Pipeline	16-training-pipeline.md	training-pipeline
17. Machine Translation	17-machine-translation.md	machine-translation

---

A Taste of the Code: The Chain Rule in C

High level languages obscure the calculus. In this framework, the math is explicit. Here is how we implemented the exact derivative of a Linear Layer ($Y = XW + b$) during the backward pass in Project 02:

/* dL/dW = Xᵀ · dL/dY */
Matrix *x_T    = matrix_transpose(layer->x);
Matrix *grad_W = matrix_matmul(x_T, grad_output);
matrix_free(x_T);

/* dL/db = Row-Sum of dL/dY */
Matrix *grad_b = matrix_create(1, layer->out_features);
for (size_t i = 0; i < grad_output->rows; ++i)
    for (size_t j = 0; j < grad_output->cols; ++j)
        grad_b->data[j] += grad_output->data[i * grad_output->cols + j];

/* dL/dX = dL/dY · Wᵀ */
Matrix *W_T    = matrix_transpose(layer->W->data);
Matrix *grad_x = matrix_matmul(grad_output, W_T);
matrix_free(W_T);

---

"Architecture is about understanding the boundaries between components."
Welcome to the Foundation.

Engineering Design © 2026 Shahid Ul Islam.
Built with passion for Mathematical Rigor and Technical Excellence.