Skip to content

11-encoder-stack.md

Latest commit

 

History

History
153 lines (107 loc) · 6.86 KB

11-encoder-stack.md

File metadata and controls

153 lines (107 loc) · 6.86 KB
Raw

Project 11: Encoder Stack

Explore the Encoder Stack Implementation on GitHub

Project Metrics

File Lines Bytes Role
include/encoder_stack.h 92 2.9 KB Public API — defines the N-layer stack struct
src/encoder_stack.c 134 4.1 KB Dynamic allocation and strict O(1) memory pipeline
src/demo.c 74 2.0 KB Visual pipeline checkpoint tracer
tests/test_encoder_stack.c 231 7.0 KB 11-test suite (unique parameters, single-layer equivalence)
Makefile 108 3.1 KB Complex linking of 7 previous standalone modules
README.md — — API reference and architecture visualizations
Analysis.md — — This file — design rationale and roadmap readiness
Total (new code) ~639 ~19.1 KB Transformer Encoder Model Integration

Reused from prior projects (100% of internal logic):

  • matrix_* (Project 01: mini-tensor)
  • encoder_layer_* (Project 10: encoder-layer)

Test result: 11 / 11 PASS — zero compiler warnings under -Wall -Wextra -Werror -pedantic

Architecture Decision Record

ADR-1: Layer Independence

Decision: Every encoder layer within the stack owns completely independent parameter allocations. Weights are intentionally not shared across layers.

Rationale: This directly mirrors the original Attention Is All You Need architecture. Each layer is designed to learn different, increasingly complex representations. A shallow copy (pointer sharing) would violate the mathematical capacity of the deep network, forcing it to act as a recurrent neural network unrolled in time rather than a true deep transformer.

ADR-2: Sequential Composition

Decision: The encoder stack is intentionally implemented as a sequential, iterative pipeline (a simple for loop) rather than a recursive structure or a complex computation graph.

Rationale:

  • Deterministic execution: We process layer $i$ fully before beginning layer $i+1$.
  • Simpler ownership: Makes the $O(1)$ intermediate matrix memory model trivial to implement.
  • Easier debugging: Facilitates clean checkpoints and introspection via utilities like encoder_stack_get_layer().
  • Mirrors the paper: The math defines it simply as $H_{i+1} = \text{EncoderBlock}(H_i)$.

ADR-3: Fixed Stack Depth

Decision: The stack depth ($N$) is fixed exactly at construction time via encoder_stack_create(num_layers, ...). Layers are never dynamically inserted, removed, or re-ordered post-allocation.

Rationale:

  • Mirrors the paper: The original architecture does not dynamically alter its depth.
  • Deterministic ownership: Simplifies parameter lifecycle tracking.
  • Memory safety: Avoids the immense latency and fragmentation overhead of realloc-ing the heavy structural arrays.
  • Simpler API: Ensures the module remains conceptually simple and easy to reason about.

Architectural Note: Iterative Refinement

The primary conceptual purpose of stacking multiple EncoderLayers is the iterative refinement of token representations.

Because encoder-stack explicitly ensures independent parameters (ADR-1), the model can learn a hierarchy of features:

  1. Layer 1 generally learns highly local, syntactic relationships (e.g., immediate adjacencies or part-of-speech dependencies).
  2. Layer $N/2$ begins resolving longer-range dependencies, disambiguating pronouns, and merging context.
  3. Layer $N$ outputs highly robust, deeply contextualized semantic representations representing the full meaning of the sentence.

Each layer linearly projects, non-linearly transforms, and normalizes the representation produced by the previous one. The stack is what elevates the network from a basic sequence mixer into a powerful language model.

Test Suite Analysis

Total: 11 structural and behavioral tests

Test What is validated
test_stack_create Struct and dynamic num_layers allocation
test_layer_count Verifies encoder_stack_depth API
test_forward_shape End-to-end dimensional integrity over depth
test_parameter_count Deterministic parameter verification ($840 \times 6 = 5040$)
test_invalid_configuration Rejects zero values and invalid dimensions
test_unique_layers Layers are independently allocated pointers
test_independent_parameters W matrices in layers $0$ and $1$ are mathematically distinct
test_component_reuse Ensures all internal components are valid
test_sequence_preserved Sequence $7 \rightarrow 7$ preserved across layers
test_model_dimension_preserved Features $16 \rightarrow 16$ preserved across layers
test_single_layer_equivalence $N=1$ stack evaluates identical to a raw EncoderLayer

Key Feature Test: test_single_layer_equivalence By seeding the internal pseudo-random generator via rng_seed(42) before creating a raw EncoderLayer and an EncoderStack (with $N=1$), we guarantee identical Xavier weight initializations. Comparing their outputs element-by-element mathematically proves that the encoder_stack_forward orchestrator injects exactly zero invisible mathematical overhead.

Demo Output Analysis

Encoder Layer 1/6 complete
  Shape      : 3x8
  Parameters : 840
...
Encoder Layer 6/6 complete
  Shape      : 3x8
  Parameters : 840

The demo traces a token sequence through all 6 layers of the architecture. Instead of dumping enormous multi-dimensional matrices, it tracks the shape and parameter count of the representation as it passes through each checkpoint. This proves the iterative refinement pipeline completes successfully without dimensionality collapse or memory corruption.

Part 11 Audit — Readiness for Part 12

Does encoder-stack fulfill all requirements to unblock the rest of the Transformer?

Requirement Status
Deep Model Integration ✅ Dynamically scales to arbitrary depth $N$
Clean parameter lifecycle ✅ Safe cascade deallocation through $N$ sub-layers
No state leakage ✅ $O(1)$ memory model frees volatile matrices immediately

Status: Verified and complete. Project 11 clears the path for Project 12 (Decoder Layer).

Full Dependency Map

We have successfully transitioned from building a Block to building an Encoder Model.

Embedding
   │
   ▼
Positional Encoding
   │
   ▼
Encoder Stack  ← WE ARE HERE
   │
   ├── Encoder Layer
          │
          ├── Multi-Head Attention
          ├── Feed Forward
          ├── LayerNorm
          └── Residual

With the Encoder branch complete, the next architectural milestone (Project 12) shifts focus to the Decoder branch, where the architecture finally breaks symmetry.

➡️ Next Project

Proceed to Project 12: Decoder Layer