Skip to content

CODE-INDEX.md

Latest commit

 

History

History
353 lines (257 loc) · 9.53 KB

CODE-INDEX.md

File metadata and controls

353 lines (257 loc) · 9.53 KB
Raw

Code Index

Quick Reference for All Code Examples

Complete index of code examples organized by chapter and topic.

Organization

All code is located in /Users/admin/ps/ML/llm/book-trm/code/

Structure:

code/
├── ch01-neural-networks/
├── ch02-backpropagation/
├── ch03-embeddings/
...
├── ch25-capstone/
├── utils/          # Shared utilities
└── tests/          # Test suites

Part I: Foundations

Chapter 1: Neural Networks

Location: code/ch01-neural-networks/

  • basic_neuron.py - Single neuron implementation
  • mlp.py - Multi-layer perceptron
  • mnist_classifier.py - MNIST digit classifier
  • activations.py - Activation functions (ReLU, sigmoid, tanh)
  • forward_prop.py - Forward propagation examples

Chapter 2: Backpropagation

Location: code/ch02-backpropagation/

  • manual_backprop.py - Backpropagation from scratch
  • optimizers.py - SGD, momentum, Adam implementations
  • training_loop.py - Complete training pipeline
  • gradient_checking.py - Numerical gradient verification
  • loss_functions.py - MSE, cross-entropy

Chapter 3: Word Embeddings

Location: code/ch03-embeddings/

  • tokenizer.py - Basic tokenization
  • word2vec.py - Skip-gram and CBOW
  • embedding_vis.py - t-SNE visualization
  • semantic_similarity.py - Word analogy tasks

Chapter 4: Sequence Modeling

Location: code/ch04-sequences/

  • simple_rnn.py - Vanilla RNN
  • lstm_cell.py - LSTM implementation
  • gru_cell.py - GRU implementation
  • char_rnn.py - Character-level language model

Chapter 5: Attention

Location: code/ch05-attention/

  • attention_basic.py - Scaled dot-product attention
  • multi_head_attention.py - Multi-head attention
  • attention_viz.py - Attention weight visualization
  • seq2seq_attention.py - Seq2seq with attention

Chapter 6: Transformer

Location: code/ch06-transformer/

  • transformer_block.py - Complete transformer block
  • positional_encoding.py - Sinusoidal and learned
  • layer_norm.py - Layer normalization
  • full_transformer.py - Complete implementation
  • mini_gpt.py - Minimal GPT-style model

Part II: Core TRM

Chapter 7: TRM Introduction

Location: code/ch07-trm-intro/

  • minimal_trm.py - Simplest TRM implementation
  • parameter_counting.py - Parameter analysis utilities
  • trm_skeleton.py - TRM architecture skeleton
  • efficiency_benchmark.py - Speed/memory benchmarks

Chapter 8: Recursive Layers

Location: code/ch08-recursive-layers/

  • recursive_attention.py - Recursive self-attention layer
  • weight_sharing.py - Parameter sharing strategies
  • depth_recursion.py - Variable recursion depth
  • cache_optimization.py - Computation reuse

Chapter 9: Tokenization

Location: code/ch09-tokenization/

  • bpe_tokenizer.py - Byte-pair encoding
  • vocabulary_optimizer.py - Vocab size optimization
  • embedding_layer.py - Embedding initialization
  • output_head.py - Output projection variants

Chapter 10: Training

Location: code/ch10-training/

  • trm_trainer.py - TRM training loop
  • lr_schedulers.py - Learning rate schedules
  • data_pipeline.py - Efficient data loading
  • training_utils.py - Training utilities

Chapter 11: Loss Functions

Location: code/ch11-objectives/

  • mlm_loss.py - Masked language modeling
  • causal_lm_loss.py - Causal language modeling
  • perplexity.py - Perplexity calculation
  • custom_objectives.py - Custom loss functions

Chapter 12: Generation

Location: code/ch12-generation/

  • generation_loop.py - Text generation pipeline
  • sampling.py - Temperature, top-k, nucleus
  • beam_search.py - Beam search implementation
  • kv_cache.py - KV-cache optimization

Part III: Advanced

Chapter 13: Context Length

Location: code/ch13-context/

  • rope.py - Rotary position embeddings
  • alibi.py - ALiBi position bias
  • sliding_window.py - Sliding window attention
  • flash_attention.py - Flash attention integration

Chapter 14: Multi-Task

Location: code/ch14-multitask/

  • multitask_trainer.py - Multi-task training
  • task_embeddings.py - Task conditioning
  • gradient_balancing.py - Loss balancing
  • curriculum.py - Curriculum learning

Chapter 15: Distillation

Location: code/ch15-distillation/

  • distillation_loss.py - Distillation losses
  • teacher_student.py - Teacher-student training
  • attention_distill.py - Attention transfer
  • online_distill.py - Online distillation

Chapter 16: Fine-Tuning

Location: code/ch16-finetuning/

  • lora.py - LoRA implementation
  • adapters.py - Adapter layers
  • prefix_tuning.py - Prefix tuning
  • peft_utils.py - PEFT utilities

Chapter 17: Compression

Location: code/ch17-compression/

  • quantization.py - INT8/INT4 quantization
  • qat.py - Quantization-aware training
  • pruning.py - Magnitude and structured pruning
  • mixed_precision.py - Mixed precision inference

Part IV: Extensions

Chapter 18: Small Transformers

Location: code/ch18-transformers/

  • compact_transformer.py - Efficient transformer variants
  • parameter_sharing.py - Transformer weight sharing
  • architecture_search.py - NAS for small models
  • benchmarks.py - Performance comparisons

Chapter 19: Hybrid Architectures

Location: code/ch19-hybrid/

  • memory_augmented.py - External memory integration
  • retrieval_augmented.py - RAG implementation
  • knowledge_graph.py - KG integration
  • hybrid_attention.py - Hybrid attention mechanisms

Chapter 20: Specialized

Location: code/ch20-specialized/

  • domain_tokenizer.py - Domain-specific tokenization
  • domain_adaptation.py - Domain transfer
  • medical_trm.py - Medical domain example
  • code_trm.py - Code generation example

Chapter 21: Multilingual

Location: code/ch21-multilingual/

  • multilingual_tokenizer.py - Multilingual tokenization
  • language_adapters.py - Language-specific adapters
  • cross_lingual.py - Cross-lingual transfer
  • zero_shot_transfer.py - Zero-shot evaluation

Chapter 22: Emerging Architectures

Location: code/ch22-emerging/

  • mamba_layer.py - Mamba implementation
  • rwkv_layer.py - RWKV implementation
  • retention.py - Retentive network
  • architecture_comparison.py - Comparative analysis

Part V: Production

Chapter 23: Deployment

Location: code/ch23-deployment/

  • fastapi_server.py - REST API serving
  • onnx_export.py - ONNX conversion
  • tensorrt_optimize.py - TensorRT optimization
  • tflite_convert.py - Mobile deployment

Chapter 24: Evaluation

Location: code/ch24-evaluation/

  • benchmark_suite.py - GLUE/SuperGLUE evaluation
  • custom_metrics.py - Custom metrics
  • profiling.py - Performance profiling
  • bias_detection.py - Fairness evaluation

Chapter 25: Capstone

Location: code/ch25-capstone/

  • project_template/ - Complete project structure
  • training_config.py - Configuration management
  • deployment_scripts/ - Deployment automation
  • monitoring.py - Production monitoring

Utilities

Shared Utilities

Location: code/utils/

  • data_utils.py - Data loading and preprocessing
  • model_utils.py - Model helpers
  • training_utils.py - Training helpers
  • eval_utils.py - Evaluation helpers
  • viz_utils.py - Visualization utilities

Testing

Location: code/tests/

  • test_attention.py - Attention mechanism tests
  • test_trm.py - TRM implementation tests
  • test_training.py - Training pipeline tests
  • test_generation.py - Generation tests

Quick Links by Topic

Attention Mechanisms

  • Basic attention: ch05-attention/attention_basic.py
  • Multi-head: ch05-attention/multi_head_attention.py
  • Recursive: ch08-recursive-layers/recursive_attention.py
  • Flash attention: ch13-context/flash_attention.py

Training

  • Basic loop: ch02-backpropagation/training_loop.py
  • TRM training: ch10-training/trm_trainer.py
  • Multi-task: ch14-multitask/multitask_trainer.py
  • Distillation: ch15-distillation/teacher_student.py

Generation

  • Sampling: ch12-generation/sampling.py
  • Beam search: ch12-generation/beam_search.py
  • KV-cache: ch12-generation/kv_cache.py

Optimization

  • LoRA: ch16-finetuning/lora.py
  • Quantization: ch17-compression/quantization.py
  • Pruning: ch17-compression/pruning.py

Deployment

  • API serving: ch23-deployment/fastapi_server.py
  • ONNX export: ch23-deployment/onnx_export.py
  • Mobile: ch23-deployment/tflite_convert.py

Usage Examples

Running Code Examples

# Navigate to code directory
cd /Users/admin/ps/ML/llm/book-trm/code

# Run specific example
python ch07-trm-intro/minimal_trm.py

# Run with module import
python -m ch07_trm_intro.minimal_trm

# Run tests
pytest tests/test_trm.py

Importing in Your Code

# Add book code to path
import sys
sys.path.append('/Users/admin/ps/ML/llm/book-trm/code')

# Import modules
from ch08_recursive_layers import RecursiveAttention
from ch10_training import TRMTrainer
from utils.model_utils import count_parameters

Code Standards

All code examples follow these standards:

  • Type hints: Full type annotations
  • Docstrings: Comprehensive documentation
  • Testing: Unit tests provided
  • Comments: Explanatory comments for complex logic
  • Style: PEP 8 compliant

Additional Resources

Last updated: October 9, 2025