autodiff-mini

An educational project to understand how backpropagation works. This implementation demonstrates the core concepts of automatic differentiation and computational graphs by implementing backpropagation on scalar values from scratch.

Features

• Value Class: Wraps scalar values and tracks gradients through computation
• Automatic Differentiation: Computes gradients via backpropagation
• Computational Graphs: Visualizes computation flow and gradient propagation
• Supported Operations:
  • Arithmetic: addition, subtraction, multiplication, division, power
  • Activation Functions: tanh, sigmoid, ReLU, exponential
  • Chain Rule: Automatic gradient computation through nested operations

Installation

git clone https://github.com/meditatoire/autodiff-mini.git
cd autodiff-mini

Quick Start

from autodiff_mini.backprop import Value

# Create variables
a = Value(2.0, label='a')
b = Value(3.0, label='b')

# Build computation
c = a * b + a ** 2

# Compute gradients
c.backward()

# Access gradients
print(f"dc/da = {a.grad}")  # 7.0
print(f"dc/db = {b.grad}")  # 2.0

Core Components

backprop.py

Implements the Value class, the fundamental building block:

Operators:

• __add__, __radd__: Addition
• __mul__, __rmul__: Multiplication
• __sub__, __rsub__: Subtraction
• __truediv__, __rtruediv__: Division
• __pow__: Power
• __neg__: Negation

Activation Functions:

• exp(): Exponential function
• tanh(): Hyperbolic tangent
• sig(): Sigmoid function
• ReLU(): Rectified Linear Unit

Backpropagation:

• backward(): Computes gradients using topological sort and reverse-mode differentiation

graph.py

Provides computational graph visualization:

• computational_graph: Builds and visualizes the computation DAG
• draw_dot(): Generates SVG visualization showing nodes, operations, and gradients

Usage Examples

Simple Computation

from autodiff_mini.backprop import Value

x = Value(3.0, label='x')
y = Value(4.0, label='y')

z = x * y + x ** 2
z.backward()

print(x.grad)  # 11.0 (dz/dx = y + 2*x = 4 + 6)
print(y.grad)  # 3.0  (dz/dy = x)

Complex Expression

a = Value(2.0, label='a')
b = Value(-3.0, label='b')
c = Value(10.0, label='c')

e = a * b
d = e + c
f = d * e

f.backward()

print(f"a.grad = {a.grad}")
print(f"b.grad = {b.grad}")
print(f"c.grad = {c.grad}")

Visualization

Note: Visualization requires graphviz to be installed on your system:

from autodiff_mini.backprop import Value
from autodiff_mini.graph import computational_graph

a = Value(2.0, label='a')
b = Value(3.0, label='b')
c = a * b
c.backward()

# Visualize the computation graph
graph = computational_graph(c)
graph.draw_dot().view()

Testing

Run the comprehensive test suite:

python3 -m test.test_backprop

Tests cover:

• All arithmetic operations
• All activation functions
• Complex expressions and chain rule
• Edge cases (negative inputs, zero, etc.)

How It Works

Forward Pass

Each operation creates a new Value node that stores:

• data: The computed result
• _children: Input values
• _op: Operation name
• _backward: Gradient computation function

Backward Pass

1. Build topological order of computation graph
2. Set output gradient to 1.0
3. Traverse nodes in reverse topological order
4. Apply chain rule: child.grad += parent.grad * local_gradient

Example: Multiplication

For c = a * b:

• Forward: c.data = a.data * b.data
• Backward: a.grad += c.grad * b.data, b.grad += c.grad * a.data

Project Structure

autodiff-mini/
├── autodiff_mini/
│   ├── __init__.py
│   ├── backprop.py          # Core Value class and operations
│   └── graph.py             # Computational graph visualization
│  
├── test/
│   └── test_backprop.py     # Comprehensive test suite
└── README.md

Dependencies

• graphviz (for visualization)

References

This implementation is inspired by:

• Andrej Karpathy's micrograd