Network Pruning via L1 Regularized Gating

This repository contains an implementation of neural network parameter pruning leveraging an $L_1$ penalty on custom gate activations. By wrapping standard linear weights in a trainable gating mechanism, the network actively learns which pathways to emphasize and which to sever, resulting in a quantifiable accuracy vs. sparsity trade-off.

---

🚀 The Core Concepts

1. The PrunableLinear Layer

Instead of using PyTorch's native nn.Linear and attempting to push naked weights to exactly zero, we built PrunableLinear. For every weight in the layer, we introduce a companion parameter (gate_scores). During the forward pass, we apply a sigmoid to bound these scores between $0$ and $1$, and element-wise multiply them by the original weights: $$ \text{effective_weight} = \text{weight} \times \text{sigmoid}(\text{gate_scores}) $$

2. The Loss Algorithm

To force the network to shut off unnecessary gates, we define our loss function as: $$ \text{Total Loss} = \text{Classification Loss (CrossEntropy)} + \lambda \times \text{Sparsity Loss} $$ Where the Sparsity Loss is the absolute sum ($L_1$ norm) of all active gate values across the network.

1. L1 Pressure: The $L_1$ penalty provides a constant mathematical pressure pushing all gate values toward zero.
2. Optimizer Dynamics: The optimizer receives an explicit scale of $\lambda$ on the sparsity gradient. High $\lambda$ values induce massive updates that drive the underlying gate_scores into deep negative territory, effectively muting the weight (since $\sigma(-\text{large}) \approx 0$).

---

🧠 Experimental Results

We evaluate our model (PrunableNet, an MLP architecture structured as 512 → 256 → 10) on the full 50,000 image CIFAR-10 training set over 10 epochs.

We mapped the network's behavior across three distinct constraint severities $\lambda$:

Lambda ($\lambda$)	Accuracy	Sparsity (< 1e-2)
0.0001 (Low)	54.30%	0.00%
0.001 (Medium)	53.82%	0.00%
0.01 (High)	24.37%	100.00%

Analytical Breakdown

• Low $\lambda$ ($0.0001$): The sparsity penalty is virtually nonexistent compared to the primary classification objective. Accuracy peaks at $54.30%$ as the network is allowed to use every single connection.
• Medium $\lambda$ ($0.001$): The network successfully balances the regularizer. Even under a 10x stricter penalty on connection usage, the network isolates and preserves its most vital nodes—costing it extremely minimal accuracy ($53.82%$). Note: While hard strict pruning (a gate value mathematically <0.01) sits at 0.00% at this epoch count, the global distribution has structurally shifted heavily toward smaller values.
• High $\lambda$ ($0.01$): The penalty on active gates overwhelms the classification gradient. The model prioritizes avoiding the massive sparsity loss, plunging all gate structures to $100%$ sparsity. With effectively zero viable weights remaining, the accuracy tanks to $\sim24%$.

---

📊 Distribution Analysis

The following overlay details how the thresholding fundamentally shifts the architecture's parameter histogram. Lower lambda scores allow weight distribution to fall naturally, while strict higher lambdas crush the parameter count against the zero axis.

---

⚙️ How to Run

Running the experiment requires zero manual configuration. The main script automatically downloads CIFAR-10, constructs the custom layers, runs the training iteration suite across the three lambda parameters, and outputs the graph.

# Clone the repository and run:
python prune.py

Depending on hardware, iterating the 50,000 images will comfortably finish running in 10-15 minutes on standard compute.