The Philosophy
Project Bonsai embodies the philosophy that neural network pruning should be deliberate and thoughtful—akin to tending a bonsai tree. Rather than crude magnitude-based approaches, Bonsai emphasizes careful, iterative reduction of network parameters based on statistical sensitivity analysis.
Core Metaphor
Bonsai Art: Like cultivating a bonsai, pruning should be a disciplined practice that enhances rather than diminishes the essential structure. Each cut should be informed, purposeful, and reversible if proven wrong.
Technical Methodology
Bonsai introduces several novel concepts for principled neural network pruning:
Neural Impact Metric (NIM)
The foundation of Bonsai is the NIM, defined as the partial derivative of the loss function with respect to the post-activation value of a neuron. This is essentially the saliency of each neuron, easily captured using PyTorch's gradient hooks.
Forward Advanced Neuron Impact Metric (FANIM)
FANIM extends NIM by incorporating temporal dynamics:
FANIM = NIM × (a[i+1] - a[i]) / L
Where:
- NIM: Neural Impact Metric (saliency)
- a[i+1] - a[i]: Change in activation between consecutive epochs/batches
- L: Current loss (for normalization)
Statistical Framework
Bonsai applies rigorous statistical testing to pruning decisions:
- Data Collection: Accumulate FANIM matrices across multiple training steps
- Wilcoxon Testing: Apply Wilcoxon Signed-Rank Test to each neuron's FANIM values
- Significance Threshold: Prune neurons with p-value < 5% for "different from zero in wrong direction"
- Conservative Approach: Only prune when statistical evidence is overwhelming
Algorithmic Process
The Bonsai pruning algorithm follows a disciplined iterative approach:
- Training Phase: Train network while collecting FANIM data
- Statistical Analysis: Run Wilcoxon tests on accumulated FANIM matrices
- Pruning Decision: Remove neurons failing statistical significance tests
- Fine-tuning: Train pruned network for 5 epochs to stabilize
- Convergence Check: Repeat until no neurons meet pruning criteria
BANIM: Backward Analysis
While FANIM looks forward in time, BANIM (Backward Advanced Neuron Impact Metric) examines past behavior:
BANIM = NIM × (a[i] - a[i-1]) / L
BANIM is mathematically less clean (not a pure Taylor expansion) but computationally simpler since it doesn't require "phantom" forward steps.
Challenges and Lessons
The AFL Discovery
Unexpected Result: Bonsai consistently pruned more conservatively than random pruning
Performance Gap: Random pruning to 70%+ sparsity often outperformed Bonsai
Strategic Pivot: AFL research revealed that the problem space is different than assumed
Current Status and Future Direction
Project Bonsai is on hold pending resolution of AFL research findings:
- Conservative Nature: Bonsai pruning falls well below AFL thresholds
- Potential Role: May be useful as a refinement step after random pruning to AFL
- Sequential Strategy: Random prune to AFL, then apply Bonsai for fine-tuning
- Research Question: Does statistical pruning add value after aggressive random pruning?
Technical Implementation
Bonsai's implementation leverages modern PyTorch capabilities:
- Gradient Hooks: Automatic NIM collection during training
- Activation Tracking: Efficient storage of activation sequences
- Statistical Computing: Integration with SciPy for Wilcoxon testing
- Reproducible Research: Centralized configuration management
Philosophical Impact
Beyond technical contributions, Bonsai represents a methodological approach:
- Discipline Over Heuristics: Replace rules of thumb with statistical rigor
- Principled Methodology: Every pruning decision backed by evidence
- Iterative Refinement: Gradual improvement rather than dramatic changes
- Reversible Decisions: Statistical framework allows for confidence assessment