On Hold

Project Bonsai

Pruning neural networks with precision and discipline

The Philosophy

Project Bonsai embodies the philosophy that neural network pruning should be deliberate and thoughtful — akin to tending a bonsai tree. Rather than crude magnitude-based approaches, Bonsai emphasizes careful, iterative reduction of network parameters based on statistical sensitivity analysis.

Core Metaphor

Bonsai Art: Like cultivating a bonsai, pruning should be a disciplined practice that enhances rather than diminishes the essential structure. Each cut should be informed, purposeful, and reversible if proven wrong.

Technical Methodology

Bonsai introduces several novel concepts for principled neural network pruning:

Neural Impact Metric (NIM)

The foundation of Bonsai is the NIM, defined as the partial derivative of the loss function with respect to the post-activation value of a neuron. This is essentially the saliency of each neuron, easily captured using PyTorch's gradient hooks.

Forward Advanced Neuron Impact Metric (FANIM)

FANIM extends NIM by incorporating temporal dynamics:

FANIM = NIM × (a[i+1] - a[i]) / L

Where:

  • NIM: Neural Impact Metric (saliency)
  • a[i+1] - a[i]: Change in activation between consecutive epochs/batches
  • L: Current loss (for normalization)

Statistical Framework

  • Data Collection: Accumulate FANIM matrices across multiple training steps
  • Wilcoxon Testing: Apply Wilcoxon Signed-Rank Test to each neuron's FANIM values
  • Significance Threshold: Prune neurons with p-value < 5% for "different from zero in wrong direction"
  • Conservative Approach: Only prune when statistical evidence is overwhelming

Algorithmic Process

  1. Training Phase: Train network while collecting FANIM data
  2. Statistical Analysis: Run Wilcoxon tests on accumulated FANIM matrices
  3. Pruning Decision: Remove neurons failing statistical significance tests
  4. Fine-tuning: Train pruned network for 5 epochs to stabilize
  5. Convergence Check: Repeat until no neurons meet pruning criteria

BANIM: Backward Analysis

While FANIM looks forward in time, BANIM (Backward Advanced Neuron Impact Metric) examines past behavior:

BANIM = NIM × (a[i] - a[i-1]) / L

BANIM is mathematically less clean (not a pure Taylor expansion) but computationally simpler since it doesn't require "phantom" forward steps.

Challenges and Lessons

The AFL Discovery

Unexpected Result: Bonsai consistently pruned more conservatively than random pruning
Performance Gap: Random pruning to 70%+ sparsity often outperformed Bonsai
Strategic Pivot: AFL research revealed that the problem space is different than assumed

Current Status and Future Direction

Project Bonsai is on hold pending resolution of AFL research findings:

  • Conservative Nature: Bonsai pruning falls well below AFL thresholds
  • Potential Role: May be useful as a refinement step after random pruning to AFL
  • Sequential Strategy: Random prune to AFL, then apply Bonsai for fine-tuning
  • Research Question: Does statistical pruning add value after aggressive random pruning?

Philosophical Impact

  • Discipline Over Heuristics: Replace rules of thumb with statistical rigor
  • Principled Methodology: Every pruning decision backed by evidence
  • Iterative Refinement: Gradual improvement rather than dramatic changes
  • Reversible Decisions: Statistical framework allows for confidence assessment
Discuss Methodology ← Back to Research