SpikingMoE: SDPrompt-Guided Dynamic Expert Fusion in Spiking Neural Networks

Paper: arXiv:2605.23188 (May 2026)
Authors: Yukai Yang, Chenxi Qin, Jungang Li, Xin Zhang, Wenwei Shao, Liqun Chen
Code: Project Page (open-source)

Summary

SpikingMoE integrates a spike-driven Transformer with a Mixture-of-Experts (MoE) framework for dynamic computation in Spiking Neural Networks. Inspired by the lateral geniculate nucleus (LGN), it introduces a spike-driven prompt (SDprompt) mechanism that enables input-dependent expert routing in a biologically plausible manner. This is the first open-source SNN framework integrating MoE into a spike-driven Transformer with LGN-inspired routing.

Key Innovations

1. LGN-Inspired SDprompt Routing: The lateral geniculate nucleus (LGN) selectively modulates sensory signals before routing to cortical areas. SpikingMoE mimics this with a spike-driven prompt that conditions expert selection on the input, enabling context-aware dynamic computation.

2. Spike-Compatible MoE: Replaces standard MLPs in the Spike-driven Transformer with spike-compatible expert modules. Enforces binary spike communication throughout, ensuring compatibility with neuromorphic hardware.

3. Fully Binary Communication: All inter-module signals remain binary spikes (0/1), replacing energy-intensive multiply-accumulate (MAC) operations with low-power accumulation (AC) operations.

Architecture

Input → Spike Embedding → Spike-Driven Transformer Blocks (×N)
  Each block:
    → Spiking Self-Attention (SDSA)
    → Spike-Compatible MoE (replacing MLP)
      → SDprompt Generator (LGN-inspired)
      → Top-K Expert Router (spike-compatible)
      → Expert Networks (spiking neurons)

SDprompt Mechanism

The spike-driven prompt is generated from the input spike sequence and conditions the expert routing decision. Unlike soft prompts in ANNs, SDprompt operates entirely in the spike domain:

• Input: Spike-form features from the preceding self-attention layer
• Processing: Lightweight spiking network that produces routing logits
• Output: Binary routing decisions for expert selection

Expert Architecture

Each expert is a dedicated spiking neuron pathway that processes its assigned inputs:

• Maintains binary spike communication
• Uses Leaky-Integrate-and-Fire (LIF) or similar spiking neurons
• Sparsely activated (only top-K experts fire per input)

Experimental Results

Dataset	Top-1 Accuracy	Reference
CIFAR-10	94.09%	Baseline SNN Transformer ~93%
CIFAR-100	74.54%	Baseline SNN Transformer ~72%

• Demonstrates that modular expert routing can be incorporated into SNNs while retaining reasonable performance
• Achieves comparable or better accuracy than standard SNN Transformers with fewer active parameters per forward pass
• Energy-efficient due to sparse expert activation + binary spike communication

Key Formulas

SDprompt Routing

The routing decision for input x to expert i is:

g_i(x) = softmax(topK(r(x)))_i

where r(x) is the SDprompt router output and topK selects the K highest-activating experts.

Spike-Driven Expert Output

Each expert processes input through spiking neurons:

y_expert = Σ_i g_i(x) · SNN_expert_i(x)

where SNN_expert_i is a spiking neuron module and g_i(x) is the routing weight (binary or soft).

Use Cases

1. Neuromorphic Visual Recognition: Deploy on Loihi/TrueNorth hardware with dynamic computation
2. Energy-Efficient Edge AI: Selective expert activation reduces computation at inference time
3. Scalable SNN Architectures: MoE enables scaling model capacity without proportional compute increase

Implementation Notes

• Compatible with the Spike-driven Transformer architecture (SDSA - Spike-Driven Self-Attention)
• Uses Hadamard product and sparse additions instead of matrix multiplications
• All activations are binary spikes → suitable for neuromorphic deployment
• Activation: spiking MoE, spike-driven transformer, SNN expert routing, LGN spiking, neuromorphic MoE

Pitfalls

• MoE introduces routing overhead; the router itself must be spike-compatible
• Training may require surrogate gradients (standard SNN training technique)
• Expert load balancing is critical — unbalanced routing reduces effective capacity
• Current evaluation limited to CIFAR-10/100; ImageNet-scale results not yet available