Mixture of Experts
MoE
Neural network architecture using multiple specialised "expert" subnetworks with a gating mechanism that routes inputs to the most relevant experts, enabling larger models with efficient compute.
In-Depth Explanation
Mixture of Experts (MoE) architectures achieve scale efficiency by activating only a subset of parameters for each input. A gating network routes inputs to relevant expert subnetworks.
How MoE works:
- Multiple "expert" networks (feed-forward layers)
- Gating/router network decides which experts process each token
- Only selected experts activate (sparse activation)
- Total parameters large, active parameters small
Benefits:
- Scale to massive parameter counts
- Efficient inference (fewer active params)
- Experts can specialise in different patterns
- Better performance-per-compute ratio
MoE in modern LLMs:
- Mixtral 8x7B: 47B total, 12.9B active
- GPT-4 reportedly uses MoE
- Switch Transformer pioneered trillion-param MoE
Challenges:
- Training stability
- Load balancing across experts
- More complex infrastructure
- Memory for full model still needed
Business Context
MoE enables more powerful models without proportional compute increase. Models like Mixtral offer excellent quality-per-cost, relevant for cost-conscious deployments.
How Clever Ops Uses This
We evaluate MoE models like Mixtral for Australian businesses seeking strong performance at lower inference costs than dense models.
Example Use Case
"Deploying Mixtral 8x7B which matches GPT-3.5 quality but only activates 12B parameters per token, reducing inference costs significantly."