Mixture of Experts

Cross-source synthesis of sparse MoE architecture — how frontier models store hundreds of billions of parameters while activating only a fraction per token. Complements Transformers. Updated as new research is ingested.

Overview

A dense Transformer activates every parameter for every token. A Mixture of Experts (MoE) model replaces some feed-forward layers with multiple parallel expert networks. A router (gate) selects which experts process each token — typically top-2 out of 8–256+ experts. The rest stay idle.

What if you could have 671 billion parameters but only pay to run 37 billion?

This is the architecture behind GPT-4, Mixtral, DeepSeek-V4, Qwen3.6 MoE, Kimi K2.7, and MiniMax M3. It lets models cheat the scaling laws from Scaling Laws Compute Optimal Explained — more capacity without linear compute growth.

See Mixture Of Experts Sparse Models Explained for the beginner-friendly explainer covering three landmark papers.

Core mechanics

Dense:  Token → [all 70B parameters] → Output

MoE:    Token → Router → Expert 3  ┐
                         Expert 7  ┴→ weighted sum → Output
                         (other experts idle)

The router

The gate network outputs a probability distribution over experts:

1. Compute gate scores for each expert
2. Select top-k experts (typically k=2)
3. Weighted combination of expert outputs

Load balancing

Without intervention, routers send most tokens to a few "popular" experts — a monopoly problem. Training adds an auxiliary loss that penalises uneven expert utilisation, encouraging specialisation without collapse.

Key terminology

Term	Meaning
Total parameters	All weights stored in the model
Active parameters	Weights used per token at inference
Sparsity ratio	Active / total (e.g. 37B / 671B ≈ 5.5%)
Expert	A separate feed-forward sub-network
Router / gate	Network that selects experts per token

Historical evolution

Era	Milestone
1991	Jacobs et al. — original Mixture of Experts ensemble concept
2017	Shazeer et al. — sparsely-gated MoE (137B params, top-k routing)
2021	Switch Transformer — 1.6T params, 1 expert active per token
2023	Mixtral 8×7B — open MoE proves production viability
2024–2026	DeepSeek, Qwen, Kimi, MiniMax — MoE dominates open-weight frontier

See Sparse Moe Architecture Evolution Deployment 2026 04 18 for the full architectural evolution including Gated DeltaNet + MoE hybrids.

Dense vs sparse in 2026

The frontier has bifurcated — neither architecture universally dominates.

	Dense	Sparse MoE
Strengths	Multimodal simplicity, predictable latency, mature tooling	Parameter efficiency, local deployment, lower API cost
Weaknesses	Linear compute with scale	Routing complexity, load balancing, deployment overhead
Typical vendors	OpenAI, Anthropic, Google (closed frontier)	Qwen, DeepSeek, MiniMax, Moonshot (open-weight)
Example	Gemma 4 31B (all 31B active)	Qwen3.6-35B-A3B (35B total, 3B active)

Key finding: Closed-source frontier models lean dense; open-source leaders embrace sparse MoE — reflecting deployment and monetization strategy as much as raw technical superiority.

See Dense Transformers Vs Sparse Moe Architecture 2026 04 20 for the full trade-off analysis.

2026 frontier deployments

Model	Total params	Active params	Notable design
DeepSeek-V4-Pro	1.6T	49B	Hybrid attention (CSA+HCA) + MoE
Qwen3.6-35B-A3B	35B	3B	Gated DeltaNet + MoE, runs on M3 Pro
MiniMax M2.7	~230B	10B	Fine-grained routing, $0.30/M tokens
Kimi K2.7 Code	1T	32B	384 experts, coding-specialised MoE
MiniMax M3	undisclosed	sparse	MiniMax Sparse Attention (MSA)
Microsoft MAI-Thinking-1	undisclosed	35B active	Enterprise MoE from Build 2026

DeepSeek-V4-Pro

Open-source MoE leader for reasoning and long context. 1M-token context, three reasoning modes. See Deepseek V4 Pro Frontier Analysis 2026 04 24.

Kimi K2.7 Code

1T-parameter coding-specialised MoE with forced preserve-thinking and ~30% fewer reasoning tokens than K2.6. 384 routed experts, 256K context. See Kimi K27 Code Coding Specialised 1t Moe 2026 06 12.

MiniMax M3

MiniMax Sparse Attention (MSA) — claims 1/20th per-token compute at 1M context vs predecessor. 59% SWE-Bench Pro at 12× lower cost than Opus 4.8, with licensing and geopolitical caveats. See Minimax M3 Open Weight Challenger Analysis 2026 06 03.

When dense wins

MoE is not always the answer. Qwen3.6-27B (dense, 27B all active) outperforms Alibaba's own 397B MoE on agentic coding benchmarks:

• 77.2% SWE-Bench Verified vs 76.2% for the 397B MoE sibling
• Perfect 100/100 on tool-eval-bench
• Fits on a single H100; Apache 2.0 license

This challenges the narrative that bigger sparse models always beat smaller dense ones. Architecture quality, training data, and task-specific optimisation matter as much as parameter count.

See Qwen36 27b Dense Beats Moe Agentic Coding Analysis 2026 06 03.

When to choose dense:

• Multimodal tasks needing encoder-free simplicity (Gemma 4 12B)
• Edge deployment with predictable memory (27B dense on 16GB)
• Tasks where routing overhead exceeds savings

When to choose MoE:

• Maximum capability per dollar at API scale
• Very large total parameter count needed (1T+ coding models)
• Local deployment where active params fit hardware (3B–32B active)

Production deployment

MoE adds deployment complexity beyond dense models:

• Expert parallelism — experts may live on different GPUs; all-to-all communication at scale
• Load imbalance — hot experts create bottlenecks
• Quantization — quantizing 256 experts vs one dense FFN
• Serving frameworks — vLLM and SGLang have MoE-specific optimisations

Open-source MoE deployment comparisons: Open Source Agents Showdown Qwen36 27b V4pro Gemma4 2026 05 19 and Open Source Agents Comparison Qwen V4 Gemma4 2026 04 29.

Related how-tos: Howto Vllm Deployment Guide, Howto Multi Model Routing Layer.

Relationship to other concepts

• Transformers — MoE replaces FFN layers inside Transformer blocks
• Agentic Coding — model choice tables reference MoE vs dense trade-offs
• Inference Optimization Quantization Sparsity Speculative Decoding 2026 05 12 — MoE is one layer in the inference optimization stack

Key source summaries

Topic	Article
Foundational explainer	Mixture Of Experts Sparse Models Explained
Architecture evolution	Sparse Moe Architecture Evolution Deployment 2026 04 18
Dense vs sparse trade-offs	Dense Transformers Vs Sparse Moe Architecture 2026 04 20
Dense beats MoE (Qwen3.6-27B)	Qwen36 27b Dense Beats Moe Agentic Coding Analysis 2026 06 03
Kimi K2.7 Code MoE	Kimi K27 Code Coding Specialised 1t Moe 2026 06 12
MiniMax M3 sparse attention	Minimax M3 Open Weight Challenger Analysis 2026 06 03
DeepSeek V4 architecture	Deepseek V4 Pro Frontier Analysis 2026 04 24

Open questions

• Routing quality ceiling — Can better routers close the gap between MoE total params and effective capability?
• Dense resurgence — Will task-specific dense models (Qwen3.6-27B pattern) erode MoE's cost advantage for coding?
• Expert interpretability — Do experts learn human-meaningful specialisations, or arbitrary partitions?
• Hybrid futures — Gated DeltaNet + MoE + thinking preservation: is this the default 2027 architecture?