Moonshot Kimi K3: The 2.8T LatentMoE Model With Kimi Delta Attention and Open Frontier Intelligence

Executive Summary

On July 16, 2026, Moonshot AI announced Kimi K3 — a 2.8-trillion-parameter Mixture-of-Experts model that represents the largest open-weight model ever released. With only 104 billion parameters activated per token (selecting 16 out of 896 experts), K3 is built on two novel architectural innovations: Kimi Delta Attention (KDA), a hybrid linear attention mechanism, and Attention Residuals (AttnRes), which enables selective retrieval of representations across model depth. The full weights were released on Hugging Face on July 27, 2026, under the Kimi K3 License.

Kimi K3 is designed for frontier intelligence across long-horizon coding, knowledge work, and reasoning. It features a 1-million-token context window, native multimodal understanding (text, images, video) via the MoonViT-V2 vision encoder, and quantization-aware training with MXFP4 weights and MXFP8 activations for broad hardware compatibility. The model achieves approximately 2.5× improvement in overall scaling efficiency over its predecessor, Kimi K2.

The benchmark results place K3 as the strongest open model to date, consistently outperforming all non-flagship proprietary models and competing closely with Claude Fable 5 and GPT-5.6 Sol. On BrowseComp, K3 scores 91.2 (best in the official evaluation suite), 88.3 on Terminal-Bench 2.1, 81.2 on FrontierSWE, and 93.5 on GPQA Diamond. Moonshot acknowledges that K3 still trails the two most powerful proprietary models overall, but the gap is narrowing significantly, particularly in coding and agentic tasks.

This article provides a comprehensive analysis of the Kimi K3 release, the KDA and AttnRes architecture, the Stable LatentMoE framework, training methodology, benchmark performance, deployment guidance, pricing, and strategic implications for the open frontier landscape.

---

1. The Release: First Open 3T-Class Model

1.1 The Timeline

Kimi K3's rollout followed a rapid announcement-to-weights pattern:

Date	Release	Key Changes
Jul 16, 2026	Kimi K3 Announcement	2.8T MoE, KDA + AttnRes, 1M context, native vision
Jul 16, 2026	API & Product Launch	Available on Kimi.com, Kimi Work, Kimi Code, Kimi API
Jul 27, 2026	Open Weights Release	Full model weights on Hugging Face (moonshotai/Kimi-K3)
Jul 27, 2026	Technical Report	Full architecture, training, and evaluation details

1.2 The Open Frontier Trajectory

Kimi K3 is the latest step in Moonshot's sustained push at the scaling frontier. For nine of the past twelve months (July 2025 – July 2026), Kimi models have set the upper bound of open-model sizes. This trajectory connects to the broader trend of increasingly capable open models, including the recent releases of Nvidia Nemotron 3 5 Lightning 30b Moe Mamba Hybrid Agentic Execution Nvfp4 Speculative Decoding 2026 08 24 (30B MoE-Mamba hybrid) and Qwen3 8 27b Dense Multimodal Hybrid Attention Local Deployment Apache 2026 08 20 (27B dense multimodal), though K3 operates at a fundamentally different scale.

1.3 Availability

Kimi K3 is accessible through multiple channels:

Channel	Access	Details
Kimi.com	Free / Paid	Web interface with max thinking effort
Kimi Work	Desktop app	Windows + Apple Silicon, v3.1.0+
Kimi Code	Terminal CLI	/model command to select K3
Kimi API	Pay-per-use	$0.30/$3.00/$15.00 per MTok
Hugging Face	Open weights	moonshotai/Kimi-K3, ~1.56TB
Kimi Enterprise	Organization	Data privacy, member management

---

2. Architecture: Kimi Delta Attention and Attention Residuals

2.1 The Core Innovation

Kimi K3's architecture represents a departure from the standard Transformer-MoE recipe. Two structural innovations form the backbone:

Kimi Delta Attention (KDA) is a hybrid linear attention mechanism designed to scale efficiently across sequence length. Unlike full attention's quadratic complexity, KDA provides linear-time sequence mixing while preserving the representational capacity needed for long-context reasoning. KDA uses a lower-bounded log-decay for numerical stability and is periodically interleaved with Gated MLA (Multi-head Latent Attention) layers for tasks requiring global context awareness.

Attention Residuals (AttnRes) enables the model to selectively retrieve representations from all preceding layers, rather than accumulating them uniformly. AttnRes uses learned pseudo-queries to determine which past representations are most relevant at each layer, creating a form of depth-aware information flow. Block AttnRes with 8 blocks is used throughout the model.

2.2 Architectural Specifications

Parameter	Value
Total parameters	2.8 trillion
Activated parameters	104 billion
Architecture type	Mixture-of-Experts (MoE)
Number of layers	93
Dense layers	1
Attention composition	69 × KDA + 24 × Gated MLA
Attention hidden dimension	7,168
Number of attention heads	96
Latent MoE dimension	3,584
MoE hidden dimension (per expert)	3,072
Number of experts	896 routed + 2 shared
Selected experts per token	16
Vocabulary size	160,000
Context length	1,048,576 (1M tokens)
Activation function	SiTU-GLU (Sigmoid Tanh Unit)
Vision encoder	MoonViT-V2 (27 layers, 401M params)
Quantization	MXFP4 weights / MXFP8 activations (QAT)
Input modalities	Text, images, video
Output modality	Text

2.3 Stable LatentMoE

The MoE routing in K3 is the most aggressive yet attempted in an open model:

• 896 routed experts with 16 activated per token (sparsity ratio of 56×)
• 2 shared experts always active for stable gradient flow
• Quantile Balancing replaces heuristic load-balancing updates, deriving expert allocation directly from router-score quantiles
• Per-Head Muon extends the Muon optimizer by optimizing attention heads independently for adaptive learning at scale
• SiTU-GLU (Sigmoid Tanh Unit) activation improves stability at extreme sparsity levels

Compared to Kimi K2's 384 experts with 8 active, K3 increases the expert pool by 133% while doubling the activation count, yet achieves 2.5× better scaling efficiency through the combined effect of KDA, AttnRes, and improved routing.

---

3. Training Methodology

3.1 Data and Pre-Training

Kimi K3 was trained on a large-scale corpus spanning:

• Web text across multiple languages and domains
• Code from diverse programming languages and frameworks
• Mathematics and scientific literature
• Vision corpus including captions, interleaved image-text, OCR, and video
• Programmatic multimodal data (code + rendered visuals)
• Long-context data synthesized through permutation and concatenation of multimodal documents

Pre-training progressed from 8K to 64K token context, followed by a cooldown phase extending to 256K and finally 1M tokens.

3.2 Optimization and Infrastructure

Training K3 required several infrastructure innovations:

Innovation	Purpose
Per-Head Muon	Independent head optimization for adaptive learning at scale
Quantile Balancing	Eliminates heuristic load-balancing hyperparameters
MoonEP	Fully balanced expert-parallel training with static shapes and zero host synchronization on the critical path
KDA Systems Co-design	Fused kernels, KDA Context Parallelism, state-aware prefix caching
Pipeline ZeRO-2	Gradient sharding for memory-efficient training
Quantization-Aware Training	MXFP4/MXFP8 from SFT stage onward for hardware compatibility

3.3 Post-Training: Three Stages

Stage	Description
Supervised Fine-Tuning (SFT)	Expanded dataset for complex agentic tasks, with QAT from this stage onward
Reinforcement Learning (RL)	Across general, agentic, and coding domains at low, high, and max reasoning effort levels (9 experts total). Uses partial rollout for long-horizon tasks with external KV-cache retention and adaptive throttling
Multi-Teacher On-Policy Distillation (MOPD)	Consolidates specialized domain policies into a unified model

The RL system uses co-located infrastructure with resumable microVM sandboxes for persistent state, enabling training on tasks that span hours of autonomous execution.

---

4. Benchmark Performance

4.1 Reasoning & Knowledge

Benchmark	Kimi K3	Claude Fable 5	GPT-5.6 Sol	Claude Opus 4.8	GPT-5.5	GLM-5.2
GPQA Diamond	93.5	92.6	94.1	91.0	93.5	91.2
CritPt	23.4	28.6	32.3	20.9	27.1	20.9
AA-LCR	74.7	70.0	73.7	67.7	74.3	71.3
HLE-Full (w/o tools)	43.5	53.3	44.5	49.8	41.4	—
HLE-Full (w/ tools)	56.0	63.0	58.0	57.9	52.2	—

4.2 Coding

Benchmark	Kimi K3	Claude Fable 5	GPT-5.6 Sol	Claude Opus 4.8	GPT-5.5	GLM-5.2
DeepSWE	67.5	70.0	73.0	59.0	67.0	46.2
ProgramBench	77.8	76.8	77.6	71.9	70.8	63.7
Terminal-Bench 2.1	88.3	88.0	88.8	84.6	83.4	82.7
FrontierSWE	81.2	86.6	71.3	66.7	64.9	67.3
SWE-Marathon	42.0	35.0	39.0	40.0	14.0	13.0
PostTrainBench	36.6	41.4	34.6	34.1	28.4	34.3
Kimi Code Bench 2.0	72.9	76.9	64.8	71.7	69.0	64.2

4.3 Agentic Tasks

Benchmark	Kimi K3	Claude Fable 5	GPT-5.6 Sol	Claude Opus 4.8	GPT-5.5	GLM-5.2
BrowseComp	91.2	88.0	90.4	84.3	84.4	—
DeepSearchQA (F1)	95.0	94.2	—	93.1	—	—
ResearchRubrics	76.2	—	73.8	73.5	64.0	71.1
GDPval-AA v2 (Elo)	1686	1747	1736	1593	1491	1510
MCPMark-Verified	94.5	87.4	92.9	76.4	92.9	—
AutomationBench	30.8	29.1	29.7	27.2	22.7	12.9
OSWorld-Verified	84.8	85.0	83.0	83.4	79.0	—
Harvey Lab-AA	94.6	93.6	87.2	91.1	86.3	91.0

4.4 Vision

Benchmark	Kimi K3	Claude Fable 5	GPT-5.6 Sol	Claude Opus 4.8	GPT-5.5
WorldVQA ForceAnswer	51.0	56.7	41.8	39.1	38.5
OmniDocBench	91.1	89.8	85.8	87.9	89.4
PerceptionBench	58.5	57.2	59.7	47.2	55.8
Video-MME (w/ sub)	90.0	—	89.5	86.0	89.3
MMMU-Pro (w/o tools)	81.6	81.2	83.0	78.9	81.2
MMMU-Pro (w/ tools)	83.4	86.5	84.6	82.7	83.2
MathVision (w/o CoT)	94.3	94.8	95.8	86.7	92.2
MathVision (w/ CoT)	97.8	98.6	97.8	97.1	96.8

Key observations:

• K3 achieves the best BrowseComp score (91.2) in the official evaluation suite, surpassing both Fable 5 and GPT-5.6 Sol
• MCPMark-Verified (94.5) and AutomationBench (30.8) are best-in-class
• SWE-Marathon (42.0) significantly outperforms all competitors, including Fable 5 (35.0)
• On coding benchmarks, K3 trails GPT-5.6 Sol on DeepSWE (67.5 vs 73.0) but leads on FrontierSWE (81.2 vs 71.3)
• Vision performance is competitive but not yet best-in-class; WorldVQA and MMMU-Pro with tools show room for improvement

---

5. Case Studies: Long-Horizon Agentic Capability

5.1 GPU Kernel Optimization

Kimi K3 was tested on optimizing GPU kernels across NVIDIA Hopper GPUs and alternative GPGPU vendors. Given up to 24 hours in an identical sandbox, K3 profiled, rewrote, and benchmarked four tasks spanning AttnRes, KDA, and a 512-head-dimension MLA kernel. K3 performed competitively with Claude Fable 5 and substantially outperformed Opus 4.8, GPT-5.6 Sol, and GPT-5.5.

Notably, during the late stages of K3's own development, an early version handled the majority of the team's kernel optimization work.

5.2 GPU Compiler Development (MiniTriton)

Kimi K3 developed MiniTriton — a compact Triton-like compiler with its own tile-level IR layer over MLIR, optimization passes, and a PTX code-generation pipeline. MiniTriton delivers performance on par with or better than Triton and torch.compile on certain workloads, and sustains end-to-end nanoGPT training with stable convergence.

5.3 Chip Design

In a 48-hour autonomous run, K3 designed a chip to serve a nano model built on its own architecture. Using open-source EDA tools on the Nangate 45nm library, the chip:

• Closes timing at 100 MHz within 4 mm²
• Sustains over 8,700 tokens/s decode throughput in simulation
• Packs 1.46M standard cells, 0.277 MB SRAM, and an INT4 MAC array with fused dequantization

5.4 Scientific Research (I-Love-Q Universal Relations)

Kimi K3 completed in approximately two hours what typically requires one to two weeks by an experienced researcher: reviewing 20+ papers, implementing the full numerical pipeline for I-Love-Q universal relations in computational astrophysics, evaluating 300+ equations of state, identifying inconsistencies in published formulas, generating 3,000+ lines of Python, and producing an interactive HTML dashboard.

5.5 Interactive Research Visualization

In Kimi Work, K3 produced a 42-year ASIC industry research website through 120+ rounds of recursive self-improvement, pulling data via 2.8K+ web searches, 1.1K+ terminal data pulls, across 11K+ pages spanning 87 quarterly reports and 99 original PDFs.

---

6. Deployment and Implementation

6.1 API Usage

Kimi K3 is available via the Kimi API with OpenAI-compatible endpoints:

import os
from openai import OpenAI

# Set MOONSHOT_API_KEY environment variable before running
client = OpenAI(
    api_key=os.getenv("MOONSHOT_API_KEY"),
    base_url="https://api.moonshot.ai/v1",
)

completion = client.chat.completions.create(
    model="kimi-k3",
    reasoning_effort="max",
    messages=[{"role": "user", "content": "Analyze this codebase for security vulnerabilities."}],
)

6.2 Pricing

Tier	Cache-Hit Input	Cache-Miss Input	Output
Kimi K3	$0.30 / MTok	$3.00 / MTok	$15.00 / MTok

The official Kimi API achieves a cache hit rate above 90% in coding workloads, powered by Mooncake's disaggregated inference architecture. With KDA's state-aware prefix caching, the effective cost per token is significantly reduced for long-context workflows.

Context caching is automatic: no cache ID, TTL, or extra parameter is required. Keep the long prefix unchanged so later requests automatically attempt a cache hit.

6.3 Local Deployment

The full model weights (~1.56TB on Hugging Face) require substantial infrastructure:

Configuration	Est. Requirements
Minimum serving	8× datacenter GPUs (MXFP4)
Recommended	32× GPUs for full throughput
Optimal	64+ accelerators (supernode configuration)
Quantization	MXFP4 weights / MXFP8 activations (QAT)
Inference framework	vLLM (with KDA prefill cache support)

Moonshot recommends deploying on supernode configurations with 64 or more accelerators to maximize the high-bandwidth communication domains that benefit inference efficiency.

6.4 Context and Caching

• 1M token context window with automatic prefix caching
• Cache hit requires previous request's prompt tokens > 256
• KDA with prefill cache enables competitive token pricing despite model scale
• vLLM community implementation for KDA context parallelism released alongside model weights

---

7. Limitations

Moonshot explicitly documents several limitations:

1. Sensitivity to thinking history — K3 was trained in preserved thinking history mode. If the agent harness fails to pass back all historical thinking content, or if a session is switched from another model to K3, generation quality may become highly unstable. Moonshot recommends using a verified-compatible harness like Kimi Code.

2. Excessive proactiveness — Due to emphasis on long-horizon challenging tasks, K3 may make unexpected decisions on the user's behalf when encountering minor issues or ambiguous intent. Explicit behavioral constraints in the system prompt or AGENTS.md are recommended for applications requiring well-defined boundaries.

3. User experience gap — Despite being highly competitive overall, K3 exhibits a noticeable gap in user experience compared with Claude Fable 5 and GPT-5.6 Sol.

---

8. Strategic Implications

8.1 The Open Frontier Accelerates

Kimi K3's release at 2.8T parameters — with 104B active — demonstrates that open models can now operate at scales previously reserved for proprietary systems. The 2.5× scaling efficiency improvement over K2 suggests that architectural innovation (KDA, AttnRes, LatentMoE) is as important as raw parameter count.

This connects to the broader trend of increasingly capable open models. While Nvidia Nemotron 3 5 Lightning 30b Moe Mamba Hybrid Agentic Execution Nvfp4 Speculative Decoding 2026 08 24 focused on efficiency at 30B with a MoE-Mamba hybrid, and Qwen3 8 27b Dense Multimodal Hybrid Attention Local Deployment Apache 2026 08 20 brought frontier vision-language to 27B dense, K3 pushes the scale frontier to 2.8T — bridging the gap between open and proprietary at the highest levels.

8.2 Architectural Diversification

The KDA + AttnRes combination represents a meaningful departure from the standard Transformer-MoE recipe. KDA's hybrid linear attention addresses the quadratic scaling problem without sacrificing representational capacity, while AttnRes's depth-aware information flow improves how representations propagate through 93 layers. These innovations suggest that the architecture landscape is still evolving, even at the frontier.

8.3 Agentic Coding at Scale

Kimi K3's case studies — GPU kernel optimization, compiler development, chip design, and scientific research — demonstrate that long-horizon agentic capability is emerging as a key differentiator. The model's ability to sustain multi-hour autonomous sessions, navigate massive repositories, and orchestrate terminal tools positions it as a serious tool for research and engineering workflows.

8.4 Pricing and Accessibility

At $0.30/$3.00/$15.00 per MTok (with 90%+ cache hit rates in coding), K3 offers a competitive price point for frontier-class reasoning. The open weights release ensures that organizations with sufficient infrastructure can self-host, avoiding ongoing API costs entirely.

---

9. References and Resources

• Official Tech Blog: Kimi K3: Open Frontier Intelligence
• Hugging Face Model Card: moonshotai/Kimi-K3
• Technical Report: MoonshotAI/Kimi-K3 on GitHub
• API Documentation: Kimi K3 Quickstart
• Paper: Kimi K3: Open Frontier Intelligence (arXiv 2607.24653)
• Kimi Code: Kimi Code Documentation
• Kimi Work: Kimi Work Desktop App

---

10. Future Directions

Several areas warrant monitoring in the months ahead:

1. Low and high-effort modes — Currently K3 defaults to max thinking effort. Moonshot has indicated that low- and high-effort modes will be introduced in subsequent updates, enabling finer-grained cost control.

2. Community fine-tuning — With open weights available, the community will likely produce specialized fine-tunes for specific domains (legal, medical, scientific) and quantized variants for smaller hardware.

3. vLLM integration — The KDA prefill cache implementation in vLLM will determine how easily K3 can be deployed in existing inference stacks.

4. Scaling beyond 3T — If K3's 2.5× efficiency gain holds, the path to 5T+ open models becomes technically feasible, potentially narrowing the gap with proprietary flagships even further.

5. Architecture adoption — KDA and AttnRes may be adopted by other model families, similar to how MoE and RoPE became standard components. The open nature of K3 accelerates this potential diffusion.

---

This article was researched and written using official sources only: the Kimi tech blog, Hugging Face model card, GitHub technical report, and Kimi API documentation. All benchmark figures are cited from the official Kimi K3 evaluation suite.