Kimi K3 scales to 2.8 trillion parameters through a combination of Kimi Delta Attention, Attention Residuals, and a highly sparse Stable LatentMoE system with 896 experts. Only 16 experts are activated for a token, allowing total model capacity to grow much faster than active computation.
Moonshot says these architectural and training changes produce roughly 2.5× the overall scaling efficiency of Kimi K2. That does not mean K3 is cheap to run. The model is still designed for supernode-scale inference with 64 or more accelerators.
Architecture information checked July 21, 2026. Moonshot has published a launch overview, while the complete technical report and weights are still planned for release. This article labels open questions instead of inventing missing implementation details.
Architecture at a glance
| Component | Published Kimi K3 detail |
|---|---|
| Total parameters | 2.8 trillion |
| Attention | Kimi Delta Attention, with Gated MLA in the architecture |
| Depth connection | Attention Residuals |
| MoE | Stable LatentMoE |
| Experts | 896 routed experts |
| Active experts | 16 per token |
| Balancing | Quantile Balancing |
| Optimizer technique | Per-Head Muon |
| Activation component | Sigmoid Tanh Unit |
| Weight format | MXFP4 through quantization-aware training |
| Activation format | MXFP8 |
| Context | 1 million tokens |
| Deployment direction | Supernode with 64+ accelerators |
What is Kimi Delta Attention?
Kimi Delta Attention, or KDA, is Moonshot's hybrid linear-attention mechanism. Its role is to provide a more efficient foundation for processing long sequences than relying only on conventional full attention at every layer and token.
Linear-attention methods generally seek to avoid the quadratic growth associated with comparing every token to every other token. The word “hybrid” matters: the exact K3 architecture combines attention components rather than reducing the model to one simple linear formula.
KDA is central to K3's 1M-token positioning, but context length is a system property. Memory layout, kernels, KV or state handling, caching, parallelism, and serving infrastructure all affect whether a long context is practical.
KDA and prefix caching
Conventional serving systems often cache attention state for a repeated prefix. KDA introduces different state and prefill requirements, so ordinary prefix-cache implementations cannot simply be assumed to work efficiently.
Moonshot says it contributed KDA prefill-cache support to the vLLM community for release alongside K3. This helps explain how the official API can offer a cache-hit input price far below uncached input despite the model's scale.
Infrastructure teams should wait for explicit K3 support in the relevant vLLM release and kernels rather than modifying a model name in an existing deployment.
What are Attention Residuals?
Traditional deep networks pass information through residual connections that accumulate layer by layer. Moonshot describes Attention Residuals, or AttnRes, as selectively retrieving representations across depth rather than uniformly accumulating everything.
The intended benefit is smoother information flow through a very deep model. Instead of forcing every layer to depend only on the immediately accumulated state, AttnRes can emphasize useful earlier representations.
This is conceptually important at 2.8T scale, where optimization and signal flow become difficult. The complete equations, ablations, and training details should be evaluated when the technical report is public.
Stable LatentMoE and 896 experts
K3 is a Mixture-of-Experts model. A router selects a small subset of expert networks for each token.
K3 has:
- 896 routed experts;
- 16 experts active for a token;
- shared and routed expert components in the published diagram;
- a Stable LatentMoE framework designed for extreme sparsity.
The activation ratio is approximately:
16 / 896 ≈ 1.8%
That percentage does not directly reveal active parameter count. Shared layers, expert sizes, routing, attention, and other components contribute computation outside the simple ratio. Treat claims that K3 has an exact active-parameter number as unverified unless Moonshot publishes it.
Why extreme MoE sparsity is difficult
More experts create more capacity, but also harder routing and infrastructure problems:
- tokens may overload popular experts;
- underused experts waste capacity;
- expert placement affects communication;
- load imbalance reduces accelerator utilization;
- dynamic shapes and host synchronization can hurt throughput;
- routing decisions must remain stable during training.
K3 addresses these issues with balancing, static-shape execution, and large communication domains.
Quantile Balancing
Moonshot describes Quantile Balancing as deriving expert allocation from router-score quantiles. The goal is to avoid heuristic update rules and a sensitive balancing hyperparameter.
In plain language, allocation adapts to the distribution of routing scores rather than depending on one manually tuned threshold. This is relevant when 896 experts must remain useful and evenly served at scale.
Independent reproduction and ablation results are still needed to quantify how much Quantile Balancing contributes relative to the rest of the training system.
Per-Head Muon
Muon is an optimization approach used during training. K3 extends it on a per-attention-head basis, allowing heads to be optimized more independently.
Moonshot's stated motivation is more adaptive optimization at large scale. This is a training technique rather than an API feature: users do not configure Per-Head Muon during inference.
SiTU and Gated MLA
The launch overview also identifies:
- Sigmoid Tanh Unit (SiTU): intended to improve activation control;
- Gated MLA: intended to improve attention selectivity.
These components sit inside a larger block design that combines KDA, latent experts, shared experts, convolutional elements, and gating. Their exact contribution requires the technical report and controlled ablations.
MXFP4 weights and MXFP8 activations
K3 uses quantization-aware training from the supervised fine-tuning stage onward, with MXFP4 weights and MXFP8 activations.
Quantization-aware training exposes the model to lower-precision behavior during training instead of converting a finished high-precision model as an afterthought. This can preserve more quality at an efficient inference format.
MXFP4 substantially reduces theoretical weight storage. At four bits per parameter, 2.8T raw parameters correspond to roughly 1.4 TB before scales, metadata, non-quantized components, runtime buffers, and redundancy. It remains far beyond consumer hardware.
Fully balanced expert-parallel training
Moonshot says K3 uses fully balanced expert-parallel training with static shapes and no host synchronization on the critical path. These choices target predictable accelerator work and reduce pauses caused by CPU coordination.
At inference time, the same broad challenge remains: experts distributed across many devices must communicate quickly. Moonshot recommends a supernode with at least 64 accelerators so more communication occurs inside a high-bandwidth domain.
How K3 reaches a 1M-token context
No single component explains the 1M window. The capability depends on:
- efficient KDA sequence processing;
- stable information flow through AttnRes;
- serving kernels and memory management;
- prefill-cache support;
- distributed accelerator infrastructure;
- training data and objectives that teach the model to use long context.
A model accepting one million tokens does not mean every token receives equal attention or that a million-token prompt is always the best design. Long-context evaluation should test retrieval, distant dependencies, distraction, latency, and cost.
Kimi K3 versus Kimi K2
| Area | Kimi K3 direction | Kimi K2 baseline |
|---|---|---|
| Scale | 2.8T total parameters | Smaller prior generation |
| Attention | KDA hybrid linear attention | Earlier architecture |
| Depth | Attention Residuals | Earlier residual design |
| MoE | 896 experts, 16 active | Less extreme sparsity |
| Efficiency | Claimed ~2.5× scaling efficiency | Baseline |
| Vision | Native visual understanding | Different prior capability surface |
| Context | 1M tokens | Lower prior limit depending on model |
The 2.5× figure is Moonshot's overall scaling-efficiency claim, not a promise of 2.5× API speed or 2.5× benchmark performance.
Architecture limitations and open questions
Important unanswered questions include:
- exact active parameter count;
- full layer and expert dimensions;
- training-token count and data mixture;
- complete long-context training method;
- independent architecture ablations;
- supported hardware and measured throughput by configuration;
- exact weight file footprint;
- final runtime and license support.
These questions should be answered from Moonshot's technical report and official repositories after release.
Why the architecture matters to developers
Most API users do not need to implement KDA or MoE routing. They do need to understand the operational consequences:
- K3 is strong at context-heavy and agentic work;
- preserving conversation state is essential;
- automatic caching can materially reduce cost;
- self-hosting requires extreme infrastructure;
- model behavior cannot be inferred from total parameter count alone;
- new runtimes and kernels must explicitly support K3.
Read Is Kimi K3 Open Source? for deployment status and Kimi K3 API Guide for application integration.
Frequently asked questions
How many parameters does Kimi K3 have?
K3 has 2.8 trillion total parameters in a sparse Mixture-of-Experts architecture.
How many experts are active?
Moonshot says 16 of 896 routed experts are activated. Shared and non-expert components also contribute to computation.
What is Kimi Delta Attention?
KDA is Moonshot's hybrid linear-attention mechanism for efficient sequence processing and long-context scaling.
Why does Kimi K3 support 1M context?
The window is enabled by architecture, training, caching, kernels, memory management, and distributed serving rather than one feature alone.
Can Kimi K3 run on consumer GPUs?
The complete model cannot realistically run on consumer GPUs. Moonshot recommends supernode deployments with 64 or more accelerators.