Kimi K3 is the flagship Mixture-of-Experts (MoE) model from Moonshot AI, featuring 2.8 trillion parameters, of which only 104 billion are activated per token. The architecture natively supports text, images, and video via the MoonViT-V2 encoder (401M parameters). The model was trained using Quantization-Aware Training (QAT): weights are natively quantized to MXFP4 and activations to MXFP8, ensuring substantial infrastructure savings.
The first key feature is a novel hybrid attention mechanism. The model consists of 93 layers: 69 layers implement Kimi Delta Attention (KDA)—a linear-recurrent attention mechanism with a fixed state size—while 24 layers utilize full Gated MLA (Multi-head Latent Attention) for precise global control. This approach enables the processing of contexts up to 1 million tokens long without the exponential growth of KV-cache memory.
The second key innovation of K3 is Attention Residuals technology, which replaces the classic residual connection. Instead of simply adding a block's input to its output, AttnRes employs a learnable depth-wise attention mechanism: each layer generates a pseudo-query and weights the RMS-normalized residual states from previous blocks (within an attn_res_block_size=12 window). Thus, information from the depths of the model is selectively extracted. This improves the flow of information through the network's depth and allows the model to more flexibly combine representations from different levels, which is especially crucial when scaling beyond a trillion parameters.
The third important aspect is Stable LatentMoE. This is a framework for stable training and inference of Mixture-of-Experts at extreme sparsity levels, with the core idea borrowed from NVIDIA. Unlike classic MoE models, where tokens pass directly through full-dimension experts, LatentMoE first projects activations into a compressed latent space (dimensionality of 3584, which is half the size), then applies routed experts, and finally projects them back. This reduces computation and communication overhead, allowing the use of more experts at a similar inference cost.
Compared to the Kimi K2 series, which relied on the MLA architecture, K3 demonstrates an approximately 2.5x improvement in scaling efficiency thanks to the aforementioned KDA, AttnRes, and Stable LatentMoE. In fact, it is the world's first open-weight 3T-class model available for research and deployment under the Kimi K3 License.
On benchmarks, Kimi K3 demonstrates frontier-level performance among open-weight models and competes with the best proprietary solutions: it leads in SWE-Marathon—a marathon software engineering benchmark with long development sessions—surpassing Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8, and also takes top spots in BrowseComp (web navigation and search), MCPMark-Verified (tool usage via the MCP protocol), AutomationBench (workflow automation), Video-MME (video content understanding), and MMVU. Furthermore, the model shows outstanding absolute scores in GPQA Diamond (93.5)—the most challenging PhD-level questions in exact sciences, ProgramBench (77.8)—code generation from specifications, Terminal-Bench 2.1 (88.3)—terminal environment operation, and DeepSearchQA (95.0)—deep search and information synthesis from multiple sources.
Kimi K3 is optimal for long-term software development and systems programming—ranging from GPU kernel optimization and compiler writing to website design and game development within long-running agent sessions with minimal human oversight. The model is also highly effective in the multimodal task segment; it can generate in-depth research reports with interactive visualizations and dashboards based on the analysis of documents, financial data, and scientific literature, as well as tackle a wide range of video processing tasks. Finally, K3 reliably handles business process automation and integration with enterprise SaaS tools, which is critical for enterprise agents.
| Model Name | Context | Type | GPU | Status | Link |
|---|
There are no public endpoints for this model yet.
Rent your own physically dedicated instance with hourly or long-term monthly billing.
We recommend deploying private instances in the following scenarios:
There are no configurations for this model, context and quantization yet.
There are no configurations for this model, context and quantization yet.
There are no configurations for this model, context and quantization yet.
Contact our dedicated neural networks support team at nn@immers.cloud or send your request to the sales department at sale@immers.cloud.