NVIDIA-Nemotron-3-Super-120B-A12B

The NVIDIA Nemotron 3 Super 120B-A12B represents a flagship model in NVIDIA's family of open LLMs, designed to tackle tasks requiring deep reasoning, complex tool interaction, and processing large volumes of data. The model utilizes an innovative hybrid architecture, combining sparse Mixture-of-Experts (MoE) layers, Mamba-2 state-space blocks, and a limited number of traditional attention layers. This approach allows it to scale the total number of parameters to 120 billion while maintaining low inference costs by activating only 12 billion of them when processing each token.

A key architectural innovation is the use of Latent MoE. The model consists of 88 layers organized in a periodic, alternating structure: the majority of layers are Mamba-2 blocks with linear complexity relative to sequence length. Strategically placed global attention layers (Grouped-Query Attention, 32 query / 2 KV heads) act as "anchors," ensuring the preservation of important dependencies and information exchange between distant parts of the context. A unique feature is the presence of MoE layers without an attention mechanism: in these layers, routing and expert computations are performed exclusively in a compressed latent space (projecting from 4096 down to 1024 dimensions), allowing 22 out of 512 experts to be activated while minimizing overhead. All resource-intensive operations—routing, expert computation, and all-to-all communication—are performed in this compressed space, further reducing memory consumption and accelerating inference without sacrificing quality. This hybrid design balances speed, memory, and accuracy, optimizing long-context performance and making the model ideal for Retrieval-Augmented Generation (RAG) tasks and large-scale document analysis.

The uniqueness of Nemotron 3 Super is underscored by the application of techniques rarely seen in open models. First, it is one of the first models to undergo a full pre-training cycle on 25 trillion tokens, where a large portion of the data was represented in the 4-bit floating point format NVFP4. Second is Multi-Token Prediction (MTP). The model is trained to predict multiple future tokens simultaneously, which not only improves training quality and prediction but also serves as a built-in speculative decoding system to accelerate response generation. Third, the post-training process (RLHF) utilized a specially trained generative reward model—Qwen3-Nemotron-235B-A22B-GenRM-2603—built upon Qwen3-235B-A22B-Thinking-2507 and specifically trained to evaluate response quality. This enabled fine-tuning of the model's behavior, enhancing its "helpfulness" and instruction-following capabilities.

Based on benchmark results, Nemotron-3-Super demonstrates outstanding performance. The model achieves top positions in mathematical reasoning benchmarks (AIME25, HMMT), where it outperforms much larger models. In programming tasks (LiveCodeBench) and agentic capabilities (SWE-Bench), it significantly surpasses counterparts like GPT-OSS-120B, confirming its superiority in solving practical tasks. The model's key advantage is its inference speed (up to 2.2x faster than comparable models) while maintaining competitive quality. Its long-context performance is particularly noteworthy: in the RULER test on 1 million tokens, it achieves 91.75%, substantially ahead of competitors.

Thanks to its efficient architecture and support for a 1-million-token context, the model becomes an ideal choice for developing autonomous AI agents, automating IT ticketing, code writing and review, and building complex RAG systems that operate on massive amounts of unstructured information.

Announce Date: 10.03.2026
Parameters: 124B
Experts: 512
Activated at inference: 12B
Context: 263K
Layers: 88, using full attention: 8, using no attention: 40
Attention Type: Hybrid Attention
Mamba Type: Mamba 2
Developer: NVIDIA
Transformers Version: 4.57.6
vLLM Version: 0.17.1
License: NVIDIA Nemotron Open Model License

Use our pre-built public endpoints for free to test inference and explore NVIDIA-Nemotron-3-Super-120B-A12B capabilities. You can obtain an API access token on the token management page after registration and verification.

Rent your own physically dedicated instance with hourly or long-term monthly billing.

We recommend deploying private instances in the following scenarios:

• maximize endpoint performance,
• enable full context for long sequences,
• ensure top-tier security for data processing in an isolated, dedicated environment,
• use custom weights, such as fine-tuned models or LoRA adapters.

Contact our dedicated neural networks support team at support@immers.cloud or send your request to the sales department at sale@immers.cloud.