Qwen2.5-7B-1M

Qwen2.5-7B-1M is an advanced compact model built on a state-of-the-art Transformer architecture, integrating key innovations such as: Rotary Positional Embeddings (RoPE) for efficient encoding of positional information, the SwiGLU activation function for nonlinear transformations, RMSNorm with pre-normalization for stable training, and QKV bias in the attention mechanism. The architecture includes 28 layers with Grouped Query Attention (GQA), which enables optimal KV-cache utilization and reduced computational costs.

The main feature of the model is its ability to process up to 1,010,000 tokens of input context! This is equivalent to processing 10 full-length novels, 150 hours of speech transcripts, or 30,000 lines of code within a single request. Dual Chunk Attention (DCA) divides sequences into chunks and redistributes relative positions, ensuring stable performance on ultra-long contexts. Integration with YaRN attention scaling further enhances focus on critical information even when processing extremely long sequences. As a result, the model demonstrates outstanding accuracy in information retrieval tasks from very large documents, achieving over 80% accuracy even with a one-million-token context.

Qwen2.5-7B-1M opens up new possibilities in document processing, automated analysis, and intelligent assistants. It is ideally suited for legal analysis of lengthy contracts and documentation, scientific research, software development involving analysis of large codebases, and building technical support systems with access to extensive knowledge bases. In education, the model can analyze entire textbooks and generate comprehensive learning materials, while in business analytics, it can process voluminous reports and extract key insights for informed decision-making.

Announce Date: 26.01.2025
Parameters: 7.62B
Context: 1010K
Attention Type: Full Attention
VRAM requirements: 17.0 GB using 4 bits quantization
Developer: Alibaba
Transformers Version: 4.47.1
License: Apache 2.0

Use our pre-built public endpoints to test inference and explore Qwen2.5-7B-1M capabilities.

Rent your own physically dedicated instance with hourly or long-term monthly billing.

We recommend deploying private instances in the following scenarios:

• maximize endpoint performance,
• enable full context for long sequences,
• ensure top-tier security for data processing in an isolated, dedicated environment,
• use custom weights, such as fine-tuned models or LoRA adapters.

Contact our dedicated neural networks support team at nn@immers.cloud or send your request to the sales department at sale@immers.cloud.