DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

What the report is

The paper is credited to DeepSeek-AI, with Anyi Xu, Bangcai Lin and Bing Xue leading the author list, and was submitted to arXiv on 26 April 2026 as 2606.19348 ^[1]. It is a preview of the DeepSeek-V4 series rather than a description of a single system, and it covers two members. DeepSeek-V4-Pro has 1.6 trillion total parameters, of which 49 billion are activated; DeepSeek-V4-Flash has 284 billion total parameters, of which 13 billion are activated ^[1]. Both support a context length of one million tokens ^[1].

The problem: length is expensive

In a standard transformer, the attention layer lets every token look at other tokens, so the compute needed grows quickly as the context gets longer, and the key-value cache — the stored per-token states that attention reuses — grows in memory at the same time. For ordinary chat turns this is manageable. At a million tokens, a single request can hold an entire code repository or a long document set, and both the compute and the memory become the binding constraint. The report's question is therefore not only whether a model can accept a million tokens, but whether doing so can be made cheap enough to be practical ^[1].

How it works

The architectural answer is a hybrid attention scheme that combines Compressed Sparse Attention (CSA) with Heavily Compressed Attention (HCA), added to improve long-context efficiency ^[1]. As the names indicate, both schemes work by compressing what the attention layer has to keep and read, instead of holding and comparing full per-token states across the whole window.

A second change concerns the residual connections, the shortcut paths that carry a signal around each block of the network and are what make very deep stacks trainable. The report introduces Manifold-Constrained Hyper-Connections (mHC) to enhance these conventional residual connections ^[1].

Training changes too: the authors switch to the Muon optimizer, reporting faster convergence and greater training stability ^[1]. Optimizers are the rules that decide how far and in which direction each parameter moves after a gradient is computed; swapping one for another affects both how quickly a run reaches a given quality and how often it becomes unstable. Both models are pre-trained on more than 32 trillion tokens, followed by a comprehensive post-training pipeline ^[1].

Results

On capability, the report focuses on DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro. That mode is reported to set a new state of the art for open models and to outperform its predecessors on core tasks ^[1].

The efficiency numbers are the more unusual part of the claim. At one million tokens of context, DeepSeek-V4-Pro needs only 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2 ^[1]. In other words, the long window is not only supported but made substantially cheaper per generated token than in the earlier model, which is what the compression in CSA and HCA is meant to buy.

What is released, and why it matters

Model checkpoints are published online together with the report ^[1]. That matters for how the claims can be read: a preview paper describing a new series is one thing, but released weights let others run the models themselves, measure long-context behaviour under their own workloads, and check whether the efficiency gains hold outside the authors' setup.

The framing — a Pro model and a Flash model, sharing a one-million-token window but differing by more than five times in total parameter count — also says something about where large-model work is heading: the same long-context machinery is expected to serve both the largest model and a much smaller, cheaper one, with the smaller model carrying 13 billion activated parameters rather than 49 billion ^[1]. Whether that pairing works in practice is exactly what a released checkpoint makes testable.