DeepSeek Sparse Attention

Select tokens before full attention .

DeepSeek Sparse Attention (DSA) uses a small learned indexer to rank earlier tokens. Main attention then reads the top-k selected positions. The indexer still scans the available context.

lightning indexer · token-level top-k selection

Read the paper

Some tokens are more relevant than others

Choose a question to see which tokens supply its answer. These examples illustrate why selecting a subset can be useful.

Inspect the supporting tokens

Select a question to highlight the tokens used in the example answer.

ProjectOrionbeganinMarch.Mayaapprovedtherevisedbudgetof$2.4M.ThedesignteammeetsinLisbon.FinaldeliveryisdueonOctober18.

ordinary attention checks

all 25 words

answer came from

Maya

Make a shortlist first

The indexer makes a quick first pass, like a reader marking passages to check more closely. It ranks token positions. Main attention then uses the shortlist.

Make a shortlist of tokens

The indexer makes a first pass through the text. Change k to see how many marked positions reach the shortlist.

tokens to keep6

shortlist

best 6 of 32 tokens

cutoff score · 0.45

This represents the lightning indexer. Its scoring operation is cheaper than main attention.

Select individual positions

Useful tokens may be far apart. Compare selecting individual tokens with reading every token in a selected block.

Compare block and token selection

The relevant positions in this example are scattered. Compare reading their full blocks with selecting only those tokens.

useful tokens

tokens actually read

extra work

DSA selects individual token positions. Selected positions can come from different parts of the context.

Indexing and main attention are separate

The indexer selects positions. Main attention computes its output using the keys and values at those positions.

Step through token selection

The indexer selects positions. Main attention computes weights and combines their values.

stage1 of 3

Score everything cheaply

The indexer scores the available earlier tokens.

Set the top-k budget

Larger selections use more computation and may retain more useful context. Any token the indexer excludes is unavailable to this main attention step.

Compare attention reads

Set the context length and top-k count. The indexer scans the context. Main attention reads the selected positions.

earlier tokens

tokens kept

full attention · expensive reads65,536

DSA main attention · expensive reads2,048

context searched

100%

full attention used

reduction

32×

Memory use

With fixed k, main attention reads a bounded number of positions. Indexing cost still grows with the context.

the risk

An excluded token cannot contribute to this main attention step. A lower k can omit useful context.

Original paper

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models DeepSeek-AI · 2025

Loading diagram…

Reference

← All guides

DeepSeek Sparse Attention

DeepSeek Sparse Attention (DSA) uses a small learned indexer to select earlier token positions. Main attention reads the selected top-k keys and values.

Overview

How it works

The indexer scans the available context with a cheaper scoring operation. Its top-k scores determine which positions enter main attention. A token that is not selected cannot contribute to that attention step.

Details

Why select tokens

Full attention scores all available positions. Some may contribute little to the current output. DSA trains an indexer to identify a smaller set worth processing with main attention.

Details

The lightning indexer

The indexer computes relevance scores for candidate tokens. It uses a lighter calculation than main attention and is trained to make useful selections. Its scores choose positions. They are not the final attention weights.

Details

Token-level selection

Selected positions can be scattered across the context. Token-level selection avoids loading every neighboring token in a chosen block. It also requires the implementation to gather the selected cache entries efficiently.

Details

The top-k budget

A larger k increases main-attention work and gives it access to more context. With fixed k, that selected attention step reads a bounded number of positions. The indexer still scans the context, and sparse selection does not by itself remove the growing KV cache.

Sources

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models ↗DeepSeek-AI · 2025