Multi-head Latent Attention

Cache a compressed KV representation .

Multi-head Latent Attention (MLA) stores a compact latent vector for each token. Learned projections supply the keys and values used by each head. A small positional key is cached separately.

low-rank KV compression · separate positional key

Read the paper

Separate heads require separate KV vectors

Standard multi-head attention caches a key and value vector for each head and token. More heads increase the amount stored.

Compare per-token storage

Standard multi-head attention caches separate keys and values for each head. Change the head count to compare storage.

attention heads6

ordinary cache

K1 · V1

K2 · V2

K3 · V3

K4 · V4

K5 · V5

K6 · V6

12 saved head-vectors per token

MLA cache

one latent

compressed content

The example keeps the latent dimension fixed as head count changes.

Pack the information before storing it

Think of the cached latent as a small suitcase containing information for several heads. A learned projection packs the token representation into fewer numbers.

Choose how much the suitcase holds

A smaller suitcase has less room. The model learns how to pack the input into a smaller vector, called the latent.

latent width4

→

input details

cached numbers

compression

4.0×

This is a low-rank projection. The projected latent is stored in place of expanded per-head keys and values.

Unpack a view for each head

Each head unpacks the same suitcase in a different way. In the model, learned projections supply that head’s key and value information.

Unpack the same suitcase for different heads

Each head gets its own view of the packed information. Learned projections define these views. Efficient decoding can combine the projections without explicitly unpacking every KV vector.

same latent

cached once

→

Example head feature people

retrieves · Maya

Keep a separate position tag

The suitcase also needs a position tag. MLA stores a small positional key separately from the content latent, so rotary position encoding can operate on it.

Separate content and position

Move the token to change the positional key. This example holds the content vector fixed.

token position5

compressed content · cached

Tokyo

Content is held fixed for this illustration.

position key · kept separately

RoPE applies the position-dependent rotation here.

MLA uses a separate positional key. The cache stores both the content latent and the smaller positional key.

Choose a latent dimension

A smaller latent reduces cache storage. The trained model must still encode the information each head needs within that dimension.

Compare suitcase and cache sizes

This example uses 128 cached scalars per token for standard attention. MLA stores a latent and a separate positional key.

context

latent width24

expanded K/V cache per token128 units

MLA latent + position key32 units

tokens

64K

cache reduction

4.0×

head views

unchanged

DeepSeek-V2 reported a 93.3% KV-cache reduction for its model configuration. This calculator uses different, simplified dimensions.

Original paper

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI · 2024

Loading diagram…

Reference

← All guides

Multi-head Latent Attention

Multi-head Latent Attention (MLA) caches a compressed vector for each token, plus a small positional key. Learned projections provide the key and value information used by each head.

Overview

How it works

The compressed vector is called a latent. MLA learns to encode the information needed by multiple heads in this smaller representation, reducing the data stored for each earlier token.

Details

Per-head storage

Standard multi-head attention saves separate key and value vectors for every head and token. The cache grows with the context. MLA reduces the size of each token’s cached representation.

Details

Compression and projection

A learned down-projection produces the latent. Separate up-projections define the content keys and values for each head. The diagrams show this expanded view. Efficient decoding can combine projections algebraically rather than explicitly rebuilding every cached head vector.

Details

A separate positional key

Rotary position encoding applies position-dependent rotations. MLA uses a small, separate positional key so those rotations do not obstruct the projection operations used for the compressed content path. Both the latent and positional key are cached.

Details

Choosing the latent dimension

A smaller latent needs less cache storage but gives the model less representational capacity. The dimension is an architectural choice learned during training. DeepSeek-V2 reported a 93.3% KV-cache reduction against its comparison model. The interactive calculator uses a smaller example.

Sources

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model ↗DeepSeek-AI · 2024