Kimi Delta Attention

Control decay for each channel .

Kimi Delta Attention (KDA) updates a fixed-size recurrent state with the delta rule. Its decay gate has a separate value for each key channel, so different parts of the state can retain information at different rates.

delta-rule updates · channel-wise decay

Read the paper

One memory board, updated for each token

Think of the state as a memory board with several channels. Each token changes parts of the board. The board keeps the same dimensions.

Update the same memory board

Each token changes the same board. The labeled channels make the example easier to follow. Real channels do not have fixed roles.

word arrivingMaya

sentence so far

MayaflewtoTokyoyesterday.

The current token updates the example name feature.

→

One fixed board4 channels

people

grammar

actions

places

The board represents the recurrent state. Its dimensions stay fixed as new tokens arrive.

One dimmer controls the whole board

A scalar decay gate works like one dimmer for the whole board. Turning it down fades both example features together.

Try one dimmer for both features

Suppose you want to keep the name and clear an old filler word. With one dimmer, both fade at the same rate.

keep old memory70%

want to keep

person · Maya

still remembered70%

want to clear

filler · “actually”

still remembered70%

Turning the dimmer up keeps more of both features. Turning it down reduces both. Gated DeltaNet uses one decay value per head and token.

Give each channel its own dimmer

KDA gives each key channel its own decay value. Think of separate dimmers: one feature can stay while another fades. Adjust the controls to compare them.

Try a separate dimmer for each channel

Choose a preset, then adjust the dimmers to keep different amounts of each example feature.

keep people92%

memory: Maya

keep small words18%

memory: actually

keep actions28%

memory: flew

keep places35%

memory: Tokyo

Channel-wise decay: each channel has its own retention factor. The named features are illustrative. Real channels do not have fixed semantic roles.

The delta rule corrects the stored value

Read the current value for a key, compare it with the new value, and write a scaled correction into the state.

Apply a delta-rule update

Read the current value, compute the error, and scale the correction with the write gate.

1 · Read the state

capital of France

Lyon

→

2 · Compute the error

Target value

Paris

→

3 · Apply the correction

Updated value

Lyon→Paris

Write gate85%

old · Lyonnew · Paris

This is the delta rule: read, compute the value error, then write a scaled correction. KDA combines this update with channel-wise decay.

State size stays fixed

Channel-wise gates provide more control over a limited state. They do not make it an exact record of every earlier token.

Compare recurrent memory use

Full attention stores per-token keys and values. KDA updates a fixed-size recurrent state.

Context length

KV cache · relative size16×

KDA state · relative size1× fixed

State size

For fixed head dimensions, a KDA decoding step uses constant state memory and work per token.

State capacity

The state has limited capacity and may lose earlier information. Channel-wise decay controls which parts are retained.

In the Kimi Linear experiments, the full hybrid model used up to 75% less KV cache and reached up to 6× decoding throughput at one-million-token context. Those are model-level results, not a promise for every KDA layer.

Original paper

Kimi Linear: An Expressive, Efficient Attention Architecture Kimi Team · 2025

Loading diagram…

Reference

← All guides

Kimi Delta Attention

Kimi Delta Attention (KDA) combines delta-rule updates with channel-wise decay gates. Different key channels can retain information at different rates within a fixed-size recurrent state.

Overview

How it works

Gated DeltaNet applies one decay value per head and token. KDA uses a vector of decay values across the key channels. This gives the model finer control over which parts of its state persist.

Details

Channel-wise decay

A scalar gate scales the whole head’s previous state by one amount. A channel-wise gate applies different decay values along the key dimension. The named features in the demo make this operation visible. Real channels do not have fixed labels such as people or places.

Details

The recurrent update

KDA decays the old state, reads its prediction for the incoming key, and writes a scaled correction toward the new value. The matrix keeps the same dimensions throughout the sequence.

Details

Capacity and recall

Separate decay gates improve control over a limited state. They do not provide an exact archive of earlier tokens. Information that is overwritten or lost cannot be retrieved directly from a token cache in that recurrent layer.

Details

Layer cost and model results

A KDA layer uses bounded recurrent state during decoding. Kimi Linear combines KDA with full-attention layers. Its reported cache and throughput improvements describe that hybrid model under specific test conditions, not a universal speedup for any KDA layer.

Sources

Kimi Linear: An Expressive, Efficient Attention Architecture ↗Kimi Team · 2025