Gated DeltaNet

Update a fixed-size state .

Gated DeltaNet stores past information in a state matrix. Each token reads and updates that matrix. The recurrent state stays the same size as the sequence grows.

delta-rule updates · learned decay and write gates

Read the paper

Cache growth and state size

Full attention saves keys and values for each token. A recurrent DeltaNet layer updates a fixed-size state. Compare their memory use as tokens arrive.

Compare cache and state growth

Each square represents a token.

Start the sequence.

transformer

Store per-token keys and values

0.0 MB

Each token adds a cache entry.

deltanet

Update a fixed state

1.0 MB

Tokens update the matrix without changing its dimensions.

Memory scaling: the KV cache grows with the token count. The recurrent state has fixed dimensions.

Store facts in one small table

Think of the state as a small table. A key acts like a lookup address, and a value is what you want to store there. Each new fact updates that table without making it larger.

Look up a fact in the table

A key acts like an address in a small table. Select one to see the value retrieved from the state. New facts update the same table.

nextcapital · France→Paris

waitingcapital · Japan→Tokyo

waitingplanet · red→Mars

waitingocean · largest→Pacific

shared statealways 4 × 4

Highlighted cells show the state update.

Correct an existing entry

Look up the current answer, compare it with the new answer, and write the difference. This is the delta rule. It updates what the table already stores.

Step through a delta update

Read the state, compute the value error, then apply the update.

1 · read

Ask the memory

capital · France

current answer?

→

2 · predict

Memory replies

what it knows now

Lyon

→

3 · compare

Find the error

new answer − old answer

Paris − Lyon

→

4 · correct

Update the state

Write the correction

Paris

Read the state using the incoming key.

Delta update: use the incoming key to read a value, then write its prediction error.

Control decay and write strength

The decay gate scales the old state. The write gate scales the correction. Change either gate to see how it affects recall.

Similar keys can interfere

Increase key similarity to compare an additive write with a delta-rule correction. Similar keys can cause interference in either case.

address overlap25%

Add the full value

Additive update

near

0.06

near

0.50

target

1.13

near

0.50

target share · 51%

Read, then write the error

Delta-rule update

near

-0.01

near

0.03

target

0.88

near

0.50

target share · 62%

1. Read: Read the current value.

→

2. Correct: Write a scaled correction.

Adjust the decay and write gates

The model learns both gates. One scales the old state. The other scales the new correction. Adjust them to compare the resulting recall.

Decay gate · retain old state92%

Write gate · scale correction72%

recall of an early fact62%

recall of the newest fact88%

Lower decay values reduce the old state. Lower write values apply less of the correction. In a trained model, both values depend on the token.

Fixed memory has limited capacity

A fixed-size state compresses the sequence history and can lose or mix details. Its size stays constant during recurrent decoding.

Choose a context length

Change the token count to compare memory use.

past tokens

64K

transformer cache

8.0 GB

delta state

1MB

Full-attention KV cache8.0 GB

Recurrent state1 MB

Memory use

Here, the growing cache is about 8192× larger. The delta layer keeps updating the same small state.

Recall limit

A fixed state can lose or mix information. Hybrid models retain full-attention layers for direct access to earlier tokens.

Original paper

Gated Delta Networks: Improving Mamba2 with Delta Rule Yang et al. · 2024

Loading diagram…

Reference

← All guides

Gated DeltaNet

Gated DeltaNet is a linear recurrent attention mechanism. It stores sequence information in a fixed-size state matrix and updates it with the delta rule and learned gates.

Overview

How it works

At each decoding step, the token reads and updates the current state. The layer does not keep a separate KV-cache entry for every earlier token. Its recurrent memory size is independent of sequence length.

Details

The state matrix

The state associates key vectors with value vectors. Reading with a key returns an estimate of its stored value. New tokens update the same matrix, so their information can overlap rather than occupy separate records.

Details

The delta rule

Read the value predicted by the current state for a key. Subtract that prediction from the incoming value, then write a scaled correction. This reduces redundant writes but does not eliminate interference between similar keys.

Details

Decay and write gates

The decay gate scales the old state before the update. The write gate controls the strength of the correction. Both depend on the current token and are learned during training.

Details

Memory and recall

For a fixed head size, recurrent decoding uses constant state memory and constant work per token. The state has limited capacity and may lose details. Hybrid models combine recurrent layers with full-attention layers, so the whole model can still have a growing cache.

Sources

Gated Delta Networks: Improving Mamba2 with Delta Rule ↗Yang et al. · 2024