Sliding-window attention

Limit attention to a local window .

Sliding-window attention lets each token attend to a fixed range of nearby tokens. Stacked layers can pass information beyond one window. Some models also include full-attention layers.

local attention · optional full-attention layers

Read the paper

Move a window over the sentence

The window works like a spotlight over recent tokens. Move the current token and resize the window to see which positions this layer can use.

current token11

window size4

The spotlight follows the current token. Positions outside the window are excluded from this layer’s attention.

More layers increase the receptive field

A token can receive information through other tokens across multiple layers. Add layers to see the possible reach in this example.

Compare receptive fields

Information can pass through neighboring tokens across layers. This shows the possible reach, not guaranteed recall.

layers stacked3

After 3 layers, information can travel about 6 tokens through this toy window.

Distant tokens may be out of reach

A local layer cannot directly attend to a fact outside its window. Increase the window until the earlier project code is included.

Check access to an earlier token

The project code is near the start. Increase the window to include it at the query position.

window size5

ORION

code?

The fact is outside the window.

Information from this position would need an indirect path through earlier layers or a full-attention layer.

Mix local and global layers

A global layer allows direct attention across the full context. Change how often one appears in the stack.

Choose the global-layer interval

Local layers attend within a window. Global layers attend across the full context.

1local

2local

3local

4local

5global

6local

7local

8local

9local

10global

11local

12local

13local

14local

15global

16local

17local

18local

local layers

global layers

Full-context access

every 5 layers

Compare the number of attention pairs

With N tokens and a window of W tokens, local attention uses O(NW) pairs rather than O(N²). Smaller windows reduce work and direct access to older tokens.

Compare local and full attention

Set the context and window lengths to compare the number of positions read.

context

window

global layer reads65,536

local layer reads1,024

local-layer saving

64× fewer

Only 2% of the context is read directly.

trade-off

A smaller window uses fewer attention pairs. Full-attention layers provide direct access to older positions and use more computation.

Original papers

Longformer: The Long-Document Transformer Beltagy et al. · 2020 Mistral 7B Jiang et al. · 2023

Loading diagram…

Reference

← All guides

Sliding-window Attention

Sliding-window attention limits each token to a fixed range of nearby tokens. This reduces the number of attention pairs computed in a local layer.

Overview

How it works

In a causal local layer, a token attends only to recent positions within its window. Information from farther away can arrive indirectly through earlier layers. Some models also use full-attention layers for direct access to the whole context.

Details

Window size and cost

For N tokens and window size W, local attention computes O(NW) pairs instead of O(N²). At decoding time, a local layer can retain a rolling cache limited to its window. Full-attention layers in a hybrid model still need their own longer caches.

Details

Depth and receptive field

An earlier token’s representation can already contain information from its neighbors. Stacking local layers therefore expands the set of positions that can influence an output. A larger possible receptive field does not guarantee that every distant fact is retained.

Details

Access to distant tokens

A token outside the window is unavailable for direct attention in that layer. A wider window or a full-attention layer can restore direct access. These choices affect both compute cost and the model’s ability to use older context.

Details

Local and global layers

A hybrid stack mixes local layers with full-attention layers. Local layers reduce cost. Full-attention layers allow direct connections across the sequence. Window size, layer depth and the global-layer schedule are model design choices.

Sources