Haoran Zhang - Blog

Haoran Zhang - Blog Haoran Zhang is an Applied Research Scientist at NVIDIA working on machine learning systems, distributed systems, and efficient deep learning. Zola 2026-07-25T00:00:00+00:00 https://snowchord.com/blog/atom.xml Linear Attention, Visualized 2026-07-25T00:00:00+00:00 2026-07-25T00:00:00+00:00 Haoran Zhang https://snowchord.com/blog/linear-attention-visualized/ <p>In July 2026, Moonshot released <a rel="external" href="https://www.kimi.com/blog/kimi-k3">Kimi K3</a>, a 2T-parameter model with a 1M-token context window. K3 uses Moonshot's linear-attention variant, Kimi Delta Attention (KDA). Moonshot reports that it trails only Claude Fable 5 and GPT-5.6 Sol across its evaluation suite.</p> <p>NVIDIA had already made a similar choice in <a rel="external" href="https://arxiv.org/abs/2512.20856">Nemotron 3</a>, which combines softmax attention with Mamba-2 recurrent layers. KDA and Mamba-2 use different recurrences. Both offer linear-time sequence mixing during training and fixed-state decoding, although this efficiency comes with some loss of expressiveness.</p> <p>In this post, we visualize the path from self-attention to the linear-attention variants used in recent models.</p>  <h2 id="self-attention-recap">Self-attention recap</h2> <p>Given hidden states $H$, three learned projections produce queries, keys, and values:</p> <div class="eqn"> $$Q=HW_Q,\qquad K=HW_K,\qquad V=HW_V,$$ </div> <p>where $Q,K\in\mathbb R^{n\times d_k}$ and $V,O\in\mathbb R^{n\times d_v}$, with $n$ denoting the sequence length.</p> <div class="eqn"> $$O = \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\!\left( \frac{QK^{\mathsf T}}{\sqrt{d_k}}+M_{\mathrm{add}} \right)V, \qquad M_{\mathrm{add}} = \begin{bmatrix} 0&-\infty&-\infty\\ 0&0&-\infty\\ 0&0&0 \end{bmatrix}.$$ </div> <p>Each query $q_t$ represents a linear functional on the key space:</p> <div class="eqn"> $$\phi_{q_t}:\mathbb R^{d_k}\to\mathbb R, \qquad k_j\mapsto\langle q_t,k_j\rangle.$$ </div> <p>Evaluating it at $k_j$ produces the scalar score $\langle q_t,k_j\rangle$. Softmax normalizes these scores into weights, and $o_t$ is the weighted sum of the corresponding values.</p> <aside class="note-card"> <div class="note-card-title">Notation</div> <div class="note-card-body"><p>From here on, we assume $d_k=d_v=d$ and represent each token's $q_t$, $k_t$, and $v_t$ as row vectors (some RNN and linear-attention literature instead uses column vectors), so $Q,K,V\in\mathbb R^{n\times d}$. We also omit the scale $1/\sqrt d$, which can be absorbed into either $Q$ or $K$.</p> </div> </aside> <h2 id="self-attention-is-quadratic">Self-attention is quadratic</h2> <p>The attention operation uses two matrix multiplications over the sequence:</p> <div class="eqn"> $$\begin{aligned} QK^{\mathsf T}:\quad (n\times d)(d\times n)&\longrightarrow(n\times n), &\approx 2n^2d\ \text{FLOPs},\\ \operatorname{softmax}(\cdots)V:\quad (n\times n)(n\times d)&\longrightarrow(n\times d), &\approx 2n^2d\ \text{FLOPs}. \end{aligned}$$ </div> <p>Masking and softmax add $\mathcal O(n^2)$ work, so the attention operation costs</p> <div class="eqn"> $$4n^2d+\mathcal O(n^2)=\mathcal O(n^2d),$$ </div> <p>which is quadratic in the sequence length.</p> <p>A direct implementation stores the $n\times n$ score matrix. FlashAttention avoids materializing this matrix in off-chip memory, but the computation remains quadratic.</p> <aside class="note-card"> <div class="note-card-title">Optional: Attention vs. FFN</div> <div class="note-card-body"><p>For model width $d$ and FFN expansion factor $e$:</p> <div class="eqn"> $$\underbrace{8nd^2}_{Q,K,V,\text{ and output projections}} + \underbrace{4n^2d}_{\text{attention operation}}, \qquad \operatorname{FLOPs}_{\mathrm{FFN}}\approx 4end^2.$$ </div> <p>For $e=4$, the full attention layer (attention operation + input/output projections) costs more than the FFN when $n>2d$; the attention operation alone does so when $n>4d$.</p> </div> </aside> <h2 id="naive-linear-attention">Naïve linear attention</h2> <p>The quadratic cost comes from multiplying $Q$ by $K^{\mathsf T}$ first, which produces an $n\times n$ score matrix. If we remove softmax, matrix multiplication is associative, so we can change the order of computation:</p> <div class="eqn"> $$O=(QK^{\mathsf T})V=Q(K^{\mathsf T}V).$$ </div> <p>The left-hand order compares every query with every key before mixing the values. The right-hand order first summarizes all key–value pairs in the $d\times d$ matrix $K^{\mathsf T}V$, then applies each query to that summary. It never constructs an $n\times n$ matrix.</p> <p>The two matrix multiplications now cost</p> <div class="eqn"> $$\begin{aligned} K^{\mathsf T}V:\quad (d\times n)(n\times d)&\longrightarrow(d\times d), &\approx 2nd^2\ \text{FLOPs},\\ Q(K^{\mathsf T}V):\quad (n\times d)(d\times d)&\longrightarrow(n\times d), &\approx 2nd^2\ \text{FLOPs}. \end{aligned}$$ </div> <p>Together, they require approximately $4nd^2=\mathcal O(nd^2)$ work, which is linear rather than quadratic in the sequence length.</p> <p>This computation is not causal: the shared summary $K^{\mathsf T}V$ contains key–value pairs from the entire sequence, so every query can read the future. The causal version is</p> <div class="eqn"> $$O=\bigl(QK^{\mathsf T}\odot M\bigr)V, \qquad M= \begin{bmatrix} 1&0&0\\ 1&1&0\\ 1&1&1 \end{bmatrix}, \qquad \odot:\ \text{Hadamard product}.$$ </div> <p>At first glance, the causal mask appears to prevent direct reassociation of the full matrix product. However, row $t$ of $O$ depends only on $q_t$, $K_{1:t}$, and $V_{1:t}$, as shown below.</p> <div class="row eq" id="rg-form"></div> <p>Once the masked columns are removed, row $t$ becomes an ordinary matrix product over a prefix:</p> <div class="eqn"> $$o_t =q_tK_{1:t}^{\mathsf T}V_{1:t} =q_t\!\left(K_{1:t}^{\mathsf T}V_{1:t}\right).$$ </div> <p>We call the product of the key and value prefixes the state $S_t$:</p> <div class="eqn"> $$S_t :=K_{1:t}^{\mathsf T}V_{1:t} \in\mathbb R^{d\times d}.$$ </div> <p>$S_t$ is a fixed-size linear map from key space to value space. Moving from one token to the next adds exactly one outer product, so we can build this state recurrently instead of recomputing the entire prefix:</p> <div class="eqn"> $$S_0=0, \qquad S_t=S_{t-1}+k_t^{\mathsf T}v_t, \qquad o_t=q_tS_t.$$ </div> <p>Each recurrent step costs $\mathcal O(d^2)$, so processing $n$ tokens costs $\mathcal O(nd^2)$. The cost is linear in the sequence length, hence the name linear attention.</p> <aside class="note-card"> <div class="note-card-title">Optional: Linear attention as an RNN</div> <div class="note-card-body"><p>This recurrence defines an RNN with a matrix-valued hidden state $S_t$. Each key–value pair updates $S_t$. The query $q_t$ reads $S_t$ to produce $o_t$. This recurrent view is the idea behind <a rel="external" href="https://arxiv.org/abs/2006.16236"><em>Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention</em></a>.</p> </div> </aside> <p>The recurrent form reduces the arithmetic complexity, but it uses GPUs inefficiently:</p> <ol> <li><strong>Large activation memory.</strong> A direct autograd implementation saves every $d\times d$ state $S_t$ for the backward pass. The $n$ saved states require $\mathcal O(nd^2)$ memory.</li> <li><strong>Sequential token updates.</strong> Computing $S_t$ requires $S_{t-1}$. This dependency prevents the GPU from updating different tokens in parallel.</li> <li><strong>Poor Tensor Core utilization.</strong> Tensor Cores accelerate tiled matrix multiply-accumulate operations. The outer product $k_t^{\mathsf T}v_t$ has shape $(d\times1)(1\times d)$, so its reduction dimension is only one. This shape does not fill the Tensor Core tiles along the reduction dimension.</li> </ol> <h2 id="chunkwise-algorithm">Chunkwise algorithm</h2> <p>To solve these problems, we introduce the chunkwise algorithm. It uses the parallel form $\bigl(QK^{\mathsf T}\odot M\bigr)V$ within each chunk and the recurrent form $S_{\mathrm{out}}=S_{\mathrm{in}}+K^{\mathsf T}V$ between chunks.</p> <p>Assume $C$ divides $n$. Split the sequence into $n/C$ chunks of $C$ consecutive tokens. Let $Q_r,K_r,V_r\in\mathbb R^{C\times d}$ denote chunk $r$.</p> <h3 id="block-the-causal-matrix">Block the causal matrix</h3> <p>Let $A=QK^{\mathsf T}\odot M$. Partition $A$ into $C\times C$ blocks and partition $V$ and $O$ into $C\times d$ blocks. Let $A_{r,s}$ denote block $(r,s)$ of $A$. In block row $r$, blocks to the left contain earlier chunks, the diagonal block contains the current chunk, and blocks to the right contain future tokens.</p> <div class="cw-block-matmul" id="cw-block-matmul"> <div class="cw-matrix-column"> <div class="cw-figure-title"><b>A</b> <span>n×n in C×C blocks</span></div> <div class="cw-score-blocks" id="cw-score-blocks"></div> </div> <div class="cw-operator">×</div> <div class="cw-matrix-column"> <div class="cw-figure-title"><span class="vc">V</span> <span>C×d blocks</span></div> <div class="cw-vector-blocks" id="cw-value-blocks"></div> </div> <div class="cw-operator">=</div> <div class="cw-matrix-column"> <div class="cw-figure-title"><span class="oc">O</span> <span>C×d blocks</span></div> <div class="cw-vector-blocks" id="cw-output-blocks"></div> </div> </div> <div class="cw-output-composition" id="cw-output-composition"></div> <div class="cw-legend"> <span><i class="cw-swatch cw-history"></i>history</span> <span><i class="cw-swatch cw-local"></i>local</span> <span><i class="cw-swatch cw-future"></i>future</span> </div> <p>The output for chunk $r$ splits into history and local contributions:</p> <div class="eqn"> $$O_r =\underbrace{\sum_{s<r}A_{r,s}V_s}_{\text{history}} +\underbrace{A_{r,r}V_r}_{\text{local}}.$$ </div> <p>The local block keeps a $C\times C$ causal mask because tokens in the chunk have different prefix lengths. The history blocks need no mask because every token in chunk $r$ may read every token in an earlier chunk.</p> <h3 id="compress-the-history">Compress the history</h3> <p>Let $K_{<r}$ and $V_{<r}$ contain all chunks before chunk $r$. Associativity compresses the dense history:</p> <div class="eqn"> $$\sum_{s<r}A_{r,s}V_s =Q_rK_{<r}^{\mathsf T}V_{<r} =Q_rS_r, \qquad S_r:=K_{<r}^{\mathsf T}V_{<r}.$$ </div> <p>$S_r$ contains all key–value updates before chunk $r$. Let $M_c$ denote the $C\times C$ causal mask inside the chunk. The chunk output is</p> <div class="eqn"> $$O_r =\underbrace{Q_rS_r}_{\text{history}} +\underbrace{\bigl(Q_rK_r^{\mathsf T}\odot M_c\bigr)V_r}_{\text{local}}.$$ </div> <h3 id="update-the-state">Update the state</h3> <p>After computing $O_r$, add the current chunk to the history:</p> <div class="eqn"> $$S_{r+1}=S_r+K_r^{\mathsf T}V_r.$$ </div> <p>The matrix product $K_r^{\mathsf T}V_r$ batches $C$ rank-one updates into a dense GEMM. Its reduction dimension is $C$ rather than $1$, so it uses Tensor Cores efficiently.</p> <p>For the recurrent state, autograd now saves one boundary state per chunk. These $n/C$ states require $\mathcal O((n/C)d^2)$ memory. Chunk boundaries still form a sequential chain, but the chain now contains $n/C$ steps instead of $n$.</p> <h3 id="cost">Cost</h3> <p>Each chunk requires</p> <div class="eqn"> $$\mathcal O\!\left( \underbrace{C^2d}_{\text{local}} +\underbrace{Cd^2}_{\text{state update}} \right).$$ </div> <p>Across $n/C$ chunks, the total cost is</p> <div class="eqn"> $$\mathcal O\!\left(nCd+nd^2\right).$$ </div> <p>Larger chunks increase the local $nCd$ term but shorten the sequential state chain. $C=1$ recovers the recurrent form, while $C=n$ recovers the full parallel form. Every valid $C$ provides exact results. <a rel="external" href="https://github.com/fla-org/flash-linear-attention">FLA</a> uses $C=64$, while <a rel="external" href="https://github.com/MoonshotAI/FlashKDA">FlashKDA</a> uses $C=16$.</p> <h2 id="retnet-scalar-decay">RetNet: scalar decay</h2> <p>In Naïve linear attention, each outer-product write $k_t^{\mathsf T}v_t$ enters the state with coefficient one and never decays. New and old writes therefore receive the same age-independent coefficient. <a rel="external" href="https://arxiv.org/abs/2307.08621">RetNet</a> introduces a recency bias by shrinking the existing state before adding the current write:</p> <div class="eqn"> $$S_0=0, \qquad S_t=\gamma S_{t-1}+k_t^{\mathsf T}v_t, \qquad o_t=q_tS_t, \qquad 0<\gamma\le 1.$$ </div> <p>$\gamma$ is a fixed scalar, so it scales the entire state uniformly. The figure follows one key–value write through the recurrence:</p> <div class="strip" id="leak"></div> <p>A write from $s$ steps earlier has weight $\gamma^s$. The weight decreases exponentially with age.</p> <h3 id="parallel-form">Parallel form</h3> <p>To compute the recurrence in parallel, note that a write at position $j$ undergoes $t-j$ decay steps before query $q_t$ reads it. Its coefficient is therefore $\gamma^{t-j}$. Collect these coefficients in a decayed causal matrix:</p> <div class="eqn"> $$\Gamma[t,j] = \begin{cases} \gamma^{t-j}, & j\le t,\\ 0, & j>t. \end{cases}$$ </div> <div class="row crow" id="par"></div> <p>The diagonal is one because the current write has not decayed. Moving one column to the left adds one decay step.</p> <p>The parallel form replaces the binary causal mask $M$ with $\Gamma$:</p> <div class="eqn"> $$O=\bigl(QK^{\mathsf T}\odot\Gamma\bigr)V.$$ </div> <h3 id="chunkwise-form">Chunkwise form</h3> <p>The chunkwise form again separates incoming history from interactions inside the chunk. Let $S_r$ denote the state immediately before chunk $r$, and let $\ell=1,\ldots,C$ index positions in the current chunk.</p> <h4 id="decay-incoming-history">Decay incoming history</h4> <p>Before $q_{r,\ell}$ (the query at local position $\ell$ in chunk $r$) reads $S_r$, the state crosses $\ell$ decay steps. Define the boundary-to-query factors</p> <div class="eqn"> $$\lambda_\ell:=\gamma^\ell, \qquad \lambda=(\gamma,\gamma^2,\ldots,\gamma^C)^{\mathsf T}.$$ </div> <p>The first illustration applies one entry of $\lambda$ to each row of $Q_rS_r$. (Hover a row of $Q_r$ to trace its decay factor and the resulting history row.)</p> <div class="row crow" id="g1raw"></div> <p>We can move the row scale to $Q_r$ and define the decayed query:</p> <div class="eqn"> $$\overleftarrow{Q}_r := \operatorname{diag}\!\left(\gamma,\gamma^2,\ldots,\gamma^C\right)Q_r.$$ </div> <p>The left arrow denotes left-boundary decay: row $\ell$ includes the factor $\gamma^\ell$ from the chunk boundary to $q_{r,\ell}$.</p> <div class="row crow" id="g1a"></div> <p>This gives the history term</p> <div class="eqn"> $$O_r^{\mathrm{hist}}=\overleftarrow{Q}_rS_r.$$ </div> <h4 id="compute-the-local-block">Compute the local block</h4> <p>Partitioning the full parallel computation into $C\times C$ blocks exposes the same history and local split:</p> <div class="bmwrap"> <div class="bmcol"> <div class="bmlbl">(<span class="q">Q</span><span class="k">K</span><sup>T</sup> ⊙ <span class="g">Γ</span>)<span class="bmsub">decayed scores · n×n → C×C blocks</span></div> <div class="hgrid" id="hero"></div> </div> <div class="bmop">×</div> <div class="bmcol"><div class="bmlbl"><span class="v">V</span><span class="bmsub">C×d blocks</span></div><div class="vstack" id="vcol"></div></div> <div class="bmop">=</div> <div class="bmcol"><div class="bmlbl"><span class="o">O</span><span class="bmsub">C×d blocks</span></div><div class="vstack" id="ocol"></div></div> </div> <div class="legend" style="margin-top:12px"> <div class="leg-item"><span class="sw" style="background:color-mix(in srgb,var(--intra) 42%,#fff)"></span>diagonal = <b>local</b></div> <div class="leg-item"><span class="sw" style="background:color-mix(in srgb,var(--inter) 44%,#fff)"></span>below diagonal = <b>history</b></div> <div class="leg-item"><span class="sw masked-sw"></span>above diagonal = future</div> </div> <p>A write at local position $m$ crosses $\ell-m$ decay steps before $q_{r,\ell}$ reads it. The local decay matrix is</p> <div class="eqn"> $$\Gamma_C[\ell,m] = \begin{cases} \gamma^{\ell-m}, & m\le\ell,\\ 0, & m>\ell. \end{cases}$$ </div> <div class="row crow" id="gloc"></div> <p>The local term is</p> <div class="eqn"> $$O_r^{\mathrm{local}} =\bigl(Q_rK_r^{\mathsf T}\odot\Gamma_C\bigr)V_r.$$ </div> <h4 id="update-the-state-1">Update the state</h4> <p>We now have the chunk output $O_r$. Next, we construct the state passed to the next chunk. The incoming state crosses all $C$ steps, while a write at local position $\ell$ crosses only $C-\ell$ steps. Define the write-to-boundary factors</p> <div class="eqn"> $$\rho_\ell:=\gamma^{C-\ell}, \qquad \rho=(\gamma^{C-1},\gamma^{C-2},\ldots,1)^{\mathsf T}.$$ </div> <p>Because token rows of $K_r$ become columns of $K_r^{\mathsf T}$, $\rho$ scales columns in the state update. (Hover a column of $\Delta S_r$ to trace the decayed keys and the corresponding column of $V_r$.)</p> <div class="row crow" id="g3raw"></div> <p>Write that column scaling as a diagonal matrix and absorb it into the keys:</p> <div class="eqn"> $$\overrightarrow{K}_r := \operatorname{diag}\!\left(\gamma^{C-1},\gamma^{C-2},\ldots,1\right)K_r.$$ </div> <p>The right arrow denotes right-boundary decay: key row $\ell$ includes the factor $\gamma^{C-\ell}$ from its position to the chunk's right boundary.</p> <div class="row crow" id="g3a"></div> <h4 id="complete-chunkwise-form">Complete chunkwise form</h4> <p>Combining the history and local terms gives the complete chunkwise equations:</p> <div class="eqn"> $$\begin{aligned} O_r &=\underbrace{\overleftarrow{Q}_rS_r}_{\text{history}} +\underbrace{\bigl(Q_rK_r^{\mathsf T}\odot\Gamma_C\bigr)V_r}_{\text{local}},\\ S_{r+1} &=\gamma^CS_r+\overrightarrow{K}_r^{\mathsf T}V_r. \end{aligned}$$ </div> <p>The arrows identify the relevant boundary. Queries absorb decay from the left boundary, while keys absorb decay toward the right boundary. The incoming history state decays from the left boundary to each query position; each key–value write made inside the chunk decays from its position to the right boundary. This exact regrouping keeps the same $\mathcal O(nCd+nd^2)$ asymptotic cost as the Naïve chunkwise algorithm.</p> <h2 id="mamba-2-input-dependent-scalar-decay">Mamba-2: input-dependent scalar decay</h2> <p>RetNet applies the same decay $\gamma$ at every position. <a rel="external" href="https://arxiv.org/abs/2405.21060">Mamba-2</a> replaces it with a token-dependent scalar $a_t$. Its recurrent form is</p> <div class="eqn"> $$S_t=a_tS_{t-1}+k_t^{\mathsf T}v_t, \qquad o_t=q_tS_t, \qquad 0<a_t<1.$$ </div> <p>Each token supplies one scalar for the entire state. A value near one preserves history; a value near zero nearly clears it.</p> <aside class="note-card"> <div class="note-card-title">Optional: Constructing the decay gate</div> <div class="note-card-body"><p>The recurrence assumes $0<a_t<1$. During training, the model must learn $a_t$ while preserving this constraint. Mamba-2 constructs the gate by discretizing continuous-time exponential decay:</p> <div class="eqn"> $$\frac{dS(\tau)}{d\tau}=-\lambda S(\tau), \qquad \lambda>0.$$ </div> <p>Over one token interval $\Delta_t$, the exact solution is</p> <div class="eqn"> $$S_t=e^{-\lambda\Delta_t}S_{t-1}.$$ </div> <p>First, Mamba-2 learns a positive decay rate. It stores an unconstrained log-rate $A_{\log}$ and exponentiates it:</p> <div class="eqn"> $$\lambda=e^{A_{\log}}>0.$$ </div> <p>The rate $\lambda$ specifies the decay <strong>per unit time</strong>. Discretization converts the continuous system into one transition per token. Mamba-2 computes an input-dependent step size $\Delta_t$ for the transition at token $t$, so different tokens advance the continuous system by different amounts. A token with a larger $\Delta_t$ advances the system farther and produces stronger decay; a token with a smaller $\Delta_t$ produces weaker decay. From the token representation $h_t$, a learned projection $W_\Delta$ produces an unconstrained step $\widetilde\Delta_t$. The model adds a learned bias $b_\Delta$ and applies softplus to make the step positive:</p> <div class="eqn"> $$\widetilde\Delta_t=h_tW_\Delta, \qquad \Delta_t=\operatorname{softplus}(\widetilde\Delta_t+b_\Delta)>0.$$ </div> <p>Combining the positive rate and interval gives</p> <div class="eqn"> $$a_t=e^{-\lambda\Delta_t} =\exp\!\left(-e^{A_{\log}}\operatorname{softplus}(\widetilde\Delta_t+b_\Delta)\right)\in(0,1).$$ </div></div> </aside> <h3 id="parallel-form-1">Parallel form</h3> <p>The outer-product write $k_j^{\mathsf T}v_j$ enters the state at position $j$. To reach $S_t$, subsequent transitions multiply it by $a_{j+1},a_{j+2},\ldots,a_t$. Its survival coefficient is</p> <div class="eqn"> $$a_{j+1}a_{j+2}\cdots a_t.$$ </div> <p>Collect these survival coefficients in the decay matrix $\Gamma$. Entry $\Gamma[t,j]$ scales the score between $q_t$ and $k_j$. (Hover an entry of $\Gamma$ to trace the corresponding query, key, value, and output.)</p> <div class="row crow" id="gProd"></div> <p>For an inclusive interval, define</p> <div class="eqn"> $$\lambda[i:j]:=\prod_{s=i}^{j}a_s, \qquad \text{empty product }(i>j)\text{ returns }1.$$ </div> <p>The decay matrix becomes</p> <div class="eqn"> $$\Gamma[t,j] = \begin{cases} \lambda[j+1:t], & j\le t,\\ 0, & j>t. \end{cases}$$ </div> <div class="row crow" id="gGam"></div> <p>The parallel equation keeps the same form as RetNet:</p> <div class="eqn"> $$O=\bigl(QK^{\mathsf T}\odot\Gamma\bigr)V.$$ </div> <h3 id="chunkwise-form-1">Chunkwise form</h3> <p>The chunkwise derivation follows RetNet. Each fixed power of $\gamma$ is replaced by a product of token-dependent gates.</p> <p>Within chunk $r$, let $a_{r,\ell}$ denote the gate at local position $\ell$, and define</p> <div class="eqn"> $$\lambda_r[i:j]:=\prod_{s=i}^{j}a_{r,s}.$$ </div> <h4 id="decay-incoming-history-1">Decay incoming history</h4> <p>Before $q_{r,\ell}$ reads the incoming state $S_r$, the recurrence multiplies $S_r$ by $\lambda_r[1:\ell]$. As in RetNet, absorb this scalar into the query row:</p> <div class="eqn"> $$\overleftarrow q_{r,\ell} :=\lambda_r[1:\ell]q_{r,\ell}, \qquad O_r^{\mathrm{hist}}=\overleftarrow Q_rS_r.$$ </div> <div class="row crow" id="gHist"></div> <p>The left arrow again denotes decay from the chunk's left boundary to each query position.</p> <h4 id="compute-the-local-block-1">Compute the local block</h4> <p>The write $k_{r,m}^{\mathsf T}v_{r,m}$ is multiplied by $\lambda_r[m+1:\ell]$ before it contributes to the output at local position $\ell$. Therefore,</p> <div class="eqn"> $$\Gamma_{C,r}[\ell,m] = \begin{cases} \lambda_r[m+1:\ell], & m\le\ell,\\ 0, & m>\ell. \end{cases}$$ </div> <div class="row crow" id="gLocal"></div> <p>The local output is</p> <div class="eqn"> $$O_r^{\mathrm{local}} =\bigl(Q_rK_r^{\mathsf T}\odot\Gamma_{C,r}\bigr)V_r.$$ </div> <h4 id="update-the-state-2">Update the state</h4> <p>We now have $O_r$. To construct the state for the next chunk, multiply the incoming state by $\lambda_r[1:C]$. The write at local position $\ell$ is multiplied only by the subsequent gates, whose product is $\lambda_r[\ell+1:C]$. Absorb this suffix product into its key:</p> <div class="eqn"> $$\overrightarrow k_{r,\ell} :=\lambda_r[\ell+1:C]k_{r,\ell}.$$ </div> <div class="row crow" id="gState"></div> <p>The right arrow denotes decay from each write position to the chunk's right boundary. The state update is</p> <div class="eqn"> $$S_{r+1} =\lambda_r[1:C]S_r+\overrightarrow K_r^{\mathsf T}V_r.$$ </div> <h4 id="complete-chunkwise-form-1">Complete chunkwise form</h4> <p>Combining the output and state equations gives</p> <div class="eqn"> $$\begin{aligned} O_r &=\underbrace{\overleftarrow Q_rS_r}_{\text{history}} +\underbrace{\bigl(Q_rK_r^{\mathsf T}\odot\Gamma_{C,r}\bigr)V_r}_{\text{local}},\\ S_{r+1} &=\lambda_r[1:C]S_r+\overrightarrow K_r^{\mathsf T}V_r. \end{aligned}$$ </div> <p>Mamba-2 therefore keeps RetNet's scalar state transition but lets each token choose how much history survives. The gate remains scalar, so it scales every entry of the state equally.</p> <aside class="note-card"> <div class="note-card-title">Optional: Discretization convention</div> <div class="note-card-body"><p>Discretization affects both the decay and the new write. Mamba-2 also multiplies the write by the step size $\Delta_t$, so the literal recurrence is</p> <div class="eqn"> $$S_t=a_tS_{t-1}+k_t^{\mathsf T}(\Delta_t v_t^{\mathrm{raw}}).$$ </div> <p>To keep the same notation as the preceding sections, define</p> <div class="eqn"> $$v_t:=\Delta_t v_t^{\mathrm{raw}}.$$ </div> <p>All equations above use this rescaled $v_t$.</p> </div> </aside> <h2 id="deltanet-correcting-memory-collisions">DeltaNet: correcting memory collisions</h2> <p>Mamba-2 decays the entire history at once. Sometimes we instead want to correct the association addressed by the current key without uniformly decaying the entire memory. <a rel="external" href="https://arxiv.org/abs/2406.06484">DeltaNet</a> provides this more selective update.</p> <h3 id="why-additive-writes-collide">Why additive writes collide</h3> <p>Treat $S$ as a key–value associative memory. For each association $(k_i,v_i)$, we would like to retrieve $v_i$ by reading the memory with $k_i$:</p> <div class="eqn"> $$k_iS\approx v_i.$$ </div> <p>However, for a unit-norm key $k_i$, the read contains</p> <div class="eqn"> $$\begin{aligned} k_iS &=\sum_{j=1}^{m}(k_i k_j^{\mathsf T})v_j\\ &=v_i+\underbrace{\sum_{j\ne i}(k_i k_j^{\mathsf T})v_j}_{\text{cross-talk}}. \end{aligned}$$ </div> <p>The first term is the intended value. The other terms come from keys that overlap with $k_i$. Orthogonal keys eliminate this interference, but learned keys generally are not mutually orthogonal. A raw outer-product write can therefore collide with associations already stored in $S$.</p> <p>DeltaNet first asks what value the current state associates with $k_t$:</p> <div class="eqn"> $$\widehat v_t=k_tS_{t-1},$$ </div> <p>The residual $v_t-\widehat v_t$ points from this prediction toward the desired value $v_t$. DeltaNet writes a step in that direction at $k_t$:</p> <div class="eqn"> $$S_t =S_{t-1}+\beta_t k_t^{\mathsf T}(v_t-\widehat v_t), \qquad 0<\beta_t\le 1.$$ </div> <h3 id="an-intuitive-example">An intuitive example</h3> <p>Consider a simplified memory with two associations. The labels below stand for key and value vectors. Assume the Alice and Bob keys are orthogonal and unit norm, so a write at Bob's key does not change the value read at Alice's key.</p> <div class="dn-memory-example" aria-label="Delta rule memory update example"> <div class="dn-memory-card"> <div class="dn-memory-title">Current memory</div> <div class="dn-memory-row"><strong>Alice</strong><span>at school</span></div> <div class="dn-memory-row"><strong>Bob</strong><span>at home</span></div> </div> <div class="dn-memory-arrow">+</div> <div class="dn-memory-card dn-memory-new"> <div class="dn-memory-title">New association</div> <div class="dn-memory-row"><strong>Bob</strong><span>at the office</span></div> </div> </div> <p>Reading the state with Bob's key first returns “at home.” The target is “at the office,” so DeltaNet writes the difference between those two value vectors at Bob's key. Reading Bob after the update gives</p> <div class="eqn"> $$k_{\mathrm{Bob}}S_{\mathrm{new}} =(1-\beta)\,v_{\mathrm{home}} +\beta\,v_{\mathrm{office}}.$$ </div> <p>If $\beta=1$, the new value replaces the old prediction exactly. A smaller $\beta$ makes a partial correction.</p> <p>More generally, when $\lVert k_t\rVert_2=1$,</p> <div class="eqn"> $$k_tS_t=(1-\beta_t)\widehat v_t+\beta_t v_t.$$ </div> <p>The unit-norm condition matters. Without it, reading the update back with $k_t$ introduces an additional factor $\lVert k_t\rVert_2^2$.</p> <aside class="note-card"> <div class="note-card-title">Optional: Test-time training (TTT) interpretation</div> <div class="note-card-body"><p>Another interpretation treats $S$ as a small linear model that maps keys to values. The pair $(k_t,v_t)$ supplies one online training example with squared-error loss</p> <div class="eqn"> $$\mathcal L_t(S)=\frac12\lVert k_tS-v_t\rVert_2^2.$$ </div> <p>Its gradient is</p> <div class="eqn"> $$\nabla_S\mathcal L_t(S)=k_t^{\mathsf T}(k_tS-v_t).$$ </div> <p>One gradient step from $S_{t-1}$ with learning rate $\beta_t$ gives</p> <div class="eqn"> $$S_t =S_{t-1}-\beta_t\nabla_S\mathcal L_t(S_{t-1}) =S_{t-1}+\beta_tk_t^{\mathsf T}(v_t-k_tS_{t-1}),$$ </div> <p>which is exactly the DeltaNet update. This view connects the delta rule to <a rel="external" href="https://arxiv.org/abs/2407.04620"><em>Learning to (Learn at Test Time)</em></a>: the forward pass updates the recurrent state as a fast-weight model.</p> <p>The same view also recovers the preceding recurrences. With a unit learning rate:</p> <div class="ttt-table-wrap"> <table class="ttt-table"> <thead> <tr> <th>Method</th> <th>Online loss $\mathcal L_t(S)$</th> <th>State update</th> </tr> </thead> <tbody> <tr> <td>Naïve</td> <td>$-\langle k_tS,v_t\rangle$</td> <td>$S_t=S_{t-1}+k_t^{\mathsf T}v_t$</td> </tr> <tr> <td>RetNet</td> <td>$-\langle k_tS,v_t\rangle+\dfrac{1-\gamma}{2}\lVert S\rVert_F^2$</td> <td>$S_t=\gamma S_{t-1}+k_t^{\mathsf T}v_t$</td> </tr> <tr> <td>Mamba-2</td> <td>$-\langle k_tS,v_t\rangle+\dfrac{1-a_t}{2}\lVert S\rVert_F^2$</td> <td>$S_t=a_tS_{t-1}+k_t^{\mathsf T}v_t$</td> </tr> </tbody> </table> </div> <p>From the TTT view, Naïve linear attention has an unbounded objective: increasing $S$ along $k_t^{\mathsf T}v_t$ can drive the linear loss toward $-\infty$. RetNet adds a fixed $L_2$ penalty on the state, while Mamba-2 makes the strength of that penalty token-dependent through $a_t$.</p> </div> </aside> <p>The example explains one token update. We now derive the recurrent and full-sequence matrix forms.</p> <h3 id="recurrent-form">Recurrent form</h3> <p>Define the scaled correction</p> <div class="eqn"> $$u_t:=\beta_t(v_t-\widehat v_t).$$ </div> <p>The complete token recurrence is</p> <div class="eqn"> $$\begin{aligned} \widehat v_t&=k_tS_{t-1},\\ u_t&=\beta_t(v_t-\widehat v_t),\\ S_t&=S_{t-1}+k_t^{\mathsf T}u_t,\\ o_t&=q_tS_t. \end{aligned}$$ </div> <div class="flow" id="flow"></div> <p>Once $u_t$ is known, the state has the same additive form as Naïve linear attention:</p> <div class="eqn"> $$S_t=\sum_{i=1}^{t}k_i^{\mathsf T}u_i.$$ </div> <p>Unlike the raw values $v_t$, the corrections $u_t$ are causally coupled: each $u_t$ depends on earlier corrections through $S_{t-1}$. We therefore cannot form all rows of $U$ independently as in Naïve linear attention. We first collect this dependency into a triangular system and solve for $U$; only then can we reuse the linear-attention readout.</p> <h3 id="the-triangular-correction-system">The triangular correction system</h3> <p>Substitute</p> <div class="eqn"> $$S_{t-1}=\sum_{i<t}k_i^{\mathsf T}u_i$$ </div> <p>into the prediction. Then</p> <div class="eqn"> $$u_t =\beta_t\left(v_t-\sum_{i<t}(k_tk_i^{\mathsf T})u_i\right).$$ </div> <p>For the first three tokens,</p> <div class="eqn"> $$\begin{aligned} u_1&=\beta_1v_1,\\ u_2&=\beta_2\left(v_2-(k_2k_1^{\mathsf T})u_1\right),\\ u_3&=\beta_3\left(v_3-(k_3k_1^{\mathsf T})u_1-(k_3k_2^{\mathsf T})u_2\right). \end{aligned}$$ </div> <p>Only earlier corrections appear on the right. Define the strictly lower-triangular matrix</p> <div class="eqn"> $$L[t,i] =\begin{cases} k_tk_i^{\mathsf T},&i<t,\\ 0,&i\ge t. \end{cases}$$ </div> <div class="dnrow" id="g-Lliteral"></div> <p>Let $M$ be the inclusive causal mask and $M^-:=M-I$ its strictly lower-triangular part. Then</p> <div class="eqn"> $$L=(KK^{\mathsf T})\odot M^-.$$ </div> <div class="dnrow" id="g-Lfactor"></div> <p>Stack $u_t$ into $U\in\mathbb R^{n\times d}$ and define $B:=\operatorname{diag}(\beta)$. The row recurrences become (as you should verify):</p> <div class="eqn"> $$\begin{aligned} U&=B(V-LU),\\ (I+BL)U&=BV. \end{aligned}$$ </div> <p>Equivalently,</p> <div class="eqn"> $$U=(I+BL)^{-1}BV.$$ </div> <h3 id="solving-for-u">Solving for $U$</h3> <p>We do not compute $(I+BL)^{-1}$ explicitly. Forming a dense $n\times n$ inverse costs $\mathcal O(n^3)$.</p> <p>Set $A:=I+BL$ and $R:=BV$. Since $A$ is unit lower triangular, <strong>forward substitution</strong> solves</p> <div class="eqn"> $$AU=R$$ </div> <p>one row at a time. At row $t$, the earlier rows of $U$ are already known, and $u_t$ is the only new row. (Hover a row to follow the known inputs and current unknown.)</p> <div class="row fs-row" id="g-forward"></div> <p>For $U\in\mathbb R^{n\times d}$, row $t$ costs $\mathcal O(td)$ and the full solve costs $\mathcal O(n^2d)$.</p> <h3 id="parallel-output">Parallel output</h3> <p>After solving for $U$, the state is a sum of key–correction outer products. The causal output therefore reuses the Naïve linear-attention equation with $U$ in place of $V$:</p> <div class="eqn"> $$\begin{aligned} O_{\mathrm{naïve}}&=(QK^{\mathsf T}\odot M)V,\\ O_{\mathrm{DeltaNet}}&=(QK^{\mathsf T}\odot M)U, \qquad U=[I+B(KK^{\mathsf T}\odot M^-)]^{-1}BV. \end{aligned}$$ </div> <h3 id="chunkwise-algorithm-1">Chunkwise algorithm</h3> <p>The full $n$-token triangular system still requires quadratic work. Divide the sequence into chunks of $C$ tokens instead. Each chunk starts from its incoming boundary state. Local dependencies then fit in a $C\times C$ system, while a $d\times d$ state carries history between chunks. This regrouping is exact.</p> <h4 id="one-chunk-as-an-affine-recurrence">One chunk as an affine recurrence</h4> <p>Let $r$ index chunks and $\ell\in{1,\ldots,C}$ index tokens in the current chunk. $S_r$ is the state before the chunk, and $S_{r+1}$ is the state after it. Rewrite the DeltaNet update as</p> <div class="eqn"> $$\begin{aligned} A_{r,\ell} &:=I-\beta_{r,\ell}k_{r,\ell}^{\mathsf T}k_{r,\ell},\\ X_{r,\ell} &:=\beta_{r,\ell}k_{r,\ell}^{\mathsf T}v_{r,\ell},\\ S_r^{(\ell)} &=A_{r,\ell}S_r^{(\ell-1)}+X_{r,\ell}, \qquad S_r^{(0)}=S_r,\quad S_r^{(C)}=S_{r+1}. \end{aligned}$$ </div> <p>For example, a chunk with $C=2$ contains two local transitions:</p> <div id="dn-chunk-rail" class="dn-chunk-rail" aria-label="DeltaNet recurrence through one two-token chunk"></div> <aside class="note-card"> <div class="note-card-title">Lemma: Explicit form of an affine recurrence</div> <div class="note-card-body"><p>So far, we have always described $S_t$ through a recurrence relation. For a numerical sequence, we often seek an explicit formula that gives its $t$-th term directly. Can we do the same for $S_t$?</p> <p>First, consider a constant transition $A$:</p> <div class="eqn"> $$S_t=AS_{t-1}+X_t.$$ </div> <p>The first two steps are</p> <div class="eqn"> $$\begin{aligned} S_1&=AS_0+X_1,\\ S_2&=A^2S_0+AX_1+X_2. \end{aligned}$$ </div> <p>The pattern gives the explicit formula</p> <div class="eqn"> $$S_t = A^tS_0 + \sum_{i=1}^{t}A^{t-i}X_i.$$ </div> <p>The same formula can start from any earlier state $S_\tau$:</p> <div class="eqn"> $$S_t = A^{t-\tau}S_\tau + \sum_{i=\tau+1}^{t}A^{t-i}X_i.$$ </div> <p>For $S_t$, the first term carries the earlier state $S_\tau$ forward. The second term is the <strong>convolution sum</strong>. Since $X_i$ enters at position $i$, the remaining $t-i$ transitions act on it.</p> <p>How about the general affine recurrence, where the transition matrix changes with $t$?</p> <div class="eqn"> $$S_t=A_tS_{t-1}+X_t.$$ </div> <p>Powers of one matrix no longer suffice. For $i\le j$, define the ordered transition product</p> <div class="eqn"> $$\Lambda[i:j]:=A_jA_{j-1}\cdots A_i,$$ </div> <p>and let $\Lambda[i:j]=I$ when $i>j$. The constant-transition formula becomes (as you should verify)</p> <div class="eqn"> $$S_t = \Lambda[1:t]S_0 + \sum_{i=1}^{t}\Lambda[i+1:t]X_i.$$ </div> <p>Starting from any earlier state $S_\tau$ gives</p> <div class="eqn"> $$S_t = \Lambda[\tau+1:t]S_\tau + \sum_{i=\tau+1}^{t}\Lambda[i+1:t]X_i.$$ </div> <p>Here $\Lambda[i+1:t]$ is the time-varying counterpart of $A^{t-i}$.</p> </div> </aside> <p>Use $\Lambda_r[i:j]$ for the ordered transition product over local positions $i$ through $j$ in chunk $r$. Applying the lemma to the incoming state $S_r$ gives</p> <div class="eqn"> $$\begin{aligned} S_{r+1} &=\Lambda_r[1:C]S_r+H_r,\\ H_r &:=\sum_{i=1}^{C}\Lambda_r[i+1:C]X_{r,i}. \end{aligned}$$ </div> <p>The first term carries the incoming state across the chunk. $H_r$ is the within-chunk <strong>convolution sum</strong> of local writes. Computing either expression directly would require many dense $d\times d$ transition products.</p> <h4 id="compact-the-transition-product-with-wy">Compact the transition product with WY</h4> <aside class="note-card"> <div class="note-card-title">Householder transformations and WY</div> <div class="note-card-body"><p>For a unit row vector $h$, the matrix</p> <div class="eqn"> $$I-2h^{\mathsf T}h$$ </div> <p>is a Householder reflection. It reflects a vector across the hyperplane orthogonal to $h$. Now consider the more general factors</p> <div class="eqn"> $$G_i=I-k_i^{\mathsf T}v_i.$$ </div> <p>Each $G_i$ is the identity plus a rank-one update, a special case of <strong>diagonal plus low rank (DPLR)</strong>, a common transition structure in structured state-space models.</p> <p>The <strong>WY representation</strong> says that the ordered product can be written as</p> <div class="eqn"> $$G_mG_{m-1}\cdots G_1 = I-K^{\mathsf T}W$$ </div> <p>where $K$ stacks the rows $k_1,\ldots,k_m$ and $W\in\mathbb R^{m\times d}$. WY replaces the growing transition product with the two thin matrices $K$ and $W$.</p> </div> </aside> <p>For DeltaNet, the two vectors in each DPLR correction are $k_{r,\ell}$ and $\beta_{r,\ell}k_{r,\ell}$. We absorb $\beta_{r,\ell}$ into the triangular factor below. WY then gives</p> <div class="eqn"> $$\Lambda_r[1:C]=I-K_r^{\mathsf T}W_r.$$ </div> <p>To construct $W_r$, maintain the partial form</p> <div class="eqn"> $$\Lambda_r[1:\ell] = I-\sum_{i=1}^{\ell}k_{r,i}^{\mathsf T}w_{r,i}.$$ </div> <p>The first three rows of $W_r$ follow by substituting one transition at a time:</p> <div class="eqn"> $$\begin{aligned} w_{r,1} &=\beta_{r,1}k_{r,1},\\ w_{r,2} &=\beta_{r,2}\!\left( k_{r,2}-(k_{r,2}k_{r,1}^{\mathsf T})w_{r,1} \right),\\ w_{r,3} &=\beta_{r,3}\!\left( k_{r,3} -(k_{r,3}k_{r,1}^{\mathsf T})w_{r,1} -(k_{r,3}k_{r,2}^{\mathsf T})w_{r,2} \right). \end{aligned}$$ </div> <p>The general row is</p> <div class="eqn"> $$w_{r,\ell} = \beta_{r,\ell}\!\left( k_{r,\ell} -\sum_{i<\ell} (k_{r,\ell}k_{r,i}^{\mathsf T})w_{r,i} \right).$$ </div> <p>Let $M_c$ be the inclusive $C\times C$ causal mask, $M_c^-:=M_c-I$, and</p> <div class="eqn"> $$B_r:=\operatorname{diag}(\beta_r), \qquad L_r:=(K_rK_r^{\mathsf T})\odot M_c^-.$$ </div> <p>As in the full-sequence parallel form, stacking these row recurrences produces:</p> <div class="eqn"> $$(I+B_rL_r)W_r=B_rK_r, \qquad \Lambda_r[1:C]=I-K_r^{\mathsf T}W_r.$$ </div> <h4 id="compact-the-local-writes">Compact the local writes</h4> <p>The local convolution sum $H_r$ is the state produced by the tokens in chunk $r$ when the incoming state is zero. The full-sequence parallel form already showed that DeltaNet writes its state as $K^{\mathsf T}U$. Applying the same result within the chunk gives</p> <div class="eqn"> $$H_r=K_r^{\mathsf T}U_r.$$ </div> <p>The first three rows of $U_r$ are (as you may verify)</p> <div class="eqn"> $$\begin{aligned} u_{r,1} &=\beta_{r,1}v_{r,1},\\ u_{r,2} &=\beta_{r,2}\!\left( v_{r,2}-(k_{r,2}k_{r,1}^{\mathsf T})u_{r,1} \right),\\ u_{r,3} &=\beta_{r,3}\!\left( v_{r,3} -(k_{r,3}k_{r,1}^{\mathsf T})u_{r,1} -(k_{r,3}k_{r,2}^{\mathsf T})u_{r,2} \right). \end{aligned}$$ </div> <p>The general row is</p> <div class="eqn"> $$u_{r,\ell} = \beta_{r,\ell}\!\left( v_{r,\ell} -\sum_{i<\ell} (k_{r,\ell}k_{r,i}^{\mathsf T})u_{r,i} \right).$$ </div> <p>This is exactly the full-sequence correction recurrence, applied only within chunk $r$. Stacking the rows gives:</p> <div class="eqn"> $$(I+B_rL_r)U_r=B_rV_r.$$ </div> <aside class="note-card"> <div class="note-card-title">Optional: Use UT to share the triangular transform</div> <div class="note-card-body"><p>Solving for $W_r$ and $U_r$ separately would repeat the same lower-triangular solve. UT lets us solve once for a shared transform, then obtain both matrices with dense matrix multiplications.</p> <p><strong>UT representation.</strong> The WY form leaves $W$ implicit. For the DPLR product above, let $V$ stack the right factors $v_1,\ldots,v_m$. Because row $i$ of $W$ depends only on $v_1,\ldots,v_i$, we can write</p> <div class="eqn"> $$W=TV$$ </div> <p>for a lower-triangular $T\in\mathbb R^{m\times m}$. The WY representation then becomes</p> <div class="eqn"> $$G_mG_{m-1}\cdots G_1 = I-K^{\mathsf T}TV.$$ </div> <p>This equivalent form is the <strong>UT representation</strong>. It exposes the triangular transform from the original right factors $V$ to their accumulated form $W$.</p> <p>The derivations of $W_r$ and $U_r$ produced</p> <div class="eqn"> $$\begin{aligned} (I+B_rL_r)W_r&=B_rK_r,\\ (I+B_rL_r)U_r&=B_rV_r. \end{aligned}$$ </div> <p>The coefficient matrix is identical; only the right-hand side changes. Following UT, define $T_r\in\mathbb R^{C\times C}$ by</p> <div class="eqn"> $$(I+B_rL_r)T_r=B_r.$$ </div> <p>Right multiplication by $K_r$ or $V_r$ then gives</p> <div class="eqn"> $$\begin{aligned} W_r&=T_rK_r,\\ U_r&=T_rV_r. \end{aligned}$$ </div> <p>The compact transition product and convolution sum are therefore</p> <div class="eqn"> $$\Lambda_r[1:C]=I-K_r^{\mathsf T}T_rK_r, \qquad H_r=K_r^{\mathsf T}T_rV_r.$$ </div></div> </aside> <h4 id="correct-the-incoming-state">Correct the incoming state</h4> <p>Substitute the compact forms of $\Lambda_r[1:C]$ and $H_r$ into the unrolled recurrence:</p> <div class="eqn"> $$\begin{aligned} S_{r+1} &=(I-K_r^{\mathsf T}W_r)S_r+K_r^{\mathsf T}U_r\\ &=S_r+K_r^{\mathsf T}(U_r-W_rS_r). \end{aligned}$$ </div> <p>Define the actual correction rows as</p> <div class="eqn"> $$Z_r=U_r-W_rS_r.$$ </div> <p>$U_r$ contains the corrections obtained with an empty incoming state. $W_rS_r$ is the transformed contribution of the predictions from $S_r$. Their difference gives the corrections produced by the actual chunk:</p> <div class="eqn"> $$S_{r+1}=S_r+K_r^{\mathsf T}Z_r.$$ </div> <p>The update now has the same additive form as Naïve linear attention. We can therefore reuse its chunkwise output formula with $Z_r$ in place of $V_r$:</p> <div class="eqn"> $$O_r = Q_rS_r + (Q_rK_r^{\mathsf T}\odot M_c)Z_r.$$ </div> <details class="algorithm-fold"> <summary>Complete per-chunk algorithm</summary> <div class="algorithm-fold-body"> <p>For each chunk $r$,</p> <div class="eqn"> $$\begin{aligned} L_r &=(K_rK_r^{\mathsf T})\odot M_c^-,\\ (I+B_rL_r)T_r &=B_r,\\ W_r &=T_rK_r,\\ U_r &=T_rV_r,\\ Z_r &=U_r-W_rS_r,\\ O_r &=Q_rS_r+(Q_rK_r^{\mathsf T}\odot M_c)Z_r,\\ S_{r+1} &=S_r+K_r^{\mathsf T}Z_r. \end{aligned}$$ </div> <pre><code><p>The local quantities $L_r,T_r,W_r,$ and $U_r$ can be computed for all chunks in parallel. The boundary pass then threads $S_r$ through the chunks to form $Z_r$, $O_r$, and $S_{r+1}$.</p> </code></pre> </div> </details> <h2 id="gated-deltanet-gdn">Gated DeltaNet (GDN)</h2> <p><a rel="external" href="https://arxiv.org/abs/2412.06464">Gated DeltaNet (GDN)</a> extends DeltaNet by adding a token-dependent scalar decay to the state transition:</p> <div class="eqn"> $$\begin{aligned} \text{DeltaNet:}\qquad S_t &=(I-\beta_tk_t^{\mathsf T}k_t)S_{t-1} +\beta_tk_t^{\mathsf T}v_t,\\ \text{GDN:}\qquad S_t &=a_t(I-\beta_tk_t^{\mathsf T}k_t)S_{t-1} +\beta_tk_t^{\mathsf T}v_t. \end{aligned}$$ </div> <p>GDN adds scalar decay to DeltaNet in the same way that Mamba-2 adds scalar decay to Naïve linear attention. DeltaNet also has the same chunkwise form as Naïve linear attention after replacing $V_r$ with $Z_r$.</p> <div class="ttt-table-wrap"> <table class="ttt-table gdn-comparison-table"> <thead> <tr> <th>Method</th> <th>Chunkwise readout</th> <th>Chunkwise state update</th> </tr> </thead> <tbody> <tr> <td>Naïve</td> <td>$O_r=Q_rS_r+(Q_rK_r^{\mathsf T}\odot M_c)V_r$</td> <td>$S_{r+1}=S_r+K_r^{\mathsf T}V_r$</td> </tr> <tr class="group-end"> <td>Mamba-2</td> <td>$O_r=\overleftarrow Q_rS_r+(Q_rK_r^{\mathsf T}\odot\Gamma_{C,r})V_r$</td> <td>$S_{r+1}=\lambda_r[1:C]S_r+\overrightarrow K_r^{\mathsf T}V_r$</td> </tr> <tr class="group-start"> <td>DeltaNet</td> <td>$O_r=Q_rS_r+(Q_rK_r^{\mathsf T}\odot M_c)Z_r$</td> <td>$S_{r+1}=S_r+K_r^{\mathsf T}Z_r$</td> </tr> <tr> <td>GDN</td> <td>$O_r=\overleftarrow Q_rS_r+(Q_rK_r^{\mathsf T}\odot\Gamma_{C,r})Z_r$</td> <td>$S_{r+1}=\lambda_r[1:C]S_r+\overrightarrow K_r^{\mathsf T}Z_r$</td> </tr> </tbody> </table> </div> <aside class="note-card"> <div class="note-card-title">Decay also changes the correction rows</div> <div class="note-card-body"><p>GDN scales the entire correction by $\beta_t$:</p> <div class="eqn"> $$u_t=\beta_t\bigl(v_t-a_tk_tS_{t-1}\bigr).$$ </div> <p>Thus, $\beta_t$ scales both the target and the prediction. Within a chunk, $\Gamma_{C,r}$ also applies the scalar decay to the key–key coefficients in the triangular system, while $B_r$ incorporates $\beta_t$ into $T_r$.</p> <p>The prediction from the incoming state has decayed from the left chunk boundary. Therefore, replace</p> <div class="eqn"> $$W_r=T_rK_r \qquad\longrightarrow\qquad \overleftarrow W_r=T_r\overleftarrow K_r$$ </div> <p>and define</p> <div class="eqn"> $$Z_r=U_r-\overleftarrow W_rS_r.$$ </div> <p>The prediction uses the <strong>left arrow</strong> because it starts from the incoming state. The state update uses $\overrightarrow K_r$ because each correction decays from its own position to the right boundary.</p> </div> </aside> <details class="algorithm-fold"> <summary>Complete per-chunk algorithm</summary> <div class="algorithm-fold-body"> <p>For each chunk $r$, define the decay products and the correction gate:</p> <div class="eqn"> $$\begin{aligned} \lambda_r[i:j] &:=\prod_{s=i}^{j}a_{r,s}, \qquad \lambda_r[i:j]=1\ \text{when }i>j,\\ \Gamma_{C,r}[\ell,i] &:= \begin{cases} \lambda_r[i+1:\ell],&i\le\ell,\\ 0,&i>\ell, \end{cases}\\ B_r &:=\operatorname{diag}(\beta_{r,1},\ldots,\beta_{r,C}). \end{aligned}$$ </div> <div class="eqn"> $$\begin{aligned} \overleftarrow q_{r,\ell} &:=\lambda_r[1:\ell]q_{r,\ell},& \overleftarrow k_{r,\ell} &:=\lambda_r[1:\ell]k_{r,\ell},& \overrightarrow k_{r,\ell} &:=\lambda_r[\ell+1:C]k_{r,\ell}. \end{aligned}$$ </div> <div class="eqn"> $$\begin{aligned} L_r &=(K_rK_r^{\mathsf T}\odot\Gamma_{C,r})\odot M_c^-,\\ (I+B_rL_r)T_r &=B_r,\\ U_r &=T_rV_r,\\ \overleftarrow W_r &=T_r\overleftarrow K_r,\\ Z_r &=U_r-\overleftarrow W_rS_r,\\ O_r &=\overleftarrow Q_rS_r +(Q_rK_r^{\mathsf T}\odot\Gamma_{C,r})Z_r,\\ S_{r+1} &=\lambda_r[1:C]S_r+\overrightarrow K_r^{\mathsf T}Z_r. \end{aligned}$$ </div> </div> </details> <h2 id="gated-delta-product-gdp">Gated Delta Product (GDP)</h2> <p><a rel="external" href="https://arxiv.org/abs/2502.10297">Gated Delta Product (GDP)</a> replaces GDN's single generalized Householder transition per token with an ordered product of $n_h$ transitions.</p> <p>Each transition has its own key, value, and correction gate, all projected from the same input token. The transitions act along different learned key directions, increasing computation and expressiveness without changing the shape of $S_t$.</p> <p>For substep $j$, define</p> <div class="eqn"> $$A_{t,j}:=I-\beta_{t,j}k_{t,j}^{\mathsf T}k_{t,j}, \qquad X_{t,j}:=\beta_{t,j}k_{t,j}^{\mathsf T}v_{t,j}.$$ </div> <p>For example, with $n_h=3$, the token update is</p> <div class="eqn"> $$\begin{aligned} S_{t,0} &=a_tS_{t-1},\\ S_{t,1} &=A_{t,1}S_{t,0}+X_{t,1},\\ S_{t,2} &=A_{t,2}S_{t,1}+X_{t,2},\\ S_{t,3} &=A_{t,3}S_{t,2}+X_{t,3},\\ S_t &:=S_{t,3}, \qquad o_t=q_tS_t. \end{aligned}$$ </div> <p>The decay $a_t$ is applied before the first substep, so the previous state decays only once. The query reads the state only after all three updates have finished.</p> <h3 id="gdp-as-gdn-over-a-virtual-sequence">GDP as GDN over a virtual sequence</h3> <p>GDP can be written as an ordinary GDN recurrence over a longer sequence. Treat its $n_h$ substeps as consecutive positions in that sequence. We call the original positions <strong>real tokens</strong> and the expanded positions <strong>virtual steps</strong>.</p> <p>For $n_h=3$, one real token expands into three virtual steps. Two real tokens therefore become six:</p> <div class="gdp-table-wrap"> <table class="gdp-table"> <thead> <tr><th></th><th colspan="3" class="token-end">real token $t=1$</th><th colspan="3">real token $t=2$</th></tr> </thead> <tbody> <tr><th>virtual step $m$</th><td>1</td><td>2</td><td class="token-end">3</td><td>4</td><td>5</td><td>6</td></tr> <tr><th>substep $(t,j)$</th><td>$(1,1)$</td><td>$(1,2)$</td><td class="token-end">$(1,3)$</td><td>$(2,1)$</td><td>$(2,2)$</td><td>$(2,3)$</td></tr> <tr><th><span class="k">$\tilde k_m$</span></th><td>$k_{1,1}$</td><td>$k_{1,2}$</td><td class="token-end">$k_{1,3}$</td><td>$k_{2,1}$</td><td>$k_{2,2}$</td><td>$k_{2,3}$</td></tr> <tr><th><span class="v">$\tilde v_m$</span></th><td>$v_{1,1}$</td><td>$v_{1,2}$</td><td class="token-end">$v_{1,3}$</td><td>$v_{2,1}$</td><td>$v_{2,2}$</td><td>$v_{2,3}$</td></tr> <tr><th><span class="bt">$\tilde\beta_m$</span></th><td>$\beta_{1,1}$</td><td>$\beta_{1,2}$</td><td class="token-end">$\beta_{1,3}$</td><td>$\beta_{2,1}$</td><td>$\beta_{2,2}$</td><td>$\beta_{2,3}$</td></tr> <tr><th><span class="g">$\tilde a_m$</span></th><td>$a_1$</td><td>1</td><td class="token-end">1</td><td>$a_2$</td><td>1</td><td>1</td></tr> <tr><th><span class="q">$\tilde q_m$</span></th><td>0</td><td>0</td><td class="token-end">$q_1$</td><td>0</td><td>0</td><td>$q_2$</td></tr> <tr class="keep-row"><th>retained output</th><td>—</td><td>—</td><td class="token-end">$o_1$</td><td>—</td><td>—</td><td>$o_2$</td></tr> </tbody> </table> </div> <p>The decay appears at the first virtual step of each real token; the remaining steps use $1$. The query appears at the last virtual step; the earlier steps use zero queries because they do not produce outputs.</p> <p>To write this construction generally, flatten token $t$ and substep $j$ into</p> <div class="eqn"> $$m=n_h(t-1)+j.$$ </div> <p>Map the GDP inputs to this virtual sequence by setting</p> <div class="eqn"> $$\begin{aligned} (\tilde k_m,\tilde v_m,\tilde\beta_m) &=(k_{t,j},v_{t,j},\beta_{t,j}),\\ \tilde a_m &= \begin{cases} a_t,&j=1,\\ 1,&j>1, \end{cases}\\ \tilde q_m &= \begin{cases} q_t,&j=n_h,\\ 0,&j<n_h. \end{cases} \end{aligned}$$ </div> <p>Every virtual step then follows the GDN recurrence:</p> <div class="eqn"> $$\begin{aligned} \tilde S_m &=\tilde a_m (I-\tilde\beta_m\tilde k_m^{\mathsf T}\tilde k_m)\tilde S_{m-1} +\tilde\beta_m\tilde k_m^{\mathsf T}\tilde v_m,\\ S_t &=\tilde S_{tn_h}, \qquad o_t=\tilde q_{tn_h}\tilde S_{tn_h}. \end{aligned}$$ </div> <p>Flattening preserves the update order. We can therefore run the GDN chunkwise algorithm on a virtual sequence of length $nn_h$ and retain every $n_h$-th output. Only the sequence becomes longer; the recurrent state keeps the same shape.</p> <h2 id="gated-linear-attention-gla">Gated Linear Attention (GLA)</h2> <p>Mamba-2 lets each token choose how much of the previous state survives, but its gate is scalar:</p> <div class="eqn"> $$S_t=a_tS_{t-1}+k_t^{\mathsf T}v_t.$$ </div> <p>The same $a_t$ scales every row of $S_{t-1}$. It can shorten or extend the lifetime of the memory as a whole, but it cannot preserve one channel while rapidly forgetting another.</p> <p><a rel="external" href="https://arxiv.org/abs/2312.06635">Gated Linear Attention (GLA)</a> replaces the scalar gate with a vector</p> <div class="eqn"> $$\boldsymbol a_t=(a_{t,1},\ldots,a_{t,d})\in(0,1)^d,$$ </div> <p>and uses the recurrence</p> <div class="eqn"> $$S_t=\operatorname{Diag}(\boldsymbol a_t)S_{t-1}+k_t^{\mathsf T}v_t, \qquad o_t=q_tS_t.$$ </div> <p>The two gates scale the same state differently:</p> <div id="gla-gate-compare" class="gla-gate-compare" aria-label="Comparison of scalar state decay in Mamba-2 and channelwise state decay in GLA" ></div> <p>GLA therefore gives each channel its own forgetting rate.</p> <h3 id="chunkwise-form-2">Chunkwise form</h3> <p>Multiplying diagonal decay matrices is equivalent to multiplying their diagonal vectors elementwise:</p> <div class="eqn"> $$\operatorname{Diag}(\boldsymbol a_{r,i}) \cdots \operatorname{Diag}(\boldsymbol a_{r,j}) = \operatorname{Diag}\left( \boldsymbol a_{r,i}\odot\cdots\odot\boldsymbol a_{r,j} \right).$$ </div> <p>We therefore replace Mamba-2's scalar prefix product with a vector prefix product. Within chunk $r$, define</p> <div class="eqn"> $$\boldsymbol\lambda_r[i:j] :=\bigodot_{s=i}^{j}\boldsymbol a_{r,s}, \qquad \boldsymbol\lambda_r[i:j]=\boldsymbol 1\ \text{when }i>j.$$ </div> <p>Here, $\bigodot$ denotes a repeated Hadamard (elementwise) product.</p> <p>After a write enters the state, later tokens decay each of its channels independently:</p> <div id="gla-channel-decay" class="gla-channel-decay" aria-label="One state write decaying at different rates across three channels" ></div> <p><strong>Incoming state.</strong> The incoming state reaches local position $\ell$ through $\boldsymbol\lambda_r[1:\ell]$. Stack these vectors and absorb them into $Q_r$:</p> <div class="eqn"> $$\overleftarrow q_{r,\ell} := \boldsymbol\lambda_r[1:\ell]\odot q_{r,\ell}, \qquad \overleftarrow Q_r := \begin{bmatrix} \boldsymbol\lambda_r[1:1]\\ \boldsymbol\lambda_r[1:2]\\ \vdots\\ \boldsymbol\lambda_r[1:C] \end{bmatrix} \odot Q_r.$$ </div> <p><strong>Local writes.</strong> A write at position $m$ reaches position $\ell$ through $\boldsymbol\lambda_r[m+1:\ell]$. Its local score is therefore</p> <div class="eqn"> $$A_r[\ell,m] = \begin{cases} q_{r,\ell} \bigl(\boldsymbol\lambda_r[m+1:\ell]\odot k_{r,m}\bigr)^{\mathsf T}, &m\le\ell,\\ 0,&m>\ell. \end{cases}$$ </div> <p>For the state update, absorb the decay from each write to the right boundary into its key:</p> <div class="eqn"> $$\overrightarrow k_{r,\ell} :=\boldsymbol\lambda_r[\ell+1:C]\odot k_{r,\ell}, \qquad \overrightarrow K_r := \begin{bmatrix} \boldsymbol\lambda_r[2:C]\\ \boldsymbol\lambda_r[3:C]\\ \vdots\\ \boldsymbol\lambda_r[C+1:C] \end{bmatrix} \odot K_r.$$ </div> <p>The complete chunkwise equations are</p> <div class="eqn"> $$\begin{aligned} O_r &=\overleftarrow Q_rS_r+A_rV_r,\\ S_{r+1} &=\operatorname{Diag}\bigl(\boldsymbol\lambda_r[1:C]\bigr)S_r +\overrightarrow K_r^{\mathsf T}V_r. \end{aligned}$$ </div> <h2 id="kimi-delta-attention-kda">Kimi Delta Attention (KDA)</h2> <p><a rel="external" href="https://arxiv.org/abs/2510.26692">Kimi Delta Attention (KDA)</a> adds GLA's channelwise decay to GDN's delta rule:</p> <div class="eqn"> $$\begin{aligned} S_t &=(I-\beta_tk_t^{\mathsf T}k_t) \operatorname{Diag}(\boldsymbol a_t)S_{t-1} +\beta_tk_t^{\mathsf T}v_t\\ &=\operatorname{Diag}(\boldsymbol a_t)S_{t-1} +\beta_tk_t^{\mathsf T} \left( v_t-k_t\operatorname{Diag}(\boldsymbol a_t)S_{t-1} \right),\\ o_t&=q_tS_t. \end{aligned}$$ </div> <p>KDA first decays the state channel by channel, then applies the delta correction. If $\boldsymbol a_t=a_t\boldsymbol 1$, it reduces exactly to GDN.</p> <h3 id="reuse-gla-and-gdn">Reuse GLA and GDN</h3> <p>The GLA and GDN sections give most of the chunkwise algorithm. GLA gives the decayed matrices $\overleftarrow Q_r$ and $\overrightarrow K_r$, together with the local score matrix $A_r$. GDN shows that the delta rule replaces the value rows $V_r$ with correction rows $Z_r$. Therefore,</p> <div class="eqn"> $$\begin{aligned} O_r &=\overleftarrow Q_rS_r+A_rZ_r,\\ S_{r+1} &=\operatorname{Diag}\bigl(\boldsymbol\lambda_r[1:C]\bigr)S_r +\overrightarrow K_r^{\mathsf T}Z_r. \end{aligned}$$ </div> <p>It remains to compute $Z_r$. KDA reuses GDN's correction solve but replaces scalar decay with channelwise decay. In this solve, decay appears in two quantities: $L_r$ and $\overleftarrow K_r$.</p> <aside class="note-card"> <div class="note-card-title">Recap: GDN correction solve</div> <div class="note-card-body"><p>With scalar decay, GDN defines</p> <div class="eqn"> $$\begin{aligned} \overleftarrow k_{r,\ell} &:=\lambda_r[1:\ell]k_{r,\ell},\\ L_r[\ell,m] &:= \begin{cases} \lambda_r[m+1:\ell] \bigl(k_{r,\ell}k_{r,m}^{\mathsf T}\bigr),&m<\ell,\\ 0,&m\ge\ell. \end{cases} \end{aligned}$$ </div> <p>These quantities enter the triangular correction solve:</p> <div class="eqn"> $$\begin{aligned} (I+B_rL_r)T_r&=B_r,& U_r&=T_rV_r,\\ \overleftarrow W_r&=T_r\overleftarrow K_r,& Z_r&=U_r-\overleftarrow W_rS_r. \end{aligned}$$ </div></div> </aside> <p>KDA keeps this structure and replaces the scalar decay products with channelwise products:</p> <div class="ttt-table-wrap"> <table class="ttt-table kda-gdn-table"> <thead> <tr> <th>Quantity</th> <th>GDN: scalar decay</th> <th>KDA: channelwise decay</th> </tr> </thead> <tbody> <tr> <td>Decay from $i$ through $j$</td> <td>$\lambda_r[i:j]=\prod_{s=i}^{j}a_{r,s}$</td> <td>$\boldsymbol\lambda_r[i:j]=\bigodot_{s=i}^{j}\boldsymbol a_{r,s}$</td> </tr> <tr> <td>Incoming prediction key</td> <td>$\overleftarrow k_{r,\ell}=\lambda_r[1:\ell]k_{r,\ell}$</td> <td>$\overleftarrow k_{r,\ell}=\boldsymbol\lambda_r[1:\ell]\odot k_{r,\ell}$</td> </tr> <tr> <td>$L_r[\ell,m]$ for $m<\ell$</td> <td>$\lambda_r[m+1:\ell](k_{r,\ell}k_{r,m}^{\mathsf T})$</td> <td>$k_{r,\ell}\bigl(\boldsymbol\lambda_r[m+1:\ell]\odot k_{r,m}\bigr)^{\mathsf T}$</td> </tr> </tbody> </table> </div> <p>Using the new $L_r$ and $\overleftarrow K_r$, reuse GDN's triangular transform. Let $B_r:=\operatorname{diag}(\beta_{r,1},\ldots,\beta_{r,C})$. Then</p> <div class="eqn"> $$\begin{aligned} (I+B_rL_r)T_r &=B_r,\\ U_r &=T_rV_r,\\ \overleftarrow W_r &=T_r\overleftarrow K_r,\\ Z_r &=U_r-\overleftarrow W_rS_r. \end{aligned}$$ </div> <details class="algorithm-fold"> <summary>Complete per-chunk algorithm</summary> <div class="algorithm-fold-body"> <p>For each chunk $r$, define the channelwise decay products and correction gate:</p> <div class="eqn"> $$\begin{aligned} \boldsymbol\lambda_r[i:j] &:=\bigodot_{s=i}^{j}\boldsymbol a_{r,s}, \qquad \boldsymbol\lambda_r[i:j]=\boldsymbol 1\ \text{when }i>j,\\ B_r &:=\operatorname{diag}(\beta_{r,1},\ldots,\beta_{r,C}). \end{aligned}$$ </div> <div class="eqn"> $$\begin{aligned} \overleftarrow q_{r,\ell} &:=\boldsymbol\lambda_r[1:\ell]\odot q_{r,\ell},& \overleftarrow k_{r,\ell} &:=\boldsymbol\lambda_r[1:\ell]\odot k_{r,\ell},& \overrightarrow k_{r,\ell} &:=\boldsymbol\lambda_r[\ell+1:C]\odot k_{r,\ell}. \end{aligned}$$ </div> <p>Stack the transformed rows into $\overleftarrow Q_r$, $\overleftarrow K_r$, and $\overrightarrow K_r$. Define the two causal within-chunk matrices:</p> <div class="eqn"> $$\begin{aligned} A_r[\ell,m] &:= \begin{cases} q_{r,\ell} \bigl(\boldsymbol\lambda_r[m+1:\ell]\odot k_{r,m}\bigr)^{\mathsf T}, &m\le\ell,\\ 0,&m>\ell, \end{cases}\\ L_r[\ell,m] &:= \begin{cases} k_{r,\ell} \bigl(\boldsymbol\lambda_r[m+1:\ell]\odot k_{r,m}\bigr)^{\mathsf T}, &m<\ell,\\ 0,&m\ge\ell. \end{cases} \end{aligned}$$ </div> <div class="eqn"> $$\begin{aligned} (I+B_rL_r)T_r &=B_r,\\ U_r &=T_rV_r,\\ \overleftarrow W_r &=T_r\overleftarrow K_r,\\ Z_r &=U_r-\overleftarrow W_rS_r,\\ O_r &=\overleftarrow Q_rS_r+A_rZ_r,\\ S_{r+1} &=\operatorname{Diag}\bigl(\boldsymbol\lambda_r[1:C]\bigr)S_r +\overrightarrow K_r^{\mathsf T}Z_r. \end{aligned}$$ </div> </div> </details> <h2 id="sources-and-further-reading">Sources and further reading</h2> <h3 id="related-papers">Related papers</h3> <p><a rel="external" href="https://arxiv.org/abs/2006.16236"><em>Transformers are RNNs</em></a>, <a rel="external" href="https://arxiv.org/abs/2307.08621"><em>RetNet</em></a>, <a rel="external" href="https://arxiv.org/abs/2312.06635"><em>Gated Linear Attention (GLA)</em></a>, <a rel="external" href="https://arxiv.org/abs/2405.21060"><em>Mamba-2 / SSD</em></a>, <a rel="external" href="https://arxiv.org/abs/2406.06484"><em>DeltaNet</em></a>, <a rel="external" href="https://arxiv.org/abs/2412.06464"><em>Gated DeltaNet (GDN)</em></a>, <a rel="external" href="https://arxiv.org/abs/2502.10297"><em>DeltaProduct (GDP)</em></a>, and <a rel="external" href="https://arxiv.org/abs/2510.26692"><em>Kimi Linear / Kimi Delta Attention (KDA)</em></a></p> <h3 id="blogs">Blogs</h3> <ul> <li>Tri Dao's <em>State Space Duality (Mamba-2)</em> series: <a rel="external" href="https://tridao.me/blog/2024/mamba2-part1-model/">Part I</a>, <a rel="external" href="https://tridao.me/blog/2024/mamba2-part2-theory/">Part II</a>, <a rel="external" href="https://tridao.me/blog/2024/mamba2-part3-algorithm/">Part III</a>, and <a rel="external" href="https://tridao.me/blog/2024/mamba2-part4-systems/">Part IV</a></li> <li>Songlin Yang's <em>DeltaNet Explained</em> series: <a rel="external" href="https://sustcsonglin.github.io/blog/2024/deltanet-1/">Part I</a>, <a rel="external" href="https://sustcsonglin.github.io/blog/2024/deltanet-2/">Part II</a>, and <a rel="external" href="https://sustcsonglin.github.io/blog/2024/deltanet-3/">Part III</a></li> <li>Jianlin Su's <a rel="external" href="https://kexue.fm/archives/11033"><em>《线性注意力简史：从模仿、创新到反哺》</em></a> <em>(in Chinese)</em></li> </ul>