Kronecker Delta Rule for Long Context Linear Attention

The paper proposes the Kronecker Delta Rule, which treats key-value pairs as joint addresses, reducing memory revise interference and stabilizing long context retrieval. When compared to other models, KronLSTM consistently achieves the highest aggregate score on language tasks, reducing LAMBADA perplexity and improving recall in hybrid settings. The Kronecker Delta Rule enhances memory capacity up to square theta values due to the unique built-in normalization reducing the state update size, thus optimizing throughput. related_questions Problem setting and central contribution “Kronecker Delta Rule for Long Context Linear Attention” (2610.26834) addresses a central failure mode of recurrent linear-attention models: a fixed-size associative memory must revise stored information without either accumulating unbounded state norms or perturbing unrelated associations. Standard Hebbian linear attention writes outer products additively, so memory magnitude can grow with sequence length. Delta-rule architectures constrain memory growth through error-correcting updates, but their vector-valued correction can interfere with all associations sharing key directions. The paper proposes the Kronecker Delta Rule, which treats a key–value pair as a joint address and corrects a scalar bilinear association rather than an entire retrieved value vector. Given memory $\BS \in \mathbb{R}^{d_v \times d_k}$, key $\Bk$, value $\Bv$, and target cc, the update is $\BS_{t+1}

\BS_t

\beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$ The corrected quantity is the scalar $\Bv_t^\top\BS_t\Bk_t$, and the update direction is fixed by the current pair $\Bv_t\Bk_t^\top$, rather than by the current retrieval residual $\BS_t\Bk_t-\Bv_t$. The resulting KronLSTM architecture embeds this rule in an xLSTM-style sequence mixer with separate forget, erase, write, and output gates. The paper’s principal claim is that targeted scalar correction provides lower interference than DeltaNet while retaining the bounded-memory and parallelization properties of delta-rule attention. The claim is both theoretical and empirical: the authors derive stability and convergence results for the stochastic recurrence and evaluate 1.5B-parameter models on language modeling, real-world recall, synthetic retrieval, context extension, state tracking, and throughput. From Hebbian writing to pair-addressed correction Linear attention represents the prefix of a sequence through a matrix state. In …