Bifurcations in Continuous Feedback of Language Models

The paper studies the dynamical behavior of transformer feedback maps, revealing periodic windows and chaotic regimes through non-linear dynamics techniques on various models from 0.5B to 72B parameters. The exact Jacobian factorization of the feedback map decomposes how temperature, token competition, and steering influence the map's dynamics, leading to qualitative predictions of periodic and chaotic behaviors. The analysis shows that chaos is a generic feature of transformer models and is sensitive to numerical execution, affecting the reproducibility of periodic and chaotic trajectories. Problem formulation and scope “Chaos and bifurcations in continuous language-model feedback” (2610.31692) studies a specific inference construction in which a pretrained transformer receives, at every iteration, the expected embedding of its own next-token distribution. With a fixed prompt and a one-token feedback window, this procedure defines an autonomous discrete-time map on the embedding space rather than an ordinary autoregressive process with growing context. The distinction is essential: the paper’s dynamical claims concern repeated application of a fixed transformer operator, not conventional sampled generation. For a model with input embedding matrix EE, logits zα(x)z_\alpha(x), temperature TT, and an internal steering intervention of strength α\alpha, the map is FT,α(x)=E⊤softmax⁡ ⁣(zα(x)T).F_{T,\alpha}(x)

E^\top \operatorname{softmax}!\left(\frac{z_\alpha(x)}{T}\right). The state xx is therefore continuous, even though the model ultimately defines a distribution over discrete tokens. Since the output is always a convex combination of token embeddings, every trajectory enters the compact convex hull of the embedding vectors after one iteration. For dense transformers, the resulting map is real analytic under the stated architectural assumptions. These properties make standard tools from nonlinear dynamics applicable: fixed points, periodic orbits, multipliers, Lyapunov exponents, continuation of bifurcation curves, center-manifold normal forms, and interval-arithmetic validation. The empirical study spans Qwen2.5, OLMo-2, and Llama-3 models from 0.5B to approximately 72B parameters. The detailed bifurcation analysis is concentrated on Qwen2.5-0.5B-Instruct, with additional confirmation in Qwen2.5-1.5B and larger models. Temperature and activation steering act as control parameters. The central claim is not merely that feedback can produce sensitivity, …