StateBench is a new benchmark for long-context state tracking: how well language models keep track of information about entities in a document as that information changes. The motivation is practical—state tracking underlies reading code with reassigned variables, following characters as they move and objects change hands, and updating game scores—but the authors argue not all state-tracking problems have the same difficulty. Some only require finding the latest update; harder cases require following a chain of changes, where one missed step corrupts the final answer. Transformers have documented theoretical limits on hard variants with long dependency chains, which has motivated recurrent and hybrid architectures such as Gated DeltaNet that carry a running memory forward. Since existing tests focus on the hardest cases, StateBench is designed to cover a spectrum of complexity.

The benchmark defines four difficulty levels. r-trivial problems use immutable assignment, so the model just recovers the first value. Aperiodic problems allow reassignment by value, so the model finds the most recent update. Soluble problems add self-referential updates, such as x = x + 5, requiring the model to carry changes forward. General finite problems add assignment by reference, e.g., x = y or swapping variables, which creates dependencies across updates; the S₅ cup-and-ball swapping problem is an example. These levels are theoretically motivated through Krohn–Rhodes theory, which treats state-tracking problems as word problems over finite monoids and decomposes them into atomic primitives. To make levels comparable, the authors derive four problem versions from the same general finite sequence of variable assignments, holding variables, values, and assignment positions fixed and changing only the operation type. The hierarchy aligns with theory: under standard assumptions, transformers can handle r-trivial robustly, and soluble and aperiodic to some degree, but are predicted to fail on general problems, while Gated DeltaNet with a more flexible memory update can in principle handle all four.

The evaluation covers three settings. First, open pretrained models: Olmo 3 (a 7B transformer), Olmo Hybrid (which adds Gated DeltaNet layers), and smaller 275M and 2.7B models, all pretrained on the same data and none on contexts longer than 64,000 tokens; most were tested beyond that length. Most models did well on r-trivial, lost accuracy on aperiodic as inputs grew, and stayed at chance on general finite problems. Format mattered: the 275M hybrid and Gated DeltaNet models were near ceiling on RULER-style r-trivial problems but near chance on spreadsheet-edit versions, which is why StateBench includes multiple formats for the same underlying problem. Second, four frontier models—Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna—performed well on r-trivial and aperiodic, but accuracy on soluble and general finite problems fell sharply with longer inputs, dropping to 13% or lower on the longest general finite problems. Sensitivity is one likely factor: harder levels have more steps that can affect the final answer.

Third, the authors trained 275M-parameter models from scratch on StateBench: two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory-update constraints. Models trained on at most 256 assignments were tested on sequences as long as 64,000 assignments, a 250x length extrapolation. Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems out to the longest sequences; transformers learned easier levels but degraded on longer sequences and never learned general finite problems. On the hardest level, the Gated DeltaNet variant with a more flexible memory update generalized farthest, staying near perfect out to roughly 10 times its training length. This suggests that for recurrent and hybrid models the pretraining gap is not architectural, while transformers still cannot handle the hardest level even with direct task training.

The report also states its boundaries. A new proof shows that Gated DeltaNet with a more limited memory update (non-negative eigenvalues) can solve all aperiodic tasks and certain periodic tasks, such as even/odd repetition counts, under fixed-precision arithmetic with rounding; its exact place in the hierarchy remains open, including whether it can solve all soluble tasks and how numerical precision or attention layers affect its capabilities. StateBench assumes a fixed number of variables, whereas real conversations and documents introduce new entities over time. The authors frame the benchmark as a finer-grained diagnostic for where models break down and a north star for developing architectures with more robust state tracking.