I recently read two Google Research works on test-time memory and continual learning [1, 2], and revisited the seminal paper that first made linear attention explicit [3]. The process reminded me once again how a shift in perspective can endow a theoretical framework with entirely new imaginative space. If you haven't experienced such a shift, it is easy to underestimate its significance. Consider the history of quantum mechanics:

(This is intentionally schematic: matrix and wave mechanics are equivalent formulations of the same quantum theory; and in quantum field theory, the operator/second-quantized and path-integral formulations are complementary languages, not a strict chronological sequence.)

Each revolution was not merely rewriting existing knowledge with new formulas; it was a re-examination of theory through a new lens and language. In this sense, mathematical language is not just the boundary of thought; it embeds part of the "geometry" of thinking—what becomes salient, manipulable, and even askable.

What space does Linear Attention open up?

Traditional attention mechanisms essentially perform two tasks:

1. Map the input \(X\) to Query, Key, and Value via learned projections: \(Q=XW_Q,\,K=XW_K,\,V=XW_V\).
2. Compute the output (softmax applied row-wise): \[ \mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V, \] ignoring masking for brevity.

The bottleneck is not "a global positional matrix," but the all-pairs interaction itself: global attention forms an \(n\times n\) matrix of query–key scores (often augmented with positional encodings or relative position biases), which incurs \(O(n^2)\) time and memory in the sequence length.

Kernelized (a.k.a. linear) attention replaces the softmax kernel with a feature map \(\phi:\mathbb{R}^{d_k}\to\mathbb{R}^{m}\) such that the similarity factorizes: \(\kappa(q,k)=\phi(q)^\top \phi(k)\). (A related but distinct line of work approximates the exponential kernel underlying softmax using random/explicit features; the algebra below covers the kernelized form and clarifies what must be normalized.)

In normalized kernel attention, the attention can be written as:

where \(\mathbf{1}\) is an all-ones vector and the division is row-wise. In practice, one often requires \(\phi(\cdot)\ge 0\) (or otherwise ensures positivity) so the denominator is well-behaved.

In the autoregressive (causal) setting, define the recurrent accumulators

then the per-token readout becomes

Here \(S_t\) has a fixed parameterization (constant memory footprint) that does not grow with the context length. The normalization accumulator \(z_t\) is essential for numerical stability and preventing scale drift as \(t\) grows. However, normalization does not eliminate information loss: compressing an ever-growing history into a fixed-size state inevitably induces recency bias and retrieval degradation unless the update rule has an explicit mechanism for selective write/forget. Katharopoulos et al. emphasize that this recursive implementation reveals an explicit RNN structure in causal linear attention: attention transforms from "scanning the entire KV list" to "updating a fixed-size recurrent state and reading out" [3].

This reformulation brings a new perspective. The intermediate state \(S_t\) plays the role of a matrix-valued hidden state: it is a fast-weight associative memory that encodes a (compressed) mapping from key features to values. Given a new query, the readout extracts the value associated with \(\phi(q_t)\) via the linear operator \(S_t\).

Associative memory is a central thread here: Hopfield's contribution is explicitly framed as an associative memory in the 2024 Nobel Prize press materials, while the broader prize citation credits foundational discoveries enabling machine learning with neural networks^*. Hebbian learning and synaptic plasticity—though far richer than any single toy model—remain foundational concepts in neuroscience.

The Wave Function Analogy

The introduction of the intermediate state \(S_t\) reminds me of the historical debate between Matrix Mechanics and Wave Mechanics. Heisenberg, deeply influenced by Machian empiricism/positivism, insisted that physical theory should be formulated in terms of observables—famously, spectral transition data—rather than unobservable classical trajectories.

Meanwhile, de Broglie, starting from the wave–particle duality of light, proposed that matter has a wave nature that invites a wave-function description. Historically, before Born's probabilistic interpretation became standard, it was natural to entertain ontological readings of this "matter wave." Although Schrödinger's and Heisenberg's formulations were soon shown to be mathematically equivalent, the wave function \(\psi(x)=\langle x|\psi\rangle\) retains geometric intuition as a particular representation of the abstract state.

In my view, the intermediate state \(S_t\) in linear attention is not an additional "ontological object," but a representation switch: it makes explicit the linear mapping (the association from key-feature space to value space) that was previously implicit in the one-shot attention computation, turning it into a recursively updatable operator/state. Just as the wave function provides an intuition-friendly representation of a quantum state in a chosen basis, making \(S_t\) explicit turns "what memory is," "how to write/forget," and "how to layer time-scales" into objects that can be directly designed and analyzed.

From Static to Dynamic: The Physics of "Surprise"

Building on this, Titans [1] proposes a deeper perspective shift: treat long-term memory as a parameterized module that can be updated at test time, and interpret "writing" as online optimization of an associative-memory objective. In particular, Titans argues that events that violate the model's expectations are more "memorable." It operationalizes this surprise using gradients of an associative-memory loss (used as a signal for what should be written), and combines it with an explicit decay mechanism for memory management; the paper further draws connections to mini-batch gradient descent with momentum and weight decay.

We are used to the forward direction: specify an objective and train parameters offline on a dataset (supervised or self-supervised) to minimize it. The online-learning lens flips what feels "given" versus "derived": we take the update dynamics of a particular state/parameter subset (the memory) seriously as a first-class object, and make the objective and regularization that the dynamics implements explicit. The surprise narrative is one concrete way to interpret that objective: routine events are compressed away; unexpected events receive disproportionate write bandwidth.

Anyone with training in theoretical physics will be sensitive to this move: modern physics often prefers scalar-valued functionals—Hamiltonians, Lagrangians, actions, partition functions—as organizing principles. They can encode symmetries and constraints cleanly, handle boundary conditions elegantly, and support powerful approximation tools. They may feel less "mechanistic" than Newton's vector dynamics, but they frequently make the structure of a theory more transparent.

Once we endow linear attention with the interpretations of associative memory and online optimization, we can systematically generalize the two-step picture. MIRAS [4] categorizes key degrees of freedom into four classes:

1. Memory Architecture: Replace the simple matrix-valued association (e.g., an outer-product accumulator) with a richer parameterization (e.g., a neural memory), trading off capacity, inductive bias, and regularization.
2. Objective: Design the internal loss ("attentional bias") that defines what the memory should fit.
3. Retention / Forget Gate: Memory and forgetting are two sides of the same coin; retention/decay can be learnable and can depend on surprise, capacity, or context.
4. Optimizer: Choose the online update rule (GD, SGD, Adam-like, learned optimizers, etc.).

To place this in a broader continual-learning narrative, Nested Learning [2] uses an analogy to anterograde amnesia: a person can retain long-past memories and experience a transient present, but cannot consolidate new experiences into long-term memory after onset. The authors argue that today's LLMs exhibit an analogous pattern: after pre-training, they can adapt within a context window, yet the information in that window typically does not get consolidated into persistent parameters.

Titans' test-time memorization breaks part of this barrier by allowing a long-term memory module to be updated online without retaining the full context buffer; it aims to distill the passing short-term context flow into a persistent state. In this framing, the foundation model (which implements \(X\mapsto Q,K,V\)) is responsible not only for broad world knowledge, but also for providing representations that make such consolidation efficient.

It is important to distinguish Titans' test-time memorization / online optimization view from Test-Time Training (TTT) layers [7]. TTT refers to Sun et al.'s approach in which the hidden state is itself a small learner (e.g., a linear model or an MLP) updated by a self-supervised learning step per token/mini-batch; Titans, in contrast, emphasizes a neural long-term memory module with surprise- and decay-modulated updates.

The Onion of Time: Nested Learning and RG

Nested Learning [2] argues that Titans-style designs are still "coarse" along one crucial axis: update frequency. A convenient caricature of many test-time memory schemes is a two-rate system:

• Slow/Outer loop (≈0 at deployment): foundation parameters are static at inference time.
• Fast/Inner loop (per-token): a memory state/module is updated with the token stream.

Nested Learning elevates update frequency into a design axis: assign each component an update rate and order nested or parallel learning problems into levels. This yields a memory system that is best viewed as a spectrum of time scales (often discretized into levels, but conceptually approaching a continuum): different modules compress and consolidate their respective context flows at different rates. In this view, Titans is a special case with essentially two update-frequency levels.

Human memory is often described as hierarchical: raw sensory registration feeds working memory, which can be encoded into episodic memory; through consolidation and abstraction, knowledge can become increasingly decontextualized into semantic knowledge and deeper cognitive schemas. Reality is messier than any single chain, but the multi-time-scale motif is robust. Nested Learning grounds this motif in neuroscience via neural oscillations spanning faster bands (e.g., gamma) to slower bands (e.g., delta/theta) that correlate with different roles in perception, cognition, and consolidation.

NL proposes an onion-like multi-level structure ordered by update rate:

• Innermost / closest-to-stream levels: operate directly on the token/context flow and update most frequently.
• Intermediate levels: consolidate and compress information from faster levels at intermediate rates.
• Outermost levels: correspond to the slowest loop (e.g., pre-training) and are effectively static at deployment.

(Titans can be viewed as an onion with only two prominent layers.)

"Hierarchical structure" triggers my physics instincts again. Physics is, to some extent, the study of scale. Our world is not scale-invariant; it contains characteristic lengths and times. Even when we lack a final theory of quantum gravity, it is meaningful to note that Planck units (built from \(c\), \(G\), and \(\hbar\)) define natural reference scales such as Planck length and Planck time.

Renormalization Group (RG) is a mathematical language for reasoning about how effective descriptions change with scale. Historically misnamed, RG refers to a composable flow across scales; in many coarse-graining settings the transformation is not strictly invertible, so it behaves more like a semigroup.

RG suggests a concrete style of question-asking: identify a coarse-graining transformation, study how effective parameters ("couplings") flow under repeated application, and characterize invariant structures (fixed points, critical manifolds, slow/invariant manifolds). Fixed points are special parameter settings that remain unchanged under the RG transformation; in statistical physics, critical behavior is associated with non-trivial fixed points where correlation lengths become large.

In deep learning, it is tempting—but nontrivial—to treat layer depth as a discrete representation "scale," and in NL to treat level index (ordered by update rate) as a time-scale axis. I do not claim a literal RG equivalence here; rather, I propose RG as an organizing language for multi-time-scale memory systems.

As an illustrative caricature (not a theorem), imagine two "trivial" endpoints of a memory system:

• IR-like endpoint (memorization): the model degenerates into a lookup table; weak generalization.
• UV-like endpoint (overwrite/noise): the system fails to retain useful information under continual updates. (Here "catastrophic forgetting" refers to continual-learning interference across tasks; earlier we used "recency bias" for within-sequence memory dilution.)

The interesting regime is the middle: an effective balance where the system compresses away nuisance variability while retaining stable, reusable structure. RG provides a principled vocabulary for such regimes via universality and the decomposition of perturbations into relevant, irrelevant, and marginal directions.

Conjecture (toward personalized AI): in personalization, the most valuable information is often neither "strongly relevant" (broad world knowledge already in the foundation model) nor "irrelevant" noise. It may live near marginal directions: aspects of a user's preferences, reasoning style, and values that should persist without destabilizing general capabilities. Making this precise would require operational definitions of (i) a scale transformation on the memory system (e.g., rescaling update intervals/retention and compensating parameters) and (ii) an observable that measures persistence vs interference. RG then suggests an empirical program: estimate which perturbations grow/decay under scale changes (effective scaling dimensions) and test whether a stable manifold exists for personalization dynamics.

The above is my recent reading experience. If you find it confusing, it's not your fault (and I hope not mine). I have not yet fully clarified how to map all degrees of freedom in [4] into a rigorous RG framework. I imagine the target reader as a scientist fluent in both physics and AI—a growing demographic. Physics still has much to offer AI, especially around "scale"—a treasure trove far richer than what today's "scaling laws" alone capture.