Symmetry in Language Model Representations

Symmetry in language statistics shapes the geometry of model representations Dhruva Karkada¹ Daniel J. Korchinski² Andres Nava³ Matthieu Wyrt³ Yasaman Bahri⁴ Abstract The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match text embedding models and large language models. Moreover, the representational geometry persists in sociocultural embeddings and phenomena when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. These results suggest that representational manifolds have a universal origin: symmetry in the statistics of natural data. In this work, we provide such an organizing principle: that representational geometry reflects pairwise co-occurrence statistics between words. In particular, we give evidence that symmetry governs these statistics and drives the formation of representational manifolds. This hypothesis underpins our theory and explains the following phenomena: • Circles in representation space for cyclical concepts. E.g. (Engels et al., 2024; Model et al., 2025) find that LLMs encode concepts such as days of the week, months of the year, or hues on the color wheel with circular geometry. Gronau (2023) and related work suggest that circular representations may enable LLMs to efficiently compute modular addition tasks (“Five months after November is”). • “Ripped” 1D manifolds for continuous sequences. Engels et al. (2024); Gurnee et al. (2025) find that LLM representations correspondingly to continuous (e.g. years of history or number lines) are organized in compact 1D manifolds with so-called “ripples,” or extrinsic curvature. The LLM exploits this manifold geometry to compute increments and decrements. • Linear decoding of spatiotemporal coordinates. Gurnee & Gemburg (2023) find that LLM representations of geographical locations and historical events linearly encode their spatial and temporal coordinates, respectively. Thus, LLMs may straightforwardly compute maps and timelines by linear transformations. To test this hypothesis, we first examine word embedding models, whose representation learning is driven exclusively by pairwise co-occurrence statistics. Surprisingly, we find that each of these geometries arise in this simple setting, we develop a mathematical theory that unifies and explains these seemingly disparate observations, deriving the representation geometries from translation symmetry in the pairwise co-occurrence statistics of words. For example, when the co-occurrence probability of two events or places depends only on their temporal or spatial distance, the models spontaneously learn Fourier representations, i.e. the embedding vectors vary sinusoidally along each principal component. We analytically predict both the amplitude and frequency of each component, showing that loops and rings in the manifold geometry correspond to the long-wavelength modes, while ripples are the higher harmonics. 1. Introduction The semantic information encoded in the trained parameters of an LLM is reflected in the vector geometry of its internal representations. A growing body of empirical work has documented a number of intriguing geometric structures in this neural code, including prisms (Mikolov et al., 2013; Park et al., 2023; Merullo et al., 2024), loops (Engels et al., 2024; Modell et al., 2025), and continuous 1D manifolds (Gurnee et al., 2025). Notably, these structures emerge consistently across a range of model architectures and tasks. Despite this apparent universality, we lack an organizing principle that explains why these patterns arise. * Work done in part during an internship at Google DeepMind. † UC Berkeley ²EPFL ³Johns Hopkins ⁴Google DeepMind. Correspondence: dhruva@berkeley.edu, daniel.korchinski@epfl.ch, anava1@jhu.edu, mwyrt@jhu.edu, yasaman.bahri@gmail.com. Preprint. February 27, 2026. arXiv:2602.15029v2 [cs.LG] 25 Feb 2026

A screenshot of the first page of an academic preprint titled “Symmetry in language statistics shapes the geometry of model representations.” The joke comes from presenting a dense, highly technical paper as a reaction to the claim that language models are merely stochastic parrots that memorize training data, implying that the paper’s mathematical analysis demonstrates deeper structure and organization in learned representations.

• @dev_meme / 7797 2026-03-06