Clock and Pizza: Neural Manifold Insights

• Clock and Pizza interpretations are algorithmic frameworks where neural networks implement modular addition using distinct attention mechanisms—softmax for Clock and linear aggregation for Pizza.
• Geometric and topological analyses reveal that both approaches are projections of a universal manifold-level solution, unifying distinct network behaviors.
• Empirical evaluations using metrics like phase alignment, Betti numbers, and MMD demonstrate that architectural biases influence representation geometry while preserving end-to-end computation.

The Clock and Pizza interpretations characterize algorithmic solutions learned by neural network architectures—particularly transformers and MLPs—trained to implement modular addition. The modular addition task $(a,b)\mapsto(a+b)\bmod n$ serves as a prototypical instance for analyzing internal circuit mechanisms and their representation geometry. Early mechanistic interpretability efforts claimed that small differences in architectural biases induce distinct algorithmic circuits, labeled "Clock" (for trainable/multiplicative attention) and "Pizza" (for uniform/linear attention). However, recent geometric and topological analyses reveal that these interpretations are not fundamentally disjoint; rather, they are projections of a universal manifold-level solution, unified within a geometric–topological framework.

1. Definitions and Mechanistic Characterization

The modular addition task involves mapping integer pairs $(a,b)\in\mathbb{Z}_n^2$ (typically $n=59$, prime) to their sum modulo $n$. Models encode each input via learned embeddings $E_t\in\mathbb{R}^d$ and process the pair $[E_a;E_b]$ using a layer of either self-attention (parameterized by attention rate $\alpha$) or purely linear MLP. Zhong et al. (2023) formalized two principal solution archetypes, each suggested by diagnostics such as gradient symmetricity and logit pattern analysis (Zhong et al., 2023):

• Pizza (Attention 0.0): Uniform (fixed) attention yields a linear summation of input embeddings

$E_{ab} = E_a + E_b$

When embeddings are constructed on the unit circle

$E_a = [\cos(2\pi f\,a/n),\; \sin(2\pi f\,a/n)]$

the summed embedding $E_{ab}$ describes a filled disk in $\mathbb{R}^2$—the "pizza slice."

• Clock (Attention 1.0): Trainable (softmax) attention enables nonlinear angle addition,

$$E_{ab} = [\cos(2\pi f(a+b)/n),\; \sin(2\pi f(a+b)/n)]$$

encoding modular addition as a point on the unit circle.

The two procedures differ in their internal mechanism: the Clock algorithm executes explicit pairwise products (realizing trigonometric identities), while Pizza relies on linear aggregation and ReLU-induced nonlinearities. Both architectures output logits for candidate sums $c$ via dot product with output embeddings $U_c$.

2. Mathematical Structures: Embedding Spaces and Manifolds

The general form of input embeddings,

$$E_a = (\cos\theta_a, \sin\theta_a),\quad \theta_a = \frac{2\pi f a}{n}$$

provides a basis for analysis. With learned phase shifts $\phi^L, \phi^R$, neuron preactivations are well-modeled by

$$N(a,b) = \cos(\theta_a + \phi^L ) + \cos(\theta_b + \phi^R )$$

For neuron clusters tuned to frequency $f$ at a given layer $\ell$, the point cloud of preactivations forms a discrete manifold,

$\mathcal{M}^\ell_f = \{ \preactivationmap{\ell}(a,b) : a, b \in \mathbb{Z}_n \} \subset \mathbb{R}^{d_{\ell,f}}$

Analogously, the logit manifold at frequency $f$ is

$\mathcal{L}_f = \{ \logitmap(a,b) : (a,b)\in\mathbb{Z}_n^2 \} \subset \mathbb{R}^n$

The geometry and topology of these manifolds are central to mechanistic interpretability.

3. Geometric and Topological Equivalence

A pivotal mathematical result, Theorem 3.1 (Disc–Torus dichotomy) (Moisescu-Pareja et al., 31 Dec 2025), distinguishes manifold geometry according to the phase alignment of constituent neurons:

• Perfect phase correlation ($\phi^L_i = \phi^R_i$) yields a vector-addition disc

$$V^\mathrm{disc}_{(a,b)} = (\cos\theta_a + \cos\theta_b,\; \sin\theta_a + \sin\theta_b)$$

The preactivation matrix $X$ thus has rank 2, and $\mathcal{M}^1_f$ is a filled disk.

• Independent phases span the 4D torus

$$V^\mathrm{torus}_{(a,b)} = (\cos\theta_a, \sin\theta_a, \cos\theta_b, \sin\theta_b )$$

$X$ has rank 4, and $\mathcal{M}^1_f$ samples $S^1 \times S^1$.

Persistent homology analysis (via Betti numbers) demonstrates:

• Pizza (Attention 0.0), Clock (Attention 1.0), and MLP-Add manifolds at layer 1 consistently yield Betti numbers $(1,0,0)$, indicative of a filled disc.
• MLP-Concat architectures form a torus $(1,2,1)$.
• Successive layers collapse both toroidal and disc structures onto a 1D circle $(1,1,0)$ corresponding to the desired logit structure.

This geometric correspondence establishes that the two architectures compute modular addition through homomorphic manifolds—disc and torus—related by linear projection, rather than fundamentally different solutions.

4. Empirical Algorithmic Phase Space and Statistical Analysis

Extensive empirical evaluation involved training 703 networks per architecture (MLP-Add, Attention 0.0, Attention 1.0, MLP-Concat) at $n=59$, recording neuron activation peaks as the Phase Alignment Distribution (PAD) (Moisescu-Pareja et al., 31 Dec 2025). Metrics for comparison include:

• Torus-distance (graph distance on $\mathbb{Z}_n \times \mathbb{Z}_n$ to diagonal $a=b$)
• Maximum Mean Discrepancy (MMD) with permutation-test $p$-values

Findings:

• Attention 0.0 (Pizza) and Attention 1.0 (Clock) manifest nearly identical PADs (MMD $\lesssim 0.03$, $p \approx 0$).
• Both are statistically distant from MLP-Concat circuits.
• Betti distributions across layers and architectures coincide within sampling error.

This large-scale analysis refutes the notion that Clock and Pizza are genuinely orthogonal algorithmic phases. Instead, architectural hyperparameters (attention strength, width, initialization) may bias the manifold structure but do not alter the end-to-end algorithm implemented.

5. Mechanistic Interpretability: Neuron Clusters and Manifold Perspective

Prior efforts focused on isolated neuron-wiring and single-weight deconstruction. Recent methodology analyzes neuron clusters by:

• Constructing $n \times n$ preactivation heatmaps per neuron
• Assigning neurons to frequency clusters via 2D DFT
• Stacking flattened heatmaps to form the cluster-wide matrix $X$
• Characterizing $X$ using PCA, matrix factorization, and topological data analysis (TDA)

This collective perspective reveals the true manifold geometry underlying learned representations. Focusing on manifold invariants and global structures obviates misleading single-neuron interpretations and enables rigorous analysis of circuit universality, phase transitions, and solution multiplicity (Moisescu-Pareja et al., 31 Dec 2025).

6. Unified Solution Landscape and Implications

The synthesis of geometric and topological analysis supports the universality hypothesis: both Clock and Pizza circuits are not rival algorithms but facets of a universal torus-to-circle solution. Uniform-attention models bias the collapsed disc solution, while softmax attention retains capacity for the full torus but learns a manifold numerically indistinguishable from the disc projection. Subsequent layers irreversibly collapse either manifold onto the task-relevant circle.

This result demonstrates the power of geometric/topological methods—Betti numbers, PCA, MMD, manifold quotient maps—for interpreting neural network solutions. It asserts that, when analyzed at the manifold level, architectural variants converge toward equivalent algorithmic circuits, reshaping mechanistic interpretability paradigms (Moisescu-Pareja et al., 31 Dec 2025, Zhong et al., 2023).

7. Implications and Future Directions

The realization that manifold equivalence subsumes perceived algorithmic diversity has several significant consequences:

• Interpretability methods should prioritize collective, manifold-based approaches rather than isolated neuron inspection.
• Phase transitions between solution archetypes (Pizza, Clock, torus, disc) reflect biasing of representation geometry rather than algorithmic divergence.
• Multiplicity and parallel ensemble phenomena, such as "accompanying pizzas," warrant systematic cataloguing.
• The geometric–topological toolkit may generalize to richer tasks and deeper architectures, facilitating robust mapping of the algorithmic phase space.

A plausible implication is the tractability of extending these methodologies to more complex forms of learned computation and to the detection of universal circuits within high-dimensional, overparameterized neural networks.