Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Abstract: Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massive datasets, they still cover only a tiny fraction of the combinatorial space of possible inputs, raising the question of what structure representations must have to support generalization to unseen combinations. We formalize three desiderata for compositional generalization under standard training (divisibility, transferability, stability) and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. This provides theoretical grounding for the Linear Representation Hypothesis: the linear structure widely observed in neural representations is a necessary consequence of compositional generalization. We further derive dimension bounds linking the number of composable concepts to the embedding geometry. Empirically, we evaluate these predictions across modern vision models (CLIP, SigLIP, DINO) and find that representations exhibit partial linear factorization with low-rank, near-orthogonal per-concept factors, and that the degree of this structure correlates with compositional generalization on unseen combinations. As models continue to scale, these conditions predict the representational geometry they may converge to. Code is available at https://github.com/oshapio/necessary-compositionality.

• The paper establishes that compositional generalization requires linearly factorized, orthogonal vision embeddings.
• It introduces a framework with divisibility, transferability, and stability, validated by empirical tests on models like CLIP and DINO.
• The findings guide architecture designs and highlight that current models only partly meet the ideal representations.

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Motivation and Framework

Compositional generalization—the ability to recognize familiar factors in novel contexts—is a central demand for foundation models in vision and vision-language domains. Despite extensive pretraining on vast but combinatorially-limited data, large models underperform when encountering unusual combinations of familiar concepts. This work investigates the geometric and structural constraints imposed on vision embeddings by the requirement of robust compositional generalization.

The authors formalize three non-negotiable desiderata for any model aiming to generalize compositionally:

• Divisibility: All parts (concepts) of an input must be accessible to a simple readout.
• Transferability: Readouts trained on a limited subset must generalize to all unseen combinations.
• Stability: Predictions for each concept should not vary erratically under retraining on other valid supports.

These desiderata are expressed in terms of concept spaces—tuples identifying objects, attributes, or scene parameters—and the mapping from raw data to embeddings.

Figure 1: Compositional generalization challenge: models must answer the same concept queries for both common and rare configurations, even with incomplete training coverage.

Necessary Geometric Constraints

The core theoretical result is that these three desiderata necessitate a specific representational geometry: linear factorization of embeddings into per-concept components, with cross-concept orthogonality of value directions. Under standard training (gradient descent with cross-entropy), this structure is both necessary and sufficient for compositional generalization.

Formally, with $k$ concepts and $n$ values each, the embedding $z_c$ corresponding to concept tuple $c$ must satisfy

$z_c = \sum_{i=1}^k u_{i,c_i},$

where $u_{i,c_i}$ are learned per-concept factors, and the difference vectors for different concepts must be orthogonal:

$$(u_{i,a} - u_{i,b}) \perp (u_{j,a'} - u_{j,b'}) \quad \forall i \neq j.$$

The minimum dimension required for compositional generalization is $d \geq k$, independent of the number of values per concept.

Figure 2: Geometry of linearly compositional models: intersections of concept-specific boundaries yield the exponential combination space.

This geometrization aligns with the Linear Representation Hypothesis, previously observed empirically in neural networks, and establishes it as an essential consequence of compositional generalization rather than a coincidental empirical artifact.

Empirical Evaluation

The authors extensively probe state-of-the-art vision and vision-LLMs (CLIP, SigLIP, DINOv1--v3, MetaCLIP) across synthetic and real-world compositional datasets (dSprites, MPI3D, PUG-Animal, ImageNet-AO).

Key empirical findings:

• Embeddings exhibit partial linear factorization: per-concept component sums explain a substantial fraction of variance, but with gaps from the ideal.
• Cross-concept orthogonality is manifest: difference vectors for distinct concepts have low cosine similarity, far exceeding random baselines.
• Correlation with compositional accuracy: Models with higher factorization scores achieve better compositional generalization on held-out combinations.

  Figure 3: Linearity and orthogonality in DINOv3 embeddings on dSprites: per-concept variation traced by nearly-orthogonal directions.

  

  Figure 4: Compositional generalization accuracy versus linear factorization score (projected $R^2$) across vision-language and vision-only models.

  

  Figure 5: Within-concept direction similarity is high; across-concept similarity is low, confirming predicted orthogonality.

Factor Dimensionality and Packing

Analysis of per-concept subspace geometry reveals that ordinal and continuous concept factors often lie in low-dimensional subspaces—critical for efficient packing when $k$ increases and $d$ is fixed. Discrete concepts may occupy higher effective rank, presumably reflecting underlying multi-attribute composition.

Figure 6: Effective rank of per-concept factor families relative to their cardinality; variance concentrates in a small number of PCs across models and datasets.

Cumulative explained variance curves for PUG-Animal factors show remarkable consistency across different model architectures, supporting claims of universal representational geometry.

Discussion and Implications

This study reframes the empirical observation of linearity and orthogonality in neural embeddings as a geometric necessity for compositional generalization under natural training regimes. The findings offer:

• A diagnostic tool: Projected linear factorization directly predicts compositional capability, suitable for model evaluation and selection.
• Design guidance: Establishing the required geometry enables principled architecture and dataset design, particularly for scaling embedding dimension to support many concepts.
• Limitations for current models: Modern models only partially satisfy the necessary structure, providing an explanation for failures on compositional benchmarks.

The theoretical analysis holds for worst-case stability across valid training supports; future work could relax stability toward average-case formulations, potentially matching practical settings more closely. Further, while the theory assumes retraining across different supports, foundation models are typically pretrained once; the impact of this difference on necessary structure remains an open question.

Conclusion

Compositional generalization in vision embedding models demands a linear, orthogonal decomposition of representations into per-concept factors. Both theory and empirical evidence corroborate this geometric requirement, which aligns with the Linear Representation Hypothesis and suggests that successful generalization to unseen combinations emerges from additive factorization and cross-concept orthogonality. This work provides a unified geometric characterization of compositional generalization, clarifies current limitations, and predicts the representational geometry toward which scaling models will converge (2602.24264).