Explain representation geometry for hierarchical attributes in language models

Develop a theoretical framework that explains the emergence of geometric structure in model representations for hierarchical attributes in large language models, complementing existing theories for continuous attributes (e.g., time and space) and binary attributes that yield linear analogies.

Background

This work provides a symmetry-based explanation for representation manifolds arising from continuous latent variables and connects to prior theory on binary attributes that explain linear analogies. However, the representational geometry observed for hierarchical attributes in LLMs lacks a comparable theoretical account.

The authors explicitly note that while continuous and binary cases are theoretically addressed, the hierarchical case remains unresolved, and a unified framework that covers all these properties would be desirable.