Preservation of Higher-Level Semantics under Ultra-High Sparsity

Determine whether ultra-sparse embeddings with very few active dimensions (e.g., k in {2,4}) preserve higher-level semantic structure such as superclasses or domains, or instead collapse into trivial, instance-specific separations across datasets.

Background

When embeddings are made extremely sparse, it is unclear if they still encode structured semantic information beyond individual instances. The authors pose this question directly before presenting empirical analyses on Banking77 and MTOPIntent to probe superclass/domain separability.

Although the paper provides empirical evidence suggesting CSRv2 retains such structure in some settings, the general question of semantic preservation under extreme sparsity is raised explicitly as open.