Algebraic Subword Embeddings

• Algebraic subword embeddings are representations that compose vectors from subword units using explicit algebraic and compositional methods to capture linguistic structure.
• They integrate techniques like linear averaging, matrix factorization, incidence algebra, and pattern-based models to derive meaningful embeddings from statistical and combinatorial properties.
• Empirical evaluations show improvements in language modeling, out-of-vocabulary handling, and morphology-sensitive tasks, demonstrating both practical and theoretical benefits.

Algebraic subword embeddings are a class of representations in which subword units—such as character $n$-grams, morphemes, or patterns—are embedded as vectors through explicit algebraic or compositional mechanisms, frequently leveraging the structure or co-occurrence statistics of the data to ground the resulting vectors. This approach enables principled, morphologically informed representations and facilitates the handling of rare and out-of-vocabulary (OOV) words, as well as connections with broader algebraic and combinatorial frameworks from mathematics and theoretical computer science.

1. Foundational Algebraic Constructions

Algebraic subword embeddings combine the principle that word meaning is compositional with explicit mechanistic formulations. Central constructions can be classified as follows:

• Linear or Averaged Composition: In models such as fastText and its counting-based analogues, a word's embedding is formed as the (normalized) sum of its subword vectors (see (Salle et al., 2018)), formally:

$$u'_w = \frac{1}{|S_w|+1} \left(u_w + \sum_{s \in S_w} q_{h(s)} \right)$$

where $u_w$ is the word vector, $S_w$ is the set of its subwords, and $q_{h(s)}$ is the (possibly hashed) vector for subword $s$.

• Matrix Solutions Grounded in Distributional Semantics: As introduced by "Lexically Grounded Subword Segmentation" (Libovický et al., 2024), subword embedding matrices are constructed such that their induced skip-gram-like distribution over words matches the empirical word-word co-occurrence statistics. For a vocabulary $\mathcal{V}$, context matrix $W$, subword incidence $A$, and co-occurrence matrix $C$, the subword embedding matrix $E_s$ is obtained by:

$$E_s = \log\bigl(\text{norm}(AC)\bigr) \cdot W_{\text{right}^{-1}}$$

where $\text{norm}(AC)$ denotes row-normalized subword-word co-occurrence and $W_{\text{right}^{-1}}$ is a right-inverse of $W$. This grounds subwords directly in the same semantic space as whole words.

• Incidence Algebra and Subword Counting: On the theoretical side, the incidence algebra $I(A^*)$ provides a framework for embedding subword-counting functions (Claesson, 2015):
  • The Pascal matrix $P$ of subword counts satisfies $P = \exp H$ for a matrix $H$ whose superdiagonal encodes cover relations among words.
  • Functions counting restricted subwords (e.g., contiguous or scattered) can be expanded (Mahler expansion) in the $\binom{w}{u}$ basis, with coefficients given via combinatorial reciprocity.
• Cluster Algebraic and Combinatorial Embeddings: There exist bijections between subwords of a binary word, antichains in an associated poset, perfect matchings in a snake graph, and terms in the Laurent expansion of a cluster variable (Bailey et al., 2019).

2. Extraction and Selection of Subword Units

Subword units suitable for algebraic embedding are obtained by strategies informed both by statistical regularity and linguistic motivation:

• Statistically Frequent Substrings: Candidate subwords are often those surpassing an occurrence threshold (e.g., in "Patterns versus Characters..." (Takhanov et al., 2017), $\Pi^\prime$ contains substrings observed $>f$ times).
• Pattern Selection via Regularized Models: An $\ell_1$-regularized pattern-based CRF selects a minimal informative set of patterns, solving

$$\mathcal{L}(c) = -\sum_{i=1}^L \log\left[A \exp\bigl(-E(\alpha_i)\bigr)\right] + C\sum_{\alpha\in\Pi'}|c^\alpha|$$

leading to sparsity and redundancy reduction.

• Unsupervised Morpheme Segmentation: Morfessor is used to segment words into candidate morphemes (Salle et al., 2018).
• BPE and Unigram Vocabularies: Subwords may be taken from established tokenization schemas but further refined by embedding-based or lexical criteria (Libovický et al., 2024).
• Algebraic Substring Grounds: All substrings up to some length $L$ can be systematically included, particularly in matrix-based approaches.

The choice of subword granularity and extraction method interacts crucially with the effectiveness of downstream compositional and algebraic embedding schemes.

3. Methods for Algebraic Composition and Grounding

The principal challenge is mapping each subword or pattern to a vector in a manner that faithfully encodes its semantic contribution and enables compositionality.

Table: Algebraic Subword Embedding Construction Methods

Method	Subword Extraction	Embedding Mechanism
FastText/LexVec-style	n-grams/morphemes	Averaged (linear sum, with normalization)
Algebraic (Skip-gram space)	Any segment inventory	Matrix solution $E_s = \log(\text{norm}(AC)) W^{-1}$
Pattern CRF-based	Frequent sub-patterns	Learned embedding; composed via sum/concat/CNN
Incidence algebra (theory)	All finite substrings	Embedding via Mahler expansion in function space
Cluster algebraic (theory)	Binary subwords	Embedded through antichain/matching correspondence

• Linear (Averaged) Aggregation: Most widely adopted for computational efficiency and differentiability. Ensures gradients flow easily through word-subword structure during parameter updates (Salle et al., 2018).
• Log-Linear/Matrix Factorization Grounding: Solving for $E_s$ in the least-squares sense aligns semantic spaces algebraically, without reliance on further network training (Libovický et al., 2024).
• Pattern-based Sequence Lifting: Words are lifted to sequences over a "pattern alphabet," then composed via sum, concatenation (with length normalization), or convolution (Takhanov et al., 2017).
• Incidence Algebra Expansion: Subword count functions are embedded into an incidence algebra via Pascal-type and Mahler expansions, offering algebraically canonical representations (Claesson, 2015).
• Structural/Algebraic Embeddings: The combinatorics of subwords is naturally encoded in cluster algebras by explicit bijection (Bailey et al., 2019).

4. Empirical and Formal Evaluation

Algebraic subword embeddings are evaluated on a range of intrinsic and extrinsic metrics:

• Language Modeling: Pattern-based embeddings in RNN LLMs reduce perplexity consistently and significantly, outperforming character-based baselines under strict parameter regimes (by 2–20 points, depending on model and corpus) (Takhanov et al., 2017).
• Similarity and Analogy Tasks: Incorporating subwords in matrix factorization improves rare word correlation (RW: +0.06), syntactic analogy (e.g., MSR: +5.4 for LV-N), and yields robust OOV embeddings, with minimal loss for word similarity (Salle et al., 2018).
• Morphological Segmentation Quality: Algebraic subword vector grounding and segmentation improves morpheme-boundary precision by 2–4 percentage points compared to BPE baselines, supported by SIGMORPHON 2018 metrics (Libovický et al., 2024).
• POS Tagging: Gains of 0.3–0.4 in normalized accuracy across seven languages indicate strong benefit for morphologically sensitive tasks (Libovický et al., 2024).
• Theoretical Properties: Embeddings in incidence algebras and cluster algebras guarantee structural bijection and compositionality, clarifying algebraic relationships between subword statistics and higher-order combinatorial objects (Claesson, 2015, Bailey et al., 2019).

5. Algorithmic and Mathematical Underpinnings

The algebraic paradigm rests on multiple concrete algorithmic frameworks:

• Dynamic Programming for Segmentation: With lexically grounded embeddings, optimal segmentations maximizing semantic similarity are efficiently searchable via dynamic programming, with complexity $O(L |x|^2)$ (Libovický et al., 2024).
• Pattern-based FST Construction: Pattern extraction produces a finite-state machine emitting "pattern-prefix" states per input character, which sequences words into pattern tokens for embedding (Takhanov et al., 2017).
• Incidence Algebra Convolution: Subword occurrence functions and their restrictions are efficiently computable via convolution in the incidence algebra (Claesson, 2015).
• Mahler Expansion and Reciprocity: Restricted subword counts are expandable via the Mahler expansion, and inversion (reciprocity) formulas relate different subword function types algebraically.
• Cluster Algebraic Embeddings: Each binary subword corresponds canonically to a perfect matching of a snake graph, and hence to a term in a Laurent expansion, embedding the combinatorial world of subwords into cluster algebra theory (Bailey et al., 2019).

6. Connections to Related Areas and Broader Significance

Algebraic subword embeddings interface with a wide spectrum of areas:

• Morphological and Linguistic Analysis: By capturing substructure at the morpheme or pattern level, these embeddings offer enhanced representational fidelity for morphologically rich languages.
• Out-of-Vocabulary Generalization: The explicit algebraic aggregation at subword level yields coherent embeddings for unseen or misspelled word forms without retraining.
• Theory–Practice Synthesis: The connection to incidence algebras, cluster algebras, and the combinatorics of words integrates algebraic, combinatorial, and application-level perspectives. This positions algebraic embeddings as a fertile ground for theoretical advances with direct NLP ramifications (Claesson, 2015, Bailey et al., 2019).
• Tokenization and Segmentation: Embedding-based segmentation can supplant frequency-only heuristics, resulting in lexically and semantically more meaningful units (Libovický et al., 2024).

A plausible implication is that future models with increasingly tight coupling between tokenization, morphology, and semantic space may further benefit from the interpretability, compositionality, and transfer properties guaranteed by these algebraic constructions.