Hierarchical Expressive Vector (HE-Vector)

• HE-Vector is a representation method that encodes hierarchical relationships and compositional structures in a vector format to enhance model expressivity.
• It is applied in multi-label learning, hypernymy detection, speech synthesis, and numerical algorithms by integrating hierarchical dependencies through specialized training objectives.
• The method demonstrates improved interpretability, data efficiency, and analogical reasoning, enabling advanced applications like zero-shot style transfer and adaptive vector compression.

Hierarchical Expressive Vector (HE-Vector) refers to a class of parameterizations and representation methods that encode hierarchical structure or compositional expressive content in a vector format. HE-Vectors have emerged independently across several research domains, including multi-label hierarchical embeddings, hypernymy detection, efficient vector compression for numerical PDEs, and, most recently, controllable speech synthesis. Despite implementation differences, these techniques share a focus on leveraging hierarchy—be it label relations, linguistic ontology, layerwise parameter injection, or adaptive data structuring—by means of vectorial representations, often leading to improved expressivity, efficiency, or interpretability.

1. Modeling Hierarchical Label Spaces with HE-Vectors in Multi-Label Learning

HE-Vector methodology for multi-label scenarios is characterized by the embedding of both hierarchical and statistical dependencies among labels. Given a dataset $\mathcal{D} = \{(x_n, Y_n)\}_{n=1}^N$, where labels $Y_n \subseteq \{1, \dots, L\}$ are drawn from a hierarchy encoded as a DAG $\mathcal{G} = (V,E)$, each label $\ell$ is associated with two vectors in $\mathbb{R}^d$: a "target" embedding $u_\ell$ and a "context" embedding $u'_\ell$. For training, two objectives are simultaneously optimized: (a) predicting all ancestors $j\in\mathcal{S}_A(i)$ for each label $i$ present, and (b) predicting all co-occurring labels $k\in Y_n, k\ne i$.

The log-likelihood is

$$\mathcal{L}(\Theta; D_Y) = \sum_n \Bigg[ \frac{1}{Z_{A,n}} \sum_{i \in Y_n} \sum_{j \in \mathcal{S}_A(i)} \log p(y_j|y_i) + \frac{1}{Z_{N,n}} \sum_{i \in Y_n} \sum_{k \in Y_n, k \neq i} \log p(y_k|y_i) \Bigg]$$

with $$p(y_j|y_i) = \frac{\exp({u'_j}^\top u_i)}{\sum_{\ell=1}^L \exp({u'_\ell}^\top u_i)}$$, typically implemented using a hierarchical softmax with a Huffman tree whose code-length reflects the number of descendants, ensuring $O(\log L)$ computation per update. Optimization proceeds via asynchronous stochastic gradient ascent, and no explicit penalty for violating the hierarchy is needed: ancestors are directly predicted.

Qualitative analysis confirms that such embeddings, when trained with both co-occurrence and hierarchical information, are capable of analogical reasoning (e.g., $A$:$a$::$B$:?) and reveal inter-group semantic directions otherwise obscured in non-hierarchical models (Nam et al., 2014).

2. Hypernymy Detection and Lexical Entailment with Hierarchical Expressive Vectors

In computational linguistics, an HE-Vector approach is used in HyperVec to encode lexical hierarchies—specifically, hypernym–hyponym relationships—into dense embeddings. The base model is the standard SGNS (skip-gram with negative sampling), which is augmented by two contrastive objectives designed to both maximize distributional similarity and enforce directionality: hyponymy embeddings are constrained to be close to their hypernyms, and the norm of the hypernym embedding is enforced to be strictly larger, yielding a hierarchical ordering in the vector space.

The unsupervised hypernymy signal, termed HyperScore, is defined for a putative hyponym $u$ and hypernym $v$ as

$$\mathrm{HyperScore}(u,v) = \cos(\vec{u}, \vec{v}) \times \frac{\|\vec{v}\|}{\|\vec{u}\|}$$

This measure successfully discerns both hypernymy and directionality, outperforming previous inclusion and informativeness baselines across benchmarks (e.g., BLESS, EVALution, HyperLex) and robustness to low-data or cross-lingual settings. The margin-based, contrastive training loss induces norm separation ($\|\vec{v}\| > \|\vec{u}\|$) and brings concept pairs with shared characteristic contexts together in the embedding space (Nguyen et al., 2017).

3. Hierarchical Expressive Vector Framework in Speech Synthesis

Recent HE-Vector methodology in expressive speech synthesis formulates style transfer as vector arithmetic over model parameters. The "expressive vector" (E-Vector) is the difference in parameters between a base model and one fine-tuned for a specific style (dialect or emotion): $\tau_i = \theta_i - \theta_\mathrm{pre}$, scaled by a coefficient ($\alpha$ for dialect, $\beta$ for emotion) to produce $\varepsilon_i = \alpha_i \tau_i$. The hierarchical expressive vector is the composition of these E-vectors, applied to disjoint sets of model layers, such as early blocks for dialect and late blocks for emotion, ensuring expressive disentanglement and reduced interference.

A two-stage regime is employed: first, single-style fine-tuning and extraction of E-vectors; second, hierarchical integration during inference without requiring multi-style labeled data. Empirical results demonstrate that such hierarchical layerwise merging achieves the highest MOS for both dialectal synthesis and emotionally expressive dialectal speech, with objective metrics confirming superior word error rate and speaker similarity compared to baselines. The effectiveness is attributed to the alignment of model hierarchy with expressive content granularity: phonetic (segmental) features in early layers and prosodic (suprasegmental) cues in deeper model components (Feng et al., 21 Dec 2025).

Stage	Operation	Result
Stage I	Single-style fine-tuning	Task vector $\tau_i = \theta_i - \theta_\mathrm{pre}$
Stage II	Layerwise vector injection	HE-Vector: early layers (dialect), late layers (emotion)

4. Hierarchical Vectorization for Efficient Numerical Linear Algebra

In numerical PDEs and large-scale linear algebra, HE-Vectors (here, hierarchical vectors) offer a hierarchically partitioned, basis-adaptive representation for high-dimensional vectors. Index sets (e.g., mesh points) are recursively subdivided into a cluster tree structure, and associated "cluster bases" of low rank $k$ allow each cluster (leaf) to represent its segment of the full vector via coefficients $\hat{x}_t \in \mathbb{R}^k$. This yields a global vector

$x|_{\hat{t}} = V_t \hat{x}_t$

requiring only $O(mk)$ storage for $m$ leaves, with inner products, linear updates, and matrix–vector multiplications implementable in $O(mk^2)$ or $O((m + \tilde m)k^2)$ time with full, error-certified adaptivity in both refinement and compression.

The adaptivity mechanism uses precise, recursively computed coarsening error certificates, enabling optimal storage–accuracy trade-offs and exact localized control over hierarchical structure as solution features evolve. This data-sparse encoding is essential for high-frequency eigenvector approximation and time-dependent PDE simulation where localized, dynamic singularities must be efficiently tracked (Börm, 2015).

5. Data-Efficiency, Expressivity, and Generalization Features

A consistent thread across HE-Vector applications is data-efficiency, attributed to the model's ability to generalize structure from local, hierarchical signals. In TTS, this obviates the need for data labeled at the full combinatorial granularity of style blends (e.g., dialect plus emotion) and allows for zero-shot multi-style production. In lexical hierarchy discovery, HE-Vector embeddings generalize from seed pairs to out-of-distribution pairs and across languages via linear mappings. In adaptive vector compression, the method rapidly refines only where needed, minimizing computational waste and storage.

A plausible implication, especially for speech and semantics applications, is that hierarchical decomposition aligns with the underlying generative or distributional processes: early network components process low-level structural information, with deeper/more abstract components encoding higher-order expressive or semantic patterns.

6. Limitations, Model-Specific Constraints, and Open Directions

HE-Vector techniques generally rely on proper alignment between imposed hierarchy (indices, labels, model layers) and the semantic or physical structure of the problem. In TTS applications, the method assumes that styles influence largely disjoint model layers; mismatched architectures, such as those with tangled style and content encoding, can lead to degraded synthesis quality. In ontology learning, reliance on external resources (e.g., WordNet for hypernyms) may propagate noise or incompleteness into the learned embeddings, and cross-POS or cross-relational entailments are not optimally encoded.

Future avenues under active investigation include nonlinear or learned hierarchical merging heuristics, adaptive or dynamic layerwise weighting across tasks/styles, integration with contextualized models for semantic tasks, and extension to more complex task hierarchies (e.g., simultaneous emotional, timbral, and phonetic control in TTS).

7. Representative Applications and Empirical Findings

HE-Vector approaches have been quantitatively and qualitatively validated:

• In label embedding, hierarchical embeddings reveal analogical regularities and group structure not observable with flat embeddings. Embeddings capture transitions such as Urban→Rural or Therapy→Disorders (Nam et al., 2014).
• In hypernymy detection, HE-Vectors yield state-of-the-art performance on multiple unsupervised and supervised tasks, excelling both in detection (AP up to 0.538 on EVALution) and directionality (accuracy 0.92 on BLESS), as well as cross-lingual transfer (Nguyen et al., 2017).
• In speech synthesis, HE-Vectors achieve highest mean opinion scores (e.g., 2.83 for emotional dialect synthesis) and outperform both dual-stage and fully-merged E-vector baselines without requiring joint labeled data (Feng et al., 21 Dec 2025).
• In numerical algorithms, HE-Vectors support eigenvector approximations and time-dependent simulations with minimal storage and rigorous error control (Börm, 2015).

The unifying principle is the harnessing of hierarchical expressive structure, either for statistical modeling, semantic knowledge organization, model parameter modulation, or data-compression, in a manner optimized for both efficiency and expressivity.