Holographic Reduced Representations (HRRs)

• HRRs are a vector-symbolic architecture that binds role-filler pairs using high-dimensional vectors and circular convolution operations.
• They leverage bundling, binding, and unbinding to support scalable neuro-symbolic reasoning and efficient memory retrieval.
• HRRs are integrated into neural networks for sequence encoding and self-attention, enhancing performance with reduced computational cost.

Holographic Reduced Representations (HRRs) provide a vector-symbolic architecture for compositional reasoning, neuro-symbolic computation, and scalable memory encoding by leveraging high-dimensional vectors and algebraic binding operations, such as circular convolution. These methods enable the binding of role-filler pairs, storage of multiple symbolic associations, and approximate retrieval through unbinding. HRRs and their descendants integrate properties from symbolic logic and distributed representation, forming a central substrate for neuro-symbolic learning, differentiable reasoning, and efficient implementation of sequence and memory models.

1. Mathematical Foundation and Core Operations

The HRR model represents each symbol as a dense, high-dimensional real vector $z \in \mathbb{R}^d$, typically drawn as $z \sim \mathcal{N}(0, I/d)$. The core operations are:

• Bundling (Superposition): $x + y$ preserves similarity of $x$ and $y$ within the same vector space.
• Binding (Association): Circular convolution is employed as

$$(x * y)_j = \sum_{k=0}^{d-1} x_k\, y_{(j-k) \bmod d}$$

or equivalently, using the discrete Fourier transform (DFT),

$$x * y = \mathcal{F}^{-1} \left( \mathcal{F}(x) \odot \mathcal{F}(y) \right)$$

• Unbinding (Approximate Inverse): Given $c = x * y$, retrieval of $y$ is performed as $c * x^*$, where $x^*$ is the involution defined by $x^*_j = x_{(-j)\bmod d}$. In the DFT, this is the complex conjugate.
• Algebraic Properties: Binding is commutative and associative; bundling is distributive over binding.

Numerical stability is addressed by projecting each vector to a unit-magnitude subspace in the Fourier domain:

$$\pi(z) = \mathcal{F}^{-1}\left( \mathcal{F}(z) / |\mathcal{F}(z)| \right)$$

ensuring invertibility and stable unbinding in learning setups (Ganesan et al., 2021, Alam et al., 2023).

2. Role in Neural and Neuro-Symbolic Architectures

HRRs serve as differentiable, constant-size substrates bridging symbolic representations and neural networks. In neuro-symbolic loss frameworks for classification, each class $n$ has a learnable key–value pair $(k_n, v_n)$, each projected to the unit-magnitude Fourier manifold. The target representation for class $n$ is $k_n * v_n$, and training minimizes

$$\mathcal{L} = \sum_{i=1}^{B} \left\| k_{n_i} * v_{n_i} - \hat{y}_i \right\|_2$$

with $\hat{y}_i$ being the tanh-activated model output (Alam et al., 2023).

At inference, unbinding by all possible keys retrieves $C$ candidate values, which are scored by cosine similarity to their respective $v_c$. This mechanism enables compositional generalization, as established in subitizing tasks, multi-label classification, and symbolic reasoning (Alam et al., 2023, Ganesan et al., 2021).

HRRs have been integrated into neural network architectures:

• As output layers with symbolic loss for multi-label classification, improving memory and compute efficiency over traditional approaches (Ganesan et al., 2021).
• As key-value associative arrays within RNN extensions such as Associative LSTM, where multiple permuted HRR traces are averaged to suppress interference (Danihelka et al., 2016).

3. Extensions: Complex, Generalized, and Non-Commutative HRRs

Fourier Holographic Reduced Representation (FHRR): Uses complex unit-modulus vectors, where binding is elementwise complex multiplication, $x * y$, and inversion is via complex conjugation. This construction preserves invertibility and similarity under binding (Yeung et al., 2024, Rachkovskij et al., 2022).

Generalized HRR (GHRR): Extends FHRR to stacks of unitary matrices $A_j \in \mathrm{U}(m)$ per position:

$$H = [A_1,\ldots,A_D]^\top, \quad H_1 \circledast H_2 = [A_{1j}A_{2j}]_{j=1}^D$$

Binding is now elementwise matrix multiplication, supporting non-commutativity as a function of the diagonality of unitary factors $Q_j$. This allows GHRRs to interpolate smoothly between FHRR (fully commutative) and full tensor-product representations (fully non-commutative). Non-commutative binding is essential for representing nested or ordered structures without explicit permutations (Yeung et al., 2024).

Geometric HRRs: By replacing convolution with geometric (Clifford) product, HRR-like properties are realized in a geometric algebra context, enabling strict invertibility and basis-free interpretation, though at the expense of exponential scaling in representation size (0710.2611).

4. Memory, Capacity, and Retrieval

HRRs encode multiple associations in a single superposed trace. For $N_{\mathrm{items}}$ key-value pairs,

$c = \sum_{k=1}^{N_{\mathrm{items}}} r_k * x_k$

Retrieval by unbinding with $r_j^*$ gives

$$\hat{x}_j = x_j + \sum_{k \neq j} (r_j^{-1} * r_k) * x_k = x_j + \mathrm{noise}$$

The retrieval noise grows linearly with $N_{\mathrm{items}}$. Capacity is thus limited by the trace dimension. Methods such as redundancy via multiple copies, as in Associative LSTM, reduce the variance of interference noise by a factor $1/N_\mathrm{copies}$, maintaining accuracy as more pairs are stored (Danihelka et al., 2016).

Empirical studies show that with complex-unit-magnitude projection, capacity for reliable retrieval becomes linear in $d$, e.g., $n_\mathrm{max} \approx 0.375 d$ at <3% error (Ganesan et al., 2021). In generalized settings (GHRR), memorization capacity for bound vectors is restored to the linear regime even for complex, non-commutative bindings (Yeung et al., 2024).

5. Applications in Sequence Encoding and Attention

In sequence processing:

• HRRs encode order via recursive binding, enabling shift-equivariant, similarity-preserving hypervector sequence representations. For a position HV $pos$ and symbol HV $e_a$,

$e_{a,i} = pos^i \odot e_a$

and summing over a local radius $R$ allows controlled similarity decay across positions. This encoding matches human word similarity data and supports fast, shift-equivariant comparisons (Rachkovskij et al., 2022).

In self-attention:

• HRR-based attention replaces quadratic dot-product interactions with binding (circular convolution) and collective unbinding, providing $O(TH \log H)$ complexity as opposed to $O(T^2H)$ in standard transformers. This construction enables efficient learning and inference on long-range sequences (up to $T=131{,}072$) with competitive accuracy and up to $280\times$ improved speed-to-convergence (Alam et al., 2023).

Operation	Formula	Algebraic Properties
Bundling	$x + y$	Preserves similarity
Binding	$x * y$ (circular convolution or FFT)	Commutative, associative
Unbinding	$c * x^*$	Approximate (HRR) or exact (FHRR, GHRR)
Projection	$$\pi(z) = \mathcal{F}^{-1}(\mathcal{F}(z)/|\mathcal{F}(z)|)$$	Ensures invertibility

6. Limitations and Theoretical Considerations

Standard HRRs are basis-dependent: convolution is not a geometric operation and lacks a basis-free interpretation (0710.2611). The approximate inverse is exact only for unitary vectors; without projection, unbinding accumulates error rapidly under learning. Memorization and retrieval degrade linearly with the number of bound items unless redundancy or higher dimensions are introduced (Danihelka et al., 2016, Ganesan et al., 2021).

Generalizations (FHRR, GHRR) cure many structural issues:

• Complex unbinding is exact up to quantization noise.
• GHRR enables flexible commutativity control and obviates explicit permutations for positional encoding.
• Geometric HRR provides projective, basis-free analogues at the cost of exponential size (0710.2611).

A remaining limitation is that all finite-dimensional HRR-based architectures are fundamentally “distributed,” lacking fine-grained interpretability for individual vector elements.

7. Impact, Generalizations, and Future Directions

HRRs and descendants occupy a foundational position in the development of neuro-symbolic systems, differentiable memory, and efficient sequence modeling. Recent advances demonstrate their capacity to:

• Enable end-to-end differentiable, constant-size output layers for large discrete label spaces, with interpretability and significant resource savings (Ganesan et al., 2021).
• Scalably implement linear-cost multi-head self-attention architectures with near state-of-the-art accuracy for long sequences (Alam et al., 2023).
• Support variable and nested structures in both commutative (FHRR) and non-commutative (GHRR) settings, with improved decoding and control over similarity preservation (Yeung et al., 2024, Rachkovskij et al., 2022).

Future directions include learning input-dependent binding factors for context-sensitive compositionality, integrating GHRR into deep pipelines, and applying these models to richer data types and reasoning tasks (Yeung et al., 2024).

• (0710.2611) Geometric Analogue of Holographic Reduced Representation
• (Danihelka et al., 2016) Associative Long Short-Term Memory
• (Ganesan et al., 2021) Learning with Holographic Reduced Representations
• (Rachkovskij et al., 2022) Recursive Binding for Similarity-Preserving Hypervector Representations of Sequences
• (Alam et al., 2023) Recasting Self-Attention with Holographic Reduced Representations
• (Alam et al., 2023) Towards Generalization in Subitizing with Neuro-Symbolic Loss using Holographic Reduced Representations
• (Yeung et al., 2024) Generalized Holographic Reduced Representations