Vector Neurons: Algebraic Neural Units

• Vector Neurons are neural units based on vector spaces with specific algebraic structures that encode symmetry and enable reduced parameter counts.
• They are implemented by mapping algebraic operations to real-valued matrix computations, supporting equivariance to groups like SO(3) for robust 3D processing.
• Applications include 3D point cloud segmentation, LiDAR place recognition, and shape reconstruction, showcasing improved generalization and training stability.

A vector neuron (VN) is a fundamental computational unit in a neural network whose activation, weights, and bias all belong to a vector space endowed with specific algebraic structure. The VN concept generalizes traditional scalar-valued neurons and forms the basis of vector-valued neural networks (V-nets). In the most prevalent applications, VNs are realized with the aim of encoding algebraic correlations across feature channels (notably, equivariances with respect to symmetries such as rotations), reducing parameter counts, and improving training robustness in multidimensional signal and geometric processing. Prominent instantiations include SO(3)-equivariant networks for 3D point cloud learning, and hypercomplex-valued networks leveraging algebras such as the complex or quaternion numbers.

1. Formal Definition and Layer Structures

Let 𝑉 be an $n$-dimensional real algebra with basis $\{e_1, ..., e_n\}$, so any $x \in 𝑉$ corresponds to a real vector $[x_1, ..., x_n]^T \in \mathbb{R}^n$. A vector neuron receives $x \in 𝑉$ as input, multiplies it by weights $w \in 𝑉$, adds a bias $b \in 𝑉$, and applies a nonlinearity $\psi: 𝑉 \to 𝑉$. A dense VN layer mapping $N$ inputs $x_j$ to $M$ outputs $y_i$ is

$$s_i = \sum_{j=1}^N w_{ij} \cdot x_j,\quad y_i = \psi(s_i + b_i),\quad i=1,\dots,M$$

where "·" denotes the algebraic multiplication in $𝑉$.

In matrix notation, with $W \in 𝑉^{M \times N}$, input $x \in 𝑉^N$, output $y \in 𝑉^M$, and bias $b \in 𝑉^M$: $s = W x \in 𝑉^M,\qquad y = \psi(s + b)$

To implement VN layers with real-valued deep learning libraries, each multiplication $a x$ is mapped to a real matrix-vector product via a left-multiplication operator $M_L(a)$: $\varphi(a x) = M_L(a)\varphi(x)$ The real-valued equivalent network is constructed by organizing all parameters with Kronecker-sum structure dictated by the algebra's basis (Valle, 2023).

VN convolutional layers perform channel mixing by replacing sum-of-scalars with algebraic products over vector channels: $$(W \star x)(p, k) = \sum_{c=1}^{C} \sum_{q\in D} W(q, c, k) \cdot x(p+q, c)$$

Nonlinearities are typically "split" activations: $\psi(x) = \sum_i \psi_\mathbb{R}(x_i) e_i$ for scalar nonlinearity $\psi_\mathbb{R}$ (e.g., ReLU).

2. Algebraic Perspective: Real, Hypercomplex, and Arbitrary Bilinear Products

VN layers encode cross-channel structure via the underlying algebra $𝑉$. Real-valued networks are recovered as the trivial case $n=1$. Hypercomplex-valued nets (e.g., complex, quaternion, Clifford) are V-nets for $𝑉$ possessing further algebraic constraints (such as specific multiplication tables). For instance, quaternion VNs use $𝑉 = \mathbb{H}$, with $[1, i, j, k]$ basis and obeying $i^2 = j^2 = k^2 = ijk = -1$.

A further generalization replaces the underlying product with any arbitrary bilinear product $$B: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}^n$$ parameterized by a third-order tensor, as in Arbitrary Bilinear Product Neural Networks (ABIPNN). This enables VNs supporting non-classical products, such as circular, skew, or reverse-time convolutions, and higher-dimensional vector or tensor products (Fan et al., 2018).

The constraints imposed by $𝑉$ or $B$ lead to a significant reduction in the number of learnable parameters: a real network of $nM\times nN$ has $n^2 MN$ parameters, whereas a VN with algebraic constraints has $nMN$, a $1/n$ reduction (Valle, 2023).

3. Equivariance, Invariance, and Group Actions

A principal motivation for VNs is their ability to enforce symmetry constraints such as SO(3)-equivariance. Given a group $G$ and representation $\rho: G \rightarrow GL(V)$, a VN network $f$ is equivariant if

$$f(\rho(g) \cdot x) = \rho(g)\,f(x),\quad \forall g \in G$$

In SO(3)-equivariant VNs, each neuron output $v \in \mathbb{R}^3$ transforms as $v \mapsto vR$ under $R \in SO(3)$. All standard network operations (linear, non-linear, pooling, batch-norm) are constructed so as to strictly commute with the action of $G$ (Deng et al., 2021).

The framework extends to SE(3)-equivariance (rotations + translations) for point cloud applications. Here, features transform as $V \mapsto V R + \mathbf{1}_C T$ with specific constraints on weight-sharing and shift-invariant nonlinearities (Katzir et al., 2022). In all cases, invariants (outputs which do not change with $g \in G$) can be constructed via inner products, norms, or appropriate readout heads.

4. Network Architectures and Examples

VN modules have been systematically integrated into various architectures:

• VN-PointNet, VN-DGCNN: Scalar MLPs and edge convolutions are replaced with VN-linear and VN-ReLU blocks, achieving strict rotation equivariance by design. Extensions yield rotation-robust classification and segmentation on ModelNet40 and ShapeNet-part (Deng et al., 2021).
• VNI-Net: A LiDAR place recognition network mapping raw point clouds to global SO(3)-invariant descriptors. Raw points are lifted into VN space; features are processed by stacks of equivariant VN-blocks (including self-attention); local invariants are extracted via Euclidean and cosine distances in the VN feature space; and GeM pooling aggregates over points. The architecture achieves full SO(3) invariance and state-of-the-art robustness in Rotation Place Recognition under arbitrary 3D rotation (Tian et al., 2023).
• VN-Transformer: Generalizes self-attention to vector neuron tokens, constructing an equivariant analog of multi-head dot-product attention. This framework supports both spatial and non-spatial input attributes, multi-scale reduction for computational efficiency, and controlled violation of equivariance (via small biases) with explicit error propagation bounds (Assaad et al., 2022).

5. Extended Feature Representations and Expressivity

A key limitation of standard VN layers is encoding only three-dimensional (or fixed $n$-dimensional) features, which can bottleneck expressivity, especially in 3D vision. Frequency-based Equivariant Representations (fer) extend VNs by mapping each 3D point to a high-dimensional feature space, maintaining equivariance via lifts of SO(3) actions to higher-dimensional irreducible representations. Each such embedding allows the network to capture arbitrarily high signal frequencies, directly addressing expressivity losses in original VN layers (Son et al., 2024).

Let $\Phi(\mathbf{u})$ denote such a multi-frequency mapping, with constructed equivariant features covering up to $\lfloor (n-1)/2 \rfloor$ harmonics. Network blocks, pooling, and invariants remain unchanged; accuracy and resilience to occlusion and pose variation are strictly improved, as evidenced in shape reconstruction, registration, and segmentation benchmarks.

6. Empirical Performance and Applications

Across multiple tasks, VNs demonstrate competitive or superior performance compared to standard or specialized equivariant approaches:

• Classification and segmentation: On ModelNet40 (classification) and ShapeNet-Part (segmentation), VN-architectures consistently outperform scalar baselines, particularly in scenarios testing generalization to unseen poses. For example, VN-DGCNN achieves $90.2\%$ accuracy under SO(3)-augmented regimes, compared to $88.6\%$ for scalar DGCNN (Deng et al., 2021).
• Reconstruction: VN-OccNet retains stable performance under arbitrary rotations, where real-valued OccNet collapses. fer-VN-OccNet further boosts shape reconstruction IoU (Son et al., 2024).
• SLAM and place recognition: VNI-Net achieves AR@1\% $\approx 94\%$ on Oxford RobotCar under SO(3) rotation, greatly surpassing SO(3)-invariant and non-invariant baselines (Tian et al., 2023).
• Shape-pose disentanglement and canonicalization: SE(3)-equivariant vector neuron autoencoders produce stable, consistent canonical frames for category-level shape-alignment, with orders-of-magnitude lower instability versus alternative approaches (Katzir et al., 2022).

VN networks are directly implementable in deep learning libraries such as PyTorch and TensorFlow, using structured matrix representations and efficient Kronecker algebra to exploit the algebraic constraints (Valle, 2023).

7. Advantages, Limitations, and Future Directions

Advantages:

• Significant parameter reduction ($\sim1/n$ relative to unconstrained real nets)
• Enforced inter-channel correlation structure, leading to faster convergence and improved generalization
• Guaranteed universal approximation under mild conditions for the algebra and nonlinearity
• Rigorous equivariance or invariance to specified transformation groups, easily composed by construction
• Flexible integration with arbitrary bilinear products, supporting both known (complex, quaternion, Clifford) and user-specified algebraic frameworks (Valle, 2023, Fan et al., 2018)

Limitations:

• For tasks where geometric symmetries are not beneficial, the structure may marginally reduce raw accuracy
• Extension to higher-order (tensor) neuron representations is required to capture all irreps of groups such as SO(3)
• Expressivity for high-frequency signals is limited unless using frequency-extended representations (Son et al., 2024)
• For heavily occluded or partial geometric data, equivarance alone is insufficient for full consistency without additional architectural adaptations (Katzir et al., 2022)

Future directions include developing real-time pipelines for SLAM using VN features, further generalizing to other symmetry groups (e.g., scaling, reflection), and expanding to higher-order representations for richer group actions.

---

Vector Neurons enable the principled imposition of algebraic symmetry structure in neural networks, providing parameter efficiency, built-in equivariance, and robust performance across multidimensional and geometric AI tasks. Hypercomplex and bilinear-product-based networks arise as strict subclasses of this general paradigm, with practical and theoretical tools now available for direct integration into modern deep learning systems (Valle, 2023, Deng et al., 2021, Fan et al., 2018, Son et al., 2024, Tian et al., 2023, Assaad et al., 2022, Katzir et al., 2022).