Equiangular Basis Vectors

Abstract: We propose Equiangular Basis Vectors (EBVs) for classification tasks. In deep neural networks, models usually end with a k-way fully connected layer with softmax to handle different classification tasks. The learning objective of these methods can be summarized as mapping the learned feature representations to the samples' label space. While in metric learning approaches, the main objective is to learn a transformation function that maps training data points from the original space to a new space where similar points are closer while dissimilar points become farther apart. Different from previous methods, our EBVs generate normalized vector embeddings as "predefined classifiers" which are required to not only be with the equal status between each other, but also be as orthogonal as possible. By minimizing the spherical distance of the embedding of an input between its categorical EBV in training, the predictions can be obtained by identifying the categorical EBV with the smallest distance during inference. Various experiments on the ImageNet-1K dataset and other downstream tasks demonstrate that our method outperforms the general fully connected classifier while it does not introduce huge additional computation compared with classical metric learning methods. Our EBVs won the first place in the 2022 DIGIX Global AI Challenge, and our code is open-source and available at https://github.com/NJUST-VIPGroup/Equiangular-Basis-Vectors.

• The paper introduces fixed equiangular basis vectors as precomputed classifiers that minimize spherical distance between input embeddings and categorical vectors.
• It reduces computational cost by replacing trainable parameters with normalized unit vectors, ensuring scalability regardless of the number of categories.
• Extensive experiments on datasets like ImageNet-1K demonstrate improved performance and potential for applications in real-time, resource-constrained environments.

An Overview of Equiangular Basis Vectors for Classification

The paper in question introduces Equiangular Basis Vectors (EBVs) as an innovative approach to improving classification tasks within the framework of deep neural networks. The authors, Shen, Sun, and Wei, propose a method that redefines classifier layers using predefined and fixed equiangular basis vectors, aiming to enhance both the accuracy and computational efficiency of traditional classification methods.

Summary and Key Contributions

Traditional classifiers in deep neural networks often involve trainable parameters that grow linearly with the number of categories, impacting both computational efficiency and memory requirements. EBVs tackle this issue by replacing these classifiers with fixed normalized vector embeddings that serve as "predefined classifiers." Each category is assigned a unique vector, and during inference, predictions are made by identifying the vector with the smallest spherical distance to the input embedding.

Key Contributions:

1. Fixed Embeddings: EBVs leverage a unit hypersphere to predefine $d$-dimensional basis vectors wherein the vectors across different categories maintain equiangular properties. These basis vectors remain unchanged during the training process, offering computational stability and invariance to the number of categories.
2. Reduction in Parameters: By maintaining a fixed set of equiangular basis vectors, EBVs significantly reduce the computational cost associated with growing category numbers. The dimension of vectors, $d$, can be manually set, allowing scalably with minimal memory footprint.
3. Spherical Optimization: The learning objective is reformulated to minimize the spherical distance between the input and its categorical basis vector, differing fundamentally from the standard cross-entropy loss used in most classification tasks.
4. Performance and Efficiency: Extensive experiments on datasets such as ImageNet-1K and onward to object detection and segmentation tasks demonstrate that EBVs surpass traditional classifiers both in performance and in lower computational complexity.

Technical and Theoretical Implications

The use of EBVs is founded on geometric principles, specifically the concepts of equiangular lines and the Tammes Problem. The authors establish relationships between $\alpha$, $d$, and $N$ (the maximum number of categories), ensuring minimal angular similarity among case vectors and maximizing orthogonality, enabling better category distinction.

This move towards using precomputed basis vectors also aligns with recent trends in AI towards decreasing reliance on large parameter models, instead favoring structural improvements at the architectural level. It invites investigation into how such fixed vector setups could be utilized in other domains of AI, from few-shot learning to managing dynamic category panels in real-time applications.

Potential Future Directions

The adoption of EBVs may serve as a prelude to further explorations into hierarchical embeddings, particularly for datasets with inherent semantic structures. Additionally, the minimal-computation characteristic of EBVs suggests potential application in environments where computational resources are constrained, such as edge devices and mobile platforms. Moreover, future work might explore method enhancements that incorporate nuances of dataset-specific hierarchies, potentially using adaptive means for real-time scalability.

In conclusion, Equiangular Basis Vectors present an innovative and efficient alternative for classification tasks within deep networks, challenging traditional paradigms with implications that span both practical and theoretical dimensions in AI research.