Archetypal Analysis++: Rethinking the Initialization Strategy

Abstract: Archetypal analysis is a matrix factorization method with convexity constraints. Due to local minima, a good initialization is essential, but frequently used initialization methods yield either sub-optimal starting points or are prone to get stuck in poor local minima. In this paper, we propose archetypal analysis++ (AA++), a probabilistic initialization strategy for archetypal analysis that sequentially samples points based on their influence on the objective function, similar to $k$-means++. In fact, we argue that $k$-means++ already approximates the proposed initialization method. Furthermore, we suggest to adapt an efficient Monte Carlo approximation of $k$-means++ to AA++. In an extensive empirical evaluation of 15 real-world data sets of varying sizes and dimensionalities and considering two pre-processing strategies, we show that AA++ almost always outperforms all baselines, including the most frequently used ones.

• The paper introduces AA++ which leverages probabilistic sampling to improve archetypal analysis initialization.
• The method, inspired by k-means++, consistently outperforms uniform and FurthestSum in reducing mean square error across diverse datasets.
• AA++ reduces computational complexity while achieving enhanced convergence and more distinct archetypal positions.

Archetypal Analysis++: Rethinking the Initialization Strategy

This paper discusses a novel initialization strategy for archetypal analysis (AA), termed Archetypal Analysis++ (AA++), inspired by $k$-means++. The proposed method seeks to address the challenges of sub-optimal starting points and poor local minima that have traditionally affected archetypal analysis due to inadequate initialization methods.

Introduction to Archetypal Analysis

Archetypal analysis is a convex matrix factorization method focusing on representing data points as convex combinations of boundary archetypes. The importance of initialization in archetypal analysis has often been underexplored, with many practitioners using random initialization methods, which can lead to sub-optimal solutions.

AA is applied across diverse domains, including single-cell gene expression, bioinformatics, apparel design, chemical spaces, geophysical data, and population genetics. Optimization and approximation methods have enhanced AA, but initialization remains crucial for achieving good convergence results.

The Proposed AA++ Method

The AA++ approach is analogous to $k$-means++, leveraging sequential sampling based on data point influence on the objective function. By sampling points in a probabilistic manner, AA++ aims to optimize the initialization of archetypes better than existing methods. This strategy reduces the susceptibility to redundant archetypes and improves the convergence to satisfactory local minima.

Figure 1: A comparison of Uniform, FurthestSum, and the proposed AA++ when consecutively initializing $k=4$ archetypes. MSE denotes the mean square error.

Theoretical Foundation

AA++ guarantees an expectation-based improvement over uniform initialization methods. Using probabilistic sampling proportional to projection distances, AA++ iteratively refines archetypal positions. The expectation that AA++ outperforms uniform sampling is supported by theoretical propositions demonstrating reduced mean squared errors through more effective sampling.

This new initialization strategy is complemented by Monte Carlo and $k$-means++ based approximations that help reduce computational complexity while maintaining performance.

Figure 2: Approximation of the distance function in two dimensions. The true distance of the green point is depicted using a solid line whereas the approximation is shown as a (larger) dashed line.

Experimental Evaluation

The paper presents an extensive empirical evaluation across 15 diverse real-world datasets using two preprocessing strategies: CenterAndMaxScale and Standardization. These datasets range from small to large, and include various feature dimensionalities, testing AA++ against existing baseline methods like Uniform and FurthestSum.

AA++ consistently demonstrated superior performance over traditional initialization methods, achieving better convergence and lower final errors. These results underscore AA++'s capability to generate more distinct and effective archetypal positions, leading to enhanced matrix factorization outcomes.

Figure 3: Results on California Housing, Covertype, FMA, KDD-Protein, Pose, RNA, and Song.

Figure 4: Aggregated statistics over 15 data sets. Each table shows how often each initialization method yields the best result for various choices of $$k under different settings.</p></p> <h2 class='paper-heading' id='implications-and-future-work'>Implications and Future Work</h2> <p>The introduction of AA++ offers significant improvements for archetypal analysis initiation, paving the way for more robust factorization solutions. Practical applications benefit from reduced convergence times and enhanced model accuracy due to better initialization. Furthermore, approximations such as AA++MC provide scalable solutions for large datasets, maintaining algorithm efficacy while reducing computational demands.</p> <p>Future research could explore theoretical guarantees akin to those found in$$k$-means++ for AA++, offering further validation of this approach. Additional explorations might also focus on extending AA++'s applicability or refining methods to address existing limitations in computational complexity.

Conclusion

Archetypal Analysis++ represents a progressive step in initializing archetypal analysis effectively. By leveraging probabilistic sampling, AA++ mitigates the pitfalls associated with traditional random or poorly separated initializations, ensuring improved convergence results across a range of empirical datasets. This work not only contributes a powerful new initialization technique but also provides comprehensive evidence of its efficacy and potential for future enhancements in data factorization strategies.