Measuring the Intrinsic Dimension of Objective Landscapes

Abstract: Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive conclusions. Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.

• The paper introduces intrinsic dimension as a novel metric that determines the minimum number of parameters needed to achieve a solution on an objective landscape.
• It demonstrates through experiments on MNIST, CIFAR-10, and reinforcement tasks that the effective parameters (intrinsic dimension) are far lower than the total count, challenging traditional complexity assumptions.
• The results suggest that extra parameters add redundancy rather than accuracy, with architectures like convolutional networks showing superior parameter efficiency in image tasks.

Measuring the Intrinsic Dimension of Objective Landscapes

Introduction

The paper "Measuring the Intrinsic Dimension of Objective Landscapes" presents a novel approach to understanding the complexity of neural network optimization problems. By introducing the concept of intrinsic dimension, the authors provide a quantitative measure of problem difficulty that transcends the traditional parameter count. The main methodology involves training networks within randomly oriented subspaces of reduced dimensionality, gradually increasing dimension until solutions appear, thereby estimating the intrinsic dimension. This concept challenges conventional notions regarding the relationship between network size and problem difficulty.

Defining Intrinsic Dimension

The intrinsic dimension is defined as the minimum number of parameters required to achieve a solution on the objective landscape. This approach is exemplified through simple yet illustrative toy problems, where the dimensionality of solutions can be calculated directly. The intrinsic dimension is estimated by confining optimization to randomly generated subspaces of varying dimensionality. Solutions begin to emerge as the dimension increases, thus providing a practical means of measuring intrinsic difficulty.

Figure 1: (left) Illustration of parameter vectors for direct optimization in the $D=3$ case. (middle) Illustration of parameter vectors and a possible random subspace for the $D=3, d=2$ case. (right) Plot of performance vs. subspace dimension.

Experimental Results

The authors evaluate intrinsic dimension across various tasks, including MNIST, CIFAR-10, ImageNet, and several reinforcement learning tasks. Notably, these experiments reveal that many problems have substantially lower intrinsic dimensions than their parameter count suggests. For instance, the intrinsic dimension for a fully connected network on MNIST is around 750, compared to the native parameter count of 199,210.

Additionally, the study demonstrates that intrinsic dimension remains relatively invariant regardless of model size within the same problem domain, thus implying that extra parameters often do not contribute to model accuracy but to the redundancy of the solution manifold.

Figure 2: Results using the policy-based ES algorithm to train agents on (left column) $\mathtt{InvertedPendulum\!-\!v1$.

Implications and Future Directions

The implication of intrinsic dimensionality is profound, offering insights into the efficiency of different architectures. For example, convolutional networks show consistently lower intrinsic dimensions compared to fully connected networks when applied to image tasks, suggesting their superior parameter efficiency. Moreover, this method proposes a new way to compress networks significantly, offering potential in scenarios where storage and computational resources are limited.

Further research could explore the application of intrinsic dimensionality in other domains and the development of subspaces tailored for specific tasks. There is potential for intrinsic dimension to guide architecture selection, network design, and even inform the trade-off between model size and complexity.

Conclusions

The paper introduces a robust framework for evaluating the intrinsic dimension of neural network optimization landscapes. Through empirical analysis, it provides a deeper understanding of the complexity inherent to various learning tasks. This metric not only offers potential for improving network design and compression but also deepens our theoretical understanding of the relationship between neural network capacity, structure, and problem difficulty. Future work may extend these findings to provide further insights into the landscape of neural network training.