The Fascinating World of 2 $\times$ 2 $\times$ 2 Tensors: Its Geometry and Optimization Challenges

Abstract: This educational article highlights the geometric and algebraic complexities that distinguish tensors from matrices, to supplement coverage in advanced courses on linear algebra, matrix analysis, and tensor decompositions. Using the case of real-valued 2 $\times$ 2 $\times$ 2 tensors, we show how tensors violate many well-known properties of matrices: (1) The rank of a matrix is bounded by its smallest dimension, but a 2 $\times$ 2 $\times$ 2 tensor can be rank 3. (2) Matrices have a single typical rank, but the rank of a generic 2 $\times$ 2 $\times$ 2 tensor can be 2 or 3 - it has two typical ranks. (3) Any limit point of a sequence of matrices of rank $r$ is at most rank $r$, but a limit point of a sequence of 2 $\times$ 2 $\times$ 2 tensors of rank 2 can be rank 3 (a higher rank). (4) Matrices always have a best rank-$r$ approximation, but no rank-3 tensor of size 2 $\times$ 2 $\times$ 2 has a best rank-2 approximation. We unify the analysis of the matrix and tensor cases using tools from algebraic geometry and optimization, providing derivations of these surprising facts. To build intuition for the geometry of rank-constrained sets, students and educators can explore the geometry of matrix and tensor ranks via our interactive visualization tool.