Polystability in positive characteristic and degree lower bounds for invariant rings

Abstract: We develop a representation theoretic technique for detecting closed orbits that is applicable in all characteristics. Our technique is based on Kempf's theory of optimal subgroups and we make some improvements and simplify the theory from a computational perspective. We exhibit our technique in many examples and in particular, give an algorithm to decide if a symmetric polynomial in $n$-variables has a closed ${\rm SL}_n$ orbit. As an important application, we prove exponential lower bounds on the maximal degree of a system of generators of invariant rings for two actions that are important from the perspective of Geometric Complexity Theory (GCT). The first is the action of ${\rm SL}(V)$ on ${\rm Sym}^3(V)^{\oplus 3}$, the space of 3-tuples of cubic forms, and the second is the action of ${\rm SL}(V) \times {\rm SL}(W) \times {\rm SL}(Z)$ on the tensor space $(V \otimes W \otimes Z)^{\oplus 5}$. In both these cases, we prove an exponential lower degree bound for a system of invariants that generate the invariant ring or that define the null cone.