The Identity Problem in nilpotent groups of bounded class

Abstract: Let $G$ be a unitriangular matrix group of nilpotency class at most ten. We show that the Identity Problem (does a semigroup contain the identity matrix?) and the Group Problem (is a semigroup a group?) are decidable in polynomial time for finitely generated subsemigroups of $G$. Our decidability results also hold when $G$ is an arbitrary finitely generated nilpotent group of class at most ten. This extends earlier work of Babai et al. on commutative matrix groups (SODA'96) and work of Bell et al. on $\mathsf{SL}(2, \mathbb{Z})$ (SODA'17). Furthermore, we formulate a sufficient condition for the generalization of our results to nilpotent groups of class $d > 10$. For every such $d$, we exhibit an effective procedure that verifies this condition in case it is true.