Boundary criticality in two-dimensional interacting topological insulators

Abstract: We study the boundary criticality in 2D interacting topological insulators. Using the determinant quantum Monte Carlo method, we present the first nonperturbative study of the boundary quantum phase diagram in the Kane-Mele-Hubbard-Rashba model. Our results reveal rich boundary critical phenomena at the quantum phase transition between a topological insulator and an antiferromagnetic insulator, encompassing ordinary, special, and extraordinary transitions. Combining analytical derivation of the boundary theory with unbiased numerically-exact quantum Monte Carlo simulations, we demonstrate that the presence of topological edge states enriches the ordinary transition that renders a continuous boundary scaling dimension and, more intriguingly, leads to a special transition of the Berezinskii-Kosterlitz-Thouless type. Our work establishes a novel framework for the nonperturbative study of boundary criticality in two-dimensional topological systems with strong electron correlations.