Evolution of Magnetic Interactions in Sb-substituted MnBi2Te4

Abstract: The Mn(Bi${1-x}$Sb$_x$)$_2$Te$_4$ series is purported to span from antiferromagnetic (AF) topological insulator at x = 0 to a trivial AF insulator at x = 1. Here we report on neutron diffraction and inelastic neutron scattering studies of the magnetic interactions across this series. All compounds measured possess ferromagnetic (FM) triangular layers and we find a crossover from AF to FM interlayer coupling near x = 1 for our samples. The large spin gap at x = 0 closes rapidly and the average FM exchange interactions within the triangular layer increase with Sb substitution. Similar to a previous study of MnBi$_2$Te$_4$, we find severe spectral broadening which increases dramatically across the compositional series. In addition to broadening, we observe an additional sharp magnetic excitation in MnSb$_2$Te$_4$ that may indicate the development of local magnetic modes based on recent reports of antisite disorder between Mn and Sb sublattices. The results suggest that both substitutional and antisite disorder contribute substantially to the magnetism in Mn(Bi${1-x}$Sb$_x$)$_2$Te$_4$.