Kitaev interaction and possible spin liquid state in CoI2 and Co2/3Mg1/3I2

Abstract: Kitaev materials are of great interest due to their potential in realizing quantum spin liquid (QSL) states and applications in topological quantum computing. In the pursuit of realizing Kitaev QSL, a Mott insulator with strong bond-dependent frustration and weak geometric frustration is highly desirable. Here we explore Kitaev physics in the van der Waals triangular antiferromagnet (AF) CoI$2$, through the spin-orbital states and Wannier function analyses, exact diagonalization and density matrix renormalization group study of the electronic structure and magnetic properties. We find that the high-spin Co$^{2+}$ ion is in the $J\mathrm{eff}=1/2$ state because of strong spin-orbit coupling, and the weak trigonal elongation and crystal field contribute to the observed weak in-plane magnetic anisotropy. The strong $t_{2g}$-$e_g$ hopping via the strong Co 3$d$-I 5$p$ hybridization gives rise to a strong Kitaev interaction ($K_1$) at the first nearest neighbors (1NN), and the long Co-Co distance and the weak $t_{2g}$-$t_{2g}$ hoppings determine a weak Heisenberg interaction $J_1$. The resultant $|K_1/J_1|$ = 6.63 confirms a strong bond-dependent frustration, while the geometric frustration due to the 3NN Heisenberg interaction $J_3$ gets involved, and they all together result in the experimental helical AF order in CoI$2$. We then propose to suppress the $J_3$ using a partial Mg substitution for Co, and indeed we find that Co${2/3}$Mg$_{1/3}$I$_2$ has the much reduced geometric frustration but hosts the robust bond-dependent frustration, and thus it would be a promising Kitaev material being so far closest to the QSL state.