Phase diagram and topological phases in the triangular lattice Kitaev-Hubbard model

Abstract: We study the half-filled Hubbard model on the triangular lattice with spin-dependent Kitaev-like hopping. Using the variational cluster approach, we identify five phases: a metallic phase, a non-coplanar chiral magnetic order, a $120^\circ$ magnetic order, a nonmagnetic insulator (NMI), and an interacting Chern insulator (CI) with a nonzero Chern number. The transition from CI to NMI is characterized by the change of the charge gap from an indirect band gap to a direct Mott gap. Based on the slave-rotor mean-field theory, the NMI phase is further suggested to be a gapless Mott insulator with a spinon Fermi surface or a fractionalized CI with nontrivial spinon topology, depending on the strength of Kitaev-like hopping. Our work highlights the rising field that interesting phases emerge from the interplay of band topology and Mott physics.