Chiral spin density wave order on frustrated honeycomb and bilayer triangle lattice Hubbard model at half-filling

Abstract: We study the Hubbard model on the frustrated honeycomb lattice with nearest-neighbor $t_1$ and second nearest-neighbor hopping $t_2$, which is isomorphic to the bilayer triangle lattice, using the SU(2)-invariant slave boson theory. We show that the Coulomb interaction $U$ induces antiferromagnetic (AF) chiral spin-density wave ($\chi$-SDW) order in a wide range of $\kappa =t_2/t_1$ where both the two-sublattice AF order at small $\kappa$ and the decoupled three-sublattice 120$^\circ$ order at large $\kappa $ are strongly frustrated, leading to three distinct phases with different anomalous Hall responses. We find a continuous transition from a $\chi$-SDW semimetal with anomalous Hall effect to a topological chiral Chern insulator exhibiting quantum anomalous Hall effect, followed by a discontinuous transition to a $\chi$-SDW insulator with zero total Chern number but anomalous ac Hall effect.The $\chi$-SDW is likely a generic phase of strongly correlated and highly frustrated hexagonal lattice electrons.