Twist-angle evolution of the intervalley-coherent antiferromagnet in twisted WSe$_2$

Abstract: Recent experimental reports of correlated physics in twisted homobilayer WSe$_2$ have spurred interest in the interplay of electronic interactions and topology in this system. Here, we explore its phase diagram using the Hartree-Fock approximation within a three-orbital Wannier model of the bilayer. Our analysis reveals a dominant intervalley-coherent antiferromagnetic instability, whose stability in the space of twist angle, interaction strength, out-of-plane displacement field, and hole density is primarily set by nesting and commensurability. At large angles or low interaction-to-bandwidth ratios, the instability arises at hole densities above half filling near a van-Hove line where the strong Fermi surface nesting occurs due to the flatness of the band in a region enclosing the van-Hove and $\kappa$ points. Increasing interaction strength or decreasing the twist angle gradually shifts the ordered phase toward half filling, where the strongest antiferromagnetic order gets pinned due to commensurability effects that enable a full gap opening. The antiferromagnetic order parameter strongly couples to the layer polarization, which makes its transition to the normal state sharp in the strong-coupling limit and carries implications for collective modes. Our Hartree-Fock phase diagram reproduces key aspects of recent experiments and the reconstructed Fermi surfaces and DOS in the antiferromagnetic phase account for subtle transport signatures observed in these studies.