Spin-valley magnetism on the triangular moiré lattice with SU(4) breaking interactions

Abstract: The discovery of correlated insulating states in moir\'e heterostructures has renewed the interest in strongly-coupled electron systems where spin and valley (or layer) degrees of freedom are intertwined. In the strong-coupling limit, such systems can be effectively described by SU(4) spin-valley models akin to Kugel-Khomskii models long studied in the context of spin-orbit coupled materials. However, typical moir\'e heterostructures also exhibit interactions that break the SU(4) symmetry down to SU(2)${}{\mathrm{spin}}\otimes$U(1)${}{\mathrm{valley}}$. Here we investigate the impact of such symmetry-breaking couplings on the magnetic phase diagram for triangular superlattices considering a filling of two electrons (or holes) per moir\'e unit cell. We explore a broad regime of couplings -- including XXZ anisotropies, Dzyaloshinskii-Moriya exchange and on-site Hund's couplings -- using semi-classical Monte Carlo simulations. We find a multitude of classically ordered phases, including (anti-)ferromagnetic, incommensurate, and stripe order, manifesting in different sectors of the spin-valley model's parameter space. Zooming in on the regimes where quantum fluctuations are likely to have an effect, we employ pseudo-fermion functional renormalization group (pf-FRG) calculations to resolve quantum disordered ground states such as spin-valley liquids, which we indeed find for certain parameter regimes. As a concrete example, we discuss the case of trilayer graphene aligned with hexagonal boron nitride using material-specific parameters.