Easy-plane multi-$\mathbf{q}$ magnetic ground state of Na$_3$Co$_2$SbO$_6$

Abstract: Na$_3$Co$_2$SbO$_6$ is a potential Kitaev magnet with a monoclinic layered crystal structure. Recent investigations of the $C_3$-symmetric sister compound Na$_2$Co$_2$TeO$_6$ have uncovered a unique triple-$\mathbf{q}$ magnetic ground state, as opposed to a single-$\mathbf{q}$ (zigzag) one, prompting us to examine the influence of the reduced structural symmetry of Na$_3$Co$_2$SbO$_6$ on its ground state. Neutron diffraction data obtained on a twin-free crystal reveal that the ground state remains a multi-$\mathbf{q}$ state, despite the system's strong in-plane anisotropy. This robustness of multi-$\mathbf{q}$ orders suggests that they are driven by a common mechanism in the honeycomb cobaltates, such as higher-order magnetic interactions. Spin-polarized neutron diffraction results show that the ordered moments are entirely in-plane, with each staggered component orthogonal to the propagating wave vector. The inferred ground state favors a so-called XXZ easy-plane anisotropic starting point for the microscopic model over a Kitaev one, and features unequal ordered moments reduced by strong quantum fluctuations.