Instability of the $U(1)$ spin liquid with a large spinon Fermi surface in the Heisenberg-ring exchange model on the triangular lattice

Abstract: It is widely believed that the $U(1)$ spin liquid with a large spinon Fermi surface(SFS state) can be realized in the spin-$\frac{1}{2}$ $J_{1}-J_{4}$ model on the triangular lattice, when the ring exchange coupling $J_{4}$ is sufficiently strong to suppress the 120 degree magnetic ordered state. This belief is supported by many variational studies on this model and seems to be consistent with the observations on the organic spin liquid materials such as $\kappa$-(BEDT-TTF)${2}$Cu${2}$(CN)${3}$ and EtMe${3}$Sb[Pd(dmit)${2}$]${2}$, which are systems close to their Mott transition and thus have large $J_{4}$. Here we show through systematic variational search that such a state is never favored in the $J_{1}-J_{4}$ model on the triangular lattice. Instead, a state with broken spatial symmetry is favored in the most probable parameter regime for the SFS state and has an energy much lower than that of the SFS state and other proposed variational states. More specifically, we find that for $J_{4}\ge 0.09J_{1}$, the model favors a valence bond solid state with a $4\times6$ period in its local spin correlation pattern and has a variational energy that is about $5\%$ lower than that of the SFS state. This state is separated from the $\pi$-flux state for $J_{4}\le 0.045J_{1}$ by an intermediate symmetry breaking phase with a zigzag pattern in its local spin correlation. We find that the variational phase diagram we got is in qualitative agreement with that obtained from exact diagonalization on a $6\times6$ cluster.