Phase diagrams of antiferromagnetic $XY$ model on a triangular lattice with higher-order interactions

Abstract: We study effects of higher-order antinematic interactions on the critical behavior of the antiferromagnetic (AFM) $XY$ model on a triangular lattice, using Monte Carlo simulations. The parameter $q$ of the generalized antinematic (ANq) interaction is found to have a pronounced effect on the phase diagram topology by inducing new quasi-long-range ordered phases due to competition with the conventional AFM interaction as well as geometrical frustration. For values of $q$ divisible by 3 the conflict between the two interactions results in a frustrated canted AFM phase appearing at low temperatures wedged between the AFM and ANq phases. For $q$ nondivisible by 3 with the increase of $q$ one can observe the evolution of the phase diagram topology featuring two ($q=2$), three ($q=4,5$) and four ($q \geq 7$) ordered phases. In addition to the two phases previously found for $q=2$, the first new phase with solely AFM ordering arises for $q=4$ in the limit of strong AFM coupling and higher temperatures by separating from the phase with the coexisting AFM and ANq orderings. For $q=7$ another phase with AFM ordering but multimodal spin distribution in each sublattice appears at intermediate temperatures. All these algebraic phases also display standard and generalized chiral long-range orderings, which decouple at higher temperatures in the regime of dominant ANq (AFM) interaction for $q \geq 4$ ($q \geq 7$) preserving only the generalized (standard) chiral ordering.