Generalized $XY$ model with competing antiferromagnetic and antinematic interactions

Abstract: We study effects of $q$-order antinematic (AN$_q$) interactions on the critical behavior of the antiferromagnetic (AF) $XY$ model on a square lattice. It is found that the evolution of the phase diagram topology of such AF-AN$_q$ models with the parameter $q$ does not follow the same line as for the corresponding FM-N$_q$ models with the ferromagnetic (FM) and $q$-order nematic (N$_q$) interactions. Their phase diagrams are similar only for odd values of the parameter $q$. In such cases the respective phases reported in the FM-N$_q$ models are observed in the AF-AN$_q$ models on each of the two AF-coupled sublattices and the corresponding phase transitions are concluded to be of the same kind. On the other hand, for even values of $q$ the phase diagrams of the AF-AN$_q$ models are different from the FM-N$_q$ models and their topology does not change with $q$. Besides the pure AF and AN$_q$ phases, observed at higher temperatures in the regions of the dominant respective couplings, at low temperatures there is a new canted (C)AF phase, which results from the competition between the AF and AN$_q$ ordering tendencies and has no counterpart in the FM-N$_q$ model. The phase transitions to the CAF phase from both AF and AN$_q$ phases appear to be of the BKT nature.