Phases and Quantum Phase Transitions in Anisotropic Antiferromagnetic Kitaev-Heisenberg-$Γ$ magnet

Abstract: We study the Kitaev-Heisenberg-$\Gamma$ model with antiferromagnetic Kitaev exchanges in the strong anisotropic (toric code) limit to understand the phases and the intervening phase transitions between the gapped $Z_2$ quantum spin liquid and the spin-ordered (in the Heisenberg limit) as well as paramagnetic phases (in the pseudo-dipolar, $\Gamma$, limit). We find that the paramagnetic phase obtained in the large $\Gamma$ limit has no topological entanglement entropy and is proximate to a gapless critical point of a system described by an equal superposition of differently oriented stacked one-dimensional $Z_2\times Z_2$ symmetry protected topological phases. Using a combination of exact diagonalization calculations and field-theoretic analysis we map out the phases and phase transitions to reveal the complete phase diagram as a function of the Heisenberg, the Kitaev, and the pseudo-dipolar interactions. Our work shows a rich plethora of unconventional phases and phase transitions and provides a comprehensive understanding of the physics of anisotropic Kitaev-Heisenberg-$\Gamma$ systems along with our paper [Phys. Rev. B 102, 235124 (2020)] where the ferromagnetic Kitaev exchange was studied.