Unconventional pairing and electronic dimerization instabilities in the doped Kitaev-Heisenberg model

Abstract: We study the quantum many-body instabilities of the $t -J_{\mathrm{K}} - J_{\mathrm{H}}$ Kitaev-Heisenberg Hamiltonian on the honeycomb lattice as a minimal model for a doped spin-orbit Mott insulator. This spin-$1/2$ model is believed to describe the magnetic properties of the layered transition-metal oxide Na$_2$IrO$_3$. We determine the ground-state of the system with finite charge-carrier density from the functional renormalization group (fRG) for correlated fermionic systems. To this end, we derive fRG flow-equations adapted to the lack of full spin-rotational invariance in the fermionic interactions, here represented by the highly frustrated and anisotropic Kitaev exchange term. Additionally employing a set of Ward identities for the Kitaev-Heisenberg model, the numerical solution of the flow equations suggests a rich phase diagram emerging upon doping charge carriers into the ground-state manifold ($\mathbb{Z}_2$ quantum spin liquids and magnetically ordered phases). We corroborate superconducting triplet $p$-wave instabilities driven by ferromagnetic exchange and various singlet pairing phases. For filling $\delta > 1/4$, the $p$-wave pairing gives rise to a topological state with protected Majorana edge-modes. For antiferromagnetic Kitaev and ferromagnetic Heisenberg exchange we obtain bond-order instabilities at van Hove filling supported by nesting and density-of-states enhancement, yielding dimerization patterns of the electronic degrees of freedom on the honeycomb lattice. Further, our flow equations are applicable to a wider class of model Hamiltonians.