Functional renormalization group study of the Kitaev-$Γ$ model on the honeycomb lattice and emergent incommensurate magnetic correlations

Abstract: The theoretical inception of the Kitaev honeycomb model has had defining influence on the experimental hunt for quantum spin liquids, bringing unprecedented focus onto the synthesis of materials with bond-directional interactions. Numerous Kitaev materials, which are believed to harbor ground states parametrically close to the Kitaev spin liquid, have been investigated since. Yet, in these materials the Kitaev interaction often comes hand in hand with off-diagonal $\Gamma$ interactions -- with the competition of the two potentially giving rise to a magnetically ordered ground state. In an attempt to aid future material investigations, we study the phase diagram of the spin-1/2 Kitaev-$\Gamma$ model on the honeycomb lattice. Employing a pseudofermion functional renormalization group approach which directly operates in the thermodynamic limit and captures the joint effect of thermal and quantum fluctuations, we unveil the existence of extended parameter regimes with emergent incommensurate magnetic correlations at finite temperature. We supplement our results with additional calculations on a finite cylinder geometry to investigate the impact of periodic boundary conditions on the incommensurate order, thereby providing a perspective on previous numerical studies on finite systems.