Topological chiral and nematic superconductivity by doping Mott insulators on triangular lattice

Abstract: The mechanism of the unconventional topological superconductivity (TSC) remains a long-standing issue. We investigate the quantum phase diagram of the extended $t$-$J$-$J_{\chi}$ model including spin chiral interactions on triangular lattice based on the state-of-the-art density matrix renormalization group simulations. We identify distinct classes of superconducting phases characterized by nonzero topological Chern numbers $C=1$ and $2$, and a nematic d-wave superconducting phase with a zero Chern number. The TSC states are shown to emerge from doping either a magnetic insulator or chiral spin liquid, which opens new opportunities for experimental discovery. In addition, we further classify the $C=2$ class of TSC phases into an isotropic and a nematic TSC phases, and present evidence of continuous quantum phase transitions from the nematic TSC phase to both isotropic TSC and nematic d-wave phases. These results provide new insight into the mechanism of TSC with an emphasis on the role played by hole dynamics, which changes spin background and drives a topological phase transition at a hole doping level around $3\%$ upon doping a magnetic insulator to enable the emergence of the TSC.