Competing spin-orbital singlet states in the 4$d^4$ honeycomb ruthenate Ag$_3$LiRu$_2$O$_6$

Abstract: When spin-orbit-entangled $d$-electrons reside on a honeycomb lattice, rich quantum states are anticipated to emerge, as exemplified by the $d^5$ Kitaev materials. Distinct yet equally intriguing physics may be realized with a $d$-electron count other than $d^5$. We found that the layered ruthenate Ag$3$LiRu$_2$O$_6$ with $d^4$ Ru$^{4+}$ ions at ambient pressure forms a honeycomb lattice of spin-orbit-entangled singlets, which is a playground for frustrated excitonic magnetism. Under pressure, the singlet state does not develop the expected excitonic magnetism but experiences two successive transitions to other nonmagnetic phases, first to an intermediate phase with moderate distortion of honeycomb lattice, and eventually to a high-pressure phase with very short Ru-Ru dimer bonds. While the strong dimerization in the high-pressure phase originates from a molecular orbital formation as in the sister compound Li$_2$RuO$_3$, the intermediate phase represents a spin-orbit-coupled $J$-dimer state which is stabilized by the admixture of upper-lying $J{\rm eff} = 1$-derived states. We argue that the $J$-dimer state is induced by a pseudo-Jahn-Teller effect associated with the low-lying spin-orbital excited states and is unique to spin-orbit-entangled $d^4$ systems. The discovery of competing singlet phases demonstrates rich spin-orbital physics of $d^4$ honeycomb compounds and paves the way for realization of unconventional magnetism.