Cohomology and relative Rota-Baxter-Nijenhuis structures on LieYRep pairs

Abstract: A LieYRep pair consists of a Lie-Yamaguti algebra and its representation. In this paper, we establish the cohomology theory of LieYRep pairs and characterize their linear deformations by the second cohomology group. Then we introduce the notion of relative Rota-Baxter-Nijenhuis structures on LieYRep pairs, investigate their properties, and prove that a relative Rota-Baxter-Nijenhuis structure gives rise to a pair of compatible relative Rota-Baxter operators under a certain condition. Finally, we show the equivalence between $r$-matrix-Nijenhuis structures and Rota-Baxter-Nijenhuis structures on Lie-Yamaguti algebras.