Transfer operators and Hankel transforms between relative trace formulas, I: character theory

Abstract: The Langlands functoriality conjecture, as reformulated in the "beyond endoscopy" program, predicts comparisons between the (stable) trace formulas of different groups $G_1, G_2$ for every morphism ${^LG}_1\to {^LG}_2$ between their $L$-groups. This conjecture can be seen as a special case of a more general conjecture, which replaces reductive groups by spherical varieties and the trace formula by its generalization, the relative trace formula. The goal of this article and its continuation is to demonstrate, by example, the existence of "transfer operators" betweeen relative trace formulas, that generalize the scalar transfer factors of endoscopy. These transfer operators have all properties that one could expect from a trace formula comparison: matching, fundamental lemma for the Hecke algebra, transfer of (relative) characters. Most importantly, and quite surprisingly, they appear to be of abelian nature (at least, in the low-rank examples considered in this paper), even though they encompass functoriality relations of non-abelian harmonic analysis. Thus, they are amenable to application of the Poisson summation formula in order to perform the global comparison. Moreover, we show that these abelian transforms have some structure --- which presently escapes our understanding in its entirety --- as deformations of well-understood operators when the spaces under consideration are replaced by their "asymptotic cones". In this first paper we study (relative) characters for the Kunzetsov formula and the stable trace formula for $\operatorname{SL}_2$ and their degenerations (as well as for the relative trace formula for torus periods in $\operatorname{PGL}_2$), and we show how they correspond to each other under explicit transfer operators.