Whittaker Support in Representation Theory

• Whittaker support is defined as the set of maximal nilpotent orbits for which generalized Whittaker models are nonvanishing for a given representation.
• It plays a crucial role in classifying representations by determining reducibility, distinction, and automorphic period relations through its connection with nilpotent orbits.
• Its applications extend globally and geometrically, influencing Fourier coefficient analysis, Jacquet modules, and the microlocal study of D-modules in the Langlands program.

Whittaker support is a fundamental invariant in the representation theory of reductive groups over local and global fields, as well as in the geometric Langlands program. It encodes which degenerate or generalized Whittaker models (or, globally, Fourier coefficients) occur nontrivially for a given (smooth, moderate growth) representation or D-module. The Whittaker support (often denoted $\operatorname{WS}(\pi)$ for a representation $\pi$) determines both the structure of the representation and its relation to the geometry of nilpotent orbits in the Lie algebra, often controlling important properties such as reducibility, distinction, and automorphic period relations.

1. Nilpotent Orbits and Degenerate Whittaker Models

Given a reductive group $G$ over a local field $F$ (either $\mathbb{R}$ or a $p$-adic field), with Lie algebra $\mathfrak{g}$ and linear dual $\mathfrak{g}^*$, the coadjoint orbits of $G$ on $\mathfrak{g}^*$ encode geometric invariants of representations. A coadjoint orbit $\mathcal{O} \subset \mathfrak{g}^*$ is called nilpotent if it contains a functional with a dense unipotent stabilizer. The identification via the Killing form yields a nilpotent element $f \in \mathfrak{g}$, which fits, by the Jacobson–Morozov theorem, into an $\mathfrak{sl}_2$-triple $(e,h,f)$.

For any rational semisimple $S \in \mathfrak{g}$ and $y \in \mathfrak{g}^*$ with $\operatorname{ad}^*(S)y = -2y$ (making $(S,y)$ a Whittaker pair), one defines a degenerate Whittaker model $\mathcal{W}_{S,y}$ as a compact or Schwartz induction of a certain character $\psi_y$ on a unipotent subgroup $L \leq G$, constructed from the eigenspace decomposition associated to $S$. This generalizes the classical (generic) Whittaker model, which is attached to the largest nilpotent orbit.

For a smooth (or Fréchet moderate-growth) representation $\pi$ of $G$, the associated (degenerate) Whittaker quotient is

$$\pi_{S,y} \coloneqq (\mathcal{W}_{S,y} \otimes \pi)^G \,.$$

When $y$ is in degree $2$ for a neutral $S = h$ (from an $\mathfrak{sl}_2$-triple), the resulting Whittaker model, written $\mathcal{W}_y$, depends only on the orbit $\mathcal{O}$ containing $y$, so the quotient is often written $\pi_{\mathcal{O}}$.

2. Definition of Whittaker Support

The Whittaker support of a representation $\pi$ is defined in terms of the nonvanishing of these generalized Whittaker quotients:

• $W(\pi) = \{\mathcal{O} : \pi_{\mathcal{O}} \ne 0\}$
• $\mathrm{WS}(\pi) =$ the set of maximal elements of $W(\pi)$ under Zariski-closure order.

Thus, a nilpotent orbit $\mathcal{O}$ belongs to $\mathrm{WS}(\pi)$ if $\pi_{\mathcal{O}} \neq 0$ but no strictly larger orbit supports a nonzero Whittaker quotient. For $G = \mathrm{GL}_n$, this is equivalent to the existence of nonvanishing "partial" Fourier coefficients attached to the corresponding Jordan form, and the closure order corresponds to the dominance order on partitions.

3. Classification of Orbits in Whittaker Support

The principal structural result describes the type of nilpotent orbits that may occur in Whittaker support.

Quasi-admissibility:

A nilpotent coadjoint orbit $\mathcal{O}$ is called quasi-admissible if, for some $y\in\mathcal{O}$, the metaplectic double cover of the centralizer $G_y$ admits a genuine finite-dimensional representation (i.e., the pullback cover admits a nontrivial finite-dimensional representation on which the nontrivial central element acts by $-1$).

Main Theorem (Gomez et al., 2016, Gourevitch et al., 2018):

If $\pi \in \operatorname{Rep}(G)$ and $\mathcal{O} \in \operatorname{WS}(\pi)$, then $\mathcal{O}$ is quasi-admissible. In the case of $G = \mathrm{GL}_n$ and $F$ non-Archimedean, these orbits are in fact special in the Lusztig sense.

$F$-Distinguished Orbits (quasi-cuspidal specialization):

For $\pi$ quasi-cuspidal (i.e., all normalized Jacquet modules $r_P(\pi)$ vanish for proper parabolics), every orbit $\mathcal{O} \in \mathrm{WS}(\pi)$ is $F$-distinguished, namely, it does not intersect the Lie algebra of any proper Levi subgroup of $G$ defined over $F$.

4. Relationship to Other Representation-Theoretic Invariants

In the $p$-adic setting, $\operatorname{WS}(\pi)$ coincides with the wave-front set $\operatorname{WF}(\pi)$, i.e., the set of maximal nilpotent orbits appearing in the character expansion of $\pi$. For admissible representations, $\dim(\pi_{\mathcal{O}})$ equals the coefficient in the character expansion for each $\mathcal{O}$ in the support.

For $G=\mathrm{GL}_n$, each degenerate Whittaker quotient $\pi_{\mathcal{O}}$ is one-dimensional if nonzero, corresponding to the uniqueness of the generalized model for each orbit. Additionally, for admissible $\pi$, the variety defined by the annihilator in the universal enveloping algebra, $V(\pi)$, equals the Zariski-closure of $\operatorname{WF}(\pi)$.

5. Global and Automorphic Aspects

For automorphic representations $\Pi$ of $G$ over the adèles of a global field $K$, the Whittaker support is defined in terms of the nonvanishing of global Fourier coefficients (period integrals) attached to Whittaker pairs. Key global results include:

• If $\mathcal{O} \in \operatorname{WS}(\Pi)$, then $\mathcal{O}$ is quasi-admissible at every local place, and for split classical $G$, these are special orbits.
• For cuspidal automorphic representations, $\operatorname{WS}(\Pi)$ consists of $K$-distinguished nilpotent orbits.
• For $\mathrm{GL}_n(K)$, the Whittaker support enjoys the "wave-front closure" property: if nonzero for a given orbit, it is nonzero for every suborbit in its closure.

Consequences for the explicit structure of Fourier coefficients, Rankin–Selberg integrals, and small representations are realized via partitions with restrictions (e.g., totally even for symplectic, totally odd for orthogonal groups).

6. Extensions to the Geometric Langlands Program

In geometric representation theory, Whittaker support is studied for $\mathcal{D}$-modules on $\operatorname{Bun}_G$ (the moduli stack of $G$-bundles). The geometric Whittaker coefficient functor $\coeff$ assigns to each $F\in \Dmod(\Bun_G)$ a vector space by an explicit (derived) integration over a twisted unipotent stack against an exponential $\mathcal{D}$-module $\Exp$. The microlocal (singular support) perspective reveals:

• $\coeff$ is $t$-exact and Verdier self-dual when restricted to sheaves with nilpotent singular support.
• The vanishing locus of Whittaker coefficients is governed by the intersection of the singular support $\SingSupp(F)$ with the global Kostant section in the nilpotent cone of the cotangent stack.
• For cuspidal and tempered $\mathcal{D}$-modules, some Whittaker coefficient is always nonzero; for very irreducible Hecke eigensheaves, the objects are perverse, of finite length, and irreducible on each connected component.

This suggests that, just as in the classical setting, Whittaker support stratifies $\mathcal{D}$-modules according to microlocal and spectral data, and that tempered/anti-tempered decompositions (and the t-exactness of functors) reflect and generalize the occurrence of various Whittaker models.

7. Proof Techniques and Structural Methods

The key proof methods for the structural theorems concerning Whittaker support include:

• Root Exchange/Stone–von Neumann Theorems: Transferring between degenerate Whittaker models associated to varying semisimple elements, tracking "jumps" and isomorphisms via induction on Heisenberg subgroups.
• Deformation Techniques: Systematically eliminating contributions from interfering orbits by sequential root exchanges, forcing nonvanishing in the neutral model associated to a maximal orbit.
• Metaplectic Centralizer Analysis: Demonstrating finite-dimensionality of the action of the metaplectic double cover of the centralizer on Whittaker quotients, confirming quasi-admissibility.
• Jacquet-Module Vanishing: For quasi-cuspidal representations, showing that support on any non-distinguished orbit would force existence of a nonzero Jacquet module, contradicting cuspidality.
• Global-Local Comparison: Interplay of local and global theta correspondence, with vanishing of local Jacobi (Whittaker) periods implying global vanishing.

These methods together yield a uniform framework that covers both $p$-adic and Archimedean fields, as well as extensions to automorphic and geometric settings, confirming and refining earlier conjectures and theorems regarding the structure of Whittaker supports for representations of reductive groups (Gomez et al., 2016, Gourevitch et al., 2018, Faergeman et al., 2022).