Orthonormal Bases in the Orbit of Square-Integrable Representations of Nilpotent Lie Groups

Abstract: Let $G$ be a connected, simply connected nilpotent group and $\pi$ be a square-integrable irreducible unitary representation modulo its center $Z(G)$ on $L^2(\mathbf{R}^d)$. We prove that under reasonably weak conditions on $G$ and $\pi$ there exist a discrete subset $\Gamma$ of $G/Z(G)$ and some (relatively) compact set $F \subseteq \mathbf{R}^d$ such that $$\bigl { |F|^{-1/2} \hspace{2pt} \pi(\gamma) 1_F \mid \gamma \in \Gamma \bigr}$$ forms an orthonormal basis of $L^2(\mathbf{R}^d)$. This construction generalizes the well-known example of Gabor orthonormal bases in time-frequency analysis. The main theorem covers graded Lie groups with one-dimensional center. In the presence of a rational structure, the set $\Gamma $ can be chosen to be a uniform subgroup of $G/Z$.