Non-separable lattices, Gabor orthonormal bases and Tilings

Abstract: Let $K\subset \Bbb R^d$ be a set with positive and finite Lebesgue measure. Let $\Lambda=M(\Bbb Z^{2d})$ be a lattice in $\Bbb R^{2d}$ with density dens$(\Lambda)=1$. It is well-known that if $M$ is a diagonal block matrix with diagonal matrices $A$ and $B$, then $\mathcal G(|K|^{-1/2}\chi_K, \Lambda)$ is an orthonormal basis for $L^2(\Bbb R^d)$ if and only if $K$ tiles both by $A(\Bbb Z^d)$ and $B^{-t}(\Bbb Z^d)$. However, there has not been any intensive study when $M$ is not a diagonal matrix. We investigate this problem for a large class of important cases of $M$. In particular, if $M$ is any lower block triangular matrix with diagonal matrices $A$ and $B$, we prove that if $\mathcal G(|K|^{-1/2}\chi_K, \Lambda)$ is an orthonormal basis, then $K$ can be written as a finite union of fundamental domains of $A({\mathbb Z}^d)$ and at the same time, as a finite union of fundamental domains of $B^{-t}({\mathbb Z}^d)$. If $A^tB$ is an integer matrix, then there is only one common fundamental domain, which means $K$ tiles by a lattice and is spectral. However, surprisingly, we will also illustrate by an example that a union of more than one fundamental domains is also possible. We also provide a constructive way for forming a Gabor window functions for a given upper triangular lattice. Our study is related to a Fuglede's type problem in Gabor setting and we give a partial answer to this problem in the case of lattices.