Rationality of twists of the Siegel modular variety of genus $2$ and level $3$

Abstract: Let $\overline{\rho}: G_{\mathbf{Q}} \rightarrow {\rm GSp}_4(\mathbf{F}_3)$ be a continuous Galois representation with cyclotomic similitude character -- or, what turns out to be equivalent, the Galois representation associated to the $3$-torsion of a principally polarized abelian surface $A/\mathbf{Q}$. We prove that the moduli space $\mathcal{A}_2(\overline{\rho})$ of principally polarized abelian surfaces $B/\mathbf{Q}$ admitting a symplectic isomorphism $B[3] \simeq \overline{\rho}$ of Galois representations is never rational over $\mathbf{Q}$ when $\overline{\rho}$ is surjective, even though it is both rational over $\mathbf{C}$ and unirational over $\mathbf{Q}$ via a map of degree $6$.