New zeta functions of Reidemeister type and twisted Burnside-Frobenius theory

Abstract: We introduce new zeta functions related to an endomorphism $\phi$ of a discrete group $\Gamma$. They are of two types: counting numbers of fixed ($\rho\sim \rho\circ\phi^n$) irreducible representations for iterations of $\phi$ from an appropriate dual space of $\Gamma$ and counting Reidemeister numbers $R(\phi^n)$ of different compactifications. Many properties of these functions and their coefficients are obtained. In many cases it is proved that these zeta functions coincide. The Gauss congruences are proved. Useful asymptotic formulas for the zeta functions are found. Rationality is proved for some examples, which give also the first counterexamples simultaneously for TBFT ($R(\phi)$=the number of fixed irreducible unitary representations) and TBFT$_f$ ($R(\phi)$=the number of fixed irreducible unitary finite-dimensional representations) for an automorphism $\phi$ with $R(\phi)<\infty$.