Exploring the z=3-4 massive galaxy population with ZFOURGE: the prevalence of dusty and quiescent galaxies

Abstract: Our understanding of the redshift $z>3$ galaxy population relies largely on samples selected using the popular "dropout" technique, typically consisting of UV-bright galaxies with blue colors and prominent Lyman breaks. As it is currently unknown if these galaxies are representative of the massive galaxy population, we here use the FourStar Galaxy Evolution (ZFOURGE) Survey to create a stellar mass-limited sample at $z=3-4$. Uniquely, ZFOURGE uses deep near-infrared medium-bandwidth filters to derive accurate photometric redshifts and stellar population properties. The mass-complete sample consists of 57 galaxies with log M $>10.6$, reaching below $M^{\star}$ at $z=3-4$. On average, the massive $z=3-4$ galaxies are extremely faint in the observed optical with median $R_{tot}^{AB}=27.48\pm0.41$ (restframe $M_{1700}=-18.05\pm0.37$). They lie far below the UV luminosity-stellar mass relation for Lyman break galaxies and are about $\sim100\times$ fainter at the same mass. The massive galaxies are red ($R-Ks_{AB}=3.9\pm0.2$; restframe UV-slope $\beta=-0.2\pm0.3$) likely from dust or old stellar ages. We classify the galaxy SEDs by their restframe $U-V$ and $V-J$ colors and find a diverse population: $46^{+6+10}_{-6-17}$% of the massive galaxies are quiescent, $54^{+8+17}_{-8-10}$% are dusty star-forming galaxies, and only $14^{+3+10}_{-3-4}$% resemble luminous blue star forming Lyman break galaxies. This study clearly demonstrates an inherent diversity among massive galaxies at higher redshift than previously known. Furthermore,we uncover a reservoir of dusty star-forming galaxies with $4\times$ lower specific star-formation rates compared to submillimeter-selected starbursts at $z>3$. With $5\times$ higher numbers, the dusty galaxies may represent a more typical mode of star formation compared to submillimeter-bright starbursts.