Where Galaxies Go to Die: The Environments of Massive Quiescent Galaxies at $3<z<5$

Abstract: At low redshift, massive quiescent galaxies (MQGs) are most frequently found in massive, rich galaxy clusters, but at high redshift the trend is less clear. Here, we present spectroscopic evidence of the effects of environment on the formation and assembly of high-redshift MQGs. We identify 25 (5) $\log (M_/\mathrm{M_\odot}\geq10.5$ ($10.0\leq\log (M_/\mathrm{M_\odot}<10.5$) spectroscopically-confirmed quiescent galaxies in the UDS and EGS fields at $3<z\<5$ with NIRSpec PRISM spectroscopy from RUBIES and other public JWST NIRSpec programs. We measure the density contrast in these fields by applying a Monte Carlo Voronoi Tesselation density mapping technique to photometric and spectroscopic redshifts of $m_\mathrm{F444W}\<27.5$ sources. We robustly detect 12 massive overdense peaks with $\log (M_\mathrm{Peak}/\mathrm{M_\odot})\geq13$ and six extended massive protoclusters ($\log (M_\mathrm{Struct}/\mathrm{M_\odot})\geq13.85$). We observe that MQGs are preferentially found in these massive peaks and within these massive structures: $\approx50\%$ of MQGs are found in massive peaks, compared to $\approx20\%$ of massive star forming galaxies (MSFGs) and $\approx15\%$ of the overall spectroscopically-confirmed population. We also find an apparent dependence on both quiescent galaxy mass and environment, with $75\%$ of the most massive ($\log (M_*/\mathrm{M_\odot}\geq10.75$) residing inside overdense peaks. We compare the star formation histories (SFHs) of the MQGs with the high-redshift galaxy stellar mass function from observations and simulated quiescent galaxies at $z\>5$, finding that the masses from the inferred MQG SFHs regularly exceed either observed or simulated high-redshift galaxies, which suggests indicates that mergers and ex-situ star formation play a key role in the mass assembly of MQGs in overdense environments.