The Luminosity Function and Clustering of H$α$ Emitting Galaxies at $z\approx4-6$ from a Complete NIRCam Grism Redshift Survey

Abstract: We study the luminosity function (LF) and clustering properties of 888 H$\alpha$ emitters (HAEs) at $3.75 < z < 6$ in the GOODS-N field. The sample, built from JWST CONGRESS and FRESCO NIRCam grism surveys using a novel redshift assignment algorithm, spans $\sim$62 arcmin$^2$ and reaches $L_{\rm H\alpha} \sim 10^{41.2} {\rm erg s^{-1}}$. We identify two prominent filamentary protoclusters at $z \approx 4.41$ and $z \approx 5.19$, hosting 98 and 144 HAEs, respectively. The observed H$\alpha$ LFs show similar shallow faint-end slopes for both protocluster and field galaxies at $z=3.75-5$, and for the protocluster at $z=5-6$ ($\alpha\approx 1.2$ to $-1.3$). In contrast, the field LF at $z=5-6$ is much steeper ($\alpha=-1.87_{-0.23}^{+0.30}$), suggesting that protocluster galaxies at $z > 5$ are more evolved, resembling those at $z=3.75-5$. The observed star formation rate density from H$\alpha$, integrated down to 0.45 ${\rm M_\odot yr^{-1}}$, is $0.050^{+0.002}_{-0.003}$ and $0.046^{+0.006}_{-0.004} M_\odot {\rm yr}^{-1} {\rm Mpc}^{-3}$ at $z=3.75-5$ and $z=5-6$, with protoclusters contributing $\sim$25% and 55%, respectively. This implies that a large fraction of star formation at $z > 4$ occurs in protoclusters. We conduct the first star-formation-rate-limited 3D clustering analysis at $z > 4$. We find the filamentary protocluster geometry flattens the power-law shape of the HAE auto-correlation functions, with slopes much shallower than typically assumed. The auto-correlation function of field HAEs have correlation lengths of $r_0 = 4.61^{+1.00}_{-0.68} h^{-1}{\rm Mpc}$ at $z \approx 4-5$ and $r_0 = 6.23^{+1.68}_{-1.13} h^{-1}{\rm Mpc}$ at $z=5-6$. Comparing the observed correlation functions with the UniverseMachine simulation, we infer the dark matter (sub-)halo masses of HAEs to be $\log (M_h/M_\odot)=11.0-11.2$ at $z\approx 4-6$, with a scatter of 0.4 dex.