Strategies for optimal sky subtraction in the low surface brightness regime

Abstract: The low surface brightness (LSB) regime ($\mu_{g} \gtrsim 26$ mag arcsec$^{-2}$) comprises a vast, mostly unexplored discovery space, from dwarf galaxies to the diffuse interstellar medium. Accessing this regime requires precisely removing instrumental signatures and light contamination, including, most critically, night sky emission. This is not trivial, as faint astrophysical and instrumental contamination can bias sky models at the precision needed to characterize LSB structures. Using idealized synthetic images, we assess how this bias impacts two common LSB-oriented sky-estimation algorithms: 1.) masking and parametric modelling, and 2.) stacking and smoothing dithered exposures. Undetected flux limits both methods by imposing a pedestal offset to all derived sky models. Careful, deep masking of fixed sources can mitigate this, but source density always imposes a fundamental limit. Stellar scattered light can contribute $\sim28$--$29$ mag arcsec$^{-2}$ of background flux even in low-density fields; its removal is critical prior to sky estimation. For complex skies, image combining is an effective non-parametric approach, although it strongly depends on observing strategy and adds noise to images on the smoothing kernel scale. Preemptive subtraction of fixed sources may be the only practical approach for robust sky estimation. We thus tested a third algorithm, subtracting a preliminary sky-subtracted coadd from exposures to isolate sky emission. Unfortunately, initial errors in sky estimation propagate through all subsequent sky models, making the method impractical. For large-scale surveys like LSST, where key science goals constrain observing strategy, masking and modelling remains the optimal sky estimation approach, assuming stellar scattered light is removed first.