Half Nikulin surfaces and moduli of Prym curves

Abstract: Let F^N_g be the moduli space of polarized Nikulin surfaces (Y,H) of genus g and let P^N_g be the moduli of triples (Y,H,C), with C in |H| a smooth curve. We study the natural map \chi_g:P^N_g -> R_g, where R_g is the moduli space of Prym curves of genus g. We prove that it is generically injective on every irreducible component, with a few exceptions in low genus. This gives a complete picture of the map \chi_g and confirms some striking analogies between it and the Mukai map m_g: P_g ->M_g for moduli of triples (Y,H,C), where (Y,H) is any genus g polarized K3 surface. The proof is by degeneration to boundary points of a partial compactification of F^N_g. These represent the union of two surfaces with four even nodes and effective anticanonical class, which we call half Nikulin surfaces. The use of this degeneration is new with respect to previous techniques.