Perverse coherent extensions on Calabi-Yau threefolds and representations of cohomological Hall algebras

Abstract: For $Y\to X$ a toric Calabi-Yau threefold resolution and $M\in \DD^b\Coh(Y)^T$ satisfying some hypotheses, we define a stack $\mf M(Y,M)$ parameterizing \emph{perverse coherent extensions} of $M$, iterated extensions of $M$ and the compactly supported perverse coherent sheaves of Bridgeland. We define framed variants $\mf M^\f(Y,M)$, prove that they are equivalent to stacks of representations of framed quivers with potential $(Q^\f,W^\f)$, and deduce natural monad presentations for these sheaves. Moreover, following Soibelman we prove that the homology $H_\bullet(\mf M^{\f,\zeta}(Y,M),\varphi_{W^\f})$ of the space of $\zeta$-stable, $\f$-framed perverse coherent extensions of $M$, with coefficients in the sheaf $\varphi_{W^\f}$ of vanishing cycles for $W^\f$, is a representation of the Kontsevich-Soibelman cohomological Hall algebra of $Y$. For $M=\mc O_Y[1]$, $\mf M^{\f}(Y,M)$ is the stack of perverse coherent systems of Nagao-Nakajima, so $\bb V_Y^\zeta=H_\bullet(\mf M^{\f,\zeta}(Y,M),\varphi_{W^\f})$ is the DT/PT series of $Y$ for $\zeta=\zeta_{\DT/\PT}$ by Szendroi and \emph{loc. cit.}, and we conjecture that $\V_Y^{\zeta_\NCDT}$ is the vacuum module for the quiver Yangian of Li-Yamazaki. For $M=\mc O_S[1]$ with $S\subset Y$ a divisor, $\mf M^{\f}(Y,M)$ provides a definition in algebraic geometry for Nekrasov's spiked instanton variant of the ADHM construction, and analogous variants of the constructions of Kronheimer-Nakajima, Nakajima-Yoshioka, and Finkelberg-Rybnikov. We conjecture that $H_\bullet(\mf M^{\f,\zeta}(Y,M),\varphi_{W^{\f}})$ is the vacuum module of the vertex algebra $\V(Y,S)$ defined by the \mbox{authors} in a companion paper, generalizing the AGT conjecture to this setting. For $Y\to X={xy-z^mw^n}$, this gives a geometric approach to the relationship between $W$-algebras and Yangians for affine $\gl_{m|n}$.