Fractional Laplacians on domains, a development of Hörmander's theory of mu-transmission pseudodifferential operators

Abstract: Let $P$ be a classical pseudodifferential operator of complex order $m$ on an $n$-dimensional smooth manifold $\Omega_1$. For the truncation $P_\Omega$ to a smooth subset $\Omega$ there is a well-known theory of boundary value problems when $P_\Omega$ has the transmission property (preserves $C^\infty (\bar\Omega)$) and is of integer order; the calculus of Boutet de Monvel. Many interesting operators, such as for example complex powers of the Laplacian $(-\Delta)^\mu $ with noninteger mu, are not covered. They have instead the mu-transmission property defined in H\"ormander's books, mapping $x_n^\mu C^\infty (\bar\Omega)$ into $C^\infty (\bar\Omega)$. In an unpublished lecture note from 1965, H\"ormander described an $L_2$-solvability theory for mu-transmission operators, departing from Vishik and Eskin's results. We here develop the theory in $L_p$ Sobolev spaces ($1<p<\infty$) in a modern setting. It leads to not only Fredholm solvability statements but also regularity results in full scales of Sobolev spaces (for $s\to \infty$). The solution spaces have a singularity at the boundary that we describe in detail. We moreover obtain results in H\"older spaces, which radically improve recent regularity results for fractional Laplacians.