Integration by parts and Pohozaev identities for space-dependent fractional-order operators

Abstract: Consider a classical elliptic pseudodifferential operator $P$ on ${\Bbb R}^n$ of order $2a$ ($0<a<1)$ with even symbol. For example, $P=A(x,D)^a$ where $A(x,D)$ is a second-order strongly elliptic differential operator; the fractional Laplacian $(-\Delta )^a$ is a particular case. For solutions $u$ of the Dirichlet problem on a bounded smooth subset $\Omega \subset{\Bbb R}^n$, we show an integration-by-parts formula with a boundary integral involving $(d^{-a}u)|_{\partial\Omega }$, where $d(x)=\operatorname{dist}(x,\partial\Omega )$. This extends recent results of Ros-Oton, Serra and Valdinoci, to operators that are $x$-dependent, nonsymmetric, and have lower-order parts. We also generalize their formula of Pohozaev-type, that can be used to prove unique continuation properties, and nonexistence of nontrivial solutions of semilinear problems. An illustration is given with $P=(-\Delta +m^2)^a$. The basic step in our analysis is a factorization of $P$, $P\sim P^-P^+$, where we set up a calculus for the generalized pseudodifferential operators $P^\pm$ that come out of the construction.