Fundamental solutions and critical Lane-Emden exponents for nonlinear integral operators in cones

Abstract: In this article we study the fundamental solutions or "$\alpha$-harmonic functions" for some nonlinear positive homogeneous nonlocal elliptic problems in conical domains, such as \begin{eqnarray*}\label{ecbir1a1} {\mathcal F }(u)=0\ \ \hbox{in} \ \ \mathcal{C}\omega,\quad u=0\ \ \hbox{in} \ \ \mathbb{R}^n\setminus \mathcal{C}\omega ,\ \ \end{eqnarray*} where $\omega$ is a proper $C^2$ domain in $S^{N-1}$ for $ N\geq 2$, $\mathcal{C}\omega:={x\,:\,x\neq 0, {|x|^{-1}}x\in \omega}$ is the cone-like domain related to $\omega$, and ${\mathcal F }$ is an extremal fully nonlinear integral operator. We prove the existence of two fundamental solutions that are homogeneous and do not change signs in the cone; one is bounded at the origin and the other at infinity. As an application, we use the fundamental solutions obtained to prove Liouville type theorems in cones for supersolutions of the Lane-Emden-Fowler equation in the form \begin{eqnarray*}\label{eq 0.2} {\mathcal F }(u)+u^p = 0\ \ \hbox{in} \ \ \mathcal{C}\omega, \quad u=0\ \ \hbox{in} \ \ \mathbb{R}^n\setminus \mathcal{C}_\omega.\ \ \end{eqnarray*} We also prove a generalized Hopf type lemma in domains with corners. Most of our results are new even when ${\mathcal F }$ is the fractional Laplacian operator.