Variational method for fractional Hamiltonian system in bounded domain

Abstract: Here we consider the following fractional Hamiltonian system \begin{equation*} \begin{cases} \begin{aligned} (-\Delta)^{s} u&=H_v(u,v) \;\;&&\text{in}~\Omega,\ (-\Delta)^{s} v&=H_u(u,v) &&\text{in}~\Omega,\ u &= v = 0 &&\text{in} ~ \mathbb{R}^N\setminus\Omega, \end{aligned} \end{cases} \end{equation*} where $s\in (0,1)$, $N>2s$, $H \in C^1(\mathbb{R}^2, \mathbb{R})$ and $\Omega \subset \mathbb{R}^N$ is a smooth bounded domain. %As the problem remains unchanged if $H(u, v)$ is replaced by $H(u, v)-H(0, 0)$, hence we always assume $H(0,0)=0$. To apply the variational method for this problem, the key question is to find a suitable functional setting. Instead of usual fractional Sobolev spaces, we use the solutions space of $(-\Delta)^{s}u=f\in L^r(\Omega)$ for $r\ge 1$, for which we show the (compact) embedding properties. When $H$ has subcritical and superlinear growth, we construct two frameworks, respectively with interpolation space method and dual method, to show the existence of nontrivial solution. As byproduct, we revisit the fractional Lane-Emden system, i.e. $H(u, v)=\frac{1}{p+1}|u|^{p+1}+\frac{1}{q+1}|v|^{q+1}$, and consider the existence, uniqueness of (radial) positive solutions under subcritical assumption.