Ground state of indefinite coupled nonlinear Schrödinger systems

Abstract: In this paper, we study the ground state solutions of the following coupled nonlinear Schrödinger system (P) $-Δu_1-τ_1 u_1 =μ_1u_1^3+βu_1u_2^2$, $ -Δu_2-τ_2 u_2 =μ_2u_2^3+βu_1^2u_2$ in $Ω$, $u_1=u_2=0$ on $\partialΩ$, where $μ_1, μ_2>0$, $β>0$ and $Ω\subset \mathbb{R}^N (N\le3)$ is a bounded domain with smooth boundary. We are concerned with the indefinite case, i.e., $τ_1, τ_2$ are greater than or equal to the principal eigenvalue of $-Δ$ with the Dirichlet boundary datum. By delicate variational arguments, we obtain the existence of ground state solution to $(P)$, and also provide information on critical energy levels for coupling parameter $β$ in some ranges.