本文主要考虑如下拟线性椭圆型问题

$$\left\{ \begin{array}{ll} -\mbox{div}(|\nabla u|^{p-2}\nabla u) +|u|^{p-2}u=\frac{1}{\alpha}F_{u}(u,v)+f(x),&x\in \Omega,\\[3mm] -\mbox{div}(|\nabla v|^{p-2}\nabla v) +|v|^{p-2}v=\frac{1}{\alpha}F_{v}(u,v)+g(x),&x\in \Omega,\\[3mm] |\nabla u|^{p-2}\frac{\partial u}{\partial n}=\lambda G_{u}(u,v),\ |\nabla v|^{p-2}\frac{\partial v}{\partial n}=\lambda G_{v}(u,v),~~ &x\in\partial\Omega,\end{array}\right. $$

(1.1)

其中$\Omega\subset{\Bbb R}^{N}$是有界光滑区域,$1<p<N$,$p<\alpha<p^{*}=\frac{Np}{N-p}$,$\lambda>0$,$\frac{\partial }{\partial n}$表示外法向导数,且函数$f$,$g$,$F$,$G$满足条件:

$(A_{1})$ $f$,$g\in C(\overline{\Omega})$,且存在非空区域$\Omega_{1}\subset \Omega$使得对任意$x\in\Omega_{1}$,$f$,$g>0$;

$(A_{2})$ $F$,$G\in C^{1}({\Bbb R}^{2},{\Bbb R}^{+})$ 分别是正的$\alpha$次,$\beta$次齐次函数,即

$$F(t u,tv)=t^{\alpha}F(u,v) (t>0,p<\alpha<p^{*}),G(t u,tv)=t^{\beta}G(u,v) (t>0,1<\beta<p). $$

近年来,带有非线性边界条件的椭圆型方程组引起了许多学者的关注,特别地,Brown和Wu在文献[13]中研究了如下问题

$$\left\{ \begin{array}{ll} -\triangle u+u=\frac{\alpha}{\alpha+\beta}f(x)|u|^{\alpha-2}u|v|^{\beta},&x\in \Omega,\\[3mm] -\triangle v+v=\frac{\beta}{\alpha+\beta}f(x)|u|^{\alpha}|v|^{\beta-2}v,&x\in \Omega,\\[3mm] \frac{\partial u}{\partial n}=\lambda g(x)|u|^{q-2}u,\ \frac{\partial v}{\partial n}=\mu h(x)|v|^{q-2}v,~~ &x\in\partial\Omega,\end{array}\right. $$

(1.2)

其中 $\Omega\subset{\Bbb R}^{N}$为有界光滑区域,$2<\alpha+\beta<2^{*}$. 利用变分法,作者得到了当$(\lambda,\mu)$属于某一特定区间时,问题(1.2)至少存在两个非平凡的非负解. 接着,Liu和Chen^[11]把以上结果推广到拟线性情形

$$\left\{ \begin{array}{ll} -\mbox{div}(a(x)|\nabla u|^{p-2}\nabla u) +b(x)|u|^{p-2}u=d^{-1}F_{u}(x,u,v),&x\in \Omega,\\[1mm] -\mbox{div}(a(x)|\nabla v|^{p-2}\nabla v) +b(x)|v|^{p-2}v=d^{-1}F_{v}(x,u,v),&x\in \Omega,\\[2mm] a(x)|\nabla u|^{p-2}\frac{\partial u}{\partial n}=\lambda h(x)|u|^{m-2}u,\ a(x)|\nabla v|^{p-2}\frac{\partial v}{\partial n}=\mu H(x)|v|^{m-2}v,~~ &x\in\partial\Omega,\end{array}\right. $$

(1.3)

且证明了当$\Omega$为一光滑的外部区域时,问题(1.3)至少存在两个非平凡的非负解.

同时,问题

$$\left\{ \begin{array}{ll} -\triangle u+u=\lambda f(x)|u|^{q-2}u,&x\in \Omega,\\[1mm]-\triangle v+v=\mu g(x)|v|^{q-2}v,&x\in \Omega,\\[2mm] \frac{\partial u}{\partial n}=\frac{\alpha}{\alpha+\beta} h(x)|u|^{\alpha-2}u|v|^{\beta},&x\in\partial\Omega,\\[3mm] \frac{\partial v}{\partial n}=\frac{\beta}{\alpha+\beta} h(x)|u|^{\alpha}|v|^{\beta-2}v,~~ &x\in\partial\Omega\end{array}\right. $$

(1.4)

也得到了广泛研究. 例如,当$2<\alpha+\beta<2^{*}$,$1<q<2$ 时,Lu^[12]研究了问题(1.4). 随后,Rasouli和Afrouzi在文献[4]中考虑了如下问题

$$\left\{ \begin{array}{ll} -\triangle_{p}u+m(x)|u|^{p-2}u=\lambda a(x)|u|^{\gamma-2}u,~~ &x\in \Omega,\\[1mm]-\triangle_{p}v+m(x)|v|^{p-2}v=\mu b(x)|v|^{\gamma-2}v,&x\in \Omega,\\[2mm] |\nabla u|^{p-2}\frac{\partial u}{\partial n}=\frac{\alpha}{\alpha+\beta} |u|^{\alpha-2}u|v|^{\beta},&x\in\partial\Omega,\\[3mm] |\nabla v|^{p-2}\frac{\partial v}{\partial n}=\frac{\beta}{\alpha+\beta} |u|^{\alpha}|v|^{\beta-2}v,~~ &x\in\partial\Omega,\end{array}\right. $$

(1.5)

其中 $2<\alpha+\beta<p<\gamma<p^{*}$.

最近,Zhou和Kim在文献[3]中研究了更一般的问题

$$\left\{ \begin{array}{ll} -\triangle_{p}u+m(x)|u|^{p-2}u=\lambda F_{u}(u,v),~~ &x\in \Omega,\\[1mm]-\triangle_{p}v+n(x)|v|^{p-2}v=\lambda F_{v}(u,v),&x\in \Omega,\\[2mm] |\nabla u|^{p-2}\frac{\partial u}{\partial n}=G_{u}(u,v),&x\in\partial\Omega,\\[3mm] |\nabla v|^{p-2}\frac{\partial v}{\partial n}=G_{v}(u,v),&x\in\partial\Omega,\end{array}\right. $$

(1.6)

其中 $F$,$G\in C^{1}({\Bbb R}\times {\Bbb R})$满足$F(t u,tv)=t^{\alpha}F(u,v)$,$G(t u,tv)=t^{\beta}G(u,v)$,$t\geq0$,$\alpha\in (p,p^{*})$,$\beta\in(1,p)$. 利用Nehari流形方法,作者证明了当$\lambda$充分小时,问题(1.6)至少存在两个非平凡的非负解.

对于带有非齐次项椭圆型问题的研究,参见文献[1, 6, 14]. 然而,据我们所知,还没有文章研究过同时含有非齐次项和非线性边界条件的问题. 此外,在本文中,我们将不再利用文献[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]中所采用的Nehari 流形方法,而是利用山路引理和Ekeland变分准则去研究问题(1.1)解的存在性与多解性. 事实上,利用Nehari流形方法很难得到相同的结果.

本文主要结果如下:

定理 1.1 假设条件 $(A_{1})$-$(A_{2})$ 成立. 则存在 $\lambda^{*}$,$L>0$ 使得当 $\lambda\in (0,\lambda^{*})$ 且 $\|f\|_{\sigma}^{\frac{p}{p-1}}+\|g\|_{\sigma}^{\frac{p}{p-1}}\in (0,L\lambda^{\frac{p(\alpha-p)}{\alpha-\beta}})$ 时,问题(1.1)至少存在两个非平凡解,其中 $\sigma=\frac{p^{*}}{p^{*}-1}.$

注 1.2 由条件$(A_{2})$可得

(i) $u F_{u}(u,v)+v F_{v}(u,v)=\alpha F(u,v),\ u G_{u}(u,v)+v G_{v}(u,v)=\beta G(u,v)$;

(ii) 存在 $M_{1}>0$ 使得对任意$(u,v)\in {\Bbb R}^{2}$,有

$$F(u,v)\leq M_{1}(|u|^{p}+|v|^{p})^{\frac{\alpha}{p}},\ G(u,v)\leq M_{1}(|u|^{p}+|v|^{p})^{\frac{\beta}{p}}, $$ 其中$M_{1}=\max\{\max_{|u|^{p}+|v|^{p}=1} F(u,v),\max_{|u|^{p}+|v|^{p}=1} G(u,v)\}$;

(iii) $F_{u}$,$F_{v}\in C({\Bbb R}^{2},{\Bbb R})$ 为正的$\alpha-1$次齐次函数,$G_{u}$,$G_{v}\in C({\Bbb R}^{2},{\Bbb R})$ 为 $\beta-1$次齐次函数.

对有界区域 $\Omega\subset{\Bbb R}^{N}$,记 $\|\cdot\|,\|\cdot\|_{p}$ 分别为 $W^{1,p}(\Omega)$ 和 $L^{p}(\Omega)$中的范数,即

$$\|u\|=\bigg(\int_{\Omega}(|\nabla u|^{p}+|u|^{p}){\rm d}x\bigg)^{\frac{1}{p}},\ \mbox{和} \ \|u\|_{p}=\bigg(\int_{\Omega}|u|^{p}{\rm d}x\bigg)^{\frac{1}{p}}. $$ 显然,$X=W^{1,p}(\Omega)\times W^{1,p}(\Omega)$ 是Banach 空间. 令 $X'$为$X$ 的对偶空间,且 $\langle ,\rangle$ 表示$X'$与$X$之间的对偶积. 空间$X$中的范数定义为 $$\|(u,v)\|=(\|u\|^{p}+\|v\|^{p})^{\frac{1}{p}}. $$

另外,我们用"$\rightharpoonup$"表示"弱收敛","$\rightarrow$" 表示"强收敛",$C$,$C_{i}$ $(i=1,2,\cdots)$ 表示不同的正数,$S$ 和 $\overline{S}$ 分别表示$W^{1,p}(\Omega)\hookrightarrow L^{p^{*}}(\Omega)$ 的最佳Sobolev常数和$W^{1,p}(\Omega)\hookrightarrow L^{\beta}(\partial\Omega)$ 的最佳Sobolev迹常数.

接下来,考虑与问题(1.1)相对应的能量泛函

$$I_{\lambda}(u,v)=\frac{1}{p}\|(u,v)\|^{p}-\frac{1}{\alpha}\int_{\Omega}F(u,v){\rm d}x-\int_{\Omega}(fu+gv){\rm d}x-\lambda\int_{\partial\Omega}G(u,v){\rm d}s. $$

由条件$(A_{1})$-$(A_{2})$,易知$I_{\lambda}\in C^{1}(X,{\Bbb R})$,且对任意$(\varphi_{1},\varphi_{2})\in X,$ 有

\begin{eqnarray*}\langle I'_{\lambda}(u,v),(\varphi_{1},\varphi_{2})\rangle&=&\int_{\Omega}(|\nabla u|^{p-2}\nabla u\nabla \varphi_{1}+|u|^{p-2}u\varphi_{1}+|\nabla v|^{p-2}\nabla v\nabla \varphi_{2}+|v|^{p-2}v\varphi_{2}){\rm d}x\\&&-\int_{\Omega}(f\varphi_{1}+g\varphi_{2}){\rm d}x-\frac{1}{\alpha}\int_{\Omega}(F_{u}(u,v)\varphi_{1}+F_{v}(u,v)\varphi_{2}){\rm d}x\\&&-\lambda\int_{\partial\Omega}(G_{u}(u,v)\varphi_{1}+G_{v}(u,v)\varphi_{2}){\rm d}s.\end{eqnarray*}特别地,由注 1.2(i)得到 $$\langle I'_{\lambda}(u,v),(u,v)\rangle=\|(u,v)\|^{p}-\int_{\Omega}F(u,v){\rm d}x-\int_{\Omega}(fu+gv){\rm d}x-\lambda\beta\int_{\Omega}G(u,v){\rm d}s. $$事实上,泛函$I_{\lambda}$的临界点即为问题(1.1)的弱解,且$(u,v)\in X$称为(1.1)的弱解,若对任意$(\varphi_{1},\varphi_{2})\in X,$都有\begin{eqnarray*}&&\int_{\Omega}(|\nabla u|^{p-2}\nabla u\nabla \varphi_{1}+|u|^{p-2}u\varphi_{1}+|\nabla v|^{p-2}\nabla v\nabla \varphi_{2}+|v|^{p-2}v\varphi_{2}){\rm d}x\\&&-\int_{\Omega}(f\varphi_{1}+g\varphi_{2}){\rm d}x-\frac{1}{\alpha}\int_{\Omega}(F_{u}(u,v)\varphi_{1}+F_{v}(u,v)\varphi_{2}){\rm d}x\\&&-\lambda\int_{\partial\Omega}(G_{u}(u,v)\varphi_{1}+G_{v}(u,v)\varphi_{2}){\rm d}s=0.\end{eqnarray*}

首先,验证 $I_{\lambda}$具有山路结构.

引理 2.1 假设条件 $(A_{1})$-$(A_{2})$ 成立. 则有

(i) 存在 $\rho$,$d_{0}>0$ 使得当$\|(u,v)\|=\rho$时,$I_{\lambda}(u,v)\geq d_{0}$;

(ii) 存在 $(\overline{u},\overline{v})\in X\setminus\{(0,0)\}$ 使得$\|(\overline{u},\overline{v})\|>\rho$ 且 $I_{\lambda}(\overline{u},\overline{v})<0$.

证 (i) 由注1.2 (ii),Sobolev嵌入定理和Sobolev迹不等式,存在常数 $C_{1}$,$C_{2}>0$使得

\begin{equation}\bigg|\int_{\Omega}F(u,v){\rm d}x\bigg|\leq M_{1}\int_{\Omega}(|u|^{p}+|v|^{p})^{\frac{\alpha}{p}}{\rm d}x\leq C_{1}\|(u,v)\|^{\alpha},\end{equation}

(2.1)

\begin{equation}\bigg|\int_{\partial\Omega}G(u,v){\rm d}s\bigg|\leq M_{1}\int_{\partial\Omega}(|u|^{p}+|v|^{p})^{\frac{\beta}{p}}{\rm d}s\leq C_{2}\|(u,v)\|^{\beta}.\end{equation}

(2.2)

利用Hölder和Young不等式,得

\begin{equation}\bigg|\int_{\Omega}fu{\rm d}x\bigg|\leq \|f\|_{\sigma}\|u\|_{p^{*}}\leq S\|f\|_{\sigma}\|u\|\leq \epsilon \|(u,v)\|^{p}+C_{\epsilon}\|f\|_{\sigma}^{\frac{p}{p-1}},\end{equation}

(2.3)

\begin{equation}\bigg|\int_{\Omega}gv{\rm d}x\bigg|\leq \|g\|_{\sigma}\|v\|_{p^{*}}\leq S\|g\|_{\sigma}\|v\|\leq \epsilon \|(u,v)\|^{p}+C_{\epsilon}\|g\|_{\sigma}^{\frac{p}{p-1}},\end{equation}

(2.4)

其中 $\sigma=\frac{p^{*}}{p^{*}-1}$,$C_{\epsilon}>0.$

于是,对$0<\epsilon\leq \frac{1}{4p}$,有

\begin{eqnarray*}I_{\lambda}(u,v)&\geq&\frac{1}{p}\|(u,v)\|^{p}-\frac{C_{1}}{\alpha}\|(u,v)\|^{\alpha}-2\epsilon\|(u,v)\|^{p}-C_{\epsilon}\Big(\|f\|_{\sigma}^{\frac{p}{p-1}}+\|g\|_{\sigma}^{\frac{p}{p-1}}\Big)-\lambda C_{2}\|(u,v)\|^{\beta}\\&\geq& \frac{1}{2p}\|(u,v)\|^{p}-\frac{C_{1}}{\alpha}\|(u,v)\|^{\alpha}-\lambda C_{2}\|(u,v)\|^{\beta}-C_{\epsilon}\Big(\|f\|_{\sigma}^{\frac{p}{p-1}}+\|g\|_{\sigma}^{\frac{p}{p-1}}\Big)\\&=& \|(u,v)\|^{p}\bigg[\frac{1}{2p}-\frac{C_{1}}{\alpha}\|(u,v)\|^{\alpha-p}-\lambda C_{2}\|(u,v)\|^{\beta-p}\bigg]-C_{\epsilon}\Big(\|f\|_{\sigma}^{\frac{p}{p-1}}+\|g\|_{\sigma}^{\frac{p}{p-1}}\Big).\end{eqnarray*} 令 $$\phi(z)=\frac{C_{1}}{\alpha}z^{\alpha-p}+\lambda C_{2}z^{\beta-p}. $$ 则有$\lim\limits_{z\rightarrow 0^{+}}\phi(z)=\lim\limits_{z\rightarrow +\infty}\phi(z)=+\infty$,且 $\phi(z)$ 在 $z_{1}>0$ 处达到其最小值. 通过直接的计算,易得 $$z_{1}=\bigg[\frac{\alpha(p-\beta)C_{2}}{(\alpha-p)C_{1}}\bigg]^{\frac{1}{\alpha-\beta}}\lambda^{\frac{1}{\alpha-\beta}} $$且 $$\phi(z_{1})=\bigg[\frac{C_{1}(\alpha-\beta)}{\alpha(p-\beta)}\bigg]\bigg[\frac{\alpha(p-\beta)C_{2}}{(\alpha-p)C_{1}}\bigg]^{\frac{\alpha-p}{\alpha-\beta}}\lambda^{\frac{\alpha-p}{\alpha-\beta}}>0. $$

显然,存在$\lambda^{*}>0$ 使得当 $0<\lambda<\lambda^{*}$时,$\phi(z_{1})< \frac{1}{2p}$. 此外,易知存在 $L>0$ (与 $\lambda$ 无关) 使得对

$$\|f\|_{\sigma}^{\frac{p}{p-1}}+\|g\|_{\sigma}^{\frac{p}{p-1}}<L\lambda^{\frac{p(\alpha-p)}{\alpha-\beta}}, $$ 有 $$z_{1}^{p}\bigg[\frac{1}{2p}-\phi(z_{1})\bigg]-C_{\epsilon}\Big(\|f\|_{\sigma}^{\frac{p}{p-1}}+\|g\|_{\sigma}^{\frac{p}{p-1}}\Big)>0. $$因此,对$0<\lambda<\lambda^{*}$且$\|f\|_{\sigma}^{\frac{p}{p-1}}+\|g\|_{\sigma}^{\frac{p}{p-1}}<L\lambda^{\frac{p(\alpha-p)}{\alpha-\beta}}$,存在$\rho$,$d_{0}>0$ 使得若$\|(u,v)\|=\rho$,$I_{\lambda}(u,v)\geq d_{0}$.

(ii) 令 $(u,v)\in X\setminus\{(0,0)\}$,有

$$I_{\lambda}(tu,tv)=\frac{t^{p}}{p}\|(u,v)\|^{p}-\frac{t^{\alpha}}{\alpha}\int_{\Omega}F(u,v){\rm d}x-t\int_{\Omega}(fu+gv){\rm d}x-\lambda t^{\beta}\int_{\partial \Omega}G(u,v){\rm d}s. $$由于$\alpha>p$,可知$\lim\limits_{t\rightarrow+\infty} I_{\lambda}(t u,tv)=-\infty$. 于是,对固定的$(u,v)\in X\setminus \{(0,0)\}$,存在$\overline{t}>0$使得$\|(\overline{t}u,\overline{t}v)\|>\rho$且$I_{\lambda}(\overline{t}u,\overline{t}v)<0.$令$(\overline{u},\overline{v})=(\overline{t}u,\overline{t}v)$. 证毕.

定义

$$c=\inf_{\gamma\in\Gamma}\max_{0\leq t\leq 1} I_{\lambda}(\gamma (t)), $$ 其中 $\Gamma=\{\gamma\in C^{1}([0, 1],X)| \ \gamma(0)=0,\ I_{\lambda}(\gamma(1))<0\}.$

接下来,我们将证明$I_{\lambda}$在$X$中满足$(PS)_{c}$条件.

引理 2.2 假设条件 $(A_{1})$-$(A_{2})$ 成立. 则 $I_{\lambda}$ 在$X$中满足 $(PS)_{c}$ 条件.

证 令 $\{(u_{n},v_{n})\}\subset X$为$I_{\lambda}$的任意$(PS)_{c}$ 序列,即

\begin{equation}I_{\lambda}(u_{n},v_{n})\rightarrow c,\ I'_{\lambda}(u_{n},v_{n})\rightarrow 0,\ n\rightarrow\infty. \end{equation}

(2.5)

首先,证明$\{(u_{n},v_{n})\}$在$X$中有界. 事实上,由(2.2)-(2.5) 式可知,当$n$充分大时,有\begin{eqnarray*}c+o(1)+o(1)\|(u_{n},v_{n})\|&\geq& I_{\lambda}(u_{n},v_{n})-\frac{1}{\alpha}\langle I'_{\lambda}(u_{n},v_{n}),(u_{n},v_{n})\rangle\\&=&\Big (\frac{1}{p}-\frac{1}{\alpha}\Big)\|(u_{n},v_{n})\|^{p}+\Big(\frac{1}{\alpha}-1\Big)\int_{\Omega}(fu_{n}+gv_{n}){\rm d}x\\&&+\lambda\Big(\frac{\beta}{\alpha}-1\Big)\int_{\partial\Omega}G(u_{n},v_{n}){\rm d}s\\&\geq& \Big(\frac{1}{p}-\frac{1}{\alpha}\Big)\|(u_{n},v_{n})\|^{p}+\Big(\frac{1}{\alpha}-1\Big)S(\|f\|_{\sigma}+\|g\|_{\sigma})\|(u_{n},v_{n})\|\\&&+\lambda\Big(\frac{\beta}{\alpha}-1\Big)C_{2}\|(u_{n},v_{n})\|^{\beta}.\end{eqnarray*}于是,$\{(u_{n},v_{n})\}$ 在 $X$ 中有界. 故存在子序列(仍记为 $\{(u_{n},v_{n})\}$) 和 $(u,v)\in X$ 满足 $$u_{n}\rightharpoonup u,\ v_{n}\rightharpoonup v \ \mbox{于} \ W^{1,p}(\Omega), $$ $$u_{n}\rightarrow u,\ v_{n}\rightarrow v \ \mbox{于} \ L^{\alpha}(\Omega) \ \mbox{和} \ L^{\beta}(\partial\Omega), $$ $$u_{n}\rightarrow u,\ v_{n}\rightarrow v \ \mbox{a.e.} \ \mbox{于} \ \Omega. $$ 因此,结合$W^{1,p}(\Omega)$中弱收敛的定义,当 $n\rightarrow\infty$时,有

\begin{eqnarray}&&\int_{\Omega}\big{[}|\nabla u|^{p-2}\nabla u\nabla (u_{n}-u)+|u|^{p-2}u(u_{n}-u)\\&&+|\nabla v|^{p-2}\nabla v\nabla (v_{n}-v)+|v|^{p-2}v(v_{n}-v)\big{]}{\rm d}x\rightarrow 0,\end{eqnarray}

(2.6)

且由(2.5)式可得到

\begin{eqnarray}&&\int_{\Omega}\big{[}|\nabla u_{n}|^{p-2}\nabla u_{n}\nabla (u_{n}-u)+|u_{n}|^{p-2}u_{n}(u_{n}-u)\\&&+|\nabla v_{n}|^{p-2}\nabla v_{n}\nabla (v_{n}-v) +|v_{n}|^{p-2}v_{n}(v_{n}-v)\big{]}{\rm d}x\\&=& \langle I'_{\lambda}(u_{n},v_{n}),(u_{n}-u,v_{n}-v)\rangle+\frac{1}{\alpha}\int_{\Omega}(F_{u}(u_{n},v_{n})(u_{n}-u)+F_{v}(u_{n},v_{n})(v_{n}-v)){\rm d}x\\&&+\int_{\Omega}(f(u_{n}-u)+g(v_{n}-v)){\rm d}x+\lambda \int_{\partial\Omega}(G_{u}(u_{n},v_{n})(u_{n}-u)+G_{v}(u_{n},v_{n})(v_{n}-v)){\rm d}s\\&=& \frac{1}{\alpha}\int_{\Omega}(F_{u}(u_{n},v_{n})(u_{n}-u)+F_{v}(u_{n},v_{n})(v_{n}-v)){\rm d}x+\int_{\Omega}(f(u_{n}-u)+g(v_{n}-v)){\rm d}x\\&&+\lambda \int_{\partial\Omega}(G_{u}(u_{n},v_{n})(u_{n}-u)+G_{v}(u_{n},v_{n})(v_{n}-v)){\rm d}s+o(1). \end{eqnarray}

(2.7)

利用注1.2 (iii),存在$M_{2}>0$使得 $$F_{u}(u,v)\leq M_{2} (|u|^{p}+|v|^{p})^{\frac{\alpha-1}{p}},\ F_{v }(u,v)\leq M_{2}(|u|^{p}+|v|^{p})^{\frac{\alpha-1}{p}}, $$ $$G_{u}(u,v)\leq M_{2}(|u|^{p}+|v|^{p})^{\frac{\beta-1}{p}},\ G_{v }(u,v)\leq M_{2}(|u|^{p}+|v|^{p})^{\frac{\beta-1}{p}}, $$其中 \begin{eqnarray*}M_{2}=\max&\Big\{& \max_{|u|^{p}+|v|^{p}=1} F_{u}(u,v),\max_{|u|^{p}+|v|^{p}=1} F_{v}(u,v),\\ && \max_{|u|^{p}+|v|^{p}=1} G_{u}(u,v),\max_{|u|^{p}+|v|^{p}=1} G_{v}(u,v)\Big\}.\end{eqnarray*}

于是,利用Hölder,Sobolev和Sobolev迹不等式,存在常数$C_{3}$,$C_{4}>0$使得

\begin{eqnarray}\bigg|\int_{\Omega}F_{u}(u_{n},v_{n})(u_{n}-u){\rm d}x\bigg|&\leq& \bigg(\int_{\Omega}|F_{u}(u_{n},v_{n})|^{\frac{\alpha}{\alpha-1}}{\rm d}x\bigg)^{\frac{\alpha-1}{\alpha}}\bigg(\int_{\Omega}|u_{n}-u|^{\alpha}{\rm d}x\bigg)^{\frac{1}{\alpha}}\\&\leq& M_{2}\bigg(\int_{\Omega}(|u_{n}|^{p}+|v_{n}|^{p})^{\frac{\alpha}{p}}{\rm d}x\bigg)^{\frac{\alpha-1}{\alpha}}\bigg(\int_{\Omega}|u_{n}-u|^{\alpha}{\rm d}x\bigg)^{\frac{1}{\alpha}}\\&\leq& C_{3}\|(u_{n},v_{n})\|^{\alpha-1}\bigg(\int_{\Omega}|u_{n}-u|^{\alpha}{\rm d}x\bigg)^{\frac{1}{\alpha}}\rightarrow 0,~~ n\rightarrow\infty,\end{eqnarray}

(2.8)

且

\begin{eqnarray}\bigg|\int_{\partial\Omega}G_{u}(u_{n},v_{n})(u_{n}-u){\rm d}s\bigg|&\leq& \bigg(\int_{\partial\Omega}|G_{u}(u_{n},v_{n})|^{\frac{\beta}{\beta-1}}{\rm d}s\bigg)^{\frac{\beta-1}{\beta}}\bigg(\int_{\partial\Omega}|u_{n}-u|^{\beta}{\rm d}s\bigg)^{\frac{1}{\beta}}\\&\leq& M_{2}\bigg(\int_{\partial\Omega}(|u_{n}|^{p}+|v_{n}|^{p} )^{\frac{\beta}{p}}{\rm d}s\bigg)^{\frac{\beta-1}{\beta}}\bigg(\int_{\partial\Omega}|u_{n}-u|^{\beta}{\rm d}s\bigg)^{\frac{1}{\beta}}\\&\leq& C_{4}\|(u_{n},v_{n})\|^{\beta-1}\bigg(\int_{\partial\Omega}|u_{n}-u|^{\beta}{\rm d}s\bigg)^{\frac{1}{\beta}}\rightarrow 0,~~ n\rightarrow\infty.\end{eqnarray}

(2.9)

类似地,易证

$$\int_{\Omega}F_{v}(u_{n},v_{n})(v_{n}-v){\rm d}x\rightarrow 0,\ n\rightarrow\infty, $$

(2.10)

$$\int_{\partial\Omega}G_{v}(u_{n},v_{n})(v_{n}-v){\rm d}s\rightarrow 0,\ n\rightarrow\infty. $$

(2.11)

此外,由条件 $(A_{1})$ 推出

$$\bigg|\int_{\Omega}f(u_{n}-u){\rm d}x\bigg|\leq \|f\|_{\infty}\int_{\Omega}|u_{n}-u|{\rm d}x\rightarrow0,\ n\rightarrow\infty, $$

(2.12)

$$\bigg|\int_{\Omega}g(v_{n}-v){\rm d}x\bigg|\leq \|g\|_{\infty}\int_{\Omega}|v_{n}-v|{\rm d}x\rightarrow0,\ n\rightarrow\infty. $$

(2.13)

于是,由(2.6)-(2.13)式得\begin{eqnarray*}&&\int_{\Omega}\big{[}(|\nabla u_{n}|^{p-2}\nabla u_{n}-|\nabla u|^{p-2}\nabla u)(\nabla u_{n}-\nabla u)+(|u_{n}|^{p-2}u_{n}-|u|^{p-2}u)(u_{n}-u)\\&&+ (|\nabla v_{n}|^{p-2}\nabla v_{n}-|\nabla v|^{p-2}\nabla v)(\nabla v_{n}-\nabla v)+(|v_{n}|^{p-2}v_{n}-|v|^{p-2}v)(v_{n}-v)\big{]} {\rm d}x\rightarrow 0.\end{eqnarray*}利用基本不等式 $$\langle |x|^{p-2}x-|y|^{p-2}y,x-y\rangle\geq C_{p}|x-y|^{p},\ p\geq 2, $$ $$\langle |x|^{p-2}x-|y|^{p-2}y,x-y\rangle\geq \frac{C_{p}|x-y|^{2}}{(|x|+|y|)^{2-p}},\ 1<p<2, $$ 得到\begin{eqnarray*}&& \int_{\Omega}\big{[}(|\nabla u_{n}|^{p-2}\nabla u_{n}-|\nabla u|^{p-2}\nabla u)(\nabla u_{n}-\nabla u)+(|u_{n}|^{p-2}u_{n}-|u|^{p-2}u)(u_{n}-u)\big{]}{\rm d}x\\&\geq& \left\{\begin{array}{lll}C\|u_{n}-u\|^{p},~~&p\geq2,\\C\|u_{n}-u\|^{2},&1<p<2&,\end{array}\right.\end{eqnarray*} \begin{eqnarray*}&& \int_{\Omega}\big{[}(|\nabla v_{n}|^{p-2}\nabla v_{n}-|\nabla v|^{p-2}\nabla v)(\nabla v_{n}-\nabla v)+(|v_{n}|^{p-2}v_{n}-|v|^{p-2}v)(v_{n}-v)\big{]}{\rm d}x\\&\geq & \left\{\begin{array}{lll}C\|v_{n}-v\|^{p},~~&p\geq2,\\C\|v_{n}-v\|^{2},&1<p<2&.\end{array}\right.\end{eqnarray*}于是,当$n\rightarrow\infty$时,易得$\|(u_{n}-u,v_{n}-v)\|\rightarrow 0$. 证毕.

定理1.1的证明 由引理2.1,2.2和山路引理(参见文献[16, 19]),存在$(u^{1},v^{1})\in X$使得$(u^{1},v^{1})$ 是问题(1.1)的解且满足

$$I_{\lambda}(u^{1},v^{1})\geq d_{0}>0. $$

接下来,利用Ekeland变分准则(参见文献[18])寻找问题(1.1)的另一个解$(u^{2},v^{2})$. 首先,选取$(\phi_{1},\phi_{2})\in C_{0}^{\infty}(\Omega)\times C_{0}^{\infty}(\Omega)$满足

$$\int_{\Omega}(f\phi_{1}+g\phi_{2}){\rm d}x>0. $$ 于是,当$t$充分小时,有\begin{eqnarray*}I_{\lambda}(t\phi_{1},t\phi_{2})&=&\frac{t^{p}}{p}\|(\phi_{1},\phi_{2})\|^{p}-\frac{t^{\alpha}}{\alpha}\int_{\Omega}F(\phi_{1},\phi_{2}){\rm d}x-t\int_{\Omega}(f\phi_{1}+g\phi_{2}){\rm d}x\\&&-\lambda t^{\beta}\int_{\partial\Omega}G(\phi_{1},\phi_{2}){\rm d}s<0.\end{eqnarray*} 因此,对任意开球$B_{r}\subset X$,有 $$-\infty < c_{r}=\inf_{\overline{B_{r}}} I_{\lambda}(u,v)<0. $$ 这就意味着,存在 $\delta>0$ 使得 $$c_{\delta}=\inf_{(u,v)\in \overline{B_{\delta}}} I_{\lambda}(u,v)<0 \ \mbox{且} \ \inf_{(u,v)\in \partial B_{\delta}} I_{\lambda}(u,v)>0. $$于是,得到

$$\inf_{(u,v)\in \partial B_{\delta}} I_{\lambda}(u,v)-\inf_{(u,v)\in B_{\delta}} I_{\lambda}(u,v)>\varepsilon_{n}>0, $$

(3.1)

其中$\varepsilon_{n}\rightarrow 0$,$n\rightarrow\infty$.

利用 Ekeland 变分准则,存在 $\{(u_{n},v_{n})\}\subset \overline{B_{\delta}}$ 使得

$$c_{\delta}\leq I_{\lambda}(u_{n},v_{n})<c_{\delta}+\varepsilon_{n}, $$

(3.2)

且

$$I_{\lambda}(u_{n},v_{n})<I_{\lambda}(u,v)+\varepsilon_{n}\|(u_{n}-u,v_{n}-v)\|, \ \forall(u,v)\in \overline{B_{\delta}},\ u\neq u_{n},\ v\neq v_{n}. $$

(3.3)

由 (3.1)-(3.2)式可得 $$I_{\lambda}(u_{n},v_{n})<c_{\delta}+\varepsilon_{n}\leq \inf_{(u,v)\in B_{\delta}} I_{\lambda}(u,v)+\varepsilon_{n}<\inf_{(u,v)\in \partial B_{\delta}} I_{\lambda}(u,v), $$从而推出$(u_{n},v_{n})\in B_{\delta}.$

考虑泛函 $\Phi: \overline{B_{\delta}}\rightarrow \bf{R}$,定义如下

$$\Phi(u,v)=I_{\lambda}(u,v)+\varepsilon_{n}\|(u_{n}-u,v_{n}-v)\|,\ \forall(u,v)\in \overline{B_{\delta}}. $$

(3.4)

由(3.3) 和 (3.4) 式得到 $$\Phi(u_{n},v_{n})<\Phi(u,v),\ \forall(u,v)\in \overline{B_{\delta}},\ u\neq u_{n},\ v\neq v_{n}. $$ 于是,$(u_{n},v_{n})$ 是$\Phi$ 的局部极小元.

并且,对$t>0$充分小和任意$(\varphi_{1},\varphi_{2})\in B_{1}$,有

$$\frac{\Phi(u_{n}+t\varphi_{1},v_{n}+t\varphi_{2})-\Phi(u_{n},v_{n})}{t}\geq 0, $$从而可得 $$\frac{I_{\lambda}(u_{n}+t\varphi_{1},v_{n}+t\varphi_{2})-I_{\lambda}(u_{n},v_{n})}{t}+\varepsilon_{n}\|(\varphi_{1},\varphi_{2})\|\geq 0. $$令$t\rightarrow 0^{+}$,有 $$\langle I'_{\lambda}(u_{n},v_{n}),(\varphi_{1},\varphi_{2})\rangle+\varepsilon_{n}\|(\varphi_{1},\varphi_{2})\|\geq 0,\ \forall (\varphi_{1},\varphi_{2})\in B_{1}. $$ 用$(-\varphi_{1},-\varphi_{2})$替代上式中的$(\varphi_{1},\varphi_{2})$,则有 $$-\langle I'_{\lambda}(u_{n},v_{n}),(\varphi_{1},\varphi_{2})\rangle+\varepsilon_{n}\|(\varphi_{1},\varphi_{2})\|\geq 0,\ \forall (\varphi_{1},\varphi_{2})\in B_{1}. $$ 故易得 $$\|I'_{\lambda}(u_{n},v_{n})\|\leq \varepsilon_{n}\rightarrow 0,~~n\rightarrow\infty. $$ 于是,存在序列$\{(u_{n},v_{n})\}\subset B_{\delta}$ 使得 $I_{\lambda}(u_{n},v_{n})\rightarrow c_{\delta}<0$,且 $I'_{\lambda}(u_{n},v_{n})\rightarrow 0$,$n\rightarrow\infty.$ 利用引理 2.2,$\{(u_{n},v_{n})\}$ 在 $X$中存在一强收敛的子序列 (仍记为 $\{(u_{n},v_{n})\}$) 满足$(u_{n},v_{n})\rightarrow (u^{2},v^{2})$. 因此,$(u^{2},v^{2})$ 是问题(1.1) 的解且满足$I_{\lambda}(u^{2},v^{2})<0$. 证毕.