USA-Chile Workshop on Nonlinear Analysis, Electron. J. Diff. Eqns., Conf. 06, 2001, pp. 343–357. http://ejde.math.swt.edu or http://ejde.math.unt.edu ftp ejde.math.swt.edu or ejde.math.unt.edu (login: ftp) Nontrivial solutions of semilinear elliptic systems in RN ∗ Jianfu Yang Abstract We establish an existence result for strongly indefinite semilinear elliptic systems in RN . 1 Introduction The main objective of this paper is to establish existence results for the semilinear elliptic system −∆u + u = g(x, v), −∆v + v = f (x, u) inRN , (1.1) u(x) → 0 and v(x) → 0 as |x| → ∞. (1.2) The existence of solutions of (1.1)-(1.2) is usually investigated by finding critical points of a related functional. Typical features of the problem are that firstly, the related functional is strongly indefinite; secondly, the growths of f in u and g in v at infinity may not be ‘symmetric’; and lastly, Sobolev embeddings in general are not compact, then the Palais - Smale condition may not be satisfied. Existence results were recently obtained in [12] and [15] in bounded domains. The arguments lie in the decomposition of Sobolev spaces by eigenfunctions of Laplacian operator and a use of linking theorems. Using spectral family theory of non-compact operator, the author and Figueiredo [13] find a suitable linking structure for the functional associate to (1.1)-(1.2) and prove that problem (1.1)(1.2) possesses at least a positive solution if f and g depend on the variable x radially. Furthermore, it is also shown in [13] that all positive solutions of problem (1.1)-(1.2) are exponentially decaying. In this paper, we establish existence results for general cases. Assume that H1) f, g : RN × R → R are measurable in the first variable, continuous in the Rt Rt second variable. Both F (x, t) = 0 f (x, s)ds and G(x, t) = 0 g(x, s)ds are increasing and strictly convex in t. H2) limt→0 f (x, t)/t = 0, limt→0 g(x, t)/t = 0 ∗ Mathematics Subject Classifications: 35J50, 35J55. Key words: indefinite, semilinear, elliptic system. c 2001 Southwest Texas State University. Published January 8, 2001. 343 uniformly in x ∈ RN . 344 Nontrivial solutions of semilinear elliptic systems H3) There is a constant c > 0 such that |f (x, t)| ≤ c(|t|p + 1) and |g(x, t)| ≤ c(|t|q + 1), where 0 < p, q < (N + 2)/(N − 2), N ≥ 3. H4) There are constants α, β > 2 such that 0 < αF (x, t) ≤ tf (x, t) and 0 < βG(x, t) ≤ tg(x, t), for t 6= 0. H5) f (x, t) → f¯(t) and g(x, t) → ḡ(t) uniformly for t bounded as |x| → ∞. |f (x, t) − f¯(t)| ≤ (R)|t| and |g(x, t) − ḡ(t)| ≤ (R)|t| whenever |x| ≥ R, |t| ≤ δ, where (R) → ∞ as R → ∞. H6) F (x, t) ≥ F̄ (t) and G(x, t) ≥ Ḡ(t), meas{x ∈ RN : f (x, t) 6≡ f¯(t)} > 0 or meas{x ∈ RN : g(x, t) 6≡ ḡ(t)} > 0. H7) Both f¯(t)/t and ḡ(t)/t are increasing in t. Our main result is as follows. Theorem 1.1 Assume (H1)-(H7). Then problem (1.1)-(1.2) possesses at least one nontrivial exponentially decaying solution. The restriction of exponents in (H3) is due to the fact that we only know the decaying law in the case. We analyze the convergence of Palais-Smale sequence of associate functional to (1.1)-(1.2) in Section 3. It is shown that the energy levels of solutions of the related autonomous system −∆u + u = ḡ(v), u(x) → 0, −∆v + v = f¯(u) in RN , v(x) → 0 as |x| → 0 . (1.3) (1.4) are obstacle levels preventing strong convergence of Palais-Smale sequences of (1.1)-(1.2). The possible critical values to be found are between obstacle levels. To retain the compactness, we have to get control of critical values. It is harder to handle critical values described by linking structure than that by the Mountain Pass Theorem. We use dual variational method as [3], [4] and [11]. The method is of the advantage avoiding the indefinite character of original functional. We start with problem (1.1)-(1.2) in bounded domains. Although existence result in the case is known, it has no control of critical values. We establish in Section 2 an existence result by the Mountain Pass Theorem and bound uniformly corresponding critical values by the energy level of ground state of problem (1.3)-(1.4). Then we construct a Palais - Smale sequence for the functional associated to problem (1.1)-(1.2). Theorem 1.1 is proved in Section 4. 2 Existence results in bounded domains Let Ω be a bounded domain. We consider the problem −∆u + u = g(x, v), −∆v + v = f (x, u) in Ω, u = 0, v = 0 on ∂Ω . (2.1) (2.2) Jianfu Yang 345 The solutions of (2.1)-(2.2) will be found by looking for critical points of associate functional. The main result in this section is as follows. Theorem 2.1 Assume (H1) − (H4). Then problem (2.1)-(2.2) possesses at least a nontrivial solution. To prove Theorem 2.1 we will need the lemmas below. First we define the dual functional associate to (2.1)-(2.2). It is well known that the inclusions ir : Wo1,r (Ω) → Lp+1 (Ω), is : Wo1,s (Ω) → Lq+1 (Ω) sN are compact if 2 < p + 1 < NrN −r , N > r and 2 < q + 1 < N −s , N > s. The 0 r operator −∆ + id : Wo1,r (Ω) → Wo−1,r (Ω) is an isomorphism, where r0 = r−1 . Hence T = i2 (−∆ + id)−1 i∗2 : L1+1/q (Ω) → Lp+1 (Ω). is continuous. Denote by X = Lp+1 (Ω)× Lq+1 (Ω), X ∗ = L1+1/p (Ω)× L1+1/q (Ω) and let 0 T 0 T −1 , K = A−1 = . A= T 0 T −1 0 For each x, the Legendre-Fenchel transformations F ∗ (x, ·) of F (x, ·), and G (x, ·) of G(x, ·) are defined by ∗ F ∗ (x, s) = sup{st − F (x, t)}, t∈R G∗ (x, s) = sup{st − G(x, t)} t∈R (2.3) respectively. Equivalently, we have and 0 F ∗ (x, s) = st − F (x, t) with s = f (x, t), t = Fs∗ (x, s) G∗ (x, s) = st − G(x, t) with s = g(x, t), t = G∗s (x, s). 0 (2.4) (2.5) In the same way, we define F̄ ∗ , Ḡ∗ for F̄ , Ḡ. By (H6) and properties of LegendreFenchel transformations, we have F ∗ (x, s) ≤ F̄ ∗ (s), G∗ (x, s) ≤ Ḡ∗ (s). (2.6) We may verify following properties of F ∗ , G∗ in Lemmas 2.2 and 2.3 as [3], [10] and [16]. Lemma 2.2 F ∗ , G∗ ∈ C 1 and F ∗ (x, s) ≥ (1 − 0 1 )sF ∗ (x, s), α G∗ (x, s) ≥ (1 − F ∗ (s, x) ≥ C|s|1+1/p − C, 0 1 )sG∗ (x, s), β G∗ (x, s) ≥ C|s|1+1/q − C. Lemma 2.3 There exist δ > 0, Cδ and Cδ0 > 0 such that if |s| ≤ δ Cδ |s|2 , Cδ |s|2 , ∗ 1 1 , G F ∗ (x, s) ≥ (x, s) ≥ 1+ p 0 Cδ |s| Cδ0 |s|1+ q , , if |s| ≥ δ if if (2.7) (2.8) |s| ≤ δ . |s| ≥ δ Nontrivial solutions of semilinear elliptic systems 346 We may verify that the dual functional Z Z 1 ∗ ∗ hw, Kwi dx, Ψ(w) = ΨΩ (w) = (F (x, w1 ) + G (x, w2 )) dx − 2 Ω Ω is well defined and C 1 on X ∗ . A critical point w of Ψ satisfies 0 0 (−∆ + id)−1 w2 = Fs∗ (x, w1 ) and (−∆ + id)−1 w1 = G∗s (x, w2 ). Let u = (−∆ + id)−1 w2 , v = (−∆ + id)−1 w1 . Then (u, v) satisfies (2.1)-(2.2). Furthermore, denoting by Z Z Z F (x, u) dx − G(x, v) dx Φ(z) = (∇u∇v + uv) dx − Ω Ω Ω the functional of (2.1) -(2.2) defined on Ho1 (Ω) × Ho1 (Ω), we deduce by (2.4) and (2.5) that Φ(z) = Ψ(w). Such a result is also valid for solutions of (1.1)-(1.2). Now we use the Mountain Pass Theorem to find critical points of Ψ. Following arguments of [6], we know that (H2) implies F ∗ (x, t)/t2 → ∞ and ∗ G (x, t)/t2 → ∞. Thus 0 is a local minmum of Ψ. Precisely, Lemma 2.4 There exist constants α, ρ > 0, independent of Ω, such that Ψ(w) ≥ α > 0 if kwkX ∗ = ρ. Lemma 2.5 There exist T > 0 and w ∈ E such that Ψ(tw) ≤ 0 whenever t ≥ T. Proof. Taking w ∈ X ∗ , w 6≡ 0 such that Z hw, Kwi dx > 0, Ω whence by (H4), for t > 0 Z Z Z β β α α 1 2 β−1 β−1 α−1 α−1 |w1 | dx + t |w2 | dx − t hw, Kwi dx. Ψ(tw) ≤ t 2 Ω Ω Ω β α , β−1 < 2, the assertion follows for t > 0 large. Since α−1 Let Γ = {g ∈ C([0, 1], X ∗ ) : g(0) = 0, g(1) = e}, where e = T w. We define c = cΩ = inf g∈Γ sup Ψ(g(t)). (2.9) t∈[0,1] The Mountain Pass Theorem will imlpy that c is a critical value of Ψ if the Palais-Smale ((PS) for short) condition holds. It is known from Lemma 2.4 that Jianfu Yang 347 corresponding critical points are nontrivial. Then the proof of Theorem 2.1 is completed. Now we verify the (PS) condition. By a (PS) condition for Ψ we mean that any sequence {wn } ⊂ X ∗ such that |Ψ(wn )| is uniformly bounded in n and Ψ0 (wn ) → 0 as n → ∞ possesses a convergent subsequence. Lemma 2.6 The (PS) condition holds for Ψ. Proof. Let {wn } be a (PS) sequence of Ψ, that is |Ψ(wn )| ≤ C Ψ0 (wn ) → 0 as n → ∞ for some constant C > 0. This inequality and lemma 2.2 yield Z [F ∗ (x, wn1 ) + G∗ (x, wn2 )] dx Ω Z 1 hwn , Kwn i dx + C ≤ 2 Ω Z 0 0 1 (Fs∗ (x, wn1 )wn1 + G∗s (x, wn2 )wn2 ) dx + o(1)kwn kX ∗ + C ≤ 2 Ω Z Z 1 β 1 α ∗ 1 F (x, wn ) dx + G∗ (x, wn2 ) dx + o(1)kwn kX ∗ . ≤ 2α−1 Ω 2β−1 Ω That is Z Ω [F ∗ (x, wn1 ) + G∗ (x, wn2 )] dx ≤ C + o(1)kwn kX ∗ . By Lemma 2.3, we obtain 1+1/p 1+1/q kwn1 kL1+1/p + kwn2 kL1+1/q ≤ C + o(1)kwn kX ∗ . It implies that kwn kX ∗ is bounded. We may assume wn → w weakly in X ∗ as n → ∞. Since the operator (−∆ + id)−1 is compact, it follows un := (−∆ + id)−1 wn2 → (−∆ + id)−1 w2 −1 vn := (−∆ + id) wn1 −1 → (−∆ + id) As a result, wn = (f (x, un ), g(x, vn )) → w pletes the present proof. 3 w 1 in X ∗ as n → ∞, ∗ as n → ∞. in X in X ∗ as n → ∞ which com- Palais-Smale sequence In this section, we prove a global compact result for problem (1.1)-(1.2). Let E = H 1 (RN ) × H 1 (RN ). By our assumptions, the functional Z Z (∇u∇v + uv) dx − (F (x, u) + G(x, v)) dx Φ(z) = RN RN is C 1 on E. The functional Φ∞ is defined with F̄ and Ḡ replacing F and G in Φ respectively. Nontrivial solutions of semilinear elliptic systems 348 Proposition 3.1 Asumme (H1)-(H6). Let {zn } be a (P S)c sequence of Φ, i.e. Φ(zn ) → c and Φ0 (zn ) → 0 as n → 0. (3.1) Then there exists a subsequence (still denoted by {zn }) for which the following holds: there exist an integer k ≥ 0, sequences {xin } ⊂ RN , |xin | → ∞ as n → ∞ for 1 ≤ i ≤ k, a solution z of (1.1)-(1.2) and solutions z i (1 ≤ i ≤ k) of (1.3)-(1.4) such that zn → z weakly in E, Pk Φ(zn ) → Φ(z) + i=1 Φ∞ (z i ), Pk zn − (z + i=1 z i (x − xin )) → 0 in (3.2) (3.3) E (3.4) as n → ∞, where we agree that in the case k = 0 the above holds without z i , xin . Proof. The result can be derived from the arguments for one equation [5]. First we remark that the boundedness of {zn } in E can be deduced as [13] by (3.1). Therefore we may assume zn → z weakly zn → z strongly in E, q+1 N N in Lp+1 loc (R ) × Lloc (R ), a.e. in RN R as n → ∞. Denote Q(z) = RN (∇u∇v + uv) dx, we have zn → z Q(zn ) = Q(zn − z) + Q(z) + o(1). It follows from Brezis & Lieb’s lemma [8] that Z Z Z F (x, un ) dx = F (x, un − u) dx + RN and RN RN G(x, vn ) dx = RN F (x, u) dx + o(1) (3.6) G(x, v) dx + o(1). (3.7) RN Z Z (3.5) Z G(x, vn − v) dx + RN Hence we obtain Φ(zn ) = Φ(zn − z) + Φ(z) + o(1), Φ0 (zn ) = Φ0 (zn − z) + Φ0 (z) + o(1) (3.8) as n → ∞. Let zn1 = zn − z. We may deduce from (H5) as [17] and [19] that Z Z u1n [f (x, u1n ) − f¯(u1n )] dx → 0 and vn1 [g(x, vn1 ) − ḡ(vn1 )] dx → 0 RN as well as Z [F (x, u1n ) − F̄ (u1n )] dx → 0 and RN RN Z RN [G(x, vn1 ) − Ḡ(vn1 )] dx → 0 Jianfu Yang 349 as n → ∞. Therefore Φ∞ (zn1 ) = Φ(zn1 ) + o(1) = Φ(zn ) − Φ(z) + o(1) Φ ∞0 (zn1 ) =Φ 0 (zn1 ) 0 0 + o(1) = Φ (zn ) − Φ (z) + o(1). (3.9) (3.10) Suppose zn1 = zn − z 6→ 0 strongly in E(otherwise we shall have finished). We want to show that there exists x1n ⊂ RN such that |x1n | → +∞ and zn1 (x+x1n ) → z 1 6≡ 0 weakly in E. We note that Φ∞ (zn1 ) ≥ α > 0 because kzn1 kE 6→ 0. In fact, were it not true, we would have 0 Φ∞ (zn1 ) → 0, < Φ∞ (zn1 ), η >= o(1)kηkE as n → ∞. (3.11) β α u1n , α+β vn1 ) =: ηn in (3.11), it follows Taking η = ( α+β Z Z β α o(1)kηn kE = u1n f¯(u1n ) dx + v 1 ḡ(v 1 ) dx α + β RN α + β RN n n Z Z (3.12) F̄ (u1n ) dx − Ḡ(vn1 ) dx. − RN RN Using hypothesis (H4) we obtain Z (F̄ (u1n ) + Ḡ(vn1 )) dx = o(1)kηkE . RN This and (3.12) yield Z u1n f¯(u1n ) dx = o(1)kηn kE , RN Z RN vn1 ḡ(vn1 ) dx = o(1)kηn kE . (3.13) It follows from assumptions (H2)-(H4) that ¯ 2 ≤ Ctf¯(t) if |f(t)| |t| ≤ 1, 0 |f¯(t)|(p+1) ≤ Ctf¯(t) if |t| > 1. (3.14) Taking η = (φ, 0) in (3.11) and using (3.14) and Hölder’s inequality, we obtain Z (∇φ∇vn1 + φvn1 ) dx| | RN Z Z + φf¯(u1n ) dx| (3.15) ≤ | {|u1n |≤1} Z ≤ C( RN {|u1n |>1} Z 0 0 dx) kφkL2 + C( |f¯(u1n )|(p+1) dx)1/(p+1) kφkLp+1 N R Z 0 1 1 ¯ 1 un f (un ) dx) 2 + C( u1n f¯(u1n ) dx)1/(p+1) ]. |f (u1n )|2 Z ≤ CkφkH s [( RN 1 2 RN which with (3.13) imply that kvn1 kH 1 = o(1). (3.16) Nontrivial solutions of semilinear elliptic systems 350 Similarly, we show that ku1n kH 1 = o(1). (3.17) kzn1 kE → 0 , we get a contradiction. (3.16) and (3.17) yield We decompose RN into N-dimensional unit hypercubes Qj with vertices having integer coordinates and put dn = maxj (ku1n kLp+1 (Qj ) + kvn1 kLq+1 (Qj ) ). We claim that there is a β > 0 such that dn ≥ β > 0 ∀n ∈ N. (3.18) Suppose, by contradiction, that dn → 0 as n → ∞. Since 0 Φ∞ (zn1 ) → 0 (3.19) is bounded, we have by (H2) and (H3) that Z Z ∞ 1 1 ¯ 1 ≤ Φ (zn ) ≤ un f (un ) dx + vn1 ḡ(vn1 ) dx + o(1) noting that 0 as n → ∞, kzn1 kE RN RN ≤ C (ku1n kp+1 + kvn1 kq+1 ) + (ku1n k2L2 (RN ) + kvn1 k2L2 (RN ) ) Lp+1 (RN ) Lq+1 (RN ) X 1 q+1 1 2 1 2 ≤ C (ku1n kp+1 Lp+1 (Qj ) + kvn kLq+1 (Qj ) ) + (kun kL2 (RN ) + kvn kL2 (RN ) ) j ≤ C dp−1 n X j ≤ C dp−1 n X j ku1n k2Lp+1 (Qj ) + C dq−1 n ku1n k2H 1 (Qj ) + C dq−1 n X j X j kvn1 k2Lq+1 (Qj ) + C kvn1 k2H 1 (Qj ) + C 1 2 q−1 1 2 ≤ C dp−1 n kun kH 1 + C dn kvn kH 1 + C. Let n → ∞ and then → 0, we obtain Φ∞ (zn1 ) → 0 as n → ∞. This and (3.19) imply as above that kzn1 kE → 0 as n → ∞, a contradiction, hence we have (3.18). Let {x1n } be the center of a hypercube Qj in which dn = ku1n kLp+1 (Qj ) + kvn1 kLq+1 (Qj ) . Now we show that |x1n | → ∞ as n → ∞. (3.20) {x1n } were bounded, by passing to a subsequence if necessary we should If find that x1n would be in the same Qj and so they should coincide. Therefore in that Qj , for every n > no , no fixed and large enough, we should have Z Z (∇ū1n ∇v̄n1 + ū1n v̄n1 ) dx − (F̄ (ū1n ) + Ḡ(v̄n1 )) dx + o(1) Φ∞ |E(Qj ) (z̄n1 ) = Qj Z Qj F̄ (ū1n ) dx + (β − 1) Z Ḡ(v̄n1 ) dx + o(1) ≥ (α − 1) ≥ 1 β C(kū1n kα Lα (Qj ) + kv̄n kLβ (Qj ) ) + o(1) ≥ 1 β C(kū1n kα Lp+1 (Qj ) + kv̄n kLq+1 (Qj ) ) + o(1), RN RN Jianfu Yang and 351 0 Φ∞ (z̄n1 ) → 0 as n → 0, where z̄n1 (x) = zn1 (x) 0 z ∈ Qj x ∈ RN \Qj . Hence z̄n1 should converge strongly in E(Qj ) to a nonzero function, contradicting to zn1 → 0 weakly in E, so we have (3.20). Let zn1 (· + x1n ) → z 1 weakly in E. Denote by Q̄ the unit hypercube centered at the origin, we have kzn1 kE(Q̄) ≥ β > 0, thus z 1 6≡ 0 and 0 hΦ∞ (z 1 ), ηi = 0, ∀η ∈ E. (3.21) Iterating the procedure, we obtain sequences xln , |xln | → ∞ and znl (x) = znl−1 (x + xm ) − z l−1 (x), j ≥ 2 znl (x + xln ) → z l (x) weakly in E as n → 0, where each z l satisfying (3.21) and by induction kznl kE = kznl−1 k2E − kz l−1 k2E = kzn k2E − kzk2E − l−1 X kz i k2E + o(1). i=1 Φ∞ (znl ) = Φ∞ (znl−1 ) − Φ∞ (z l−1 ) + o(1) = Φ(zn ) − Φ(z) − l−1 X Φ(z i ) + o(1). i=1 Since z l is a solution of (1.3)-(1.4) and z l 6≡ 0, we may prove as Lemma 4.1 below that kz l kE ≥ C > 0. Thus the iteration will terminate at some index k ≥ 0. The assertion follows. 4 Uniform bounds and proof of Theorem 1.1 We shall bound critical values defined in (2.9) by the energy of the ground state of problem (1.3)-(1.4). By a ground state of problem (1.3)-(1.4) we mean a minimizer of the variational problem Φ∞ = inf{Φ∞ (u, v) : (u, v) ∈ E is a solution of (1.3)-(1.4), (u, v) 6≡ (0, 0)}. (4.1) It is shown in [13] that problem (1.3)-(1.4) has a positive radially decaying solution, so the variational problem (4.1) is well defined. Lemma 4.1 Variational problem (4.1) is assumed by a nontrivial solution of (1.3)-(1.4). Nontrivial solutions of semilinear elliptic systems 352 Proof. Let zn = (un , vn ) be a minimizing sequence of Φ∞ . It is obvious that {zn } is a (PS) sequence of Φ∞ . We deduce by Proposition 3.1 that Φ∞ = Φ(zn ) + o(1) = k X Φ∞ (zj ) + o(1), j=1 where zj is a solution of (1.3)-(1.4). By the definition of Φ∞ , k = 1. The proof will be completed if we show z1 6= 0. To this end, we bound solutions of (1.3)-(1.4) in H 1 norm below by a positive constant. Suppose z = (u, v) is a solution of (1.3)-(1.4), we have Z Z 2 2 uḡ(v) dx, kvkH 1 = v f¯(u) dx, (4.2) kukH 1 = RN and RN Z Z (∇u∇v + uv) dx = RN Z ¯ dx. uf(u) vḡ(v) dx = RN (4.3) RN By assumptions (H2), (H3) and (H5), we obtain N +2 f¯(u) ≤ C |u| N −2 + u, N +2 ḡ(v) ≤ C |v| N −2 + v. (4.4) We deduce by (4.2)-(4.4) and Hölder’s inequality that ∗ 2 −1 ∗ kuk2H 1 ≤ C kvkL 2∗ kukL2 + kukL2 kvkL2 , where 2∗ = 2N N −2 . Using Young’s inequality and Sobolev embedding, we obtain ∗ ∗ ∗ ∗ kuk2H 1 ≤ C (kuk2H 1 + kvk2H 1 ) + kvk2H 1 . Similarly, kvk2H 1 ≤ C (kuk2H 1 + kvk2H 1 ) + kuk2H 1 . So for small, we have ∗ ∗ kuk2H 1 + kvk2H 1 ≤ C(kuk2H 1 + kvk2H 1 ). It yields that kukH 1 or kvkH 1 ≥ C > 0, uniformly for solutions of (1.3)-(1.4), and where C > 0 is independent of z = (u, v). Consequently, z1 = (u1 , v1 ) 6≡ 0. Let Rn → ∞, Bn = BRn (0). Taking Ω = Bn in problem (2.1)-(2.2), we infer from Theorem 2.1 that there exists a solution zn of problem (2.1)- (2.2) defined on Bn for each n. Moreover, Φ(zn ) = Ψ(wn ) = cn ≥ α > 0, (4.5) where zn = Kwn , Φ = ΦRN and Ψ = ΨRN . We have extended zn to RN by letting zn = 0 outside Bn . Proposition 4.1 cn < Φ∞ for n large. Jianfu Yang 353 Proof. Since each element w in Xn∗ = L1+1/p (Bn ) × L1+1/q (Bn ) can be extended to an element of X ∗ by letting w = 0 outside Bn , we shall denote ΨBn as Ψ in brief. By Lemma 4.1, Φ∞ is assumed. Let zo = (uo , vo ) be a minimizer of Φ∞ . Choosing w1o = f¯(uo ), w2o = ḡ(vo ) R and using (H4)-(H5) and equations (1.3)-(1.4), one has RN < wo , Kwo > dx > 0, where wo = (w1o , w2o ). Moreover, we know as Lemma 2.5 that there are t1 , t2 > 0 such that maxt≥0 Ψ(two ) = maxt1 ≤t≤t2 Ψ(two ). Suppose that to ∈ (t1 , t2 ) and Ψ(to wo ) = maxt1 ≤t≤t2 Ψ(two ). Because F (x, t) ≥ F̄ (t) and G(x, t) ≥ Ḡ(t), one has F ∗ (x, s) ≤ F̄ ∗ (s) and G∗ (x, s) ≤ Ḡ(s). By the assumption (H6), Ψ(to wo ) < Ψ∞ (to wo ), it follows sup Ψ(two ) < sup Ψ∞ (two ). t≥0 t≥0 (4.6) The density of real number field implies that there exists > 0 such that sup Ψ(two ) + 2 < sup Ψ∞ (two ). t≥0 t≥0 (4.7) Let φ ∈ Co∞ (RN ), 0 ≤ φ ≤ 1 and φ ≡ 1 if |x| ≤ 12 ; φ ≡ 0 if |x| > 1; φn (x) = φ( Rxn ). Then zn := (φn uo , φn vo ) converges to (uo , vo ) in E. Let w1n = f¯(φn uo ), w2n = ḡ(φn vo ). We also have wn → wo in X ∗ . Suppose Ψ(tn wn ) = sup Ψ(twn ), t≥0 then {tn } is bounded. Indeed, if tn → ∞, arguments in Lemma 2.5 would yield supt≥0 Ψ(twn ) → −∞. It is not possible because the value is not negative. Suppose tn → t̄o , the continuity of the functional Ψ gives Ψ(tn wn ) → Ψ(t̄o wo ). We claim that Ψ(t̄o wo ) = supt≥0 Ψ(two ). In fact, for every > 0 there exists δ > 0 such that Ψ(to wo ) − ≤ Ψ(two ) whenever |t − to | < δ. By the continuity of Ψ, we may find no > 0 such that if n ≥ no Ψ(two ) ≤ Ψ(twn ) + , Ψ(tn wn ) ≤ Ψ(t̄o wo ) + . Nontrivial solutions of semilinear elliptic systems 354 Therefore if n ≥ no we have Ψ(to wo ) − ≤ Ψ(tn wn ) + ≤ Ψ(t̄o wo ) + 2 ≤ Ψ(to wo ) + 2. Since is arbitrary, the conclusion holds. By the same arguments, we find that there exists sn such that sn → s̄o and Ψ∞ (sn wn ) = sup Ψ∞ (twn ) → Ψ∞ (s̄o wo ) = sup Ψ∞ (two ) t≥0 t≥0 as n → ∞. By (4.7), we obtain Ψ(tn wn ) + < Ψ∞ (sn wn ) for n large enough. We may assume sn > 0, and then dΨ∞ (twn ) |t=sn = 0, dt that is Z Z 0 0 [F̄s∗ (sn w1n )w1n + Ḡ∗s (sn w2n )w2n ] dx − sn RN RN < wn , Kwn > dx = 0. (4.8) By the definition of Legendre - Fenchel transformation, we obtain Z [F̄ ∗ (sn w1n ) + Ḡ∗ (sn w2n )] dx RN Z 0 0 [F̄s∗ (sn w1n )sn w1n + Ḡ∗s (sn w2n )sn w2n ] dx = RN Z [F̄ (f¯−1 (sn w1n )) + Ḡ(ḡ −1 (sn w2n ))] dx (4.9) − RN Z Z < wn , Kwn > dx − [F̄ (f¯−1 (sn w1n )) + Ḡ(ḡ −1 (sn w2n ))] dx. = s2n RN RN Consider (−∆ + id)−1 w2n = uo + σn , (−∆ + id)−1 w1n = vo + µn in RN , we obtain (−∆ + id)σn = ḡ(φn vo ) − ḡ(vo ), (−∆ + id)µn = f¯(φn uo ) − f¯(uo ). In terms of Lp −estimates, σn → 0 and µn → 0 in H 2,2 as n → ∞. Furthermore, we infer from (4.8) that Z f¯−1 (sn w1n ) f¯−1 (w1n ) sn (w1n )2 [ − ] dx sn w1n w1n RN Z ḡ −1 (sn w2n ) ḡ −1 (w2n ) sn (w2n )2 [ − ] dx + sn w2n w2n RN Z = [w1n σn + w2n µn + (1 − φn )(w1n + w2n )] dx = o(1) RN Jianfu Yang 355 as n → ∞. The equality and assumption (H7) imply sn → 1 as n → ∞. hence we deduce by (4.8) and (4.9) that sup Ψ∞ (twn ) t≥0 Z Z 1 (uo f¯(uo ) + vo ḡ(vo )) dx − (F̄ (uo ) + Ḡ(vo )) dx + n ≤ 2 RN RN = Ψ ∞ + n , where n = Z 1 2 ¯ o ) + vo ḡ(vo )) dx (sn − 1) (uo f(u 2 RN Z [(F̄ (φn uo ) − F̄ (uo )) + (Ḡ(φn vo ) − Ḡ(vo ))] dx − N ZR [(F̄ (φn uo ) − F̄ (f¯−1 (sn w1n )) + (Ḡ(φn vo ) − Ḡ(ḡ −1 (sn w2n ))] dx. + RN The above estimates imply n = o(1) as n → ∞. Therefore sup Ψ(twn ) < sup Ψ(twn )∞ − ≤ Ψ∞ − + o(1), t≥0 t≥0 the assertion follows for n large. Lemma 4.2 zn is a (PS) sequence of Φ in E. Proof. It is readily to verify that cn = Φ(zn ) ≤ cn−1 = Φ(zn−1 ), thus by Proposition 4.2 α ≤ c n ≤ c1 < Φ∞ , (4.10) we obtain cn = Φ(zn ) → c, α ≤ c ≤ c1 < Φ∞ . (4.11) as n → ∞. (4.12) Now we show that Φ0 (zn ) → 0, In fact, ∀(φ, ψ) ∈ Co∞ (RN )×Co∞ (RN ), there is no > 0 such that suppφ, suppψ ⊂ Bn whenever n ≥ no and Φ0 (zn )(φ, ψ) = 0, if n ≥ no . This implies that Φ0 (zn )z → 0 as n → ∞ for all z ∈ Co∞ (RN ) × Co∞ (RN ). Hence (4.12) follows because Co∞ (RN ) × Co∞ (RN ) is dense in H 1 (RN ) × H 1 (RN ). 356 Nontrivial solutions of semilinear elliptic systems Completion of the proof of Theorem 1.1 We may prove that the (PS) sequence zn of Φ is bounded in E as [13], and assume zn → zo weakly in E. Obviously, zo solves (1.1)-(1.2). We claim that zo is nontrivial. In fact, Lemma 2.4, Proposition 3.1 and Proposition 4.2 give that X α ≤ Φ(zn ) = Φ(zo ) + Φ∞ (zj ) + o(1) < Φ∞ . j If j = 0, Φ(zo ) ≥ α > 0, zo is a nontrivial solution; if j ≥ 1, then Φ(zo ) < 0, also implying zo 6≡ 0. The decaying law of zo at infinity was proved in [13]. ACKNOWLEDGEMENTS The work was partially supported by 21CSPJ, NSFJ and NSFC in China. References [1] R.A. Adams, Sobolev Spaces Academic Press 1975. [2] A. Ambrosetti and P. H. Rabinowitz, Dual variational methods in critical point theory and applications, J. Funct. Anal., 14 (1973), 349 - 381. [3] A. Ambrosetti and P.N.Srikanth, Superlinear elliptic problems and the dual principle in critical point theory, J. Math. & Phys., 18 (1984), 441-451. ∗ [4] A. Ambrosetti and M. Struwe, A note on the problem −∆u = λu+u|u|2 Manus. Math., 54 (1986), 373-379. −1 , [5] V. Benci and G. Cerami, Positive solutions of some nonlinear elliptic problems in exterior domains, Arch. Rational Mech.Anal., 99 (1987), 283-300. [6] V. Benci and D. Fortunato, The dual method in critical point theory: multiplicity results for indefinite functional, Ann. Mat. Pura Appl., 134 (1982), 215-242. [7] Berestycki and P.L. Lions, Nonlinear scalar field equations, I and II, Arch. Rational Mech. Anal., 82 (1983), 313-376. [8] H. Brezis and E. Lieb, A relation between pointwise convergence of functions and convergence of functionals, Proc. Amer. Math.Soc., 88 (1983), 486-490. [9] D.G. Costa, On a class of elliptic systems in RN , Eleton. J. Diff. Eqn. 1994 (1994), No. 7, 1-14. (http://ejde.math.swt.edu) [10] K.C. Chang, Critical Point Theory and Its Applications, Shanghai Sci. & Tech. Press, 1986. Jianfu Yang 357 [11] Ph. Clément and R.C.A.M.van der Vorst, On a semilinear elliptic system, Diff.& Int.Equas., 8 (1995),1317-1329. [12] D.G. de Figueiredo and P.L. Felmer, On supequadratic elliptic systems, Trans. Amer. Math. Soc., 343 ( 1994), 99-116. [13] D.G. de Figueiredo and Yang Jianfu, Decay, symmetry and existence of solutions of semilinear elliptic systems Nonlinear Anal. TMA, 33 (1998), 211-234. [14] D. Gilbarg and N.S. Trudinger Elliptic Partial Differential Equations of Second Order, Springer-Verlag, 1983. [15] J. Hulshof and R. van der Vorst, Differential systems with strongly indefinite variational structure, J.Funct.Anal., 114 (1993), 32-58. [16] J. Mawhin and M. Willem, Critical Point Theory and Hamiltonian Systems, Springer-Verlag, 1993. [17] P.L. Lions, The concentration-compactness principle in the calculus of variations, Ann.I.H.Anal.Nonli., 1 (1984), 109-283. [18] P. Rabinowitz, Minimax methods in critical point theory with applications to differential equations, CBMS Regional Conf.Ser. in Math.,No.65, Amer. Math.Soc., Providence, R.I., 1986. [19] Jianfu Yang and Xiping Zhu, On the existence of nontrivial solution of a quasilinear elliptic boundary value problem for unbounded domains, Acta Math.Sci., 7 (1987), 341-359. Jianfu Yang Department of Mathematics, IMECC-Unicamp Caixa Postal 6065 Campinas 13081-970, SP, Brazil e-mail: jfyang@ime.unicamp.br and Department of Mathematics, Nanchang University Nanchang, Jiangxi 330047 P. R. of China