Electronic Journal of Differential Equations, Vol. 2013 (2013), No. 109, pp. 1–16. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu ftp ejde.math.txstate.edu EXISTENCE OF SOLUTIONS FOR CRITICAL ELLIPTIC SYSTEMS WITH BOUNDARY SINGULARITIES JIANFU YANG, LINLI WU Abstract. This article concerns the existence of positive solutions to the nonlinear elliptic system involving critical Hardy-Sobolev exponent −∆u = 2λα uα−1 v β − up , α + β |π(x)|s in Ω, uα v β−1 2λβ − v p , in Ω, α + β |π(x)|s u > 0, v > 0, in Ω, −∆v = u = v = 0, on ∂Ω, C1 where N ≥ 4 and Ω is a bounded domain in RN , 0 < s < 2, α + β = 2(N −s) 2∗ (s) = N −2 , α, β > 1, λ > 0 and 1 ≤ p < NN . −2 Let P be a linear subspace of RN such that k = dimR P ≥ 2, and π be the orthogonal projection on P with respect to the Euclidean structure. We consider mainly the case when P ⊥ ∩ Ω = ∅ and P ⊥ ∩ ∂Ω 6= ∅. We show that there exists λ∗ > 0 such that the system above possesses at least one positive solution for 0 < λ < λ∗ provided that at each point x ∈ P ⊥ ∩ ∂Ω the principal curvatures of ∂Ω at x are non-positive, but not all vanish. 1. Introduction Let P be a linear subspace of RN such that k = dimR P ≥ 2, and π be the orthogonal projection on P with respect to the Euclidean structure. In this paper, we are concerned with the existence of positive solutions of the following nonlinear elliptic system involving critical Hardy-Sobolev exponent 2λα uα−1 v β − up , α + β |π(x)|s 2λβ uα v β−1 −∆v = − vp , α + β |π(x)|s u > 0, v > 0, in Ω, −∆u = u = v = 0, in Ω, in Ω, on ∂Ω, 2000 Mathematics Subject Classification. 35J25, 25J50, 35J57. Key words and phrases. Existence; compactness; critical Hardy-Sobolev exponent; nonlinear system. c 2013 Texas State University - San Marcos. Submitted October 16, 2012. Published April 29, 2013. 1 (1.1) 2 J. YANG, L. WU EJDE-2013/109 where N ≥ 4 and Ω is a C 1 bounded domain in RN . We assume in this paper that −s) N 0 < s < 2, α + β = 2∗ (s) = 2(N N −2 , α, β > 1, λ > 0 and 1 < p < N −2 . For the one equation case, if k = N , the problem is related to the CaffarelliKohn-Nirenberg inequalities. It was discussed in [5] the existence of minimizer of the best constant of the Caffarelli-Kohn-Nirenberg inequalities and related subjects. In particular, it shows that if 0 ∈ Ω, the best Hardy-Sobolev constant R |∇u|2 dx Ω (1.2) µ2∗ (s),s (Ω) = inf 2/2∗ (s) R ∗ u2 (s) u∈H01 (Ω)\{0} dx Ω |x|s is never attained unless Ω = RN and µ2∗ (s),s (Ω) = µ2∗ (s),s (RN ). If s = 0, the quantity µ2∗ (s),s (Ω) is the best Sobolev constant R |∇u|2 dx Ω S = S(Ω) = inf ∗, R 2∗ dx 2/2 u∈H01 (Ω)\{0} |u| Ω where 2∗ = N2N −2 is the critical Sobolev exponent and S is achieved if and only if Ω = RN , see [19]. Related results can also be found in [8] and [18]. In contrast with the case 0 ∈ Ω, if 0 ∈ ∂Ω, the problem is closely related to the properties of the curvature of ∂Ω at 0. Ghoussoub and Kang showed in [9] that there exists a solution of the problem ∗ u2 (s)−1 −∆u = + λup , |x|s u>0 in Ω; u = 0, on ∂Ω, N +2 where λ > 0, 1 < p < N −2 , 0 ∈ ∂Ω and the mean curvature of ∂Ω at 0 is negative. Such a result was proved by the global compactness method. Moreover, Ghoussoub and Robert in [10] have proved that µ2∗ (s),s (Ω) is achieved if 0 ∈ ∂Ω and the mean curvature of ∂Ω at 0 is negative. For the elliptic equation with two critical exponents ∗ − ∆u = N +2 u2 (s)−1 + λu N −2 , |x|s u>0 in Ω; u = 0, on ∂Ω, (1.3) using the blow-up method, Hsai et al [13] prove that problem (1.3) possesses at least one positive solution. In [17], the Hardy-Sobolev inequality Z Z |u|2∗ (s) 2∗2(s) ≤ |∇u|2 dx, ∀u ∈ D1,2 (RN ) (1.4) µ2∗ (s),P dx s |π(x)| N N R R was established for all u ∈ D1,2 (RN ). Then, it was shown in [7] that µ2∗ (s),P (Ω) ≥ µ2∗ (s),P (RN ) > 0 for all smooth domain Ω ⊂ RN . The attainability of µ2∗ (s),P (Ω) depends on the position between P and Ω, this was discussed in [12]. In this article, we study the existence of positive solutions of (1.1). In [14], positive solutions of problem (1.1) were found in non-contractible domains if λ = 0, k = N and s = 0. In [20], the existence of sign-changing solutions was obtained for (1.1) with k = N and s = 0. For further results for the system we refer the references in [14] and [20]. Equation (1.1) involves the Hardy type potential, that is s 6= 0 and possibly, k ≤ N , and the lower order terms are negative, which will push the energy up. EJDE-2013/109 EXISTENCE OF SOLUTIONS 3 We will prove that (1.1) possesses at least one positive solution by the blow up argument. The limiting problem after blowing up is as follows. 2α uα−1 v β , α + β |π(x)|s 2β uα v β−1 , −∆v = α + β |π(x)|s −∆u = u > 0, v > 0, u = v = 0, in RN +, in RN +, (1.5) in RN +, on ∂RN +. Denote R µα,β,P (Ω) = inf 1 (u,v)∈(H0 (Ω))2 \{0} Ω (|∇u|2 + |∇v|2 )dx 2 R uα v β dx 2∗ (s) Ω |π(x)|s (1.6) for a domain Ω ⊂ RN . The solution of (1.4) will be obtained by showing that N µα,β,P (RN + ) is achieved. The minimizer of µα,β,P (R+ ) is the least energy solution of (1.4) up to a multiplicative constant. It was observed in [2] that µα,β,P (Ω) and µα+β,P (Ω) are closely related. Precisely, we have α β α −α µα,β,P (Ω) = ( ) α+β + ( ) α+β µα+β,P (Ω) β β (1.7) for α + β ≤ 2∗ . Moreover, if w0 realizes µα+β,s (Ω), then u0 = Awq 0 and v0 = Bw0 A realizes µα,β,P (Ω) for any real constants A and B such that B = α β. N In the case Ω = RN + , it was proved in [12] that µ2∗ (s),P (R+ ) is achieved by a ⊥ N 1 N function u ∈ H0 (R+ ) provided that P ⊂ ∂R+ . This implies that µα,β,P (RN + ) is . Hence, there exists a least energy entire achieved if α + β = 2∗ (s) and P ⊥ ⊂ ∂RN + solution of (1.4) in this case. To deal with (1.1), we consider a related subcritical problem, and obtain a sequence of solutions of the subcritical problems. Then, we analyse the blow up behavior of the approximating sequence. Since the coefficient of lower order terms are negative, the energy of the corresponding functional becomes larger, it makes difficult to find the upper compact bound. Our main result is as follows. Theorem 1.1. Let Ω be a smooth bounded domain of RN , N ≥ 3, and let P be a linear subspace of RN such that k = dimR P ≥ 2. Suppose s ∈ (0, 2), then we have (i) If P ⊥ ∩Ω 6= ∅, problem (1.1) possesses at least one positive solution provided that s = 1. (ii) If P ⊥ ∩ Ω̄ = ∅, problem (1.1) possesses at least one positive solution. (iii) If P ⊥ ∩Ω = ∅ and P ⊥ ∩∂Ω 6= ∅, there exists λ∗ > 0 such that for 0 < λ < λ∗ problem (1.1) possesses at least one positive solution provided that at each point x ∈ P ⊥ ∩ ∂Ω the principle curvatures of ∂Ω at x are non-positive, but not all vanish. In section 2, we find a suitable upper bound for the mountain pass level and prove (i) and (ii) of Theorem 1.1, then using this bound and the blow-up argument, we prove (iii) of Theorem 1.1 in section 3. 4 J. YANG, L. WU EJDE-2013/109 2. Preliminaries We recall that |∇u|2 dx 2∗2(s) , R u2∗ (s) dx s Ω |π(x)| R µ2∗ (s),P (Ω) = Ω inf 1 u∈H0 (Ω)\{0} (2.1) −2) where 2∗ (s) = 2(N N −2 , s ∈ (0, 2) and π is the orthogonal projection on P with respect to the Euclidean structure. The attainability of µ2∗ (s),P (Ω) depends on the position between Ω and P. Actually, it was proved in [12] that if P ⊥ ∩ Ω 6= ∅, µ2∗ (s),P (Ω) = µ2∗ (s),P (RN ). Therefore, µ2∗ (s),P (Ω) is not achieved. If P ⊥ ∩ Ω̄ = ∅, the problem becomes subcritical without singularities, thus µ2∗ (s),P (Ω) is attained. Finally, if P ⊥ ∩ Ω = ∅ and P ⊥ ∩ ∂Ω 6= ∅, µ2∗ (s),P (Ω) is achieved provided that the principle curvatures of ∂Ω at x ∈ P ⊥ ∩ ∂Ω are non-positive, and do not all vanish. Furthermore, the following lemma was also shown in [12]. N 1 N Lemma 2.1. There exists a minimizer u ∈ C 1 (R̄N + ) ∩ H0 (R+ ) of µ2∗ (s),P (R+ ) such that ∗ u2 (s)−1 −∆u = in RN +, |π(x)|s (2.2) u>0 in D0 (RN + ) satisfying R in RN +, u=0 on ∂RN + 2∗ (s) RN + 2∗ (s)−2 , provided that 2 ≤ k ≤ |∇u|2 dx = µ2∗ (s),P (RN +) N − 1 and P ⊥ ⊂ ∂RN +. N 1 N Let u ∈ C 1 (R̄N + ) ∩ H0 (R+ ) be the minimizer of µ2∗ (s),P (R+ ). We have the following estimates. Lemma 2.2. There exists C > 0 such that |∇u(x)| ≤ C(1 + |x|)−N |u(x)| ≤ C(1 + |x|)1−N , (2.3) for x ∈ RN +. Proof. Let u∗ (x) = |x|−(N −2) u( x ), |x|2 x ∈ RN +, ∗ be the Kelvin transformation of u. Since u ∈ D01,2 (RN + ), we may verify that the u R R ∗ 2∗ (s) | also satisfies equation (2.2), and both RN |∇u∗ |2 dx and RN |u|π(x)| s dx are finite. + + Next, by a regularity result in [12], u∗ ∈ C 1 (R̄N + ). It implies in a standard way that (2.3) holds. The proof is complete. By (1.7), we see that µα,β,P (Ω) and µ2∗ (s),P (Ω) are closely related if α + β = 2∗ (s), which and Lemma 2.2 allow us to state the following result. Proposition 2.3. Suppose α + β = 2∗ (s). Then (i) µα,β,P (Ω) = µα,β,P (RN ) if P ⊥ ∩ Ω 6= ∅, and µα,β,P (Ω) is not achieved. (ii) If P ⊥ ∩ Ω̄ = ∅, µα,β,P (Ω) is attained. (iii) If P ⊥ ∩ Ω = ∅ and P ⊥ ∩ ∂Ω 6= ∅, µα,β,P (Ω) is achieved provided that the principle curvatures of ∂Ω at x ∈ P ⊥ ∩ ∂Ω are non-positive, and do not all vanish. EJDE-2013/109 EXISTENCE OF SOLUTIONS 5 Moreover, all components of the minimizer of µα,β,P (RN + ) satisfy the decaying law in (2.3). Proof of (i) and (ii) of Theorem 1.1. In the case (i), problem (1.1) is problem with singularities in Ω. The existence of positive solution of lem can be proved by the mountain pass theorem as [1], [4]. In the problem (1.1) is a subcritical problem without singularities, the result obtained. a critical the probcase (ii), is readily In the rest of the paper, we only consider the case (iii); that is, we assume that P ⊥ ∩ Ω = ∅ and P ⊥ ∩ ∂Ω 6= ∅. In the following, we establish the upper bound for the mountain pass level. We 1 N ⊥ N recall that by [10], µ2∗ (s),P (RN + ) is achieved by a function u ∈ H0 (R+ ) if P ⊂ ∂R+ . N ∗ This implies that µα,β,P (R+ ) is achieved if α + β = 2 (s). Hence, there exists a least energy entire solution of system (1.4). The energy functional for (1.1) is Z 1 2λ uα v β 1 1 1 + up+1 + v p+1 ) dx , Iλ (u, v) = ( |∇u|2 + |∇v|2 − ∗ s 2 2 2 (s) |π(x)| p + 1 p + 1 Ω which is well defined on H01 (Ω). It is well known that to find positive solutions of problem (1.1) is equivalent to find nonzero critical points of functional Iλ in H01 (Ω) × H01 (Ω). Now, we bound the mountain pass level for the functional Iλ . Lemma 2.4. Suppose that Ω is a C 1 bounded domain in RN with P ⊥ ∩ Ω = ∅ and P ⊥ ∩ ∂Ω 6= ∅. There exist λ∗ > 0 and nonnegative functions u0 and v0 in H01 (Ω) \ {0} such that for 0 < λ < λ∗ and 1 ≤ p < NN−2 , we have Iλ (u0 , v0 ) < 0 and 2∗ (s) −2 1 1 2∗ (s)−2 . max Iλ (tu0 , tv0 ) < (2λ) 2∗ (s)−2 ( − ∗ )µα,β,P (RN +) 0≤t≤1 2 2 (s) provided that the principle curvatures of ∂Ω at x ∈ P ⊥ ∩ ∂Ω are non-positive, and do not all vanish. Proof. Let (u, v) be the minimizer of µα,β,P (RN + ), such that Z Z Z uα v β 2 2 N |∇u| dx + |∇v| dx = µα,β,P (R+ ), dx = 1. |π(x)|s RN RN RN + + + q A Then, there exist A, B ∈ R such that u = Aw, v = Bw with B = α β , where w is a minimizer of µ2∗ (s),P (RN + ). Since |∇w(x)| ≤ C(1 + |x|)−N , |w(x)| ≤ C(1 + |x|)1−N , we obtain |u(x)| ≤ C(1 + |x|)1−N , |∇u(x)| ≤ C(1 + |x|)−N , (2.4) |v(x)| ≤ C(1 + |x|)1−N , |∇v(x)| ≤ C(1 + |x|)−N . (2.5) Moreover, (u, v) satisfies − ∆u = uα−1 v β α µα,β,P (RN , +) α+β |π(x)|s −∆v = β uα v β−1 µα,β,P (RN , +) α+β |π(x)|s in RN +. (2.6) Let x0 ∈ P ⊥ ∩ ∂Ω. Since P ⊥ ∩ Ω = ∅, we have P ⊥ ⊂ Tx0 ∂Ω, where Tx0 ∂Ω is the tangent space of the smooth manifold ∂Ω at x0 . Thus, (Tx0 ∂Ω)⊥ ⊂ P. 6 J. YANG, L. WU EJDE-2013/109 Denote k = dimR P, we choose a direct orthonormal basis (e1 , . . . , eN ) of RN such that e1 = nx0 is the outward normal of ∂Ω at x0 , span{e1 , . . . , ek } = P and span{ek+1 , . . . , eN } = P ⊥ . For any x ∈ RN , we denote x = (x1 , y, z), where x1 ∈ R, y ∈ span{e2 , . . . , ek } and z ∈ P ⊥ . Since ∂Ω is smooth, there exist open sets U, V of RN such that 0 ∈ U and x0 ∈ V , and there exist ϕ ∈ C ∞ (U, V ) and ϕ0 ∈ C ∞ (U 0 ) with U 0 = {(y, z) : there exists x1 ∈ R such that (x1 , y, z) ∈ U } such that (i) ϕ : U → V is a C ∞ diffeomorphism, ϕ(0) = x0 ; (ii) ϕ(U ∩ {x1 > 0}) = ϕ(U ) ∩ Ω and ϕ(U ∩ {x1 = 0}) = ϕ(U ) ∩ ∂Ω; (iii) ϕ0 (0) = 0 and ∇ϕ0 (0) = 0; (iv) ϕ(x1 , y, z) = (x1 − ϕ0 (y, z), y, z) + x0 for all (x1 , y, z) ∈ U . Denote ψ = ϕ−1 . We choose a small positive number r0 so that there exist neighborhoods V and Ṽ of x0 , such that ψ(V ) = Br0 (0), ψ(V ∩Ω) = Br+0 (0), ψ(Ṽ ) = B r20 (0), ψ(Ṽ ∩ Ω) = B + r0 (0). For ε > 0, we define 2 N −2 ψ(x) ψ(x) := η(x)uε , ṽε (x) = ε− 2 η(x)v := η(x)vε , ε ε where η ∈ C0∞ (V ) is a positive cut-off function with η ≡ 1 in Ṽ . In what follows, we estimate each term in Iλ (tũε , tṽε ). We have Z Z |∇ũε |2 dx = (|∇η|2 u2ε + η 2 |∇uε |2 + 2∇η∇uε ηuε ) dx. ũε (x) = ε − N 2−2 η(x)u Ω Ω Since Z Z ηuε ∇η∇uε dx = − Ω |∇η|2 u2ε dx − Z Ω Z ∇ηη∇uε uε dx − Ω η(∆η)|uε |2 dx, Ω we obtain Z |∇ũε |2 dx = Ω Z η 2 |∇uε |2 dx − Ω∩U By the change of the variable X = Z Z η(∆η)u2ε dx ≤ Cε2 0 η(∆η)|uε |2 dx. Ω∩U ψ(x) ε ∈ Br+0 /ε (0) and (2.4), we obtain Br+ /ε (0)\B + r0 (0) Ω∩U Z η(ϕ(εX))∆η(ϕ(εX))u2 (X) dX 2ε = O(ε2 ) N 0 and since ∇x uε (x) = ε− 2 ∇X u( ψ(x) ε )∇x ψ(x), we deduce for X = (X2 , . . . , XN ) 0 and ∇ = (∂X2 , . . . , ∂XN ) that Z η 2 |∇uε |2 dx Ω∩U Z Z ≤ |∇u|2 dX − 2 η 2 (ϕ(εX))∂1 u(X)∇0 u(X)(∇0 ϕ0 )(εX 0 ) dX Br+ /ε RN + 0 Z η (ϕ(εX))|∇0 u(X)|2 |(∇0 ϕ0 )(εX 0 )|2 dX = I1 + I2 + I3 . 2 + Br+ /ε 0 Using that |∇0 ϕ0 (X 0 )| = O(|X 0 |), ϕ0 (X 0 ) = N X i=2 αi Xi2 + o(1)(|X 0 |2 ) EJDE-2013/109 EXISTENCE OF SOLUTIONS 7 and (2.4), we have Z (1 + |X|)−2N |εX|2 dX = O(ε2 ). I3 ≤ C RN Integrating by parts, we infer that Z 4 I2 = η(ϕ(εX))∇0 η(ϕ(εX))∂1 u(X)∇0 u(X)ϕ0 (εX 0 ) dX ε Br+ /ε Z0 2 η 2 (ϕ(εX))∇0 ∂1 u(X)∇0 u(X)ϕ0 (εX 0 ) dX + ε Br+ /ε 0 + Z 2 ε η 2 (ϕ(εX))∂1 u(X) Br+ /ε N X ∂ii u(X)ϕ0 (εX 0 ) dX = I21 + I22 + I23 . i=2 0 By (2.4), |I21 | ≤ Cε2 Z (1 + |X|)−2N |X|2 dX ≤ CεN . Br+ /ε (0)\B r0 (0) 0 2ε In the same way, I22 = O(εN ). By equation (2.6), N X ∂ii u(X) = ∆u − ∂11 u(X) = − i=2 αλ uα−1 v β µα,β,s (RN − ∂11 u(X). +) α+β |π(X)|s Therefore, 2 ε Z 2 − ε Z I23 = − Br+ /ε η 2 (ϕ(εX))∂1 u(X) αλ uα−1 v β µα,β,s (RN ϕ0 (εX 0 ) dX +) α+β |π(X)|s (2.7) 0 Br+ /ε η 2 (ϕ(εX))∂1 u(X)∂11 u(X)ϕ0 (εX 0 ) dX := F1 + F2 . 0 Since u = Aw, F1 = − C0 ε ∗ Z η 2 (ϕ(εX)) Br+ /ε ∂1 w(X)2 (s) ϕ0 (εX 0 ) dX, |π(X)|s 0 where C0 = 2αλ N α β (2∗ (s))2 µα,β,s (R+ )A B . Z F1 = C0 Br+ /ε Integrating by parts, we obtain 2η(ϕ(εX))∂1 η(ϕ(εX))ϕ0 (εX 0 ) 2∗ (s) w dX |π(X)|s 0 C0 s − ε Z Br+ /ε η 2 (ϕ(εX))ϕ0 (εX 0 )X1 2∗ (s) w dX |π(X)|s+2 0 = F11 + F12 . N 2 −N −N s+2 N −2 ). We may verify as above that F11 = O(ε Now, we estimate F2 . Integrating by parts, we deduce Z 1 F2 = ∂1 [η 2 (ϕ(εX))ϕ0 (εX 0 )](∂1 u)2 dX ε Br+ /ε Z0 1 + η 2 (ϕ(εX))ϕ0 (εX 0 )(∂1 u)2 ν N dSX ε Br+ /ε ∩{X1 =0} 0 (2.8) 8 J. YANG, L. WU = 1 ε Z 2η(ϕ(εX))∂1 [η(ϕ(εX))]ϕ0 (εX 0 )(∂1 u)2 dX Br+ /ε 1 + ε EJDE-2013/109 Z0 Br+ /ε ∩∂RN + η 2 (ϕ(εX))ϕ0 (εX 0 )(∂1 u)2 dSX 0 = F21 + F22 . It can be shown that F21 = O(εN −1 ). Hence, I2 = F12 + F22 + O(εN −1 ). Since η(ϕ(εX)) ≡ 1 in B + r0 , we have 2ε C0 s ε Z C0 s − ε Z F12 = − η 2 (ϕ(εX))ϕ0 (εX 0 )X1 2∗ (s) w dX |π(X)|s+2 Br+ /ε \B + r0 0 2ε B+ r0 ϕ0 (εX 0 )X1 2∗ (s) w dX = J1 + J2 . |π(X)|s+2 2ε We have ∗ Z J1 ≤ Cε Br+ /ε \B + r0 0 |π(X)|3 (1 + |X|)(1−N )2 |π(X)|s+2 dX 2ε Z ≤ Cε 1 N −k (Br+ /ε \B + r0 )∩R 0 ≤ Cε (s) N (N −s) N −2 |x| 2∗ (s)(N −1) 2 |x|1−s Z dx k (Br+ /ε \B + r0 )∩R 0 2ε |x| 2∗ (s)(N −1) 2 2ε . In the same way, Z C0 s J2 = − ε RN + Z C0 s =− ε RN + ϕ0 (εX 0 )X1 2∗ (s) C0 s dX − w |π(X)|s+2 ε Z + RN + \Br /ε ∗ ϕ0 (εX 0 )X1 w(X)2 (s) dX |π(X)|s+2 0 N (N −s) ϕ0 (εX 0 )X1 2∗ (s) w dX + O(ε N −2 ) s+2 |π(X)| Z ∗ N X N (N −s) Xi2 X1 w(y)2 (s) = −εC0 s αi dX(1 + o(1)) + O(ε N −2 ) s+2 |π(X)| RN + i=2 Z ∗ N X N (N −s) sεc1 |X 0 |2 X1 w(X)2 (s) dX αi (1 + o(1)) + O(ε N −2 ) =− s+2 N − 1 RN |π(X)| + i=2 = −C0 K1 H(0)(1 + o(1))ε + O(ε N (N −s) N −2 ), where N 1 X H(0) = αi , N − 1 i=2 Z K1 = s RN + ∗ |X 0 |2 X1 w2 (s) dX. |π(X)|s+2 Similarly, F22 = dx 1 ε Z (Br+ /ε \B + r0 )∩{X1 =0} 0 2ε η 2 (ϕ(εX))ϕ0 (εX 0 )(∂1 u(X))2 dSX EJDE-2013/109 EXISTENCE OF SOLUTIONS + 1 ε Z B+ r0 ∩{X1 =0} 9 ϕ0 (εX 0 )(∂1 u(X))2 dSX = L1 + L2 . 2ε Also Z C L1 ≤ ε r { 20 |(∂1 u)(0, X 0 )|2 |ϕ0 (εX 0 )| dX 0 <|εX 0 |≤r 0} Z ≤ Cε { r0 2 |X 0 |−2N +2 dX 0 = O(εN ). <|εX 0 |≤r0 } Using that Z RN −1 \(B + r0 ∩{X1 =0}) ϕ0 (εX 0 )(∂N u(X))2 dSX = O(εN ), 2ε one finds L2 = 1 ε =ε Z ϕ0 (εX 0 )(∂1 u(X))2 dSX + O(εN −1 ) RN −1 N X Z αi RN −1 i=2 [(∂1 u)(0, X 0 )]2 Xi2 dX 0 (1 + o(1)) + O(εN −1 ) = K2 H(0)(1 + o(1))ε + O(εN −1 ), R where K2 = RN −1 |(∂N u)(0, X 0 )|2 |X 0 |2 dX 0 . Consequently, Z Z |∇u|2 dX − (C0 K1 − K2 )H(0)(1 + o(1))ε + O(ε2 ), |∇ũε |2 dx = RN + Ω and similarly, Z Z 2 |∇ṽε | dx = where C1 = |∇v|2 dX − (C1 K1 − K2 )H(0)(1 + o(1))ε + O(ε2 ). RN + Ω 2βλ α β N (2∗ (s))2 µα,β,s (R+ )A B . X = ψ(x) ε . We estimate Next, let Z Z Z Z β β β ũα uα ũα uα (X)v β (X) ε ṽε ε vε ε ṽε dx ≥ dx = dx = dX s s s | π(ϕ(εX)) |s Ω∩Ṽ |π(x)| Ω∩Ṽ |π(x)| B+ Ω |π(x)| r0 ε 2ε ⊥ since η ≡ 1 in Ω ∩ Ṽ . We recall that x0 ∈ P ∩ ∂Ω, then we may write π(ϕ(εX)) = (εx1 + ϕ0 (εy, εz), εy, 0) and ϕ2 (εX 0 ) 2X1 ϕ0 (εX 0 ) + 20 . |π(ϕ(εX))|2 = ε2 |π(X)|2 1 + 2 ε|π(X)| ε |π(X)|2 Therefore, 1 | ϕ(εX) |s ε = 1 sX1 ϕ0 (εX 0 ) sϕ20 (εX 0 ) 1 − − |π(X)|s ε|π(X)|2 2ε2 |π(X)|2 2X ϕ (εX 0 ) 1 ϕ20 (εX 0 ) 1 0 O + . + |π(X)|s ε|π(X)|2 ε2 |π(X)|2 10 J. YANG, L. WU EJDE-2013/109 This and Z + RN + \B r0 N (N −s) uα v β dX = O(ε N −2 ) |π(X)|s 2ε enable us to show that Z Z Z β s uα v β X1 ϕ(εX 0 )uα (X)v β (X) ũα ε ṽε dx = dX − dX + O(ε2 ) s s s+2 + |π(X)| ε |π(X)| B+ B Ω∩Ũ |π(x)| r0 r0 2ε 2ε Z Z α β u v s X1 ϕ(εX 0 )uα v β dX − dX + O(ε2 ). = s s+2 + |π(X)| ε |π(X)| RN B r + 0 2ε Moreover, − s ε X1 ϕ(εy 0 )uα v β dX |π(X)|s+2 Z B+ r0 2ε s = − Aα B β ε ∗ X1 ϕ(εX 0 )w2 |π(X)|s+2 Z B+ r0 (s) dX 2ε = −sε N X α αi A B β RN + i=2 =− ∗ N (N −s) X1 Xi2 w2 (s) dX(1 + o(1)) + O(ε N −2 ) |π(X)|s+2 Z sε Aα B β N −1 N ∗ Z RN + X N (N −s) X1 |X 0 |2 w2 (s) dX αi (1 + o(1)) + O(ε N −2 ). s+2 |π(X)| i=2 Hence, Z Ω∩Ũ β ũα ε ṽε dx = |π(x)|s Z RN + uα v β dX − K3 H(0)(1 + o(1))ε + O(ε2 ), |π(X)|s ∗ where K3 = sAα B β R RN + ψ(x) ε X1 |X 0 |2 w2 (s) dX = Aα B β K1 . |π(X)|s+2 ∈ Br+0 /ε (0). We estimate Finally, let X = Z Z (2−N )(p+1) 2 ũp+1 dx = ε ε Ω η 2 (x)[u( Ω∩U =ε (2−N )(p+1) +N 2 Z ψ(x) p+1 )] dx ε up+1 dX Br+ /ε 0 =ε (N −2)p N +2 2 − 2 Z up+1 dX + O(ε N (p+1) 2 v p+1 dX + O(ε N (p+1) 2 ). RN + Similarly, Z ṽεp+1 dx = ε (N −2)p N +2 2 − 2 Ω Since q < N N +2 N −2 , 2 − (N −2)p 2 Z ). RN + > 1. For t ≥ 0, we have Iλ (tũε , tṽε ) Z Z 2t2∗ (s) λ Z t2 uα v β = |∇u|2 dX + |∇v|2 dX − ∗ dX 2 2 (s) RN |π(X)|s RN RN + + + EJDE-2013/109 EXISTENCE OF SOLUTIONS 11 ∗ H(0) 4 [(2K2 − C0 K1 − C1 K1 )t2 + ∗ (λK3 + o(1))t2 (s) ]ε + O(ε2 ) 2 2 (s) H(0) = f1 (t) + εf2 (t) + O(ε2 ), 2 where ∗ t2 2λt2 (s) f1 (t) = µα,β,s (RN ) − . + 2 2∗ (s) It can be verified that 2∗ (s) −2 1 1 2∗ (s)−2 , max f1 (t) = f1 (t0 ) = (2λ) 2∗ (s)−2 ( − ∗ )µα,β,s (RN ) + 0≤t≤1 2 2 (s) + 1 1 2∗ (s)−2 . Since K > 0, with t0 = ( 2λ µα,β,s (RN 1 + )) f2 (t0 ) = (2K2 − C0 K1 − C1 K1 )t20 + = (2K2 − 2λ 2∗ (s) = 2K2 t20 + Aα B β K1 )t20 + 2λ 2∗ (s) Aα B β K1 ( 4λ 2∗ (s) 2∗ (s) K3 t0 4λ 2∗ (s) 2∗ (s) Aα B β K1 t0 µα,β,s (RN +) − 1)t20 . λ Hence, f2 (t0 ) > 0 if λ > 0 and small. Since H(0) < 0, by choosing T large enough, we have Iλ (T ũε , T ṽε ) < 0 for t ≥ T and ε ≥ 0 small. Let u0 = T ũε , v0 = T ṽε . We obtain 2∗ (s) −2 1 1 2∗ (s)−2 max Iλ (tu0 , tv0 ) < (2λ) 2∗ (s)−2 ( − ∗ )µα,β,s (RN +) 0≤t≤1 2 2 (s) and Iλ (u0 , v0 ) < 0. This completes the proof of Lemma 2.1. 3. Existence of positive solution in Ω Now we will use the blow up argument to prove (iii) of Theorem 1.1. For any ε > 0, by the mountain pass theorem, we have a positive solution pair (uε , vε ) of the subcritical system 2αλ uεα−1 vεβ−ε − up−ε , ε α + β − ε |π(x)|s β−1−ε 2βλ uα ε vε −∆vε = − vεp−ε , α + β − ε |π(x)|s uε > 0, vε > 0, in Ω, −∆uε = uε = vε = 0, in Ω, in Ω, (3.1) on ∂Ω. Using Lemma 2.4, we see that the mountain pass level cε of (3.1) satisfies 2∗ (s) −2 1 1 2∗ (s)−2 cε = Iλε (uε , vε ) < (2λ) 2∗ (s)−2 ( − ∗ )µα,β,s (RN (3.2) +) 2 2 (s) if 0 < λ < λ∗ , where β−ε 1 1 2λ uα ε vε ( |∇uε |2 + |∇vε |2 − ∗ ) dx 2 2 (s) − ε |π(x)|s Ω 2 Z Iε (uε , vε ) = 12 J. YANG, L. WU Z + ( Ω EJDE-2013/109 1 1 up+1−ε + v p+1−ε ) dx. ε p+1−ε p+1−ε ε It can be easily shown that both kuε kH01 (Ω) and kvε kH01 (Ω) are uniformly bounded for ε > 0 small. Thus, there is a subsequence {(uj , vj )} of {(uε , vε )} such that uj * u, uj → u, uj * u, in H01 (Ω), vj * v, in Lp+1 (Ω), vj → v, vj * v, ∗ in L2 (s) (3.3) (Ω, |π(x)|−s dx), with u, v ≥ 0 and (u, v) is a solution of system (1.1). If (u, v) is a nontrivial solution, by the strong maximum principle, u, v > 0, then we are done. Now, we prove (u, v) is nontrivial. This will be shown by the blowing up argument. Suppose on the contrary that u = v = 0 in Ω. By the regularity result, see for instance [12, Proposition 3.2], uε , v ∈ C 1 (Ω̄). Let xj , yj ∈ Ω be such that Mj = uj (xj ) = max uj (x), Nj = vj (yj ) = max vj (x). Ω̄ (3.4) Ω̄ Then, we have either mj → ∞ or nj → ∞ as j → ∞. Indeed, on the contrary we would have mj ≤ C and nj ≤ C for a positive constant C. By the Sobolev embedding, Z Z α β−εj uj vj uα j dx ≤ C dx → 0 s s Ω |π(x)| Ω |π(x)| as j → ∞. This implies Z Z Z Z α β−εj u j vj p+1−εj p+1−εj dx − λ u dx − λ vj dx → 0; (|∇uj |2 + |∇vj |2 ) dx = 2 j s |π(x)| Ω Ω Ω Ω that is, uj → 0, vj → 0 strongly in H01 (Ω). It yields Z 1 (|∇uj |2 + |∇vj |2 ) dx = c > 0 0 = lim j→∞ 2 Ω a contradiction. Suppose Nj ≤ Mj → ∞. Denote ũj (x) = Mj−1 uj (kj x + xj ), 2∗ (s)−2−εj − 2−s where kj = Mj satisfies ṽj (x) = Mj−1 vj (kj x + xj ), for x ∈ Ωj , and Ωj = {x ∈ RN | xj + kj x ∈ Ω}. Obviously, (ũj , ṽj ) −∆ũj = 2αλ k j s α + β − εj |π(xj )| |π( −∆ṽj = 2(β − εj )λ k j s α + β − εj |π(xj )| |π( β−εj ũα−1 ṽj j xj |π(xj )| + p−1−εj kj s |π(xj )| x)| − kj2 Mj p−εj ũj , in Ωj , p−1−εj p−εj ṽj , in Ωj , β−1−εj ũα j ṽj xj |π(xj )| + kj s |π(xj )| x)| 0 ≤ ũj , ṽj ≤ 1, ũj = ṽj = 0, − kj2 Mj in Ωj , on ∂Ωj . (3.5) Since Mj → ∞, kj → 0 as j → ∞. Furthermore, we have p−1−εj kj2 Mj 2− = kj (2−s)(p−εj −1) 2∗ (s)−2−εj →0 as j → ∞ EJDE-2013/109 EXISTENCE OF SOLUTIONS 13 (2−s)(p−ε −1) j N +2 as the facts that kj → 0 and 2 − 2∗ (s)−2−ε > 0; i.e, p < N −2 . j We will show that Mj = O(1)Nj . First, we claim that |π(xj )| = O(kj ) as j → ∞. |π(x )| Suppose on the contrary that lim supj→∞ kjj = ∞. 1 Because (ũj , ṽj ) is uniformly bounded in Cloc , we may assume that ũj → u, ṽj → 0 v in Cloc . Suppose now xj → x0 ∈ Ω̄. There are two cases: (i) x0 ∈ Ω or x0 ∈ ∂Ω and dist(xj ,∂Ω) kj dist(xj ,∂Ω) kj → σ ≥ 0. (ii) x0 ∈ ∂Ω and → ∞; and In the case (i), we have Ωj → RN as j → ∞ and (u, v) satisfies ∆u = 0, ∆v = 0 0 ≤ u, v ≤ 1, in RN , u(0) = 1. Furthermore, Z 2N N −2 ũj N εj 2∗ (s)−2−εj Z 2N N −2 dy = kj Ωj uj Z dx ≤ C, and Ω 2N ṽjN −2 dy ≤ C, Ωj which yields Z Z 2N 2N u N −2 dy < ∞, v N −2 dy < ∞. RN RN However, by the Liouville theorem, u ≡ v ≡ 1 for x ∈ RN . This is a contradiction. In case (ii), after an orthogonal transformation, we have Ωj → RN + = {x = (x1 , . . . , xN ) | x1 > 0} as j → ∞ and ũj , ṽj converge to some u, v uniformly in every compact subset of RN + . Now, u(0) = 1 and 0 ≤ v(0) ≤ 1. Hence, (u, v) satisfies ∆u = 0, ∆v = 0 0 ≤ u, v ≤ 1 in RN +, in RN +, on ∂RN +. u=v=0 By the boundary condition and the maximum principle, u ≡ v ≡ 0 for x ∈ RN + |π(x )| which violate to u(0) = 1. Consequently, lim supj→∞ kjj < ∞. Since kj → 0, we have π(xj ) → 0 as j → ∞. |π(x )| Next, we show that lim inf j→∞ kjj > 0. Were it not the case, we would have, up to a subsequence, that limj→∞ |π(xj )| kj = 0. Then (ũj , ṽj ) satisfies β−ε ũα−1 ṽj j 2αλ p−1−εj j −∆ũj = − kj2 Mj u˜j p−εj , α + β − εj | π(xj ) + π(x)|s kj in Ωj , β−1−ε j 2(β − εj )λ ũα j ṽj p−1−εj p−εj − kj2 Mj −∆ṽj = v˜j , π(x ) j s α + β − εj | + π(x)| kj 0 ≤ ũj , ṽj ≤ 1, ũj = ṽj = 0, in Ωj , on ∂Ωj , in Ωj , (3.6) 14 J. YANG, L. WU EJDE-2013/109 Up to a rotation, we have Ωj → RN + and ũj , ṽj converge to some u, v uniformly in compact subsets of RN respectively, where (u, v) satisfies + −∆u = 2αλ uα−1 v β , α + β |π(x)|s −∆v = in RN +, 0 ≤ u, v ≤ 1 2βλ uα v β−1 α + β |π(x)|s in RN +, on ∂RN +. u=v=0 |π(x )| The boundary condition violates to u(0) = 1. Hence, lim inf j→∞ kjj > 0. Now, we complete the proof of Theorem 1.1 by showing that problem (1.1) has a nontrivial solution. First, we remark that dist(xj , ∂Ω) = O(kj ). Indeed, since P ⊥ ∩ Ω = ∅, we have xj − π(xj ) ∈ P ⊥ ⊂ RN \ Ω. Because xj ∈ Ω, there exists tj ∈ (0, 1) such that tj xj + (1 − tj )(xj − π(xj )) ∈ ∂Ω. Therefore, d(xj , ∂Ω) ≤ |xj − (tj xj + (1 − tj )(xj − π(xj )))| = (1 − tj )|π(xj )| ≤ |π(xj )| = O(kj ). dist(x ,∂Ω) j → σ ≥ 0. By an affine transformation, we Hence, we may assume kj find (ũj , ṽj ) converges to (u, v) uniformly in any compact subset of RN + and (u, v) satisfies 2αλ uα−1 v β 2βλ uα v β−1 −∆u = , −∆v = in RN +, α + β |π(x)|s α + β |π(x)|s (3.7) in RN +; u, v > 0 u=v=0 on ∂RN + with u(0, . . . , σ) = 1. By the definition of µα,β,s (Ω), we have R (|∇ũj |2 + |∇ṽj |2 ) dx µα,β,s (Ωj ) ≤ Ω R α β−ε 2 , ũj ṽj 2∗ (s) dx s |x| Ω and then R µα,β,s (RN +) ≤ Z 2∗2(s)−2 (|∇u|2 + |∇v|2 ) dy ∗ (s) uα v β ; dx 2∗2(s) = 2λ R s N |π(x)| uα v β R+ dx RN |π(x)|s RN + + that is, Z Z 2 2 (|∇u| + |∇v| ) dx = 2λ RN + RN + 2∗ (s) −2 uα v β N 2∗ (s)−2 2∗ (s)−2 µ dx ≥ (2λ) (R ) . (3.8) α,β,s + |π(x)|s Furthermore, noting that Z Z (N −2)εj − 2∗ (s)−2−ε j lim (|∇uj |2 + |∇vj |2 ) dx = lim kj (|∇ũj |2 + |∇ṽj |2 ) dx j→∞ j→∞ Ω Ωj Z ≥ lim j→∞ Z ≥ (|∇ũj |2 + |∇ṽj |2 ) dx Ωj (|∇u|2 + |∇v|2 ) dx, RN + we derive from (3.2), (3.8), (3.9) that Z 1 1 c = ( − ∗ ) lim (|∇uj |2 + |∇vj |2 ) dx 2 2 (s) j→∞ Ω 2∗ (s) −2 1 1 2∗ (s)−2 , ≥ ( − ∗ )(2λ) 2∗ (s)−2 µα,β,s (RN +) 2 2 (s) (3.9) EJDE-2013/109 EXISTENCE OF SOLUTIONS 15 which yields a contradiction to (3.2). Thus, (u, v) is a nontrivial solution of (1.1) if Nj ≤ Mj . Now we show Mj = O(Nj ). Indeed, since u is nontrivial, so is v. Otherwise, we would have ∆u = 0 in RN +, 0 ≤ u ≤ 1, u(0, . . . , σ) = 1 u=0 in RN +, on ∂RN +. By the strong maximum principle, u would be a constant because it attains its maximum value inside RN + . This yields a contradiction between u(0, . . . , σ) = 1 and the boundary condition. Therefore, there exists y0 ∈ RN + such that v(y0 ) 6= 0. Hence, ṽj (y0 ) = m−1 j vj (xj + kj y0 ) → v(y0 ) > 0 implying vj (xj + kj y0 ) nj ≥ ≥ v(y0 ) − ε > 0 mj mj for ε > 0 small and j large, namely, Nj = O(1)Mj as j → ∞. Replacing Mj by Nj in above blow up process, we may also derive a contradiction if we assume u = v = 0. Consequently, (1.1) has a positive nontrivial solution. The proof of Theorem 1.1 is complete. 1≥ Acknowledgments. This work is supported by grant 11271170 from the NNSF of China, by GAN PO 555 program of Jiangxi, and by grant 2912BAB201008 from the NNSF of Jiangxi. References [1] A. Al-aati, C. Wang, J. Zhao; Positive solutions to a semilinear elliptic equation with a Sobolev-Hardy term, Nonlinear Analysis, 74(2011), 4847-4861. [2] C. O. Alves, D. C. de Morais Filho, M. A. S. Souto; On systems of elliptic equations involving subcritical and critical Sobolev exponents, Nonlinear Anal. TMA, 42 (2000), 771-787. [3] H. Brézis, L. Nirenberg; Positive solutions of nonlinear elliptic equations involving critical Sobolev exponents, Comm. Pure Appl. Math., 36 (1983), 437-477. [4] M.Badiale, G. Tarantello; A Sobolev-Hardy inequality with applications to a nonlinear elliptic equation arising in astrophysics, Arch. Ration. Mech. Anal. 163(2002), 259-293. [5] L. Caffarelli, R. Kohn, L. Nirenberg; First order interpolation inequality with weights, Compositio Math., 53 (1984), 259-275. [6] H. Egnell; Positive solutions of semilinear equations in cones, Trans. Amer. Math. Soc., 11 (1992), 191-201. [7] I. Fabbri, G. Mancini, K. Sandeep; Classification of solutions of a critical Hardy Sobolev operator, Jour. Diff. Equa., 224 (2006), 258-276. [8] M. Ghergu, V. Radulescu; Singular elliptic problems with lack of compactness, Annali di Matematica Pura ed Applicata, 185 (2006), 63-79. [9] N. Ghoussoub, X. S. Kang; Hardy-Sobolev critical elliptic equations with boundary singularities, Ann. Inst. H. Poincaré Anal. Non Linéaire, 21 (2004), 769-793. [10] N. Ghoussoub, F. Robert; The effect of curvature on the best constant in the Hardy-Sobolev inequalities, Geom. Funct. Anal., 16 (2006), 1201-1245. [11] N. Ghoussoub, F. Robert; Concentration estimates for Emden-Fowler equations with boundary singularities and critical growth, IMRP Int. Math. Res. Pap., 21867 (2006), 1-85. [12] N. Ghoussoub, F. Robert; Elliptic equations with critical growth and a large set of boundary singularities, Trans. Amer. Math. Soc. 361 (2009), 4843-4870. 16 J. YANG, L. WU EJDE-2013/109 [13] C. H. Hsia, C. S. Lin, H. Wadade; Revisiting an idea of Brézis and Nirenberg, J. Funct. Anal., 259 (2010), 1816-1849. [14] Haiyang He, Jianfu Yang; Positive solutions for critical elliptic systems in non-contractible domain, Nonlinear Anal. TMA 70 (2009), 952-973. [15] P. L. Lions; The concentration-compactness principle in the calculus of variations. The locally compact case, Ann. Inst. H. Poincaré Anal. Non Linéaire, 1 (1984), 109-145 and 223-283. [16] C.S. Lin, H. Wadade; Minimizing problems for the Hardy-Sobolev type inequality with the sigularity on the boundary, preprint 2011. [17] V. G. Maz’ya; Sobolev spaces, Springer Series in Soviet Mathematics, Springer-Verlag, Berlin, 1985. [18] V. Radulescu, D. Smets, M. Willem; Hardy-Sobolev inequalities with remainder terms, Topol. Meth. Nonlin. Anal., 20 (2002), 145-149. [19] G. Talenti; Best constant in Sobolev inequality, Ann. Mat. Pura Appl., 110 (1976), 353-372. [20] Zhongwei Tang; Sign-changing solutions of critical growth nonlinear elliptic systems, Nonlinear Anal. TMA, 64 (2006), 2480-2491. Jianfu Yang Department of Mathematics, Jiangxi Normal University, Nanchang, Jiangxi 330022, China E-mail address: jfyang 2000@yahoo.com Linli Wu Department of Mathematics, Jiangxi Normal University, Nanchang, Jiangxi 330022, China E-mail address: llwujxsd@sina.com