New York Journal of Mathematics New York J. Math. 20 (2014) 627–643. Compactness of products of Hankel operators on convex Reinhardt domains in C2 Željko Čučković and Sönmez Şahutoğlu Abstract. Let Ω be a piecewise smooth bounded convex Reinhardt domain in C2 . Assume that the symbols φ and ψ are continuous on Ω and harmonic on the disks in the boundary of Ω. We show that if the product of Hankel operators Hψ∗ Hφ is compact on the Bergman space of Ω, then on any disk in the boundary of Ω, either φ or ψ is holomorphic. Contents 1. Introduction 2. Some background information and lemmas 3. Proof of Theorem 3 References 627 629 635 641 1. Introduction This paper is a sequel to our two previous papers [ČŞ09, ČŞ10] on compactness of Hankel operators on Bergman spaces of domains in Cn . In the first paper we studied compactness of a single Hankel operator with a smooth symbol on quite general domains. We note that in this paper smooth means C ∞ -smooth. We used ∂ methods to relate the compactness property of Hankel operators to the behavior of the symbol on the analytic disks in the boundary of the domain. The most complete result is the following theorem in C2 . Here Hφ denotes the Hankel operator on the Bergman space A2 (Ω) with a symbol φ. Furthermore, ∂Ω and D denote the boundary of Ω and the open unit disk in the complex plane, respectively. Theorem 1 ([ČŞ09]). Let Ω be a smooth bounded convex domain in C2 and φ ∈ C ∞ (Ω). Then Hφ is compact on A2 (Ω) if and only if φ◦f is holomorphic for any holomorphic mapping f : D → ∂Ω. Received December 12, 2013. 2010 Mathematics Subject Classification. Primary 47B35; Secondary 32A07. Key words and phrases. Hankel operators, analytic disks, Reinhardt domains. ISSN 1076-9803/2014 627 628 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU In the second paper we studied compactness of products of two Hankel operators on the polydisk. Notable is the absence of ∂ methods: the domain is simple enough to be treated by reducing the dimension by one. For simplicity, we state the main result in C2 only. Theorem 2 ([ČŞ10]). Let Ω be the bidisk in C2 and the symbols φ, ψ ∈ C(Ω) such that φ◦f and ψ ◦f are harmonic for any holomorphic mapping f : D → ∂Ω. Then Hψ∗ Hφ is compact on A2 (Ω) if and only if for any holomorphic function f : D → ∂Ω, either φ ◦ f or ψ ◦ f is holomorphic. In this paper we treat domains that are more general than a polydisk (see Theorem 3). A domain Ω ⊂ Cn is called Reinhardt if (z1 , . . . , zn ) ∈ Ω and θ1 , . . . , θn ∈ R imply that (eiθ1 z1 , . . . , eiθn zn ) ∈ Ω. Namely, the domain Ω is circular in each variable. The ball and the polydisk are the best known examples of Reinhardt domains. The following theorem is the main result of our paper. As before, the analyticity of the symbols is a necessary condition for compactness of the product of Hankel operators, provided that their symbols are harmonic on the disks in the boundary. Theorem 3. Let Ω be a piecewise smooth bounded convex Reinhardt domain in C2 . Assume that the symbols φ, ψ ∈ C(Ω) are such that φ ◦ f and ψ ◦ f are harmonic for every holomorphic function f : D → ∂Ω. If Hψ∗ Hφ is compact on A2 (Ω) then for every holomorphic function f : D → ∂Ω either φ ◦ f or ψ ◦ f is holomorphic. The proof of Theorem 3 uses convexity and rotational symmetry of the domain in a significant way. If there is a disk ∆ in the boundary of a convex Reinhardt domain Ω then there are disks in Ω nearby ∆ of at least the same size. Furthermore, these disks “converge” to ∆. This geometric property is an important ingredient in our proof. Remark 1. Even though Theorem 1 is stated for symbols that are smooth up to the boundary and domains with smooth boundaries, the proof shows that the theorem is still true under reasonably weaker smoothness assumptions. In the case of the polydisk Le [Le10] studied compactness of Hankel operators with symbols continuous on the closure of the polydisk. Remark 2. Products of Hankel operators can be viewed as semicommutators of Toeplitz operators. Several authors have studied compactness of these semicommutators on the unit disk D and the polydisk Dn . Zheng [Zhe89] characterized compact semicommutators of Toeplitz operators with symbols that are harmonic on D. Later Ding and Tang [DT01], Choe, Koo, and Lee [CKL04], and Choe, Lee, Nam, and Zheng [CLNZ07] extended this result to semicommutators of Toeplitz operator acting on the Bergman space of Dn with the assumption that the symbols are pluriharmonic functions on Dn . Notice that the symbols in Theorem 2 are assumed to be continuous up to the boundary but pluriharmonic on the disks in the boundary of Ω only. COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS 629 Remark 3. The class of domains to which Theorem 3 applies includes many more domains other than the bidisk. For example, it√includes the intersection of Reinhardt domains such as (D × D) ∩ B(0, (1 + 2)/2) where B(p, r) denotes the ball centered at p with radius r. Remark 4. If there is no disk in the boundary of a convex domain then the ∂-Neumann operator is compact (see [FS98, Theorem 1.1] or [Str10, Theorem 4.26]); in turn, this implies that the Hankel operator with a symbol that is continuous on the closure of the domain is compact (see [Str10, Proposition 4.1]). Hence, if a bounded convex domain does not have a disk in the boundary then the product of Hankel operators with symbols continuous on the closure of the domain is compact. For more information about Reinhardt domains we refer the reader to [JP08, Kra01, Ran86]. 2. Some background information and lemmas Let Ω be a bounded domain in Cn and A2 (Ω) denote the Bergman space, the set of holomorphic functions that are square integrable on Ω with respect to the Lebesgue measure V . Unless we integrate on a subdomain of Ω, the norm k.kL2 (Ω) is denoted by k.k and the complex inner product h., .iL2 (Ω) by h., .i. Let P Ω denote the Bergman projection on Ω, the orthogonal projection from L2 (Ω) onto A2 (Ω). The Toeplitz and Hankel operators with symbol φ ∈ L∞ (Ω) are defined on A2 (Ω) by TφΩ f = P Ω (φf ) and HφΩ f = φf − P Ω (φf ), respectively. Notice that the range of HφΩ is a subspace of the orthogonal complement of A2 (Ω) in L2 (Ω). Then one can define the product of two Hankel operators with symbols ψ and φ as (HψΩ )∗ HφΩ : A2 (Ω) → A2 (Ω), where (HψΩ )∗ denotes the Hilbert space adjoint of HψΩ . When it is clear from the context on which domain we are working on, we will omit the domain superscripts on the operators P, Tφ , and Hφ . It is well known that this product can be written as a semicommutator of Toeplitz operators. Namely, (1) Hψ∗ Hφ = Tψφ − Tψ Tφ . For more information about these operators we suggest the reader consult [Zhu07, Axl88]. We now present and prove several key lemmas that will be used in the proof of the main theorem. They represent our idea that geometry, analysis, and approximation intertwine in an interesting manner and they enable us to prove the main result in this paper. The first lemma is simple and it allows us to rewrite the product of two Hankel operators in a different way than the semicommutator of Toeplitz operators. 630 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU Lemma 1. Let Ω be a domain in Cn and φ, ψ ∈ L∞ (Ω). Then Hψ∗ Hφ = P Mψ Hφ where Mψ denotes the product by ψ. Proof. Let f, g ∈ A2 (Ω). Then we have ∗ Hψ Hφ f, g = hHφ f, Hψ gi = hHφ f, ψgi = ψHφ f, g = P ψHφ f, g . Therefore, Hψ∗ Hφ = P Mψ Hφ . The next lemma gives us an important information about the disks in the boundary of complete Reinhard domains in C2 . It shows that piecewise smooth bounded complete Reinhardt domains in C2 can have vertical or horizontal disks only. This will allow us to use the slicing method to approach the disks by horizontal and vertical slices of the domain itself. Lemma 2. Let Ω be a piecewise smooth bounded complete Reinhardt domain in C2 and let F = (f, g) : D → ∂Ω be a holomorphic function. Then either f or g is constant. Proof. Let F (z) = (f (z), g(z)) be an analytic disk in the boundary. If |f (z)| and |g(z)| are constant then F is constant. Therefore, there are no nontrivial disks on the singular part of the boundary. Now assume that there is an analytic disk in the boundary away from singular points. Then we can assume that the domain is smooth and it is given by ρ(|z|, |w|). By convexity if there is a disk then it must be an affine disk (see, for example, [ČŞ09, Lemma 2] and [FS98, Proposition 3.2]). So there exist a, b, c, d ∈ C such that the set {(aξ + b, cξ + d) ∈ C2 : ξ ∈ D} is a disk in the boundary. We may also assume that the disk does not intersect the coordinate axes. In other words, we may assume that |aξ + b| > 0 and |cξ + d| > 0. Computing the Laplacian of r(ξ) = ρ(|aξ + b|, |cξ + d|) where ξ ∈ D and we get 0=4 ∂2r |a|2 |c|2 (ξ) = Hρ (r(ξ); W ) + ρx (r(ξ)) + ρy (r(ξ)) |aξ + b| |cξ + d| ∂ξ∂ξ where W = a aξ + b aξ + b 1/2 ,c cξ + d cξ + d 1/2 ! and Hρ (p; X) is the (real) Hessian of ρ applied to the vector X at the point p. Let (|p|, |q|) be a boundary point of Z = {(x, y) ∈ R2 : x ≥ 0, y ≥ 0, ρ(x, y) < 0}. Then the rectangle R(|p|,|q|) ⊂ R2 formed by (0, 0), (|p|, 0), (0, |q|), and (|p|, |q|) is inside Z and (ρx (|p|, |q|), ρy (|p|, |q|)) is normal to the boundary of Z at (|p|, |q|). If ρx (|p|, |q|) < 0 and ρy (|p|, |q|) > 0 (or ρx (|p|, |q|) > 0 and ρy (|p|, |q|) < 0) then the tangential vector to ∂Z at (|p|, |q|) has components COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS 631 with the same sign. Then R(|p|,|q|) ∩ ∂Z is nonempty which in turn implies that R(|p|,|q|) \ Z is nonempty. Similarly, if ρx (|p|, |q|) < 0 and ρy (|p|, |q|) < 0 then R(|p|,|q|) cannot be contained in Z. Hence, ρx ≥ 0, ρy ≥ 0 and ρy +ρy > 0, and Hρ (r(ξ); W ) ≥ 0 for any W ∈ C2 . Therefore, either a = 0 or c = 0. That is, the disk is either horizontal or vertical. Assume that F (z) = (f (z), g(z)) is a non-trivial analytic disk through a singular point in the boundary. That is, F is nonconstant and there exists p ∈ D such that F 0 (p) = 0. Then by the previous part the smooth part of the disk is either horizontal or vertical. If it is horizontal then there exists an open set U ⊂ D such that |g| is constant on U. The identity principle implies that g is constant on D. Hence, the whole disk is horizontal. As mentioned earlier, we use slicing of the domain and the resulting disks to approach horizontal or vertical disks in the boundary. The following lemma will enable us to do that in the sense that projections of these disks onto the complex plane approach the projection of the disk in the boundary. Even though this lemma is stated for horizontal disks, the result holds for vertical disks as well. Lemma 3. Let Ω be a bounded convex Reinhardt domain in C2 and ∆w = {z ∈ C : (z, w) ∈ Ω} for w ∈ C. Assume that ∅ 6= ∆w0 × {w0 } ⊂ ∂Ω for some w0 ∈ C, {wj } is a sequence of complex numbers that converges to w0 , and ∆wj is nonempty for all j. Then limj→∞ rj = r0 where rj denotes the radius of the disk ∆wj for j = 0, 1, 2, . . . . Proof. Since Ω is a convex Reinhardt domain it is also complete. Hence, all of these disks are centered at the origin and we want to prove that {rj } converges to r0 , the radius of ∆w0 . In addition, since the domain is also convex one can show that rj ≥ r0 for j ≥ 1. Hence lim inf j→∞ rj ≥ r0 On the other hand, if lim supj→∞ rj > r0 we can choose pk ∈ ∆wjk such that |pk | = rjk and limk→∞ |pk | = lim supj→∞ rj . Then the sequence {(pk , wjk )} ⊂ ∂Ω has a subsequence that converges to a point (p, w0 ) ∈ ∂Ω. This means that p ∈ ∆w0 and lim sup rj = lim |pk | = |p| ≤ r0 . j→∞ Therefore, limj→∞ rj = r0 . k→∞ The convergence of the disk in Lemma 3 brings the natural question of a convergence of the corresponding Bergman kernels and projections. Let K be a set in Cn and TK denote the characteristic function of K. That is, TK (z) = 1 if z ∈ K and TK (z) = 0 otherwise. Also for a function f defined on a set U we let EU f denote the extension of f by 0 outside U. Lemma 4. Let ψ ∈ L2 (C). Then limr→1 EDr P Dr ψ − ED P D ψ L2 (C) = 0. 632 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU Proof. Since ψ is square integrable, for every ε > 0 there exists δ > 0 such that |r − 1| < δ implies that kψkL2 (D1+δ \D1−δ ) < ε/2. Then Dr ≤ 2kψkL2 (D1+δ \D1−δ ) ≤ ε + P D (TD\D1−δ ψ) 2 P (TDr \D1−δ ψ) 2 L (D) L (Dr ) for |r − 1| < δ. Next the proof of the lemma will be completed by showing that → 0 as r → 1. EDr P Dr (TD1−δ ψ) − ED P D (TD1−δ ψ) 2 L (C) We define Gr (z, w) = F r (z, w) − F 1 (z, w) F r (z, w) = and r > 1 − δ. We note that for (z, w) ∈ C × D1−δ , where TDr (z)r2 (r2 − zw)2 r2 π(r2 −zw)2 is the Bergman kernel for Dr . Then there exists r0 > 1 such that Gr → 0 uniformly on Dr0 × D1−δ as r → 1. For 1 − δ < r < r0 we have 2 EDr P Dr (TD1−δ ψ) − ED P D (TD1−δ ψ) 2 L (C) 2 Z Z Z r 1 = F (z, w)ψ(w)dV (w) − F (z, w)ψ(w)dV (w) dV (z) C D1−δ D1−δ !2 Z Z |Gr (z, w)||ψ(w)|dV (w) ≤ C dV (z) D1−δ ≤ kψk2L2 (D) Z Dr0 Z |Gr (z, w)|2 dV (w)dV (z). D1−δ Since Gr → 0 uniformly as r → 1 we have EDr P Dr (TD1−δ ψ) − ED P D (TD1−δ ψ) L2 (C) as r → 1. →0 The lemma above and [Kra01, Lemma 1.4.1] imply the following corollary. Corollary 1. Let ψ ∈ L2 (C) and K be a compact subset of D. Then {P Dr ψ} converges uniformly to P D ψ on K as r → 1. The following lemma is stated for bounded convex domains because these domains are the focus of our paper. However, similar ideas can be used for Ap spaces on starlike domains. This has been done for Ap (D) in [DS04, Theorem 3, p.30]. Lemma 5. Let U be a bounded convex domain in C and f ∈ A2 (U ). Then for any ε > 0 there exists a holomorphic polynomial h such that kf −hkL2 (U ) < ε. COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS 633 Proof. Without loss of generality we may assume that U contains the origin. Let us define fr (z) = f (rz) for r ∈ (0, 1) and assume that ε > 0 is given. Then fr ∈ A2 (U ) ∩ C(U ) and one can show that there exists 0 < r < 1 such that ε kf − fr0 k < . 2 This can be seen as follows: First there exists 0 < δ < 1 so that ε kf kL2 (U \δU ) < . 6 The uniform continuity of f on compact subsets of U implies that there exists 12 < r < 1 such that 1+δ ε U < p sup |f (z) − f (rz)| : z ∈ 2 6 V (U ) where V (U ) denotes the volume of U. Then we have kf − fr k ≤ kf − fr kL2 (( 1+δ )U ) + kf kL2 (U \( 1+δ )U ) + kfr kL2 (U \( 1+δ )U ) 2 2 2 ε ε 1 ≤ + + kf kL2 (U \δU ) 6 6 r 2ε < . 3 On the other hand, Mergelyan’s theorem implies that there exists a holomorphic polynomial h such that ε sup{|fr (z) − h(z)| : z ∈ U } < p . 3 V (U ) Then we have kf − hk ≤ kf − fr k + kfr − hk ≤ 2ε ε + = ε. 3 3 This completes the proof of Lemma 5. The next lemma shows that when concentric disks converge, then not only the kernels and the Bergman projections converge but also the products of Hankel operators converge “weakly”. Lemma 6. For r > 0 let Dr = {z ∈ C : |z| < r}, f1 and f2 be entire functions, and φ, ψ ∈ C(C). Then D E D E Dr Dr lim HφDr (f1 ), HψDr (f2 ) = Hφ 0 (f1 ), Hψ 0 (f2 ) . r→r0 Dr Dr0 Proof. First assume that r0 ≤ r. For any 0 < δ < r0 we have D E D E Dr0 Dr0 H Dr (f1,)H Dr (f2 ) − Hφ (f1 ), Hψ (f2 ) ψ φ Dr Dr0 D E D E Dr0 Dr = φf1 , Hψ (f2 ) − φf1 , Hψ (f2 ) Dr Dr 0 634 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU D D E E Dr0 Dr − φf1 , P (ψf2 ) ≤ hφf1 , ψf2 iDr \Dr + φf1 , P (ψf2 ) 0 Dr0 Dr D D E E Dr0 Dr − φf1 , P (ψf2 ) ≤ hφf1 , ψf2 iDr \Dr + φf1 , P (ψf2 ) 0 Dr0 −δ Dr0 −δ D D E E Dr Dr0 . + φf1 , P (ψf2 ) + φf1 , P (ψf2 ) Dr \Dr0 −δ Dr0 \Dr0 −δ Therefore, we have D E D E Dr0 Dr0 Dr Dr (2) Hφ (f1,)Hψ (f2 ) − Hφ (f1 ), Hψ (f2 ) Dr0 Dr D E ≤ hφf1 , ψf2 iDr \Dr + φf1 , P Dr (ψf2 ) − P Dr0 (ψf2 ) 0 + kφf1 kL2 (Dr \Dr 0 −δ ) kψf2 kL2 (Dr ) + kφf1 kL2 (Dr0 \Dr Dr0 0 −δ −δ ) kψf2 kL2 (Dr0 ) . Then for ε > 0 one can choose 0 < δ1 < min{1, r0 } so that kφf1 kL2 (Dr 0 +δ1 \Dr0 −δ1 ) ≤ ε. Furthermore, by Corollary 1 we can choose 0 < δ2 < δ1 so that r0 ≤ r ≤ r0 + δ2 implies that Dr Dr0 P (ψf2 )(z) − P (ψf2 )(z) ≤ ε for z ∈ Dr0 −δ1 and hφf1 , ψf2 iDr \Dr ≤ ε. 0 Therefore, for r0 ≤ r ≤ r0 + δ2 we have D E E D Dr0 Dr0 H Dr (f1 ), H Dr (f2 ) − Hφ (f1 ), Hψ (f2 ) ψ φ Dr0 Dr ≤ ε 1 + kψf2 kL2 (Dr0 ) + kψf2 kL2 (Dr ) D E D D r r + φf1 , P (ψf2 ) − P 0 (ψf2 ) Dr0 −δ1 ≤ ε 1 + kψf2 kL2 (Dr0 ) + kψf2 kL2 (Dr0 +1 ) √ + εr0 πkφf1 kL2 (Dr0 ) . We note that by the Cauchy–Schwarz inequality we used the following inequality above D E √ D D r r φf1 , P (ψf2 ) − P 0 (ψf2 ) ≤ εr0 πkφf1 kL2 (D ) . r0 Dr −δ 0 Therefore, there exists a constant K > 0 independent of r D E D E Dr0 Dr0 H Dr (f1 ), H Dr (f2 ) − H (f ), H (f ) 1 2 ψ φ ψ φ Dr Dr 0 for r0 ≤ r ≤ r0 + δ2 . and ε so that ≤ εK1 COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS 635 Similarly if r ≤ r0 equation (2) is valid for r and r0 interchanged. For ε > 0 we choose 0 < δ3 < min{1, r0 /2} such that kφf1 kL2 (Dr0 \Dr 0 −δ3 ) < ε. By Corollary 1 we choose 0 < δ4 < δ23 so that so that r0 − δ4 < r ≤ r0 implies that Dr P (ψf2 )(z) − P Dr0 (ψf2 )(z) ≤ ε for z ∈ Dr − δ3 0 2 and hφf1 , ψf2 iDr 0 \Dr ≤ ε. Therefore, there exists a constant K2 > 0 independent of r and ε such that r0 − δ4 ≤ r ≤ r0 implies that D E E D Dr0 Dr0 ≤ εK2 . H Dr (f1 ), H Dr (f2 ) − Hφ (f1 ), Hψ (f2 ) ψ φ Dr0 Dr E E D D Dr Dr . = Hφ 0 (f1 ), Hψ 0 (f2 ) Thus, we have lim HφDr (f1 ), HψDr (f2 ) r→r0 Dr Dr0 3. Proof of Theorem 3 Proof of Theorem 3. Assume that Hψ∗ Hφ is a compact operator and there exists an analytic disk ∆ in ∂Ω (if not we are done), and two symbols φ and ψ that are not holomorphic “along” ∆. Namely, there exists a holomorphic function f : D → ∆ so that neither φ ◦ f nor ψ ◦ f is holomorphic on D. Since Ω is a convex Reinhardt bounded domain Lemma 2 implies that the disk ∆ is either horizontal or vertical. So without loss of generality we may assume that ∆ is horizontal and [ Ω= (∆w × {w}) w∈H where H ⊂ C, ∆w = {z ∈ C : (z, w) ∈ Ω} is a disk in C centered at the origin, and ∆ = ∆w0 for some w0 ∈ ∂H. By using a linear holomorphic map, (z, w) → (z, eiθ0 (w − w0 )) for some θ0 ∈ R, we translate the domain Ω into {(z, w) ∈ C2 : Im(w) < 0}. Hence withoutSloss of generality we may assume that H ⊂ {w ∈ C : Im(w) < 0} and Ω = w∈H (∆w × {w}) where ∆w ’s are disks centered at the origin and ∆ = ∆0 . Let us extend φ(z, 0) and ψ(z, 0) as continuous functions on C and call the extensions φ0 (z) and ψ0 (z). Since φ0 and ψ0 are harmonic and not holomorphic on ∆0 , Theorem 5 in [Zhe89] (see also [AČ01, Corollary 6]) implies that the product Hψ∆00 ∗ Hφ∆00 is a nonzero operator. Then there exist f1 , f2 ∈ A2 (∆0 ) such that Z Hφ∆00 (f1 )(z)Hψ∆00 (f2 )(z)dV (z) 6= 0. ∆0 636 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU Then by Lemma 5 we can choose f1 and f2 to be holomorphic polynomials (of one variable). For convenience, in the following calculations we will abuse the notation as follows: we will assume that φ0 , ψ0 , f1 , and f2 are functions of z only (or functions of (z, w) but independent of w). We remind the reader that in the computations below, the Bergman projection on the disk ∆w is denoted by P ∆w and Hη∆w (f ) = ηf − P ∆w (ηf ) for f ∈ A2 (∆w ) and η ∈ L∞ (∆w ). We note that functions (z, w) → P ∆w (ηf )(z) and (z, w) → Hη∆w (f )(z) are continuous on Ω. In case of the first function this can be seen as follows: |P ∆w0 (ηf )(z0 ) − P ∆w (ηf )(z)| ≤ |P ∆w0 (ηf )(z0 ) − P ∆w (ηf )(z0 )| Z + |K∆w (z0 , ξ) − K∆w (z, ξ)||η(ξ)f (ξ)|dV (ξ). ∆w As (z, w) goes to (z0 , w0 ) in Ω, the first term on the right hand side goes to zero by Corollary 1 and the second term goes to zero because sup{|K∆w (z0 , ξ) − K∆w (z, ξ)| : ξ ∈ ∆w } goes to zero. Also Fubini’s Theorem implies that these functions are square integrable. Let gj ∈ A2 (H) which will be specified later. For fixed w ∈ H and any z ∈ ∆w HφΩ0 (f1 gj )(z, w) = φ0 (z, w)f1 (z)gj (w) − P Ω (φ0 f1 gj )(z, w) and Hφ∆0w(.,w) (f1 )(z) = φ0 (z, w)f1 (z) − P ∆w (φ0 (., w)f1 )(z) imply that HφΩ0 (f1 gj )(z, w) − gj (w)Hφ∆0w (f1 ) = P Ω (φ0 f1 gj )(z, w) − gj (w)P ∆w (φ0 f1 )(z) is holomorphic in z on ∆w . Using Lemma 1 in the first equality below we get Z (HψΩ0 )∗ HφΩ0 (f1 gj ))(z, w)f2 (z)dV (z) ∆w Z = P Ω (ψ 0 HφΩ0 (f1 gj ))(z, w)f2 (z)dV (z) ∆w Z = ψ0 (z, w)HφΩ0 (f1 gj )(z, w)f2 (z)dV (z) ∆w Z − (I − P Ω )(ψ 0 HφΩ0 (f1 gj ))(z, w)f2 (z)dV (z) ∆w COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS Z 637 HφΩ0 (f1 gj )(z, w)Hψ∆0w (f2 )(z)dV (z) = ∆w Z + Z∆w HφΩ0 (f1 gj )(z, w)P ∆w (ψ0 f2 )(z)dV (z) (I − P Ω )(ψ 0 HφΩ0 (f1 gj ))(z, w)f2 (z)dV (z) Z Hφ∆0w (f1 )(z)Hψ∆0w (f2 )(z)dV (z) = gj (w) Z ∆w HφΩ0 (f1 gj )(z, w)P ∆w (ψ0 f2 )(z)dV (z) + ∆w Z − (I − P Ω )(ψ 0 HφΩ0 (f1 gj ))(z, w)f2 (z)dV (z). − ∆w ∆w If we multiply both sides by gj (w) and integrate over H we get Ω Hφ0 (f1 gj ), HψΩ0 (f2 gj ) Z Z 2 = |gj (w)| Hφ∆0w (f1 )(z)Hψ∆0w (f2 )(z)dV (z)dV (w) H ∆w Z + HφΩ0 (f1 gj )(z, w)P ∆w (ψ0 f2 )(z)gj (w)dV (z, w) Ω Z − (I − P Ω )(ψ 0 HφΩ0 (f1 gj ))(z, w)f2 (z)gj (w)dV (z, w). Ω We note that the last integral on the right hand side above is zero. Hence, we have Ω (3) Hφ0 (f1 gj ), HψΩ0 (f2 gj ) Z Z = |gj (w)|2 Hφ∆0w (f1 )(z)Hψ∆0w (f2 )(z)dV (z)dV (w) H ∆w Z + HφΩ0 (f1 gj )(z, w)P ∆w (ψ0 f2 )(z)gj (w)dV (z, w). Ω Our next goal is to show that the second integral on the right hand side of (3) goes to zero while the first one does not as j goes to infinity. Let h be an entire function on C. Then Z HφΩ0 (f1 gj )(z, w)P ∆w (ψ0 f2 )(z)gj (w)dV (z, w) Ω Z = HφΩ0 (f1 gj )(z, w)h(z)gj (w)dV (z, w) Ω Z + HφΩ0 (f1 gj )(z, w)(P ∆w (ψ0 f2 )(z) − h(z))gj (w)dV (z, w) Z Ω = HφΩ0 (f1 gj )(z, w)(P ∆w (ψ0 f2 )(z) − h(z))gj (w)dV (z, w). Ω 638 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU Using the Cauchy–Schwarz inequality we have Z |HφΩ0 (f1 gj )(z, w)(P ∆w (ψ0 f2 )(z) − h(z))gj (w)dV (z, w)| Ω ≤ HφΩ0 (f1 gj ) Z |(P ∆w 1/2 (ψ0 f2 )(z) − h(z))gj (w)| dV (z, w) . 2 Ω a Now we choose gj (w) = wαjj such that aj → 0, αj → 1− , and kgj kH = 1 as j → ∞. Then one can show that Ω H (4) φ−φ0 (gj ) ≤ k(φ − φ0 )gj k → 0 as j → ∞ because gj goes to 0 uniformly on any compact set away from ∆0 and φ − φ0 = 0 on ∆0 . Let ε > 0 be fixed. Then there exists a set Lε b ∆0 such that ε kψ0 f2 kL2 (∆0 \Lε ) ≤ . 2 Furthermore, Lemma 5 and [Kra01, Proposition 1.4.1] imply that there exists an entire function h such that ∆ P 0 (ψ0 f2 ) − h 2 ≤ε L (∆0 ) and ε sup{|P ∆0 (ψ0 f2 )(z) − h(z)| : z ∈ Lε } ≤ . 2 Then by Lemma 3 we can choose δ1 > 0 such that |w| < δ1 implies that Lε b ∆w . Furthermore, δ1 can be chosen so that ε kψ0 f2 kL2 (∆w \∆0 ) + khkL2 (∆w \∆0 ) ≤ . 2 Finally, Lemma 3 and Corollary 1 imply that there exists δ2 > 0 such that ε sup{|P ∆0 (ψ0 f2 )(z) − P ∆w (ψ0 f2 )(z)| : z ∈ Lε } ≤ , 2 for |w| < δ2 and Lemma 3 and Lemma 4 imply that there exists δ3 > 0 such that E∆w P ∆w (TLε ψ0 f2 ) − E∆ P ∆0 (TLε ψ0 f2 ) 2 ≤ε 0 L (C) for |w| < δ3 . If we put all these together we have the following: for ε > 0 there exist δ = min{δ1 , δ2 , δ3 } > 0, a set Lε b ∆0 , and an entire function h such that |w| < δ implies that: (i) Lε b ∆w , kψ0 f2 kL2 (∆w \Lε ) ≤ ε, and khkL2 (∆w \∆0 ) ≤ ε/2, (ii) P ∆0 (ψ0 f2 ) − hL2 (∆0 ) ≤ ε, (iii) sup{|P ∆w (ψ0 f2 )(z) − h(z)| : z ∈ Lε } ≤ ε, (iv) E∆w P ∆w (TLε ψ0 f2 ) − E∆0 P ∆0 (TLε ψ0 f2 )L2 (C) ≤ ε. COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS 639 Now we choose j0 so that Z |P |gj (w)| < ε 1 + ∆w 2 − 1 2 (ψ0 f2 )(z) − h(z)| dV (z, w) Ω for |w| ≥ δ and j ≥ j0 . Let us define Kδ = ∪|w|≥δ ∆w ⊂ Ω. Then we have Z |(P ∆w (ψ0 f2 )(z) − h(z))gj (w)|2 dV (z, w) Ω Z |(P ∆w (ψ0 f2 )(z) − h(z))gj (w)|2 dV (z, w) = Kδ Z |(P ∆w (ψ0 f2 )(z) − h(z))gj (w)|2 dV (z, w) + Ω∩(Lε ×B(0,δ)) Z + |(P ∆w (ψ0 f2 )(z) − h(z))gj (w)|2 dV (z, w) Ω\(Kδ ∪(Lε ×B(0,δ)) Z 2 . sup{|gj (w)| : |w| ≥ δ} |P ∆w (ψ0 f2 )(z) − h(z)|2 dV (z, w) Kδ Z + sup{|P ∆w (ψ0 f2 )(z) − h(z)|2 : z ∈ Lε , |w| ≤ δ} |gj (w)|2 dV (w) H Z Z + |gj (w)|2 |P ∆w (ψ0 f2 )(z) − h(z)|2 dV (z)dV (w) |w|<δ ∆w \Lε Z Z . ε2 + |gj (w)|2 |P ∆w (ψ0 f2 )(z) − h(z)|2 dV (z)dV (w). |w|<δ ∆w \Lε We note that (iii) is used in the last inequality. Then Z |P ∆w (ψ0 f2 )(z) − h(z)|2 dV (z) ∆w \Lε Z . |P ∆w ((1 − TLε )ψ0 f2 )(z)|2 dV (z) ∆w \Lε Z |P ∆w (TLε ψ0 f2 )(z) − E∆0 P ∆0 (TLε ψ0 f2 )(z)|2 dV (z) + ∆ \L Z w ε + |E∆0 P ∆0 (TLε ψ0 f2 )(z) − E∆0 P ∆0 (ψ0 f2 )(z)|2 dV (z) ∆ \L Z w ε + |E∆0 P ∆0 (ψ0 f2 )(z) − h(z)|2 dV (z). ∆w \Lε Let |w| < δ. Then by (i) we have ∆ P w ((1 − TLε )ψ0 f2 )2 2 ≤ k(1 − TLε )ψ0 f2 )k2L2 (∆w ) ≤ ε2 L (∆w \Lε ) and by (iv) we have 640 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU Z ∆w \Lε |P ∆w (TLε ψ0 f2 )(z) − E∆0 P ∆0 (TLε ψ0 f2 )(z)|2 dV (z) 2 ≤ E∆w P ∆w (TLε ψ0 f2 ) − E∆0 P ∆0 (TLε ψ0 f2 )L2 (C) ≤ε2 . By (i) again and the fact that ∆0 ⊂ ∆w we have Z |E∆0 P ∆0 (TLε ψ0 f2 )(z) − E∆0 P ∆0 (ψ0 f2 )(z)|2 dV (z) ∆w \Lε ≤ k(1 − TLε )ψ0 f2 )k2L2 (∆0 ) ≤ ε2 . Furthermore, (i) and (ii) imply that Z |E∆0 P ∆0 (ψ0 f2 )(z) − h(z)|2 dV (z) ∆w \Lε Z Z = |P ∆0 (ψ0 f2 )(z) − h(z)|2 dV (z) + ∆0 \Lε |h(z)|2 dV (z) . ε2 . ∆w \∆0 Therefore, we have Z |P ∆w (ψ0 f2 )(z) − h(z)|2 dV (z) . ε2 for |w| ≤ δ. ∆w \Lε R Furthermore, since H |gj (w)|2 dV (w) = 1 and Ω is bounded there exists a constant C > 0 such that kgj k < C. Therefore, Z HφΩ0 (f1 gj )(z, w)P ∆w (ψ0 f2 )(z)gj (w)dV (z, w) → 0 as j → ∞. Ω Now we will show that the first integral on the right hand side of (3) stays away from zero as j goes to infinity. We remind the reader that f1 and f2 are holomorphic polynomials such that Z Hφ∆00 (f1 )(z)Hψ∆00 (f2 )(z)dV (z) 6= 0. ∆0 Therefore, by Lemma 6, without loss of generality and by choosing a smaller δ > 0, if necessary, we may assume that there exists β > 0 such that Z ∆w ∆w Re Hφ0 (f1 )(z)Hψ0 (f2 )(z)dV (z) > β ∆w for |w| < δ. The mass of gj “accumulates” at the origin in the sense that Z |gj (w)|2 dV (w) = 1 H COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS 641 for all j while gj (w) → 0 as w stays away from ∆0 . Then there exists j0 so that j ≥ j0 implies that Z β ∆w 2 ∆w |gj (w)| Hφ0 (f1 )(z)Hψ0 (f2 )(z)dV (z)dV (w) < . 4 Kδ On the other hand, there exists j1 such that ! Z Re |gj (w)|2 Hφ∆0w (f1 )(z)Hψ∆0w (f2 )(z)dV (z, w) Ω\Kδ Z |gj (w)|2 dV (w) > >β {w∈H:|w|<δ} β 2 for j ≥ j1 . Therefore, for j ≥ max{j0 , j1 } we have Z β ∆w 2 ∆w Re |gj (w)| Hφ0 (f1 )(z)Hψ0 (f2 )(z)dV (z, w) > . 2 Ω This shows that the first integralD on the right hand side E of (3) stays away Ω Ω from zero. Hence, by (3) again, Hφ0 (f1 gj ), Hψ0 (f2 gj ) does not converge to zero as j goes to infinity. D E ∗ Now we we will show that HψΩ HφΩ (f1 gj ), f2 gj does not converge to zero which contradicts the assumption that Hψ∗ Hφ is compact. D E ∗ HψΩ HφΩ (f1 gj ), f2 gj = HφΩ (f1 gj ), HψΩ (f2 gj ) . H Ω (f1 gj ), H Ω (f2 gj φ0 ψ0 + k(φ − φ0 )f1 gj kkψ0 f2 gj k + kφ0 f1 gj kk(ψ − ψ0 )f2 gj k + k(φ − φ0 )f1 gj kk(ψ − ψ0 )f2 gj k. We note that by (4) the last three terms on the right hand side of the inequality above go to zero as j goes toD ∞ and ∗ we just showed E that the Ω Ω first term stays away from zero. Hence, Hψ Hφ (f1 gj ), f2 gj does not converge to zero. References [AČ01] [Axl88] [CKL04] Ahern, Patrick; Čučković, Željko. A theorem of Brown–Halmos type for Bergman space Toeplitz operators. J. Funct. Anal. 187 (2001), no. 1, 200–210. MR1867348 (2002h:47040), Zbl 0996.47037, doi: 10.1006/jfan.2001.3811. Axler, Sheldon. Bergman spaces and their operators. Surveys of some recent results in operator theory, Vol. I, 1–50, Pitman Res. Notes Math. Ser., 171. Longman Sci. Tech., Harlow, 1988. MR0958569 (90b:47048), Zbl 0681.47006. Choe, Boo Rim; Koo, Hyungwoon; Lee, Young Joo. Commuting Toeplitz operators on the polydisk. Trans. Amer. Math. Soc. 356 (2004), no. 5, 1727–1749 (electronic). MR2031039 (2004k:47054), Zbl 1060.47034, doi: 10.1090/S0002-9947-03-03430-5. 642 ŽELJKO ČUČKOVIĆ AND SÖNMEZ ŞAHUTOĞLU [CLNZ07] Choe, Boo Rim; Lee, Young Joo; Nam, Kyesook; Zheng, Dechao. Products of Bergman space Toeplitz operators on the polydisk. Math. Ann. 337 (2007), no. 2, 295–316. MR2262785 (2007k:47039), Zbl 1122.47022, doi: 10.1007/s00208-006-0034-6. [ČŞ09] Čučković, Željko; Şahutoğlu, Sönmez. Compactness of Hankel operators and analytic discs in the boundary of pseudoconvex domains. J. Funct. Anal. 256 (2009), no. 11, 3730–3742. MR2514058 (2010e:47054), Zbl 1171.47022, arXiv:0809.1901, doi: 10.1016/j.jfa.2009.02.018. [ČŞ10] Čučković, Željko; Şahutoğlu, Sönmez. Compactness of products of Hankel operators on the polydisk and some product domains in C2 . J. Math. Anal. Appl. 371 (2010), no. 1, 341–346. MR2661011 (2011g:47059), Zbl 1200.32001, arXiv:1004.0720. [DS04] Duren, Peter; Schuster, Alexander. Bergman spaces. Mathematical Surveys and Monographs, 100. American Mathematical Society, Providence, RI, 2004. x+318 pp. ISBN: 0-8218-0810-9. MR2033762 (2005c:30053), Zbl 1059.30001, doi: 10.1090/surv/100. [DT01] Xuanhao, Ding; Shengqiang, Tang. The pluriharmonic Toeplitz operators on the polydisk. J. Math. Anal. Appl. 254 (2001), no. 1, 233–246. MR1807898 (2001k:47037), Zbl 0987.47015, doi: 10.1006/jmaa.2000.7265. [FS98] Fu, Siqi; Straube, Emil J. Compactness of the ∂-Neumann problem on convex domains. J. Funct. Anal. 159 (1998), no. 2, 629– 641. MR1659575 (99h:32019), Zbl 0959.32042, arXiv:math/9711201, doi: 10.1006/jfan.1998.3317. [JP08] Jarnicki, Marek; Pflug, Peter. First steps in several complex variables: Reinhardt domains. EMS Textbooks in Mathematics. European Mathematical Society (EMS), Zürich, 2008. viii+359 pp. ISBN: 978-3-03719-049-4. MR2396710 (2009a:32001), Zbl 1148.32001, doi: 10.4171/049. [Kra01] Krantz, Steven G. Function theory of several complex variables. Reprint of the 1992 edition. AMS Chelsea Publishing, Providence, RI, 2001. xvi+564 pp. ISBN: 0-8218-2724-3. MR1846625 (2002e:32001), Zbl 1087.32001. [Le10] Le, Trieu. Compact Hankel operators on generalized Bergman spaces of the polydisc. Integral Equations Operator Theory 67 (2010), no. 3, 425–438. MR2660469 (2011i:47037), Zbl 1237.47031, arXiv:1004.1152, doi: 10.1007/s00020-010-1788-5. [Ran86] Range, R. Michael. Holomorphic functions and integral representations in several complex variables. Graduate Texts in Mathematics, 108. SpringerVerlag, New York, 1986. xx+386 pp. ISBN: 0-387-96259-X. MR0847923 (87i:32001), Zbl 0591.32002. [Str10] Straube, Emil J. Lectures on the L2 -Sobolev theory of the ∂-Neumann problem. ESI Lectures in Mathematics and Physics. European Mathematical Society (EMS), Zürich, 2010. viii+206 pp. ISBN: 978-3-03719-076-0. MR2603659 (2011b:35004), Zbl 1247.32003, doi: 10.4171/076. [Zhe89] Zheng, De Chao. Hankel operators and Toeplitz operators on the Bergman space. J. Funct. Anal. 83 (1989), no. 1, 98–120. MR0993443 (91b:47057), Zbl 0678.47026, doi: 10.1016/0022-1236(89)90032-3. [Zhu07] Zhu, Kehe. Operator theory in function spaces. Second edition. Mathematical Surveys and Monographs, 138. American Mathematical Society, Providence, RI, 2007. xvi+348 pp. ISBN: 978-0-8218-3965-2. MR2311536 (2008i:47064), Zbl 1123.47001. COMPACTNESS OF PRODUCTS OF HANKEL OPERATORS 643 University of Toledo, Department of Mathematics & Statistics, Toledo, OH 43606, USA Zeljko.Cuckovic@utoledo.edu Sonmez.Sahutoglu@utoledo.edu This paper is available via http://nyjm.albany.edu/j/2014/20-31.html.