No title

Vol. 101, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

APPROXIMATE SPECTRAL SYNTHESIS IN THE BERGMAN SPACE S. M. SHIMORIN

1. Introduction. From a hard-analysis point of view, the main result of the present paper is an approximation theorem for extremal (Bergman-inner) functions in terms of finite zero divisors. This corresponds to the famous Carathéodory-Schur theorem from the early 1900s, which states that every function in the unit ball of H ∞ (the uniform algebra of bounded holomorphic functions in the unit disk) can be approximated by finite Blaschke products, in the topology of uniform convergence on compact subsets of the disk. Along the way, we find a theorem about kernel functions for weighted Bergman spaces of the type that estimate them away from the diagonal. The motivation for these results is the study of z-invariant subspaces in the Bergman space. It is known that the lattice of z-invariant subspaces in Bergman spaces has a very complicated structure. But one may single out among all z-invariant subspaces those of simplest nature, the zero-based ones. What z-invariant subspaces can be approximated by zero-based ones? What z∗ -invariant subspaces can be approximated by finite-dimensional ones? In this paper, we answer these questions (see Theorems 1 and 2), and we discuss some relations with rational approximation and cyclic vectors for the backward shift. Let X be a Banach space of functions analytic in the unit disk D = {z ∈ C : |z| < 1} of the complex plane C. Suppose that X is invariant with respect to the operator Mz of multiplication by the independent variable. Many important problems concerning the structure of the lattice of subspaces of X that are closed and invariant with respect to Mz (or, simply, z-invariant) are related to problems of spectral synthesis. Let Y be the space dual to X. Then the functionals kλ of evaluation of functions in X at the points of D, kλ : f → f (λ); λ ∈ D, are eigenvectors of the operator Mz∗ : 
λI − Mz∗ kλ = 0; (n)

the functionals kλ of evaluation of the derivatives (n)

kλ : f → f (n) (λ);

n ∈ Z+ , λ ∈ D,

Received 15 May 1997. Revision received 16 December 1998. 1991 Mathematics Subject Classification. Primary 46E20; Secondary 30H05. Author’s research partially supported by Russian Foundation for Fundamental Studies grant number N 96-01-00693. 1

2

S. M. SHIMORIN

are root vectors of Mz∗ . Here Z+ is the set of nonnegative integers. Assume that all (n) root vectors of Mz∗ are of the form kλ , n ∈ Z+ , λ ∈ D. (This is not the case, e.g., (n) if X = H ∞ (D), the space of bounded analytic functions in D.) The vectors kλ ∗ form a weak -complete system in Y . The spectral synthesis of a subspace of Y that is closed and invariant with respect to Mz∗ (or, simply, z∗ -invariant) is the process of reconstructing it, starting with these root vectors. If J ⊂ Y is a finite-dimensional z∗ -invariant subspace, then the classical Kronecker theorem states that J is the linear (n) span of the root vectors kλ contained in J . For z-invariant subspaces in X, this means that if I ⊂ X is a z-invariant subspace of finite codimension, then there exists a function v : D → Z+ (a divisor of I ) such that v(λ)  = 0 only for finitely many λ ∈ D, and   I = Iv = f ∈ X : f (n) (λ) = 0, λ ∈ D, 0 ≤ n ≤ v(λ) − 1 . (1) This observation gives rise to the classical notions of a zero-based (or divisorial) z-invariant subspace and a synthesizable z∗ -invariant subspace. For any function v : D → Z+ (a divisor function), the z-invariant subspace Iv generated by v is defined as in formula (1). For any z-invariant subspace I ⊂ X, the divisor vI of I is defined as   vI (λ) := max n ∈ N : ∀f ∈ I, ∀k = 0, 1, . . . , n − 1, f (k) (λ) = 0 . (We agree that the maximum of the empty set is zero.) Here N stands for the set of positive integers. A z-invariant subspace I ⊂ X is said to be zero-based if I = IvI . In this case, if vI (λ)  = 0 only for finitely many λ ∈ D, then I is said to be finitely zero-based. A z∗ -invariant subspace J ⊂ Y is said to be synthesizable in some topology if    (n) (n) J= k λ : kλ ∈ J ,  where the symbol denotes the closure of the linear span. It is easy to check that a z-invariant subspace I ⊂ X is zero-based if and only if its annihilator I ⊥ ⊂ Y is synthesizable in the weak∗ topology of Y . It is well known that there exist z-invariant subspaces that are not zero-based (and, consequently, nonsynthesizable z∗ -invariant subspaces). The classical example is given by the subspace H p in the Hardy space H p , where  is an inner function with nontrivial singular part. This means that the procedure of forming the closure of the linear span of root vectors is too rigid to yield all z∗ -invariant subspaces, so that one needs another, more flexible, procedure of their reconstruction by root vectors; possibly, this procedure could be applicable to all z∗ -invariant subspaces. Such a procedure, called approximate spectral synthesis, was suggested by Nikolskii in 1978 [21]. His idea was to use not only the operation of closure of the linear

APPROXIMATE SPECTRAL SYNTHESIS

3

span but also the passage to the limit in order to obtain an arbitrary z∗ -invariant subspace. To describe this procedure in more precise terms, we recall the notion of the lower limit of a sequence of subspaces in a Banach space. If (En )n≥1 is a sequence of subspaces in a Banach space X, then the lower limit E of this sequence is defined as   E = lim En := x ∈ X : ∃ xn ∈ En : x = lim xn n→∞ n→∞   = x ∈ X : lim dist(x, En ) = 0 . n→∞

It is easy to verify that the lower limit of an arbitrary sequence of subspaces is always closed, and if each En is invariant with respect to a certain bounded linear operator T ∈ L(X), then E is also T -invariant. The following definitions describe different versions of approximate spectral synthesis, that is, synthesis via forming linear spans and passing to the lower limit. Definition 1. A z∗ -invariant subspace J ⊂ Y is said to admit weak approximate spectral synthesis if there exists a sequence Jn of z∗ -invariant subspaces such that dim Jn < +∞ and J = lim n→∞ Jn . Definition 2. A z-invariant subspace I ⊂ X is said to admit weak approximate spectral cosynthesis if there exists a sequence (In )n≥1 of z-invariant subspaces such that dim(X/In ) < +∞ and I = lim n→∞ In . Definition 3. A z∗ -invariant subspace J ⊂ Y is said to admit strong approximate spectral synthesis if there exists a sequence (Jn )n≥1 of z∗ -invariant subspaces such that dim Jn < ∞, J = lim n→∞ Jn , and J⊥ = lim n→∞ (Jn )⊥ . Definition 4. A z-invariant subspace I ⊂ X is said to admit strong approximate spectral cosynthesis if there exists a sequence In of z-invariant subspaces such that dim(X/In ) < +∞, I = lim n→∞ In , and I ⊥ = lim n→∞ In⊥ . Here E⊥ denotes the preannihilator of a subspace E ⊂ Y in the space X. Obviously, if J ⊂ Y is weak∗ -closed, then J admits strong approximate spectral synthesis if and only if I = J⊥ admits strong approximate spectral cosynthesis, and a z-invariant subspace I ⊂ X admits strong approximate spectral cosynthesis if and only if I ⊥ admits strong approximate spectral synthesis. However, for weak approximate spectral synthesis and cosynthesis, the similar property fails, as we see later. (The main reason is that the relation E = lim n→∞ En only implies the inclusion E ⊥ ⊃ lim n→∞ En⊥ , but not the equality.) Note that the supplementary condition Jn ⊂ Jn+1 in Definition 1 (and In ⊃ In+1 in Definition 2) implies that J is synthesizable (correspondingly, that I is zero-based). In [21], Nikolskii proposed the following conjecture. Conjecture 1. Any weak ∗ -closed z∗ -invariant subspace of Y admits weak approximate spectral synthesis.

4

S. M. SHIMORIN

In the case where X is the Hardy space H 2 , this fact was proved by Douglas, Shapiro, and Shields [6]. In this context, it is easy to verify that any z∗ -invariant subspace also admits strong approximate spectral synthesis. This result is based on the famous Beurling theorem, which states that any z-invariant subspace in H 2 is of the form H 2 , with some inner function , and on approximation of inner functions by finite Blaschke products. The calculation of lower limits of z∗ -invariant subspaces is closely related to problems of rational approximation. Indeed, suppose that Y is identified with a space of functions analytic in D in such a way that the duality between X and Y is determined by the Cauchy pairing  f, g := fˆ(n)g(n), ˆ n≥0

where fˆ(n) and g(n) ˆ are the Taylor coefficients of the functions f and g, and the infinite sum on the right is understood in an appropriate sense. In this case, the (n) functionals kλ correspond to the functions (n)

kλ (z) =

n! zn , (1 − λz)n+1

and if v : D → Z+ is a divisor such that v(λ)  = 0 only for λ belonging to some finite set  ⊂ D, then the annihilator Iv⊥ ⊂ Y is the collection of rational functions r having poles of order at most v(λ) at the points 1/λ, λ ∈  \ {0} and such that the function zr(z) has a pole of order at most v(0) at infinity. Therefore, the problem of calculating lim n→∞ (Ivn )⊥ for a sequence of divisors (vn )n≥1 is equivalent to the problem of describing those functions in Y that can be obtained as limits of rational functions with preassigned poles determined by the divisors vn . For the weighted Lp spaces on T, rational approximation with preassigned poles was studied by Tumarkin [29]–[31] and Katsnelson [17], [18]. In [10], Gribov and Nikolskii calculated the lower limits lim n→∞ (n H 2 )⊥ in H 2 for arbitrary sequences of inner functions (n )n≥1 (see also [22, Chapter 2]). In the study of rational approximation with preassigned poles, the notion of the capacity of a divisor of poles is important. (The divisor of poles of a rational function r is the function v : C → Z+ such that v(λ) = v ≥ 1 if r has a pole of order v at λ, and v(λ) = 0 otherwise.) As shown in [17], [18], and [29]–[31], for the Hardy spaces and the weighted Lp spaces on T, the following capacity is of importance:   1 , v(λ) 1 − cap(v) = |λ| λ∈D−

where D− = C \ D, and v : D− → Z+ is the divisor of poles. In this case, the calculation of the lower limits lim n→∞ (Ivn )⊥ depends on the behaviour of the capacities cap(vn ). Different notions of capacity related to arbitrary spaces Y were introduced

APPROXIMATE SPECTRAL SYNTHESIS

5

in [10]. In what follows, we use the following version of the definition: CapY (v) := inf

|λ|≤1/2



−1 (0) , distY kλ , Iv⊥

where v  (λ) = v(1/λ), and Iv  ⊂ X is defined by formula (1). In [10], it was proved that, for any sequence of divisors vn : D− → Z+ , the relation lim n→∞ (Ivn )⊥ = Y is equivalent to limn→∞ CapY (vn ) = +∞. By duality, we immediately obtain

−1
1 = sup Re g(λ) : g ∈ Iv  , g ≤ 1, |λ| ≤ . (2) CapY (v) 2 Thus, estimates of capacities related to Y are equivalent to certain qualitative versions of uniqueness theorems for the functions in X. Another problem associated with weak approximate spectral synthesis and rational approximation is that of describing the noncyclic vectors for the operator Mz∗ in the space Y . With respect to the Cauchy pairing, Mz∗ is the operator S ∗ of the backward shift:
∗  f (z) − f (0) S f (z) = . z For the Hardy space H 2 , S ∗ -noncyclic vectors were described by Douglas, Shapiro, and Shields in [6]. They obtained two versions of the description: in terms of pseudocontinuations and in terms of rational approximation. The latter description reads as follows: An element f ∈ H 2 is noncyclic for the backward shift if and only if f = limn→∞ rn and supn≥1 cap(vn ) < +∞, where rn are rational functions with poles in D− and vn is the divisor of the poles of rn . A similar description of S ∗ noncyclic vectors in the general situation is a weaker version of Conjecture 1 (which was also suggested as a problem in [21]). Conjecture 2. A function f ∈ Y is noncyclic for S ∗ if and only if f = limn→∞ rn and supn≥1 CapY vn < +∞, where rn are rational functions with poles in D− and vn is the divisor of the poles of rn . The “if” part of Conjecture 2 easily follows from the definition of the capacity CapY (see [21] or [22, Chapter 2]). The “only if” part is a consequence of Conjecture 1. Indeed, if f ∈ Y is S ∗ -noncyclic, then we can form the z∗ -invariant subspace  
n  J= Mz∗ f, n ≥ 0  = Y. We then take an approximation J = lim n→∞ Jn with dim Jn < +∞, and we choose some λ satisfying |λ| ≤ 1/2 and a subsequence (nk )k≥1 such that

 (0) inf dist kλ , Jnk > 0. k≥1

6

S. M. SHIMORIN

Finally, we write f = limk→∞ rnk , rnk ∈ Jnk . Our aim in the present paper is to study the problems related to approximate spectral (co)synthesis in the case where X is the Bergman space. For p > 0, the space p p La (D, dm2 ) (or, simply, La ) consists of functions analytic and area-integrable with power p in the disk D. If  f p := |f (z)|p dm2 (z), D

p

p

then  ·  is a norm in La for p ≥ 1, and f − gp is a metric in La for p ∈ (0, 1). Here dm2 denotes the normalized area measure in D. The first simple observation pertaining to this situation (this observation appeared in [23] as a referee comment) concerns weak approximate spectral cosynthesis. Suppose that X satisfies the additional condition (the division property) f ∈ X,

f (λ) = 0 ⇒

f ∈ X. z−λ

(3)

Definition 5. A z-invariant subspace I ⊂ X is said to be of index 1 if dim(I /zI ) = 1. Sometimes this property of I is called the codimension-1 property. As shown in [24, Lemma 3.1], under condition (3) the index-1 property of I is equivalent to the property similar to (3): f ∈ I,

f (λ) = 0 ⇒

f ∈I z−λ

for some λ such that vI (λ) = 0. Obviously, this property holds for any zero-based subspace. Moreover, it is stable under passage to the lower limit. Indeed, suppose that I = lim n→∞ In , where In are z-invariant subspaces with dim(In /zIn ) = 1. If vI (λ) = 0, then there exists g ∈ I with g(λ)  = 0 and gn ∈ In such that g = limn→∞ gn . If f ∈ I and f (λ) = 0, we choose fn ∈ In such that f = limn→∞ fn ; we have
 fn − fn (λ)/gn (λ) gn f = lim ∈ lim In = I. z − λ n→∞ z−λ n→∞ (The continuity of division by z − λ follows from the closed graph theorem.) Therefore, under condition (3), any z-invariant subspace in X admitting weak approximate spectral cosynthesis is of index 1. It is well known (see [3]) that in all radial weighted Bergman spaces L2a (D, w dm2 ) (consisting of functions analytic and square area-integrable with a radial weight w in the unit disk), there exist z-invariant subspaces I for which dim(I /zI ) > 1. (For any n ∈ N ∪ {∞}, it is possible to find a z-invariant subspace I with dim(I /zI ) = n.) Therefore, weak approximate spectral cosynthesis (and, a fortiori, strong approximate spectral synthesis and cosynthesis) is not always possible in Bergman spaces.

APPROXIMATE SPECTRAL SYNTHESIS

7

However, it turns out that in the unweighted Bergman space L2a , the index-1 property is the only obstruction for approximate spectral cosynthesis (weak and even strong), and for weak approximate spectral synthesis there are no obstructions. These facts comprise the following two theorems, which are the principal results of this paper. Theorem 1. Suppose that I ⊂ L2a is a z-invariant subspace of index 1. Then I admits strong approximate spectral cosynthesis. Theorem 2. Any z∗ -invariant subspace J ⊂ (L2a )∗ admits weak approximate spectral synthesis. We see that Theorem 2 provides another example of a Banach space X for which Conjecture 1 holds. The space dual to L2a with respect to the Cauchy pairing is the Dirichlet space D, consisting of functions analytic in D and having finite Dirichlet integral, supplied with the norm   fˆ(n)2 (n + 1). f 2D := n≥0

The discussion after Conjecture 2 and Theorem 2 results in the following consequence that describes S ∗ -noncyclic vectors in D in terms of rational approximation. Theorem 3. A function f ∈ D is noncyclic for the operator of backward shift if and only if f = limn→∞ rn and supn≥0 CapD vn < +∞, where rn are rational functions with poles in D− and vn is the divisor of the poles of rn . The hard-analysis basis for the proofs of Theorems 1 and 2 consists of, first, an estimate of weighted Bergman reproducing kernels away from the diagonal, given by Proposition 2 in §3, and, second, an approximation theorem for extremal (Bergmaninner) functions in terms of finitely zero-based extremal functions (finite zero divisors), given by Theorem 1B in §2. In §2 we also show that Theorem 2 is an easy consequence of Theorem 1. The main difficulty lies in the proof of Theorem 1. This proof is based on the concept of an extremal function introduced by Hedenmalm in [12]. A function ϕ ∈ L2a (D, dm2 ) is said to be extremal (or Bergman-inner) if ϕ = 1 and ϕ ⊥ zn ϕ for any n ≥ 1. The extremal functions may be regarded as analogs of the inner functions for the Bergman spaces. They turned out to be very important for the inner-outer factorization (in the sense of Bergman spaces) and for the study of the structure of z-invariant subspaces (see [1], [7], [8], [12], and [26]). For any z-invariant subspace I ⊂ L2a , the elements of I  zI having unit norm are extremal functions; in addition, if dim I  zI = 1, (k) then there exists a unique extremal function ϕI ∈ I zI such that ϕI (0) > 0, where k = vI (0). It is easily seen that ϕI is the solution of the following extremal problem:   sup Re g (k) (0) : g ∈ I, g ≤ 1 .

8

S. M. SHIMORIN

The function ϕI is called the extremal function for the subspace I . In the case where I is zero-based (I = Iv ), the function ϕI is also called a canonical zero divisor, because it may serve as a canonical factor (responsible for zeros) in the factorization of functions from Iv (see [12]). One of the principal results of [1] states that if I ⊂ L2a is a z-invariant subspace of index 1, then I is generated (and, hence, uniquely determined) by the extremal function ϕI ; that is, I=



 z n ϕI : n ≥ 0 .

This fact allows us to prove Theorem 1 with the help of the technique of extremal functions and to split the proof into two independent steps (as done in the case of the theorem by Douglas, Shapiro, and Shields for the Hardy space; see [22, Chapter 2] for such an approach). In the first step, we show that pointwise convergence of extremal functions implies the lower limit relations for z-invariant subspaces generated by these functions and for their annihilators. This is done in §3, where we also calculate the lower limits of z-invariant subspaces and their annihilators in the general situation. An important point in justifying the passage from convergence of extremal functions to the approximation of corresponding z-invariant subspaces is an estimate of weighted Bergman reproducing kernels, given by Proposition 2. The second step consists in approximating an arbitrary extremal function by extremal functions generating finitely zero-based z-invariant subspaces (i.e., by Bergman spaces analogs of finite Blaschke products or by canonical finite zero divisors in the case of the unweighted Bergman space). This is done in §4, where we completely describe the class of functions that can be approximated by such finitely zero-based extremal functions in the radial weighted Bergman spaces. In some sense, we obtain a Bergman spaces analog of the Carathéodory-Schur theorem stating that any function in the unit ball of H ∞ (D) is a pointwise limit of finite Blaschke products. Finally, §5 is devoted to some more questions and observations concerning approximate spectral synthesis and cosynthesis in Bergman spaces. p For a special class of z-invariant subspaces in La , weak approximate spectral cosynthesis was studied by Korenblum in [20]. He proved that κ-Beurling-type zp invariant subspaces in La generated by the singular measures supported by Carleson sets admit weak approximate spectral cosynthesis (see [15] and [20] for precise definitions). However, the examples of functions invertible but noncyclic in the Bergman spaces due to Borichev and Hedenmalm (see [5]) show that κ-Beurling-type invariant subspaces do not exhaust all invariant subspaces of index 1, so that the results of Korenblum (for the space L2a ) do not cover Theorem 1 in full generality. 2. Preliminary observations. Suppose that X and Y are as above. For any subset M ⊂ X, we denote by [M] the smallest z-invariant subspace containing M; that is, [M] =



 zn f : n ≥ 0, f ∈ M .

9

APPROXIMATE SPECTRAL SYNTHESIS

If M consists of a single element f , we simply write [f ] to denote the cyclic zinvariant subspace generated by f . It is easy to see that every cyclic z-invariant subspace is of index 1 (see, e.g., [24, Corollary 3.3]). The following proposition shows that Theorem 2 is a consequence of Theorem 1. Proposition 1. Suppose that Y is separable as a Banach space. If every cyclic z-invariant subspace in X admits strong approximate spectral cosynthesis, then every weak ∗ -closed z∗ -invariant subspace in Y admits weak approximate spectral synthesis. Proof. Let J ⊂ Y be a weak∗ -closed z∗ -invariant subspace. Then I = J⊥ ⊂ X is a z-invariant subspace in X, and J = I ⊥ . Since the space X is separable (because so is its dual Y ), we can choose a sequence (fn )n≥1 that is contained and dense in I . If In = [fn ], we have  I= In , n≥1

whence J=

 n≥1

In⊥ .

By assumption, for each n ≥ 1, we have In⊥ = lim Jnk k→∞

for some sequence (Jnk )k≥1 , with dim Jnk < +∞. It remains to use the following general lemma. Lemma 1. Suppose that Y is a separable Banach space and Jnk n, k ≥ 1 are some subspaces of Y . If Jn = lim Jnk , k→∞

then there exist sequences of indices (nj )j ≥1 and (kj )j ≥1 such that 

k

Jn = lim Jnjj .

n≥1

j →∞

Proof. We choose a sequence (y m )m≥1 contained and dense in any m, n ≥ 1 we have

 lim dist ym , Jnk = 0,

k→∞

for any n ≥ 1 there exists a number kn ∈ N such that ∀k ≥ kn , ∀m ≤ n,

 1 dist ym , Jnk ≤ n . 2



n≥1 Jn . Since for

10

S. M. SHIMORIN


 kn +lj Now, let j → nj , lj be some enumeration of N × N. Putting J (j ) := Jnj j , we show that  Jn = lim J (j ) . j →∞

n≥1

Let m ∈ N and N ≥ m. By the choice of kn , the inequality 
1 dist ym , Jnkn +l ≤ N 2

(4)

holds for any n ≥ N and l ≥ 1. For each n = 1, . . . , N − 1, the same inequality holds for all indices l ∈ N except for a finite number of them, because
 lim dist ym , Inkn +l = 0.

l→∞

Hence, the inequality (4) holds for all pairs (n, l) but a finite number of them, whence
 lim dist ym , J (j ) = 0.

j →∞

This means that ym ∈ lim J (j ) . j →∞

So, we have the inclusion



Jn ⊂ lim J (j ) .

n≥1

j →∞

On the other hand, if y ∈ lim j →∞ J (j ) , then y ∈ lim Jnkn +l = Jn , l→∞

which proves the converse inclusion and accomplishes the proof of Proposition 1. Before proving Theorem 1, we note that if X is a Hilbert space, then for a z-invariant subspace I , strong approximate spectral cosynthesis is equivalent to approximation of the orthogonal projection to I by orthogonal projections to finitely zero-based subspaces In . We have the following lemma. Lemma 2. Suppose that X is a Hilbert space and E and (E n )n≥1 are closed subspaces of X. The following statements are equivalent: (i) PEn → PE in weak operator topology, where PEn and PE are the operators of orthogonal projections to En and E, respectively; (ii) E = lim n→∞ En and E ⊥ = lim n→∞ En⊥ .

APPROXIMATE SPECTRAL SYNTHESIS

11

Proof. (1) (i) ⇒ (ii). If f ∈ E, then f = w-lim n→∞ PEn f (the limit in the weak topology of X). Since PEn f  ≤ f , we have f = limn→∞ PEn f in the norm and, therefore, E ⊂ lim n→∞ En . Similarly, E ⊥ ⊂ lim n→∞ En⊥ , because PE ⊥ = I − PE = limn→∞ PEn⊥ . Since  ⊥ ⊥ lim En ⊂ lim En n→∞

n→∞

for arbitrary subspaces En ⊂ X, we arrive at (ii). (2) (ii) ⇒ (i). For any f ∈ H , we have

 PE f − PEn f = PE f − PEn PE f + PE ⊥ f = PE f − PEn PE f − PEn PE ⊥ f. As n → ∞, the first summand tends to zero because E = lim n→∞ En , and so does the second because E ⊥ = lim n→∞ En⊥ . Now, let I ⊂ L2a be a z-invariant subspace of index 1, and let ϕI be the extremal function for I . We recall that, by the theorem of Aleman, Richter, and Sundberg [1], we have I = [ϕI ]. Theorem 1 is a compilation of the following two independent theorems. Theorem 1A. If ϕn and ϕ are extremal functions in the Bergman space L2a and ϕ = limn→∞ ϕn pointwise in D, then lim P[ϕn ] = P[ϕ]

n→∞

in the weak operator topology. Theorem 1B. If ϕ is an extremal function in L2a , then there exists a sequence of extremal functions (ϕ n )n≥1 such that ϕ = limn→∞ ϕn pointwise in D and [ϕn ] are finitely zero-based invariant subspaces. The proofs of Theorems 1A and 1B are presented in §3 and §4, respectively. Theorems 1A and 1B are analogs for the Bergman space of some results leading to the strong approximate spectral synthesis in the Hardy space H 2 . The Hardy space version of Theorem 1A follows easily from the results of [10]. The approximation of the inner functions by finite Blaschke products is a consequence of the CarathéodorySchur theorem mentioned above (or of the Frostman theorem stating that if θ is an inner function, then (θ − µ)/(1 − µθ) ¯ is a Blaschke product for all µ ∈ D \ E, where E is an exceptional set of zero logarithmic capacity). 3. Lower limits of z- and z∗ -invariant subspaces. For a function f ∈ L2a , we denote by A2f the closure of the polynomials in the space L2 (D, |f |2 dm2 );  · f stands for the norm in this space. (The notation · without indices denotes the norm in L2a .) It is easy to check that [f ] = f · A2f

12

S. M. SHIMORIN

for any function f ∈ L2a (i.e., g ∈ [f ] ⇔ g = f · h, where h ∈ A2f and g = hf ). A description of the spaces A2ϕ for the extremal functions ϕ was obtained in [1]. Fixing an extremal function ϕ ∈ L2a , consider a function . defined as follows:    2 ϕ (λ) /(z, λ) dm2 (λ), .(z) = D

where /(z, λ) is the Green function for the bilaplacian in D,    z − λ 2

  + 1 − |z|2 1 − |λ|2 . /(z, λ) = |z − λ|2 log   1 − λ¯ z It is well known that /(z, λ) is nonnegative everywhere; thus, . is also nonnegative. The function . may be viewed as the solution of the following boundary value problem:  .|T = 0, (5) 0.(z) = |ϕ(z)|2 − 1. Here T = ∂D = {ζ ∈ C : |ζ | = 1}, and 0 is the operator ∂ 2 /(∂z ∂ z¯ ), that is, one quarter of the usual Laplacian. For more information about boundary value problems of type (5), see [1], [7], [8], [12], and [19]. Proposition 4.5 in [1] states that A2ϕ consists of the functions f analytic in D for which    2 f (z) .(z) dm2 (z) < +∞. D

The space

A2ϕ

is contained in L2a , and the norm in A2ϕ is given by the formula  2  2 2 f ϕ = f L2 + f  (z) .(z) dm2 (z) a

or, equivalently,

 

f ϕ = f  + 2

2

D D

D

  2    2 f (z) ϕ (λ) /(z, λ) dm2 (z) dm2 (λ).

(6)

From this identity we see that the extremal functions in the Bergman space are expansive multipliers, that is, ϕp2 ≥ p2 for any polynomial p. For the first time, this property of extremal functions was discovered by Hedenmalm in [12]. For an extremal function ϕ ∈ L2a , let Kϕ denote the reproducing kernel of the space A2ϕ . From the definition of an extremal function it follows that for any polynomial p,  p(z)|ϕ(z)|2 dm2 (z) = p(0). D

This means that (p, 1)A2ϕ = p(0), whence, Kϕ (z, 0) ≡ 1. The following estimate is crucial for further considerations.

APPROXIMATE SPECTRAL SYNTHESIS

13

Proposition 2. There exists a constant B(λ)continuously depending only on λ∈D such that for any extremal function ϕ, the following inequality holds for any z, λ ∈ D:   Kϕ (z, λ) ≤ B(λ). (7)  Proof. For any n ∈ Z+ , we put In := [zn ϕ]. Clearly, In ⊃ In+1 and n≥1 In = {0}. Hence, we have the decomposition 
 [ϕ] = I0 = In  In+1 . n≥0

Obviously, In+1 = zIn and, since In is cyclic, dim(In  In+1 ) = 1. If ϕn is the extremal function for In , then In In+1 is the linear span of ϕn ; therefore, the system {ϕn }n≥0 is an orthonormal basis for [ϕ]. Since multiplication by ϕ is a unitary map from A2ϕ to [ϕ], we see that the system {ϕn /ϕ}n≥0 is an orthonormal basis for A2ϕ . Hence, Kϕ (z, λ) =

 ϕn (z) ϕn (λ) · . ϕ(z) ϕ(λ)

(8)

n≥0

The desired estimate (7) follows from this decomposition and the two following independent estimates of ϕn /ϕ. Estimate 1. We have

  n  ϕn (λ)  √   ≤ n + 1
|λ|  .  ϕ(λ)  1/2 1 − |λ|2

(9)

Proof. We note that

 n  z g  ≥ √ 1 g n+1 2 for any function g ∈ La . (This can be checked, e.g., by using the expression for the norm in L2a in terms of the Taylor coefficients.) Furthermore, we have ϕn ∈ [zn ϕ] = zn [ϕ], whence ϕn /ϕ = zn gn for some gn ∈ A2ϕ ⊂ L2a . We show that√the function gn is a multiplier from H 2 to L2a with multiplier norm not exceeding n + 1. For any polynomial p, we have       √ √  ϕn p  √  √      ≤ n + 1  ϕn p  ≤ n + 1  ϕn p  = n + 1 ϕn p ≤ n + 1 p 2 . H  zn ϕ   ϕ   ϕ  ϕ The last inequality holds, since any extremal function in L2a is a contractive multiplier from H 2 to L2a (see [12]). From estimates of multipliers between Bergman spaces proved in [28], it follows that √ n+1 |gn (λ)| ≤
1/2 , 1 − |λ|2 which implies (9).

14

S. M. SHIMORIN

Estimate 2. We have

   ϕn (z)     ϕ(z)  ≤ (n + 1)(n + 2).

(10)

Proof. Since ϕ is an extremal function, we have ϕ ⊥ [zn ϕ], whence  ϕn (z) ϕ(z) zm dm2 (z) = 0 D

for all m ≥ 0. Since ϕn is the extremal function for [zn ϕ], we have ϕn ⊥ [zn+1 ϕ], so that  ϕn (z) ϕ(z) zn+l dm2 (z) = 0 D

for all l ≥ 1. For k = 1, . . . , n, we put  γk := ϕn (z) ϕ(z) zk dm2 (z), D

and r(z) :=

n 

(k + 1)γk zk .

k=1

Obviously, |γk | ≤ 1, which yields rH ∞ ≤

(n + 1)(n + 2) . 2

Taking the above identities and the choice of r into account, we see that for any harmonic polynomial q,  
ϕn (z) ϕ(z) − r(z) q(z) dm2 (z) = 0. D

Now, we can perform the standard calculation used in different forms in [7], [8], [12], and [19] for deducing the expansive multiplier property of the extremal functions in the Bergman spaces. In brief, we consider the following boundary value problem in D:  7 |T = 0, 07 = ϕn ϕ − r. Since 07 is orthogonal to the harmonic polynomials in D, we have ∂7/∂n|T = 0 (in an appropriate sense), and 7 can be expressed by the integral formula   7(λ) = 02 7(z) /(z, λ) dm2 (z) = ϕn (z) ϕ  (z) /(z, λ) dm2 (z). D

D

15

APPROXIMATE SPECTRAL SYNTHESIS

Now, the application of the Green formula leads to the identity    ϕn (z) ϕ  (z)0h(λ)/(z, λ) dm2 (z) dm2 (λ), (ϕn ϕ − r) h dm2 = D

(11)

D D

which holds for any h ∈ C (2) (D). (For a detailed account of such calculations, see [8, proof of identity (4)].) Now, let p and p1 be some analytic polynomials. Then, substituting h = pp1 in (12), we get               (ϕn ϕ − r)p p1 dm2  =  ϕn (z) ϕ (z)p (λ) p1 (λ) /(z, λ) dm2 (z) dm2 (λ)    D

D D

 

≤

1/2

D D

|ϕn (z)|2 |p  (λ)|2 /(z, λ) dm2 (z) dm2 (λ)

  ·

D D

1/2 |ϕ  (z)|2 |p1 (λ)|2 /(z, λ) dm2 (z) dm2 (λ) .

Observe that, when applying the Cauchy-Schwarz-Bunyakovskii inequality, we used the positivity of the Green function /(z, λ). Taking identity (6) into account, we can rewrite the last inequality as follows:
1/2
1/2 |(ϕn p, ϕp1 ) − (rp, p1 )| ≤ ϕn p2 − p2 . ϕp1 2 − p1 2 By continuity, this inequality holds for any p1 ∈ A2ϕ (p remains a polynomial). Substituting p1 = (ϕn /ϕ)p, we obtain   2 1/2       
 ϕ ϕ 1/2 n n ϕn p2 − rp, p  ≤ ϕn p2 − p2 ϕn p2 −  p .    ϕ ϕ  Here the left-hand side is at least

   ϕn   , ϕn p − rp ·  p  ϕ  2

while the right-hand side is at most  2  1/2 (ϕn /ϕ)p2 ϕn  1 2 2  ≤ ϕn p −  p  . ϕn p 1 − 2 ϕ  ϕn p2 Therefore,

  2   ϕn  ϕn  1 2    , ϕn p − rp ·  p  ≤ ϕn p −  p  ϕ 2 ϕ  2

whence

       ϕn 2  p  ≤ 2 · rH ∞ p ·  ϕn p  , ϕ  ϕ 

16

S. M. SHIMORIN

and

   ϕn   p  ≤ (n + 1)(n + 2) p. ϕ 

This shows that the function ϕn /ϕ is a multiplier of L2a with multiplier norm not exceeding (n+1)(n+2). But it is well known (see [28]) that the space of multipliers of L2a coincides with H ∞ (D) with the equality of norms, which yields the desired estimate. Substituting estimates (9) and (10) in (8), we arrive at estimate (7) from Proposition 2 with the constant
−1  B(λ) = 1 − |λ|2 (n + 1)3/2 (n + 2)|λ|n . n≥0

Remark. Estimate 2 answers [13, problem 4.2] in the affirmative. Using the modified arguments of the proof of Estimate 2, Hedenmalm (see [14]) recently obtained a better off-diagonal estimate for weighted reproducing kernels: |Kϕ (z, λ)| ≤

2 . ¯ 2 |1 − λz|

We need the reproducing kernels Kϕ for the study of orthogonal projections to the cyclic z-invariant subspaces [ϕ], as the following lemma shows. Lemma 3. If ϕ is an extremal function in L2a , then the orthogonal projection P[ϕ] is given by the integral formula  P[ϕ] f (λ) = ϕ(λ) Kϕ (z, λ) ϕ(z)f (z) dm2 (z). (12) D

Proof. For any λ ∈ D, the kernel Kϕ (·, λ) belongs to A2ϕ . Therefore ϕKϕ (·, λ) ∈ [ϕ] ⊂ L2a , and the integral (12) is well defined. If f is orthogonal to [ϕ], then (f, ϕKϕ (·, λ)) = 0, and the integral in the right-hand side of (12) vanishes. If f ∈ [ϕ], then f = ϕg, g ∈ A2ϕ , and the reproducing property of Kϕ implies that   ϕ(λ)Kϕ (z, λ)ϕ(z)f (z)dm2 (z) = ϕ(λ) g(z)Kϕ (z, λ) |ϕ(z)|2 dm2 (z) D

D

= ϕ(λ)g(λ) = f (λ). Proof of Theorem 1A. Suppose that ϕn and ϕ are extremal functions in L2a such that ϕn → ϕ weakly. Then ϕn → ϕ in the norm, since ϕn  = ϕ = 1. Show that limn→∞ Kϕn (µ, λ) = Kϕ (µ, λ) pointwise. In [12], it was proved that H 2 (D) ⊂ A2ϕ provided that ϕ is an extremal function, so that Kϕn (·, µ) ∈ A2ϕ and Kϕ (·, λ) ∈ A2ϕn by Proposition 2. The reproducing properties of Kϕn and Kϕ lead to relations  Kϕ (z, λ)Kϕn (z, µ) |ϕn (z)|2 dm2 (z) = Kϕ (µ, λ) D

17

APPROXIMATE SPECTRAL SYNTHESIS



and

D

Kϕ (z, λ)Kϕn (z, µ) |ϕ(z)|2 dm2 (z) = Kϕn (µ, λ).

Hence,  Kϕ (µ, λ) − Kϕn (µ, λ) =

D


 Kϕ (z, λ)Kϕn (z, µ) |ϕn (z)|2 − |ϕ(z)|2 dm2 (z),

and, by Proposition 2,      Kϕ (µ, λ) − Kϕ (µ, λ) ≤ B(µ)B(λ) |ϕn (z)|2 − |ϕ(z)|2  dm2 (z) n D

n→∞

/ 0.

The pointwise convergence of the reproducing kernels Kϕn to Kϕ implies the limit relation ϕn Kϕn (·, λ) −→ ϕ Kϕ (·, λ)

(13)

in the weak topology of L2a for any fixed λ ∈ D. Indeed, we have pointwise convergence in (13), and it suffices to check uniform boundedness of norms. We have ϕn Kϕn (·, λ) = Kϕn (·, λ)ϕn , and the norm Kϕn (·, λ)ϕn is the norm of the functional f ' → f (λ) in the space A2ϕn . But for any f ∈ A2ϕn (⊂ L2a ), |f (λ)| ≤

f ϕn f  ≤ , 1 − |λ|2 1 − |λ|2

whence ϕn Kϕn (·, λ) ≤

1 . 1 − |λ|2

From (12) and (13), it follows immediately that lim P[ϕn ] f (λ) = P[ϕ] f (λ)

n→∞

for any fixed f ∈ L2a and any λ ∈ D, which is equivalent to the convergence P[ϕn ] → P[ϕ] in the weak operator topology. This completes the proof of Theorem 1A. The following proposition establishes a fact that, in view of Lemma 2, is converse to Theorem 1A. Proposition 3. Suppose that ϕn and ϕ are extremal functions in L2a such that ϕn (0) > 0 for all n ≥ 1 and ϕ(0) > 0. If lim n→∞ [ϕn ] = [ϕ] and lim n→∞ [ϕn ]⊥ = [ϕ]⊥ , then limn→∞ ϕn = ϕ in the norm.

18

S. M. SHIMORIN

Proof. The proof is a slight modification of that given in [20] (in a less general context). Suppose that some subsequence (ϕnk )k≥1 weakly converges to an element f ∈ L2a . If g ∈ z[ϕ], then g = limn→∞ gn for some gn ∈ z[ϕn ], and
 (g, f ) = lim gnk , ϕnk = 0, k→∞

which means that f ⊥ z[ϕ]. Similarly, f ⊥ [ϕ]⊥ , whence f ∈ [ϕ]z[ϕ], and f = γ ϕ for some γ ∈ [0, 1]. On the other hand, since f ∈ [ϕ] = lim n→∞ [ϕn ], we have f = limn→∞ ϕn gn , where gn ∈ A2ϕn , and   f (0) ≤ lim ϕnk (0) · gnk ϕ = lim ϕnk (0) · ϕnk gnk  = f (0) · f . nk

k→∞

k→∞

The relation f (0)= 0 is impossible (since ϕ(0)> 0 and one can write ϕ= limn→∞ ϕn hn with hn ∈A2ϕn , hn ϕn→1); thus, f = 1 and f = ϕ. It follows that w-lim n→∞ ϕn = ϕ, so that limn→∞ ϕn = ϕ in the norm. Now we pass to calculations of lower limits of cyclic z-invariant subspaces [ϕn ] and their annihilators in the case where the behaviour of the extremal functions ϕn is arbitrary. First, we consider two particular cases, namely, w-lim n→∞ ϕn = 0 and w-lim n→∞ ϕn = ψ  = 0. Proposition 4. Suppose that ϕn , n ≥ 1, are extremal functions in L2a such that w-lim n→∞ ϕn = 0. Then (1) lim n→∞ [ϕn ]⊥ = (L2a )∗ . (2) lim n→∞ [ϕn ] = {0}. Proof. To prove the relation lim n→∞ [ϕn ]⊥ = (L2a )∗ , it suffices to prove that limn→∞ P[ϕn ] = 0 in the strong operator topology. To this end, it suffices to verify that limn→∞ P[ϕn ] (zm ) = 0 in the norm for any m ≥ 0. As in the proof of Proposition 2, we (k) (k) consider the z-invariant subspaces In := [zk ϕn ]. Let ϕn be the extremal function (k) for In . We have the decomposition (see (8))    ϕn(k) (z) ϕn(k) (λ) · . Kϕn (z, λ) = ϕn (z) ϕn (λ) k≥0

By Lemma 3, we can write


P[ϕn ] z

m

    (k) (k) ϕn (λ) ϕn (z) · · ϕn (z) zm dm2 (z) (λ) = ϕn (λ) ϕn (λ) ϕn (z) D



k≥0

=

m  k=0

ϕn(k) (λ) ·



(k)

D

ϕn (z) · zm dm2 (z).

APPROXIMATE SPECTRAL SYNTHESIS

19

(k)

It suffices to show that w-lim n→∞ ϕn = 0 for any k ≥ 0, or, equivalently, limn→∞ (k) ϕn (z) = 0 pointwise in D. But limn→∞ ϕn (z) = 0 pointwise by assumption, and (k) |ϕn (z)| ≤ (k + 1)(k + 2)|ϕn (z)| by Estimate 2 from the proof of Proposition 2. The relation lim n→∞ [ϕn ]⊥ = (L2a )∗ is proved. Now, the second limit relation claimed in the proposition follows immediately. Suppose that a sequence (ϕ n )n≥1 of extremal functions in L2a weakly converges to a function ψ ∈ L2a , where ψ  ≡ 0 and ψ is not extremal. The extremal functions in the Bergman space can be characterized by the identity  |ϕ|2 h dm2 = h(0), D

which is valid for any bounded harmonic function h. Fatou’s lemma shows that if ψ(z) = limn→∞ ϕn (z) pointwise in D and ϕn are extremal, then  |ψ|2 h dm2 ≤ h(0) (14) D

for any positive harmonic function h. The functions ψ satisfying (14) for any positive harmonic h are called subextremal functions. In §4 it is shown that any subextremal function is a pointwise limit of some sequence of extremal functions. Let Pz (ζ ) = (1 − |z|2 )/(|1 − ζ¯ z|2 ) be the Poisson kernel for the disc D, and let ᏼ denote the operator of harmonic extension of functions from the boundary T = ∂D inside the disc D,  ᏼf (z) = f (ζ )Pz (ζ ) dm1 (ζ ) T

(dm1 is the normalized Lebesgue measure on T). We consider ᏼ as an operator acting from C(T) to C(D). For the conjugate operator ᏼ∗ : M(D) → M(T) (M denotes the space of finite Borel measures), it is easy to verify the following properties: (i) ᏼ∗ M+ (D) = M+ (T) (where M+ is the cone of nonnegative Borel measures); (ii) ᏼ∗ L1 (D) = L1 (T) (here L1 is identified with a subspace of M). Indeed, in order to verify (i), we take a nonnegative Borel measure µ ∈ M+ (D) and an arbitrary nonnegative function f ∈ C(T); then   ∗ f d(ᏼ µ) = ᏼf (z) dµ(z) ≥ 0, T

D

whence ᏼ∗ M+ (D) ⊂ M+ (T). On the other hand, any ν ∈ M(T) may be viewed as an element of M(D), and we have ᏼ∗ ν = ν, which shows that ᏼ∗ M+ (D) = M+ (T). To prove (ii), we consider the rotation operators Rθ defined on the Borel measures µ ∈ M(D) (or M(T)) by the formula
 (Rθ µ)(E) := µ e−iθ E .

20

S. M. SHIMORIN

(E is a Borel subset of D, and e−iθ E = {e−iθ z : z ∈ E}.) It is well known that a measure ν ∈ M(T) is absolutely continuous with respect to the Lebesgue measure dm1 if and only if limθ →0 ν −Rθ ν = 0. (The norm is taken in M(T).) On the other hand, a simple calculation shows that the operators ᏼ∗ and Rθ commute: ᏼ∗ Rθ = Rθ ᏼ∗ . Since, obviously, limθ →0 µ − Rθ µ = 0 for any µ ∈ L1 (D), the inclusion ᏼ∗ L1 (D) ⊂ L1 (T) follows. The converse inclusion is a consequence of the relations ᏼ∗ (zn )(ζ ) = ζ n /(n + 1) and ᏼ∗ (zn )(ζ ) = ζ n /(n + 1). A function ϕ ∈ L2a is extremal if and only if ᏼ∗ (|ϕ|2 dm2 ) = dm1 , and ψ is subextremal if and only if ᏼ∗ (|ψ|2 dm2 ) = h dm1 , where h ∈ L1 (T) and 0 ≤ h ≤ 1. We need the following technical lemma. Lemma 4. Suppose that ϕn ∈ L2a are extremal functions and limn→∞ ϕn (z) = ψ(z) pointwise in D. Consider the nonnegative measure dν ∈ M+ (T) defined as follows:
 dν = dm1 − ᏼ∗ |ψ|2 dm2 . Then the following hold: (i) The measures |ϕn |2 dm2 converge to the measure |ψ|2 dm2 + dν in the weak∗ topology of M(D). (ii) For any l ≥ 0,

 lim ᏼ∗ |z|2l |ϕn |2 dm2 = ᏼ∗ |z|2l |ψ|2 dm2 + dν n→∞

in the norm of M(T). (iii) If the functions x, xn ∈ H ∞ (D) are such that x = w* -limn→∞ xn (the limit is taken in the weak∗ topology of H ∞ ), then    zl xn |ϕn |2 dm2 = zl x |ψ|2 dm2 + zl x dν, l = 0, 1, . . . . lim n→∞ D

D

T

(In the second term, x means the boundary function; the integral is well defined because dν is absolutely continuous with respect to dm1 .) Proof. (i) Suppose that
 w* -lim |ϕnk |2 dm2 = dλ ∈ M(D) k→∞

for some sequence (nk )k≥1 . We show that dλ = |ψ|2 dm2 + dν. Since ϕn (z) → ψ(z) uniformly on the compact subsets of D, for any Borel subset E of D we have  λ(E) = |ψ|2 dm2 . E

Hence, it suffices to show that λ(F ) = ν(F )

21

APPROXIMATE SPECTRAL SYNTHESIS

for any Borel subset F ⊂ T. Indeed, for any h ∈ C(T) we can write    h dλ = ᏼh dλ − ᏼh · |ψ|2 dm2 T D D   2 = lim ᏼh · |ϕnk | dm2 − ᏼh · |ψ|2 dm2 k→∞ D D  
∗
2  = ᏼh(0) − h · d ᏼ |ψ| dm2 = h dν. T

(ii) The

weak∗

T

convergence

 ᏼ∗ |z|2l |ϕn |2 dm2 −→ ᏼ∗ |z|2l |ψ|2 dm2 + dν

follows immediately from (i). To see that we have norm convergence, it suffices to observe that both sides of the above limit relation are positive elements of L1 (T) and to use the following simple fact from the integration theory: If (ᐄ, µ) is a space with measure, xn and x are nonnegative integrable functions on ᐄ such that xn → x in measure, and   xn dµ −→ x dµ, ᐄ 1 L (ᐄ, dµ)

ᐄ

then xn → x in the norm of (see [11, §26]). (iii) Consider the following operators on the space H ∞ (D): (Ul x) (z) :=

l−1 

k x(k)z ˆ +

k=0

l 

x(k ˆ + l)

k=0

l − k k+l z l

(the operator of taking the de la Vallée–Poussin means), and (Vl x) (z) :=

1 (x − Ul x). zl

Then, by properties of the de la Vallée–Poussin means, Ul  ≤ 3, Vl  ≤ 4, and x = Ul x + zl Vl x for any x ∈ H ∞ . By (i), we have    l 2 l 2 z Ul xn |ϕn | dm2 = z Ul x |ψ| dm2 + zl Ul x dν, lim n→∞ D

D

T

and by (ii), we have    |z|2l Vl xn |ϕn |2 dm2 = |z|2l Vl x |ψ|2 dm2 + Vl x dν, lim n→∞ D

D

T

because we can convert the integrals over D into integrals over T by the use of ᏼ∗ .

22

S. M. SHIMORIN

Now we are ready to describe the lower limits of the subspaces [ϕn ] and [ϕn ]⊥ in the case where w-lim n→∞ ϕn = ψ. Proposition 5. Suppose that ϕn are extremal functions in L2a , and w-lim n→∞ ϕn = ψ  ≡ 0. Then lim n→∞ [ϕn ]⊥ = [ψ]⊥ . If ψ is an extremal function (this is equivalent to ψ = 1), then lim n→∞ [ϕn ] = [ψ]; otherwise, lim n→∞ [ϕ] = {0}. Proof. By Lemma 4(i), w* -limn→∞ (|ϕn |2 dm2 ) = |ψ|2 dm2 +dνψ =: dµψ , where dνψ = dm1 − ᏼ∗ (|ψ|2 dm2 ) ∈ M+ (T). Consider the Hilbert space Pψ2 , which is the closure of the analytic polynomials in

L2 (D, dµψ ). Since ψ  ≡ 0, the point evaluation functionals f → f (λ) are bounded in Pψ2 for λ ∈ D, which allows us to consider the reproducing kernel Kψ (z, λ) of Pψ2 (defined on D × D). Let us denote Kϕn , simply, by Kn . Show that limn→∞ Kn (z, λ) = Kψ (z, λ) pointwise in D × D. Fix λ ∈ D. By Proposition 2, we have |Kn (z, λ)| ≤ B(λ); hence, it suffices to check that if limj →∞ Knj (·, λ) = k(·, λ) pointwise in D for some sequence (nj )j ≥1 , then the function k(·, λ) possesses the reproducing property   zl k(z, λ) |ψ(z)|2 dm2 (z) + zl k(z, λ) dνψ (z), l = 0, 1, . . . . λl = D

T

Indeed, the reproducing property of Kn shows that  zl Kn (z, λ) |ϕn (z)|2 dm2 (z), λl = D

and it remains to apply Lemma 4(iii). By Proposition 2, the convergence Kn → Kψ is uniform on the compact subsets of D × D, and |Kψ (z, λ)| ≤ B(λ) for any λ ∈ D. The operator Mz is bounded away from zero in the space Pψ2 ; hence, zm Pψ2 is the closed subspace of Pψ2 given as   zl : l ≥ m (the closure is taken in the norm of L2 (D, dµψ )). Clearly, dim(zm Pψ2 zm+1 Pψ2 ) = 1  and m≥0 zm Pψ2 = {0}. Let e(m) be an element of zm Pψ2  zm+1 Pψ2 such that e(m) P 2 = 1 and (d/dzm )e(m) (z)|z=0 > 0. Then we have the decomposition ψ

Kψ (z, λ) =



e(m) (z) e(m) (λ).

m≥0

As in the proofs of Propositions 2 and 4, we can write the decomposition    ϕn(m) (z) ϕn(m) (λ) · , Kn (z, λ) = ϕn (z) ϕn (λ) m≥0

23

APPROXIMATE SPECTRAL SYNTHESIS (m)

where ϕn is the extremal function for [zm ϕn ]. Show that (m)

ϕn (z) = e(m) (z) n→∞ ϕn (z) lim

(15)

pointwise in D (and, by Estimate 2 from the proof of Proposition 2, in the weak∗ topology of H ∞ ). For m = 0, we have (0) Kψ (z, 0) Kn (z, 0) ϕn (z) = lim = lim 1/2 n→∞
1/2 n→∞ ϕn (z) Kψ (0, 0) Kn (0, 0)

e(0) (z) =


(≡ 1).

Suppose that we have already proved (15) for m = 0, . . . , m0 − 1. Then, uniformly on the compact subsets of D × D, (m0 )

lim Kn(m0 ) (z, λ) = Kψ

n→∞

where

(z, λ),

   ϕn(m) (z) ϕn(m) (λ) Kn(m0 ) (z, λ) := · ϕn (z) ϕn (λ) m≥m 0

and

(m0 )

Kψ



(z, λ) :=

e(m) (z) e(m) (λ).

m≥m0

We have e

(m0 )

 (z) =

∂ 2m0 ∂zm0 ∂λ



 (m0 ) m0 Kψ (z, λ) z=λ=0

−1/2 ·

∂ m0 ∂λ

m0

(m0 )

Kψ

(m )

  (z, λ)

λ=0

;

(m )

the analogous formula expresses (ϕn 0 (z)/ϕn (z)) by the kernel Kn 0 , and the con(m ) (m ) (m ) vergence ϕn 0 /ϕn → e(m0 ) follows from the convergence of kernels Kn 0 → Kψ 0 . As a consequence of (15), we deduce that w-lim ϕn(m) = e(m) ψ n→∞

in the weak topology of L2a . Consider the operators Tn , T˜ψ , and Tψ defined as follows:  (Tn g)(λ) = Kn (z, λ) ϕn (z)g(z) dm2 (z), D
 T˜ψ g (λ) = Kψ (z, λ) ψ(z)g(z) dm2 (z), D

(16)

24 and

S. M. SHIMORIN


 (Tψ g)(λ) = ψ(λ) T˜ψ g (λ) = ψ(λ)

 D

Kψ (z, λ) ψ(z) g(z) dm2 (z).

By Lemma 3, we have ϕn Tn g = P[ϕn ] g for any g ∈ L2a ; thus, Tn are contractions acting from L2a to A2ϕn . We are going to show that T˜ψ is a contraction from L2a to Pψ2 and that, for any f ∈ L2a , we have   T˜ψ f  2 = lim Tn f  2 . (17) Aϕ P n→∞

ψ

n

It suffices to check that T˜ψ f ∈ Pψ2 and to prove that (17) is true for any polynomial f . Using the above decomposition of kernels Kn and Kψ , we obtain l (m)   ϕn (m) · ϕn (z) zl dm2 (z) Tn z = ϕn D l

m=0

and T˜ψ zl =

l 

e

(m)

m=0

 ·

D

e(m) (z) ψ(z) zl dm2 (z).

These formulae and the limit relation (16) imply that

  lim Tn zl , Tn zp A2 = T˜ψ zl , T˜ψ zp P 2 n→∞

ϕn

ψ

for any l, p ∈ Z+ , which leads immediately to (17). Relation (17) implies that ker Tψ = [ψ]⊥ . The operator Tψ : L2a → L2a is the limit of the projections P[ϕn ] in the weak operator topology. This can be shown by using the same arguments as in the proof of Theorem 1A. If g ∈ [ψ]⊥ , then Tψ g = 0; and so from (17) and Lemma 3, we obtain limn→∞ P[ϕn ] g = limn→∞ Tn gA2ϕ = 0. Therefore, g ∈ lim n→∞ [ϕn ]⊥ , and we have the n

inclusion [ψ]⊥ ⊂ lim n→∞ [ϕn ]⊥ . The converse inclusion is obvious. In the case where ψ is an extremal function, the relation lim n→∞ [ϕn ] = [ϕ] follows from Theorem 1A. If ψ is not extremal, then ψ < 1 and dν > 0. If limn→∞ P[ϕn ] f = f for some nonzero f ∈ L2a , then f = Tψ f and      
   T˜ψ f 2 |ψ|2 dm2 < T˜ψ f 2 |ψ|2 dm2 + dν = T˜ψ f 2 2 ≤ f 2 , f 2 = P D

D

ψ

which is a contradiction. This means that in this case we have lim n→∞ [ϕn ] = {0}. Proposition 5 is proved. The following theorem describes the lower limits of cyclic z-invariant subspaces and of their annihilators in the general case where the behaviour of extremal functions ϕn is arbitrary.

APPROXIMATE SPECTRAL SYNTHESIS

25

Theorem 4. Suppose that (ϕ n )n≥1 is a sequence of extremal functions in L2a . Consider the set A of weak limits of subsequences of this sequence, that is,  
 A = ψ ∈ L2a : ∃ nj j ≥1 such that ψ = w-lim ϕnj . j →∞

Then the following hold:  (i) lim n→∞ [ϕn ]⊥ = ψ∈A [ψ]⊥ . (ii) If A consists only of extremal functions, then  [ψ]; lim [ϕn ] = n→∞

ψ∈A

otherwise, lim n→∞ [ϕn ] = {0}. Proof. The proof is based on the following general fact proved in [10]. Lemma 5. Suppose that Qn , n ≥ 1, are orthogonal projections in a Hilbert space H . Consider the set {Qn } , consisting of the limit points of the sequence (Qn )n≥1 in the weak operator topology. Then  ker Q. lim ker Qn = n→∞

Q∈{Qn }

In our case, we take Qn = P[ϕn ] . The set {Qn } coincides with the set {Tψ }ψ∈A . (Here Tψ is defined as in the proof of Proposition 5.) Indeed, for ψ ∈ A, the inclusion Tψ ∈ {Qn } follows from the proof of Proposition 5. On the other hand, if Q = limj →∞ P[ϕnj ] , then w-lim k→∞ ϕnjk =ψ∈A for some (j k )k≥1 , and we have Q=Tψ . Since ker Tψ = [ψ]⊥ (see the proof of Proposition 5), Lemma 5 readily shows that  [ψ]⊥ , lim [ϕn ]⊥ = n→∞

ψ∈A

and (i) is proved. To prove (ii), we observe that if there exists some ψ = w-lim j →∞ ϕnj such that ψ is not extremal, then lim j →∞ [ϕnj ] = {0} by Proposition 5, whence lim n→∞ [ϕn ] = 0. In the case where A consists only of extremal functions, the relation lim n→∞ [ϕn ] =  ψ∈A [ψ] follows from Theorem 1A and Lemma 5. For the Hardy space H 2 , a statement analogous to Theorem 4(i) was proved in [10] (see also [22, Chapter 2]). 4. Approximation of subextremal functions. First, we recall the following classical theorem. Carathéodory-Schur theorem. If ψ ∈ H ∞ (D) is such that ψH ∞ ≤ 1, then there exists a sequence of finite Blaschke products Bn such that ψ = lim Bn n→∞

26

S. M. SHIMORIN

pointwise in D. For the proof see, for example, [22, Chapter 2]. Theorem 1B is a particular case of a more general fact, namely, Theorem 5, which is a Bergman spaces version of the Carathéodory-Schur theorem. A positive finite measure µ ∈ M+ (D) is said to be radial if Rθ µ = µ for any θ ∈ R. Obviously, any radial measure can be written in the form dµ0 (r 2 ) × (dϕ/2π) (r and ϕ are the polar coordinates) for some µ0 ∈ M+ ([0, 1)). In what follows, we deal with a fixed radial measure µ ∈ M+ (D), dµ = dµ0 (r 2 ) × (dϕ/2π), satisfying the following two conditions: µ(D) = 1 and
 µ0 (1 − ε, 1) > 0.

∀ε > 0,

(18)

The radial weighted Bergman space L2a (D, dµ) consists of functions analytic in D for which  2 |f (z)|2 dµ(z) < +∞. f  := D

This space is supplied with the natural norm induced from L2 (D, dµ). The condition (18) immediately implies the continuity of the point evaluation functionals for the points in D and the completeness of L2a (D, dµ). If we introduce the moments  1  wn := r n dµ0 (r) = |z|2n dµ(z) D

0

of the measure µ0 , then the norm in L2a (D, dµ) is given by the formula   fˆ(n)2 wn . f 2 = n≥0

In what follows, we need the following properties of the moment sequence (w n )n≥0 : (i) w0 = 1; (ii) the sequence (w n )n≥0 is logarithmically convex; √ (iii) limn→∞ n wn = 1; (iv) limn→∞ (wn+1 /wn ) = 1; (v) limn→∞ (wn2 /w2n ) = 0. Property (i) is equivalent to the fact that µ(D) = 1; property (ii) is well known; properties (iii) and (iv) are consequences of (18); and (v) follows from (ii), (iv), and the fact that wk → 0 as k → ∞. Indeed, for k ≤ n, (ii) implies that w n wk wn2 ≤ . w2n wn+k

APPROXIMATE SPECTRAL SYNTHESIS

27

Fixing k, letting n → ∞, and using (iv), we obtain wn2 ≤ wk , n→∞ w2n lim

and it remains to let k → ∞. The notions of an extremal and a subextremal function in the spaces L2a (D, dµ) are analogous to those in the case of the classical (unweighted) Bergman space L2a . Definition 6. A function ϕ ∈ L2a (D, dµ) is said to be extremal if ϕ = 1 and (ϕ, zn ϕ) = 0 for any n ≥ 1. It is easy to check that ϕ is extremal if and only if  |ϕ|2 h dµ = h(0) D

for any bounded harmonic function h, or, in other words, ᏼ∗ (|ϕ|2 dµ) = dm1 . For any z-invariant subspace I ⊂ L2a (D, dµ), the extremal function for I is defined as the solution of the following extremal problem:   sup Re g (k) (0) : g ∈ I, g ≤ 1 , where k = vI (0). An extremal function ϕ ∈ L2a (D, dµ) is said to be finitely zerobased if [ϕ] is a finitely zero-based z-invariant subspace. The finitely zero-based extremal functions are Bergman spaces analogs of the finite Blaschke products. We avoid the term zero divisor in this general situation, because, in general, zero-based extremal functions fail to be good canonical factors in factorization of functions in radial weighted Bergman spaces (see [16]). Definition 7. A function ψ ∈ L2a (D, dµ) is said to be subextremal if  |ψ|2 h dµ ≤ h(0) D

for any positive harmonic function h. Clearly, this condition is equivalent to the identity ᏼ∗ (|ψ|2 dµ) = gdm1 , where g ∈ L1 (T) and 0 ≤ g ≤ 1 on T. For a function ψ ∈ H ∞ , the property ψH ∞ ≤ 1 is equivalent to  ˆ |ψ|2 h dm1 ≤ h(0) T

L1 (T).

Therefore, the subextremal functions may be viewed as for any positive h ∈ Bergman spaces analogs of the functions from the unit ball of H ∞ . Theorem 5 (Bergman spaces analog of the Carathéodory-Schur theorem). A function ψ ∈ L2a (D, dµ) is subextremal if and only if there exist finitely zero-based extremal functions ϕn ∈ L2a (D, dµ) such that ψ = limn→∞ ϕn pointwise in D.

28

S. M. SHIMORIN

Obviously, Theorem 1B is a particular case of this theorem. Another Bergman spaces version of the Carathéodory-Schur theorem was suggested by Hedenmalm (see [13, Conjecture 5.2]). He conjectured that any function ϕ analytic in D and satisfying ϕf L2a ≤ f H 2 for any f ∈ H 2 (i.e., any contractive multiplier from H 2 to L2a (D)) is the normal limit of finitely zero-based extremal functions in L2a (D). Any subextremal function ϕ in L2a (D, dµ) is a contractive multiplier from H 2 to L2a (D, dµ). Indeed, for any polynomial p, we have  

 |ϕ(z)p(z)|2 dµ(z) ≤ |ϕ(z)|2 ᏼ|p|2 (z) dµ(z) ≤ ᏼ|p|2 (0) D D = |p(ζ )|2 dm1 (ζ ), T

ϕp2

which means that ≤ p2H 2 . The question as to whether or not any contractive 2 2 multiplier from H to La (D, dµ) is subextremal in L2a (D, dµ) remains open (even in the unweighted case dµ = dm2 ). Proof of Theorem 5. The “if” part of the theorem follows immediately from Fatou’s lemma. The “only if” part is less trivial. First, from Definition 7, we see that the set of subextremal functions is convex, rotation invariant, and closed in the weak topology of L2a (D, dµ). Therefore, for any subextremal function ψ, the Fejér integral 
 ψN (z) := ψ ζ z .N (ζ ) dm1 (ζ ), z ∈ D, T

is also a subextremal function. Here


.N e

iθ




 1 sin2 (N + 1)/2 θ = N +1 sin2 (θ/2)

is the Fejér kernel. Since ψN → ψ pointwise in D, it suffices to prove the “only if” implication of Theorem 5 in the case where ψ is a polynomial. Moreover, without loss of generality, we may assume that there exists a positive δ such that  |ψ|2 h dµ ≤ (1 − δ)h(0) (19) D

for any positive harmonic polynomial h. Also, we assume that ψ(0) > 0. So, we have a polynomial ψ satisfying (19) and such that ψ(0) > 0. Let d = deg ψ. For the trigonometric polynomial g(ζ ) :=

−1 


d   n ψ, z|n| ψ ζ |n| + (zn ψ, ψ)ζ ,

n=−d

n=0

APPROXIMATE SPECTRAL SYNTHESIS

it is easy to verify that

29






ᏼ∗ |ψ|2 dµ = g dm1 .

Inequality (19) implies that 0 ≤ g(ζ ) ≤ 1 − δ on T. The function 1 − g is a positive trigonometric polynomial of degree d; therefore, by the Fejér–F. Riesz theorem (see [9, Chapter 1]), there exists an analytic polynomial G of degree d such that G has no zeros in D and |G(ζ )|2 = 1 − g(ζ )

(20)

on T. Moreover, since 1 − g(ζ ) ≥ δ on T, we have |G(z)|2 ≥ δ everywhere in D. Now, we put

(21)

zn pn (z) := ψ(z) + G(z) √ , wn

and In = [pn ]. Let ϕn be the extremal function for In . If we show that ϕn are finitely zero-based and that ψ = limn→∞ ϕn pointwise in D, the theorem will be proved. The crucial limit relation ψ = limn→∞ ϕn is based on the choice of G and pn . The polynomials pn turn out to be “almost extremal,” which means that the sequences (zl pn , pn )l≥0 are “almost” δ-sequences. In a sense, the larger is the parameter n, the √ more “independent” are the terms ψ and G(z)(zn / wn ). Combined with the choice of G, this implies that ᏼ∗ |pn |2 is close to 1. Therefore, ϕn is close to pn and, hence, to ψ. Making these arguments precise requires some technical work. It follows from the next lemma that ϕn are finitely zero-based extremal functions. Lemma 6. If p is a polynomial, then [p] ⊂ L2a (D, dµ) is the finitely zero-based z-invariant subspace determined by the divisor of zeros, lying in D, of p. Proof. It suffices to write p = p1 · p2 , where p1 is a polynomial with zeros in D and where p2 is a polynomial with zeros in C \ D. Observe that p2 is cyclic in the weak∗ topology of H ∞ (D). Lemma 7. There exists a number N0 (depending on ψ and G) such that, for any n ≥ N0 and any f ∈ L2a (D, dµ), we have pn f 2 ≥

δ   ˆ 2 wn+k . f (k) 8 wn k≥0

Proof. For each n ≥ 1, let rn be a number in (0, 1) such that  1 |z|2n dµ(z) ≤ wn 2 |z|
(22)

30

S. M. SHIMORIN



and

rn <|z|<1

|z|2n dµ(z) ≤

1 wn . 2

Clearly, limn→∞ rn = 1. We show that lim

n→∞

Fix a number B > 0. For n > B,



B 1− n

whence 1 wn ≤ 2 =

wn = 0. rn2n n

≤ e−B ,

 |z|≤rn

|z|2n dµ(z)





|z|<(1−(B/n))rn

|z|2n dµ(z) +



(1−(B/n))rn ≤|z|≤rn

|z|2n dµ(z)

2n  B ≤ 1− rn + rn2n dµ(z) n (1−(B/n))rn ≤|z|<1   2n −2B ≤ rn e + dµ(z) . (1−(B/n))rn ≤|z|<1

Since rn → 1, this yields

wn ≤ 2e−2B , n→∞ r 2n n lim

and since B can be chosen arbitrarily large, we obtain lim

n→∞

wn = 0. rn2n

Now, since |G(z)|2 ≥ δ in D, there exists N0 = N0 (ψ, G) such that if n ≥ N0 and rn ≤ |z| < 1, then |z|n ≥ 2|ψ(z)|. |G(z)| √ wn Then, in the same annulus {rn ≤ |z| < 1}, |z|n 1 δ 1/2 |z|n ≥ |pn (z)| ≥ |G(z)| √ √ , 2 wn 2 wn and, for any f ∈ L2a (D, dµ), we can write   δ |pn |2 |f |2 dµ ≥ |f (z)|2 |z|2n dµ(z). pn f 2 ≥ 4wn rn ≤|z|<1 rn ≤|z|<1

APPROXIMATE SPECTRAL SYNTHESIS

If f (z) =



ˆ

k≥0 f (k)z

k,

31

then

δ   ˆ 2 pn f  ≥ f (k) 4wn



2

k≥0

rn ≤|z|<1

|z|2n+2k dµ(z).

For any k ≥ 0, we have   2n+2k 2k 1 |z| dµ(z) ≤ rn wn ≤ |z|2n+2k dµ(z), 2 |z|
 rn ≤|z|<1

|z|2n+2k dµ(z) ≥

1 2



1 |z|2n+2k dµ(z) = wn+k . 2 |z|<1

This results in the desired inequality (22). Corollary 1. If n ≥ N0 , then, for any f ∈ L2a (D, dµ), δ pn f 2 ≥ f 2 . 8 Proof. In view of (22), it suffices to observe that the logarithmic convexity of the sequence (w n )n≥0 implies the inequality wn+k ≥

wn w k = w n wk . w0

Now we define a new norm  · n in L2a (D, dµ) by the formula f n := pn f . The above corollary shows that ·n is equivalent to the standard norm in L2a (D, dµ). Let Bn2 denote the space L2a (D, dµ) supplied with the norm  · n . It is easy to see that In = pn Bn2 and that multiplication by pn is a unitary map from Bn2 to In . We write ϕ n = p n qn with some function qn ∈ Bn2 . The function ϕn is the solution of the extremal problem   sup Re g(0) : g ∈ In , g ≤ 1 . Therefore, qn is the solution of the problem   sup Re q(0) : q ∈ Bn2 , qn ≤ 1 ,

32

S. M. SHIMORIN

which means that qn can be expressed by the reproducing kernel kn of Bn2 : kn (z, 0) qn (z) = √ . kn (0, 0) Thus, we have the identity  zn kn (z, 0) ϕn (z) = ψ(z) + G(z) √ , √ wn kn (0, 0) showing that the desired relation ϕn (z) → ψ(z) follows from the pointwise limit relation lim kn (z, 0) = 1. n→∞

Let k denote the reproducing kernel of the space L2a (D, dµ). Lemma 8. If N0 is the constant occurring in Lemma 7, then, for any n ≥ N0 , we have 8 kn (·, λ)2n ≤ k(λ, λ). δ Proof. The norm kn (·, λ)n is the norm of the functional f → f (λ) in the space Bn2 . But, for f ∈ Bn2 , |f (λ)|2 ≤ f 2 k(λ, λ), and it remains to employ Corollary 1. The following very simple lemma is needed for further calculations. Lemma 9. If p and q are analytic polynomials and f ∈ L2a (D, dµ), then  
 fˆ(l) zl p, q . f (z)p(z)q(z) dµ(z) = D

l≥0

Proof. Since q is a polynomial, (Mz∗ )l q = 0 for all sufficiently large l. Hence, in the sum on the right, only finitely many terms differ from zero, and both sides of the above identity are continuous with respect to f ∈ L2a (D, dµ). But, if f is a polynomial, this identity is obvious. Now we fix λ ∈ D. The reproducing property of kn implies that  1= 1 · kn (z, λ) |pn (z)|2 dµ(z). D

By Lemma 9, we have 1=

 l≥0


l  k
n (·, λ)(l) z pn , pn .

(23)

33

APPROXIMATE SPECTRAL SYNTHESIS

Consider the behaviour of the inner products (zl pn , pn ). Suppose that n > d (d = deg ψ = deg G) and n > N0 . Then     
l
l zn zl+n zn zl+n l + G√ ,ψ + G√ ,G√ z pn , pn = z ψ, ψ + z ψ, G √ wn wn wn wn (n)

=: al

(n)

(n)

(n)

+ b l + c l + dl .

The construction of the function g shows that (n)

al

= g(−l). ˆ

(n)

The numbers bl may be different from zero only for n − d ≤ l ≤ n + d, and they satisfy the estimate   d  (n)      zn  √ b  ≤ ψ(ν) ˆ  ·  zl+ν , G √ ≤ B 1 wn , l  w  n

ν=0

(24)

where the constant B1 depends only on ψ and G, but not on n. (The same refers to the constants B2 and B3 in what follows.) (n) (n) 2 (−l) may differ Since n > d, we have cl = 0. Further, the quantity dl − |G| from zero only for 0 ≤ l ≤ d, and 2 (−l) − |G|  d   d  d d    1  ν+l+n s+n ν+l+n s+n   ζ  ζ G(ν) z G(s)z G(ν) G(s) − , , = wn

(n)

dl

ν=1

=

d−l 

 G(ν  + l) G(ν)

ν=0



s=0

ν=0



s=0

H2

wν+l+n −1 . wn

Hence,    wn+d   (n) 2 1 − . − |G| (−l) ≤ B d   l 2 wn

(25)

Since the polynomial G was chosen in such a way that |G|2 + g ≡ 1 (see (20)), we can write

 
l (n) (n) 2 (−l) . z pn , pn = δl0 + bl + dl − |G| By (23), we have 1 − kn (0, λ) =

n+d  l=n−d

(n) k
n (·, λ)(l) bl +

d  l=0



(n)  2 (−l) . − |G| k
(·, λ)(l) d n l

34

S. M. SHIMORIN

Using Lemmas 7 and 8, we arrive at the estimate    wn 
 8 k(λ, λ) , kn (·, λ)(l) ≤ δ wn+l which implies (in view of (24) and (25)) that     2  B3 wn+d wn wn  |1 − kn (0, λ)| ≤ k(λ, λ)  + 1− . δ w2n+d wn wn+d Finally, properties (iv) and (v) of the moments wn show that the right-hand side vanishes as n → ∞, which accomplishes the proof of Theorem 5. 5. Concluding remarks. In the introduction, we already discussed the relationship between the calculation of the lower limits of z∗ -invariant subspaces and rational approximation with preassigned poles. The results of §3 show that, for the study of such rational approximation in the Dirichlet space D, we need a good knowledge of the behaviour of extremal functions in the Bergman space L2a ; in particular, we need to know the relationship between the divisor functions v : D → Z+ and the extremal functions for the corresponding z-invariant subspaces Iv . Estimates for the capacities, related to D, of the divisors of poles of rational functions are a particular aspect of this general question. As formula (2) shows, these estimates are equivalent to some qualitative versions of the uniqueness theorems for functions in L2a . Recent results of Seip [25] about the zero sets of functions in weighted Bergman spaces show that, for such uniqueness theorems, the Korenblum-type density characteristics are important. In the particular case of the unweighted Bergman space L2a , we can reformulate Seip’s results in terms of divisors of poles and their Dirichlet space capacities. With any finite subset F ⊂ T, we associate a domain GF in D− ,  z ,F , GF := z ∈ D− : |z| − 1 ≥ dist |z| and the Carleson characteristic of F , κ(F ˆ ) :=

 |Il |  l

2π

2π 1 + log . |Il |

Here dist is the normalized angular distance on T:  |t − s|
, dist eit, eis = π and {Il }l≥1 is the collection of all arcs complementary to F .

35

APPROXIMATE SPECTRAL SYNTHESIS

For any divisor v : D− → Z+ and any positive β, the Korenblum density mβ (v) is defined as follows:    
 1 mβ (v) := sup − β κ(F ˆ ) , v(λ) 1 − |λ| F λ∈GF

where the supremum is taken over all finite subsets F ⊂ T. Similarly, for positive β and p, we can define  mβ,p (v) := sup F

and

v(λ) 1 −

λ∈GF

 mβ,p (v) := sup  F










 1 − β κ(F ˆ ) + p log κ(F ˆ ) , |λ|


 1 − β κ(F ˆ ) − p log κ(F ˆ ) . v(λ) 1 − |λ| 



λ∈GF

As usual, for α > 0, we denote by A−α the space of functions f analytic in the disk D and satisfying sup |f (z)|(1 − |z|)α < +∞.

z∈D

Moreover, for α > 0 and C > 0, we introduce the space A˜ −α C consisting of the functions f analytic in D and satisfying  sup |f (z)|(1 − |z|)α log

z∈D

e 1 − |z|

C

< +∞.

Proposition 1 in [25] shows that any L2a -zero set is also a zero set for the functions in −1/2 A . Combined with the necessary condition for A−α zero sets (see [25, Theorem 1, the necessity part]) or, more precisely, its proof, this leads to the relation (which holds for any p > 2) m1/2,p (v) ≺ CapD v. Here, the sign “≺” means that if, for some sequence (v n )n≥1 of divisors, the left-hand side of the relation tends to infinity, then so does the right-hand side. −1/2 On the other hand, for any C > 1/2, we have the embedding A˜ C ⊂ L2a . At the  same time, the condition mα, C (v) < +∞ is sufficient for the divisor v  , defined by v  (λ) = v(1/λ), to be a divisor of zeros for functions in A˜ −α C . (Seip, in a private communication, stated that the proof is the same as that of the sufficiency part of Theorem 1 in [25].) This implies that, for C > 1/2, we have

36

S. M. SHIMORIN

CapD v ≺ m1/2, C . Thus, we see that the capacity CapD v admits estimates from above and from below in terms of the Korenblum-type density characteristics of v, which differ only by terms of order of log κ(F ˆ ). A natural question concerning approximate spectral synthesis in the Bergman spaces is the case of weighted spaces. Theorem 5 shows that an important step in the proofs of Theorems 1 and 2, the approximation of extremal functions by finitely zero-based ones, can be performed in any radial weighted Bergman space. However, in §3, for the proof of Theorem 1A, we needed several special facts related to the unweighted space L2a , namely, the positivity of the Green function / for the bilaplacian 02 in D (this property was used for estimating the reproducing kernels Kϕ in Proposition 2) and the theorem of Aleman, Richter, and Sundberg [1] stating that I = [ϕI ] if dim(I zI ) = 1. The analysis of the proof of that theorem shows that the positivity of the same Green function / (together with some other properties of it) is also important in this case. In the case of the weighted Bergman spaces L2a (D, wdm2 ), one has to work with the weighted biharmonic operators 0 w1 0 instead of the bilaplacian, and the positivity of the corresponding Green function /w comes into play. For the radial and logarithmically subharmonic weight functions w, the positivity of /w was proved in [27]. Apparently, in this case, theorems analogous to the theorem of Aleman, Richter, and Sundberg and Theorem 1A hold as well, and, consequently, there are analogs of our main Theorems 1 and 2. However, for arbitrary radial weight function w, the corresponding Green function /w may fail to be nonnegative everywhere, as shown by the counterexample of Hedenmalm and Zhu [16]. In this case, the study of approximate spectral synthesis requires a different approach. Finally, there is one more problem worth mentioning, which is related to approximate spectral synthesis in the Bergman space. Suppose we have two embedded z-invariant subspaces I  ⊂ I  ⊂ L2a (or two embedded z∗ -invariant subspaces J  ⊂ J  ⊂ (L2a )∗ ). What can be said about the simultaneous weak or strong approximate spectral cosynthesis (or synthesis) of these two subspaces? This problem is of particular interest, since it is closely related to the famous invariant subspace problem. It is known, from the theory of dual algebras, that the lattice of z-invariant subspaces in L2a is so rich that any strict contraction T in an abstract separable Hilbert space H is unitary equivalent to a compression of Mz in L2a , that is, to an operator of the form P(I  I  ) Mz |(I  I  ) for some z-invariant subspaces I  ⊂ I  ⊂ L2a (see [3]). A similar property holds for the operator Mz∗ and the lattice of z∗ -invariant subspaces. In particular, the abstract invariant subspace problem is equivalent to the following problem on z∗ -invariant subspaces: Is it true that, for any two z∗ -invariant subspaces J  ⊂ J  ⊂ (L2a )∗ with dim J  /J  = +∞, there exists a z∗ -invariant subspace J between J  and J  coinciding with neither J  nor J  ?

APPROXIMATE SPECTRAL SYNTHESIS

37

Suppose that we have simultaneous weak approximate spectral synthesis for J  and by embedded z∗ -invariant subspaces, that is, there exist z∗ -invariant subspaces  Jn ⊂ Jn ⊂ (L2a )∗ such that dim Jn < ∞, dim Jn < ∞ and J  = lim n→∞ Jn , J  = lim n→∞ Jn . Then we can apply arguments going back to the proofs of the theorems of Aronszajn and Smith [2] and Bernstein and Robinson [4] on the existence of nontrivial invariant subspaces for the compact and polynomially compact operators. First, application of the Schur theorem on the triangle form of a finite matrix shows that, for each n, there exists a chain J 

Jn = Jn(1) ⊂ Jn(2) ⊂ · · · ⊂ Jn(kn ) = Jn (l+1)

(l)

of z∗ -invariant subspaces such that dim(Jn /Jn ) = 1. Now we may try to find (s ) an intermediate subspace J as a lower limit of the type J = lim n→∞ Jn n for some sequence sn . Of course, the problem of finding a sequence sn , for which J does not coincide with J  and J  , remains open. Probably, certain compactness arguments related to functions of the operator Mz∗ could be applied in this situation. Acknowledgments. The main part of the present paper was written during the author’s stay at Bordeaux University, France, whose warm hospitality is very much appreciated. The author is grateful to Professor N. Nikolskii for helpful and stimulating discussions, and the author also thanks the referees for allowing these results to be appreciated by a wider audience. References [1] [2] [3]

[4] [5] [6] [7] [8] [9] [10]

A. Aleman, S. Richter, and C. Sundberg, Beurling’s theorem for the Bergman space, Acta Math. 177 (1996), 275–310. N. Aronszajn and K. T. Smith, Invariant subspaces of completely continuous operators, Ann. of Math. (2) 60 (1954), 345–350. H. Bercovici, C. Foia¸s, and C. Pearcy, Dual Algebras with Applications to Invariant Subspaces and Dilation Theory, CBMS Regional Conf. Ser. in Math. 56, Amer. Math. Soc., Providence, 1985. A. R. Bernstein and A. Robinson, Solution of an invariant subspace problem of K. T. Smith and P. R. Halmos, Pacific J. Math. 16 (1966), 421–431. A. Borichev and H. Hedenmalm, Cyclicity in Bergman-type spaces, Internat. Math. Res. Notices 1995, 253–262. R. G. Douglas, H. S. Shapiro, and A. L. Shields, Cyclic vectors and invariant subspaces for the backward shift operator, Ann. Inst. Fourier (Grenoble) 20 (1970), 37–76. P. Duren, D. Khavinson, H. S. Shapiro, and C. Sundberg, Contractive zero-divisors in Bergman spaces, Pacific J. Math. 157 (1993) 37–56. , Invariant subspaces in Bergman spaces and the biharmonic equation, Michigan Math. J. 41 (1994), 247–259. R. E. Edwards, Fourier Series: A Modern Introduction, Vol. 1, 2d ed., Grad. Texts in Math. 64, Springer-Verlag, New York, 1979. M. B. Gribov and N. K. Nikolskii, “Invariant subspaces and rational approximation” in Investigation on Linear Operators and the Theory of Functions, IX (in Russian), Zap.

38

[11] [12] [13]

[14] [15] [16] [17] [18]

[19] [20] [21]

[22] [23] [24] [25] [26]

[27]

[28] [29]

[30]

S. M. SHIMORIN Nauchn. Sem. Leningrad. Otdel. Mat. Inst. Steklov. (LOMI) 92, “Nauka” Leningrad. Otdel., Leningrad, 1979, 103–114, 320. P. Halmos, Measure Theory, Nostrand, New York, 1950. H. Hedenmalm, A factorization theorem for square area-integrable analytic functions, J. Reine Angew. Math. 422 (1991), 45–68. , “Open problems in the function theory of the Bergman space” in Festschrift in Honour of Lennart Carleson and Yngve Domar (Uppsala, 1993), Acta Univ. Upsaliensis Skr. Uppsala Univ. C Organ. Hist. 58, Uppsala Univ., Uppsala, 1995, 153–169. , An off-diagonal estimate of Bergman kernels, to appear in J. Math. Pures Appl. (9). H. Hedenmalm, B. Korenblum, and K. Zhu, Beurling type invariant subspaces of the Bergman spaces, J. London Math. Soc. (2) 53 (1996), 601–614. H. Hedenmalm and K. Zhu, On the failure of optimal factorization for certain weighted Bergman spaces, Complex Variables Theory Appl. 19 (1992), 165–176. V. E. Katsnelson, Weighted spaces of pseudocontinuable functions and approximations by rational functions with prescribed poles, Z. Anal. Anwendungen 12 (1993), 27–67. , “Description of a class of functions which admit an approximation by rational functions with preassigned poles, I” in Matrix and Operator Valued Functions, Oper. Theory Adv. Appl. 72, Birkhäuser, Basel, 1994, 87–132. D. Khavinson and H. S. Shapiro, Invariant subspaces in Bergman spaces and Hedenmalm’s boundary value problem, Ark. Mat. 32 (1994), 309–321 B. Korenblum, Outer functions and cyclic elements in Bergman spaces, J. Funct. Anal. 115 (1993), 104–118. N. K. Nikolskii, “Two problems on the spectral synthesis” in Studies in Linear Operators and Function Theory: 99 Unsolved Problems in Linear and Complex Analysis (in Russian), Zap. Nauchn. Sem. Leningrad. Otdel. Mat. Inst. Steklov. (LOMI) 81, “Nauka” Leningrad. Otdel., Leningrad, 1978, 139–141; English translation in J. Soviet Math. 26 (1984), 2185–2186. , Treatise on the shift operator: Spectral Function Theory, Grundlehren Math. Wiss. 273, Springer-Verlag, Berlin, 1986. , Distance formulae and invariant subspaces, with an application to localization of zeros of the Riemann ζ -function, Ann. Inst. Fourier (Grenoble) 45 (1995), 143–159. S. Richter, Invariant subspaces in Banach spaces of analytic functions, Trans. Amer. Math. Soc. 304 (1987), 585–616. K. Seip, On Korenblum’s density condition for the zero sequences of A−α , J. Anal. Math. 67 (1995), 307–322. S. Shimorin, Factorization of analytic functions in weighted Bergman spaces (in Russian), Algebra i Analiz 5 (1993), 155–177; English translation in St. Petersburg Math. J. 5 (1994), 1005–1022. , “On the Green function for the weighted biharmonic operator 0w −1 0 in the unit disk” in Some Questions of Mathematical Physics and Function Theory (in Russian), Probl. Mat. Anal. 16, S.-Peterburg. Gos. Univ., St. Petersburg, 1997, 266–277; English translation in J. Math. Sci. (New York) 92 (1998), 4404 – 4411. G. D. Taylor, Multipliers on Dα , Trans. Amer. Math. Soc. 123 (1966), 229–240. G. Ts. Tumarkin, Approximation with respect to various metrics of functions defined on the circumference by sequences of rational fractions with fixed poles (in Russian), Izv. Akad. Nauk SSSR Ser. Mat. 30 (1966), 721–766; English translation in Amer. Math. Soc. Transl. Ser. 2 77 (1968), 183–233. , The description of the class of functions which can be approximated by fractions with preassigned poles (in Russian), Izv. Akad. Nauk Armjan. SSR Ser. Mat. 1 (1966), 89–105.

APPROXIMATE SPECTRAL SYNTHESIS [31]

39

, Necessary and sufficient conditions for the possibility of approximating a function on a circumference by rational fractions, expressed in terms directly connected with the distribution of poles of the approximating fractions (in Russian), Izv. Akad. Nauk SSSR Ser. Mat. 30 (1966), 969–980; English translation in Amer. Math. Soc. Transl. Ser. 2 77 (1968), 235–248.

Department of Mathematics, Lund University, Box 118, S-221 00 Lund, Sweden; [email protected]

Vol. 101, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

SINGLY GENERATED PLANAR ALGEBRAS OF SMALL DIMENSION DIETMAR BISCH and VAUGHAN JONES 0. Introduction. A subfactor N ⊂ M gives rise to a powerful set of invariants that can be approached successfully in several ways. (See, for instance, [B2], [EK], [FRS], [GHaJ], [H], [Iz], [JSu], [Lo1], [Lo2], [Oc1], [Oc2], [Po1], [Po2], [Po3], [Po4], [Wa], [We1], and [We2]). A particular approach suggests a particular kind of subfactor as the “simplest.” For instance, in Haagerup’s approach [H], subfactors of small index are the simplest. In [J2], a pictorial language is developed in which the invariants appear as a graded vector space V = (Vn )n≥0 whose elements can be combined in planar, but otherwise quite arbitrary, ways. Thus, for instance, in the diagram

4

1 3 2

every time one assigns elements v1 ∈ V4 , v2 ∈ V2 , v3 ∈ V3 , and v4 ∈ V1 to the “empty boxes” 1, 2, 3, and 4, there is associated, in a multilinear and natural way, an element of V4 . The grading of the Vn ’s is given by half the number of strings attached to the boundary of the box. Various algebra and other structures are given by particular planar ways of combining elements. It is shown in [J2] that under appropriate positivity conditions on V (summed up by saying that V is a subfactor planar algebra), there is a subfactor N ⊂ M having Vn as its higher relative commutant N  ∩ Mn−1 . Any subset S of V then generates a planar subalgebra as the smallest graded vector space containing S and closed under planar operations. From this point of view, the simplest subfactors are those whose planar algebra is generated by the smallest sets S, Received 16 June 1998. 1991 Mathematics Subject Classification. Primary 46L37; Secondary 46L10. Bisch supported by a Heisenberg fellowship and National Science Foundation grant number DMS9531566. Jones supported by National Science Foundation grant number DMS-9322675 and the Marsden Fund. 41

42

BISCH AND JONES

while the index may be arbitrarily large. If S is empty, we obtain the Temperley-Lieb algebra. In [J2], planar algebras generated by elements of V1 are determined. The next case to examine is that of planar algebras generated by a single element of V2 . The problem is well posed and may even be tractable (while the problem of determining all finitely generated planar algebras certainly is not; see [J2]). In this paper, we take a first step in this program by classifying all subfactor planar algebras generated by a single element of V2 , for which dim V2 is as small as possible, that is, 3, and dim V3 is at most 12. In fact, the classification for dim V3 ≤ 15 appears to be possible, based on the techniques of [BiWe] and [Mu], but that is as yet beyond our calculations. In the course of our solution, we actually obtained a significantly stronger result, easiest stated in classical subfactor language. Namely, if N ⊂ M is a subfactor and the relative commutant N  ∩ M2 is abelian modulo the ideal generated by e2 (the projection from M1 onto M), then there is an intermediate subfactor P , N ⊂ P ⊂ M. The stronger result implies the classification, since the projection from M to P is in N  ∩ M1 (= V2 ) and generates a planar subalgebra completely analyzed in [BJ1] called the Fuss-Catalan algebra (or briefly FC algebra). We see that any subfactor planar algebra satisfying our hypothesis is thus a Fuss-Catalan algebra. (In fact there is one exception, when the index is 3, where the subfactor in question is the fixed point algebra for an outer Z3 -action.) Our first attempt at classification used the Ocneanu connection approach, and while successful, it involved many special cases and pages of opaque 3 × 3 matrix calculations. The more powerful result was inaccessible because of a large number of undetermined trace parameters. In this paper we adopt, for the first time in subfactor theory, a fully diagrammatic approach, where the “rotation” (a certain map on the higher relative commutants discussed in Section 3) plays a crucial role. Since these calculations are unfamiliar to experts, we translate several of the first diagrammatic arguments into algebraic ones, including a proof of the periodicity of the rotation in the special case of N  ∩ M2 (which is all that is needed in this paper). Finally, we show that our more powerful result applies to more than just the FC algebras by exhibiting a subfactor N ⊂ M (a free composition of two others; see [BH] and [BJ1]) with dim N  ∩ M1 = 3, N  ∩ M2 abelian modulo the ideal generated by e2 , and dim N  ∩ M2 = 14. It is obtained as the free composition of two subfactors with principal graph E6 . 1. Planar algebras and diagrams. The concept of a planar algebra was introduced in [J2], and it is shown there that the standard invariant of an extremal subfactor with finite index is a planar algebra satisfying certain positivity conditions. The proofs of our main results, described in Section 2, are motivated by the planar algebra point of view of the system of higher relative commutants associated to a subfactor, and we use planar algebra technology in the proofs. In particular, elements in the higher relative commutants and certain operators on the higher relative commutants can be described diagrammatically, and we explain this formalism carefully in the next three

SINGLY GENERATED PLANAR ALGEBRAS

43

sections. We would like to point out that all the computations in this paper which involve diagrams can also be done by replacing the diagrams with the operators they represent, and we explain to the reader in most places how to do the computations either way. We hope, however, to convince the reader that computing with diagrams leads to the desired result in a more efficient and intuitive way than manipulating traditional formulas on a purely algebraic level.
Recall that a general planar algebra (see [J2]) is a filtered algebra P = k≥0 Pk together with a homomorphism of filtered algebras  : ᏼ(L) → P , where ᏼ(L) =
k≥0 ᏼk (L) denotes the universal planar algebra on the label set L = k≥0 Lk , with Lk a label set of the k-boxes of ᏼk (L) (where the zero-boxes are just planar networks). An element in ᏼk (L) or Pk is depicted as a k-box, as in Figure 1,

R Figure 1

where the orientations of the k vertical strings on the top and bottom of the box are alternating up-down-up-down and R is a label in Lk . The homomorphism  must be such that Ker  is invariant under the action of Ꮽ(L), the set of all isotopy classes of labelled annular j -k tangles, for all j and k. (See [J2] for details and other equivalent definitions using operads.) If dim P0 = 1, we say that P is connected. (In fact, the definition of connectedness of a planar algebra is slightly more subtle; see [J2] for the precise definition). If P is a connected, general planar algebra with presenting map  : ᏼ(L) → P as above, then  applied to a labelled network ᏺ ∈ ᏼ0 (L) gives rise to a scalar, which we denote by Z (ᏺ). (More precisely, if the unbounded region of ᏺ is positively oriented, then (ᏺ) = Z (ᏺ) · 1 ∈ P0 defines the scalar Z (ᏺ). If the unbounded region is negatively oriented, add a vertical string to the left and take the partition function of the resulting tangle.) Z (ᏺ) is called the partition function of ᏼ, and it is an isotopy invariant of labelled planar networks. A connected, general planar algebra is then called a planar algebra if there is a presenting map  such that the partition function Z is multiplicative on connected components. We require the parameters δ1 = Z ( ) and δ2 = Z ( ) to be not equal to zero. It turns out that δ1 , δ2 , and the multiplicativity of Z do not depend on the presenting map . Observe that the notion of planar algebra is algebraic and topological, but we are mostly interested in certain subclasses of planar algebras that have certain positivity properties,

44

BISCH AND JONES

the so-called C ∗ -planar algebras, which we explain next. Suppose we are given an involution R ∈ L → R ∗ ∈ L on the label set L. ᏼ(L) then becomes a ∗-algebra in the obvious way. We call a ∗-algebra P a (general) planar ∗-algebra if it is a (general) planar algebra and if it is presented by  on a ᏼ(L), L with an involution ∗ in such a way that  is a filtered ∗-homomorphism. Note that the partition function Z is a sesquilinear form if P is connected.
Given a connected, planar algebra P = k≥0 Pk with partition function Z, one         can define two traces tr L and tr R by tr L R = Z R and tr R R = Z R , with R an arbitrary element in Pk , for all k ≥ 0. Z is called positive if tr L (x ∗ x) ≥ 0, for all x ∈ Pk , k ≥ 0. If, in addition, tr L (x ∗ x) > 0 whenever x  = 0 (this holds if and only if tr R (x ∗ x) > 0, for all x  = 0), then P is called nondegenerate.
If dim Pk < ∞, for all k ≥ 0, then we say that P is finite-dimensional. If P = k≥0 Pk is a finite-dimensional, nondegenerate, connected, planar ∗-algebra with positive partition function, we can make each Pk into a C ∗ -algebra. P is then called a C ∗ -planar algebra. A (connected) planar algebra is called spherical if its partition function is an invariant of planar networks on the 2-sphere S 2 . Note that in this case tr L = tr R def

and δ1 = δ2 = δ. Spherical C ∗ -planar algebras (or subfactor planar algebras) are precisely the standard invariants of extremal subfactors with finite index (see [Po3] and [J2]). We next describe how the diagram formalism applies to the analysis of the system of higher relative commutants associated to a subfactor. Let N ⊂ M be an (extremal) e1

e2

e3

inclusion of II1 factors with finite index, and denote by N ⊂ M ⊂ M1 ⊂ M2 ⊂ M3 ⊂ · · · the associated tower of II1 factors constructed from N ⊂ M by iterating the basic construction (see [J1]). We denote as usual by J : L2 (M) → L2 (M) the modular conjugation and by tr or tr M the normalized, faithful trace on M. From the tower of II1 factors associated to N ⊂ M one obtains a sequence of finite-dimensional commuting squares C = N  ∩ N ⊂ N  ∩ M ⊂ N  ∩ M 1 ⊂ N  ∩ M2 ⊂ . . . ∪ ∪ ∪    C = M ∩ M ⊂ M ∩ M1 ⊂ M ∩ M2 ⊂ . . . which is called the system of higher relative commutants, or the standard lattice (see [Po3]), or the standard invariant (see [Po2]), or the Popa system (see [BJ1]), associated to the subfactor N ⊂ M (see also [GHaJ]). Since spherical C ∗ -planar algebras are precisely the standard invariants of extremal subfactors with finite index (see [J2]), we can think of an element R ∈ N  ∩ Mk as being depicted by a labelled (k+1)-box as in Figure 1 (with k+1 vertical strings on the top and bottom of the box and orientations alternating up-down-up-down). We usually

45

SINGLY GENERATED PLANAR ALGEBRAS

omit the orientation of the vertical strings and simply write R for R ∈ N  ∩ Mk . The (unital) embedding R ∈ N  ∩Mk → N  ∩Mk+1 is given by adding to a (k +1)-box R a vertical string at the right end of the box (with the appropriate orientation) so that we get R . Note that it is not hard to see that a spherical C ∗ -planar algebra gives rise to a Popa system (see [J2]), and hence, by a theorem of Popa, is the system of higher relative commutants of an extremal subfactor (which is not necessarily hyperfinite); see [Po3]. Conversely, it can be shown that a system of higher relative commutants of an extremal subfactor gives rise to a spherical C ∗ -planar algebra (see [J2]). This requires a bit more work, and we do not need to use the result in this paper. R

If Q, R ∈ N  ∩ Mk are given by Q R , then Q · R is depicted as Q . (Caution: We follow the convention of multiplying diagrams from bottom to top, which is also the convention used in [BJ1].) As is well known, the special (k +1)-boxes Ej ∈ N  ∩Mk , 1 ≤ j ≤ k, given by Figure 2, play an important role in the theory. (The diagram in Figure 2 has k + 1 vertical strings, and the arcs connect hook j and j + 1 on the top and bottom lines of the box.)

Figure 2

We denote as above the parameter associated to a closed loop by δ, and we recall that δ = [M : N ]1/2 (see, for instance, [J1], [BJ1], and [GHaJ]). The projections ej = (1/δ)Ej , 1 ≤ j ≤ k, generate the Temperley-Lieb algebra, if δ > 2 (see, for instance, [GHaJ], [BJ1]). The projection ek+1 implements the trace-preserving conditional expectation EMk−1 : Mk → Mk−1 , that is, ek+1 xek+1 = EMk−1 (x)ek+1 , for all x ∈ Mk . When restricted to the relative commutant N  ∩ Mk , EMk−1 is equal to EN  ∩Mk−1 , the trace-preserving conditional expectation N  ∩ Mk → N  ∩ Mk−1 . Sime0 ilarly, if N ⊂ M is extremal and we let N1 ⊂ N ⊂ M be one step in the downward basic construction (see [J1]), then e0 implements the tr M -preserving conditional expectation EM  ∩Mk : N  ∩Mk → M  ∩Mk . If N ⊂ M is not extremal, e0 implements the tr N  -preserving conditional expectation (see also [B2, Proposition 2.7]). These conditional expectations can be written diagrammatically as follows: Let R ∈ N  ∩ Mk be an arbitrary element depicted as R . Then EMk−1 (R) = EN  ∩Mk−1 (R) is computed diagrammatically in Figure 3 (up to scalars).

46

BISCH AND JONES

R

=

R

Figure 3

Thus, taking scalars into account, we get that δEN  ∩Mk−1 (R) is given by R . Similarly, we get that δEM  ∩Mk (R) is given by the diagram R . Observe that the orientations of the vertical strings of the k-box R are alternating down-up-down-up. Thus elements in M  ∩Mk are depicted as k-boxes. However, the orientations are reversed; that is, the k vertical strings on the top and bottom of the box have orientations alternating down-up-down-up from left to right. The unital embedding M  ∩ Mk → N  ∩ Mk is obtained by taking a k-box representing an element in M  ∩ Mk and adding R . a vertical string to the left of the box, which is oriented upward, that is, The result is a (k + 1)-box oriented correctly as in Figure 1. Note that the ∗-operation on N  ∩ Mk is obtained by replacing the label R in a (k + 1)-box by the label R ∗ . Other important operations on the higher relative commutants are presented in Section 3. We use in Section 4 a natural (linear) basis of N  ∩ M1 and of a certain direct summand in N  ∩ M2 . Recall that (N  ∩ Mk )ek+1 (N  ∩ Mk ) is a two-sided ideal in N  ∩ Mk+1 , which we call the basic construction ideal in N  ∩ Mk+1 . It is the ideal generated by ek+1 and is isomorphic to the basic construction for N  ∩ Mk−1 ⊂ N  ∩ Mk (see [GHaJ]). If dim N  ∩ M1 = 3, then dim(N  ∩ M1 )e2 (N  ∩ M1 ) = 9. Let us write N  ∩ M1 = Ce1 ⊕ Cq ⊕ C(1 − e1 − q), where e1 , q, and 1 − e1 − q are the minimal (central) or briefly by as projections in N  ∩ M1 . We depict q by Q and E1 = δe1 by

usual (where δ = [M : N]1/2 ). Later, we also use the projection p = e1 + q, which is depicted by P . The identity in N  ∩ M1 is depicted by or briefly by . Clearly, ,   , and Q or , , and P form a (linear) basis of N  ∩ M1 . Thus the nine diagrams in Figure 4 are a basis of (N  ∩ M1 )e2 (N  ∩ M1 ). These diagrams, of course, are just the elements E1 , E2 , E1 E2 , E2 E1 , qE2 , E2 q, qE2 E1 , E1 E2 q, and qE2 q in N  ∩ M2 (from left to right), where ei = (1/δ)Ei as above.

47

SINGLY GENERATED PLANAR ALGEBRAS

Q

Q

Q

Q

Q Q

Figure 4

Later, we also extensively use the elements δ 3 EM  ∩M2 (qe2 e1 ) and δ 3 EM  ∩M2(e1 e2 q) depicted in Figure 5.

Q

Q Figure 5

Note that these are elements in M  ∩ M2 , so that the orientation of the vertical strings is down-up (from left to right). 2. The main theorems. We describe in this section the main results of this paper. Let us start with the main theorem. Theorem 2.1. Let N ⊂ M be an inclusion of II1 factors with 3 < [M : N] < ∞. Suppose that dim N  ∩ M1 = 3 and that N  ∩ M2 is abelian modulo the basic construction ideal (N  ∩ M1 )e2 (N  ∩ M1 ). Then there is an intermediate subfactor P of N ⊂ M, P  = N , M. In particular, J x ∗ J = x, for all x ∈ N  ∩ M1 . Note that if [M : N] = 3 and dim N  ∩ M1 = 3, then the subfactor is given as the fixed point algebra for an outer Z3 action; that is, it is of the form N = M Z3 , and the associated planar algebra (or equivalently the associated Popa system) is determined by the group Z3 and its representation theory. Furthermore, observe that the condition dim N  ∩ M1 = 3 implies that N ⊂ M is irreducible and, hence, extremal. We give a proof of Theorem 2.1 in Section 4. Let us point out that the conditions in Theorem 2.1 are very simple conditions on the shape of the principal graph for the tower of inclusions C = N  ∩ N ⊂ N  ∩ M ⊂ N  ∩ M1 ⊂ · · · . Theorem 2.1 says that if the principal graph of N ⊂ M is of the form shown in Figure 6, then N ⊂ M must have an intermediate subfactor.
Theorem 2.1 allows us to classify all spherical C ∗ -planar algebras V = ∞ k=0 Vk , which are generated (as spherical C ∗ -planar algebras) by a single 2-box, subject to the conditions dim V2 = 3 (which implies dim V0 = dim V1 = 1) and dim V3 ≤ 12. Note that it is shown in [J2], using a theorem of Popa (see [Po3]), that every spherical C ∗ planar algebra gives rise to an extremal subfactor N ⊂ M such that N  ∩ Mk+1 = Vk . In fact, every Popa system coming from an extremal subfactor is a spherical C ∗ -planar algebra (and therefore called a subfactor planar algebra), so that one can go freely between subfactor and planar algebra language (see [J2]).

48

BISCH AND JONES

∗

Figure 6

We call the system of higher relative commutants associated to a subfactor N = M Z3 ⊂ M the Z3 -planar algebra. Note that it is completely determined by the commuting square u u∗ ⊂ M3 (C) ∪ ∪ ⊂ , C where is the algebra of diagonal matrices in√M3 (C) and u = (uij )0 ≤ i, j ≤ 2 is the unitary 3 × 3 matrix with entries ur, s = (1/ 3)(σ r (a s ))0 ≤ r, s ≤ 2 , where Z3 = {1, a, a 2 }, with dual group Zˆ 3 = {1, σ, σ 2 }. Its description in terms of planar diagrams can be found in [J2]. Similarly, we call the system of higher relative commutants of a subfactor N ⊂ M, with principal graphs (D∞ , D∞ ), the D∞ -planar algebra. (Recall that both graphs are automatically D∞ if one is.) It is again uniquely determined by a commuting square (this time an infinite one); see, for instance, [Po3] and [H]. Recall that this system can be obtained as the Popa system of a free composition (see [BH] and [BJ1]) of the form N = P Z2 ⊂ P ⊂ P
Z2 , where the group G = Z2 , Z2 , generated by two copies of Z2 in the outer automorphism group of the II1 factor P , is the infinite dihedral group Z2 ∗ Z2 . This system is therefore a special case of the Fuss-Catalan (FC) planar algebras, discovered in [BJ1], which can be viewed as the Popa system of a subfactor obtained from the free composition of the Popa system of a subfactor with principal graphs (An , An ) and that of a subfactor with principal graphs (Am , Am ), n, m = 3, 4, . . . , ∞. The FC planar algebras can be described as colored generalizations of the Temperley-Lieb algebras, and an explicit description as planar algebras can be found in [BJ1] (see also [J2]). These colored generalizations of the Temperley-Lieb algebras turn out to be the minimal system of algebras appearing whenever an intermediate subfactor is present. We recall that their tower of inclusions is given by the Fibonacci

SINGLY GENERATED PLANAR ALGEBRAS

49

graph F (see Figure 7) in the generic case (i.e., the case of free composition of two subfactors with principal graphs A∞ ) and certain subgraphs of F , of which D∞ is a special case, in the nongeneric case. We refer the reader to [BJ1] for details.

Figure 7

Theorem 2.1 implies the following classification result. Theorem 2.2. If V is a spherical C ∗ -planar algebra generated by a threedimensional V2 , subject to the condition dim V3 ≤ 12, then it must be one of the following: (a) if dim V3 = 9, then it is the Z3 -planar algebra; (b) if dim V3 = 10, then it is the D∞ -planar algebra (a special case of (c) in fact); (c) if dim V3 = 11 or 12, it is one of the FC planar algebras in [BJ1]. Proof. Observe that if dim V2 = 3, then dim V0 = dim V1 = 1 and we are investigating spherical C ∗ -planar algebras generated by a single (nontrivial) 2-box such that dim V3 ≤ 12. Let N ⊂ M be a subfactor whose system of higher relative commutants is given by V . If dim V3 = 9, then [M : N] = 3 and we are in case (a). If 10 ≤ dim V3 ≤ 12, Theorem 2.1 implies that N ⊂ M must have an intermediate subfactor. On the other hand, it was shown in [BJ1] that the FC planar algebra (i.e., the system of algebras FC n (a, b) in the notation of [BJ1]) must be contained in the system of higher relative commutants of any subfactor that has an intermediate subfactor. Since by assumption no other conditions on the 2-box are assumed to hold, the spherical C ∗ -planar algebras generated by a single 2-box must be precisely the ones listed in the theorem. 3. Some useful results. We prove in this section several results that are needed to prove the main theorem. We also include some results about Popa systems and intermediate subfactors that are of interest in their own right. Furthermore, we show the reader how to use the diagrammatic description of elements in the higher

50

BISCH AND JONES

relative commutants, and we explain how complicated formulas become simple when rewritten by using diagrams. We start with a lemma. Lemma 3.1. (i) Let N ⊂ M be an inclusion of II1 factors with finite index. Then we have [M : N]EMk (xek+1 )ek+1 = xek+1 , for all x ∈ Mk+1 . (ii) Let N ⊂ M be an extremal inclusion of II1 factors with finite index, and let x ∈ N  ∩ Mk . Then [M : N]EM  ∩Mk (xe1 )e1 = xe1 . Proof. Let us first show (i). Since xek+1 = yek+1 for a unique y ∈ Mk (see [PiPo1]), namely, y = [M : N]EMk (xek+1 ), the result follows. To prove (ii), we let {mi }i∈I be an orthonormal basis of M over N with m1 = 1 (see [PiPo1]). Then, by [B2, Proposition 2.7], we have EM  ∩Mk (xe1 ) =  [M : N]−1 i∈I mi xe1 m∗i . Since e1 m∗i e1 = EN (m∗i )e1 = 0 if i  = 1 and = 1 if i = 1, we are done. The following proposition establishes that certain higher relative commutants are isometric, although not isomorphic in general (see [B2] and [Oc2]). Proposition 3.2. Let N ⊂ M be an extremal inclusion of II1 factors with finite index. Consider the maps ψk : N  ∩ Mk → M  ∩ Mk+1 , k ≥ 0 (M0 = M), ψk (x) = [M : N](k+2)/2 EM  ∩Mk+1 (xek+1 ek · · · e2 e1 ), x ∈ N  ∩ Mk and φk : M  ∩ Mk+1 → N  ∩ Mk , k ≥ 0, φk (x) = [M : N](k+2)/2 EMk (xe1 e2 · · · ek ek+1 ). Then ψk and φk are isometries (in the norm  · 2 ), inverse to each other. Proof. Fix a k ≥ 0, and let ψ = ψk and φ = φk . Set vk = ek ek−1 · · · e2 e1 , k ≥ 1, and note that vk vk∗ = [M : N]−(k−1) ek and vk∗ vk = [M : N]−(k−1) e1 . Let x, y ∈ M  ∩ Mk , and compute the inner product       ∗  φ(x), φ(y) L2 (M ) = tr φ(y)∗ φ(x) = [M : N]k+2 tr EMk (vk+1 y ∗ )EMk xvk+1 k+1   = [M : N]k+2 tr EMk (ek+1 vk y ∗ )xvk∗ ek+1   = [M : N]k+1 tr ek+1 vk y ∗ xvk∗     = [M : N ]k tr vk∗ vk y ∗ x = [M : N] tr e1 y ∗ x     = [M : N ] tr EM  ∩M1 (e1 )y ∗ x = tr y ∗ x , where we used Lemma 3.1(i) in the third equality and extremality in the last one. Thus φ is indeed an isometry. The calculation for ψ is similar. Let x, y ∈ N  ∩ Mk ; then     ∗   ψ(x), ψ(y) L2 (M ) = [M : N]k+2 tr EM  ∩Mk+1 vk+1 y ∗ xvk+1 k  ∗  = [M : N]k+1 tr vk+1 y ∗ xvk+1     = [M : N] tr ek+1 y ∗ x = [M : N] tr EMk (ek+1 )y ∗ x   = tr y ∗ x , where we used Lemma 3.1(ii) in the second equality. Thus ψ is an isometry as well.

SINGLY GENERATED PLANAR ALGEBRAS

51

Let us finally show that the two maps are inverses of each other. Let x ∈ M  ∩Mk+1 ; then   ∗     vk+1 ψ φ(x) = [M : N]k+2 EM  ∩Mk+1 EMk xvk+1  ∗  = [M : N]k+1 EM  ∩Mk+1 xvk+1 vk   = [M : N]EM  ∩Mk+1 xe1 = x. ∗ ) = [M : N]k+1 Similarly, if x ∈ N  ∩Mk , then φ(ψ(x)) = EMk (EM  ∩Mk+1 (xvk+1 )vk+1 ∗ EMk (xvk+1 vk+1 ) = x. Note that again we used Lemma 3.1 in both calculations.

The maps φk and ψk are usually referred to as Fourier transforms (see [B2, Section 2], [Oc1] and [Oc2]). Let us describe how φ1 and ψ1 act on 2-boxes. If an arbitrary element R ∈ N  ∩M1 is represented by R , then ψ1 (R) = δEM  ∩M2 (RE2 E1 ) is given by

=

R

R

=

Q

as an element in N  ∩ M2 . Similarly, if an arbitrary element Q ∈ M  ∩ M2 ⊂ N  ∩ M2 is represented by Q , then φ1 (Q) = δEM1 (QE1 E2 ) = δEN  ∩M1 (QE1 E2 ) is given by

Q

Using the diagrams, it is now easy to verify that the two maps are inverse to each other. We next give a diagrammatic description of the map Jk · Jk : N  ∩ Mk → N  ∩ Mk , where Jk : L2 (Mk ) → L2 (Mk ) is the modular conjugation. Recall first that the Jones ek2k+1

projection for N ⊂ Mk ⊂ M2k+1 is given by   ek2k+1 = [M : N ]k(k+1)/2 ek+1 ek · · · e1      × ek+2 ek+1 · · · e2 · · · e2k · · · ek e2k+1 · · · ek+1

52

BISCH AND JONES

(see [PiPo2]). Thus, if Ek2k+1 is given by the diagram in Figure 8

Figure 8

((k + 1) arcs at the top and bottom), then ek2k+1 = δ −(k+1) Ek2k+1 = [M : N]−(k+1)/2 Ek2k+1 . To simplify notation, set fk = ek2k+1 . It is shown in [B2, Theorem 2.6] that  M for x ∈ N  ∩ M2k+1 , we have Jk x ∗ Jk = [Mk : N] i∈I EMk2k+1 (fk mi x)fk m∗i , where  {mi }i∈I is any finite basis of Mk over N with y = i∈I mi EN (m∗i y), for all y ∈ Mk . fk

f2k+1

f2k+1 Jk x ∗ Jk = Thus, if N ⊂  Mk ⊂ M2k+1 ⊂ M3k+2 is the basic construction, then  ∗ k+1 k+1 [M : N] fk mi = [M : N ] f2k+1 fk ( i∈I mi xf2k+1 i∈I f2k+1 fk mi xf2k+1 ∗ ∗ −(k+1)   2 fk mi ). But (y) = [M : N ] i∈I mi ymi , y ∈ N = N ∩ B(L (Mk )), im  plements the tr N  -preserving conditional expectation N → Mk (see, for instance, [B2, Proposition 2.7]). Hence, if N ⊂ M is extremal, we have the following diagrams. Suppose an arbitrary element R ∈ N  ∩ M2k+1 is depicted as a (2k + 2)-box R with 2k +2 vertical strings on top and bottom. Then Rf2k+1 fk is given by Figure 9 (up to the scalar δ 2k+2 ).

}

}

2k + 2 k + 1

R }

2k + 2 k + 1 Figure 9

Hence



i∈I

mi Rf2k+1 fk m∗i is depicted as

SINGLY GENERATED PLANAR ALGEBRAS

k +1

53

k +1k +1

R k +1

k +1 k +1 Figure 10

and therefore f2k+1 fk multiple):



i∈I

mi Rf2k+1 fk m∗i is given by Figure 11 (up to a scalar 2k + 2

k +1

R k +1 k +1 k +1 Figure 11

Since Jk x ∗ Jk = [M : N]k+1 EM2k+1 (f2k+1 Jk x ∗ Jk ), x ∈ N  ∩ M2k+1 ; keeping track of the scalars, we get that Jk R ∗ Jk is depicted as

R Figure 12

We have therefore proved the following proposition.

54

BISCH AND JONES

Proposition 3.3. If the element R ∈ N  ∩ M2k+1 is depicted as R , then Jk R ∗ Jk is depicted as . R

Note that Proposition 3.3 can, in fact, be deduced from a more general map on N  ∩ M2k+1 , the rotation of order 2k + 2. (See Theorem 3.8 for the case k = 2 and [J2] for the general case.) The map Jk · Jk : N  ∩ M2k+1 → N  ∩ M2k+1 is then just a rotation of a (2k + 2)-box by 180 degrees. Let us prove another useful formula for the maps Jk · Jk on N  ∩ M2k+1 , related to the Fourier transforms in Proposition 3.2. Proposition 3.4. Let A ⊂ B be an extremal inclusion of II1 factors with finite index, and let A ⊂ B ⊂ B1 ⊂ B2 be the beginning of the tower associated to A ⊂ B. Then J x ∗ J = [B : A]3 EB1 (EB  ∩B2 (e1 e2 x)e1 e2 ), for all x ∈ A ∩ B1 . Furthermore, we have J x ∗ J = [B : A]3 EB1 (e2 e1 EB  ∩B2 (xe2 e1 )). Proof. Let {mi }i∈I be an orthonormal basis of B over A. Then we have, for x ∈ A ∩ B1 ,        3 2 ∗ [B : A] EB1 EB  ∩B2 e1 e2 x e1 e2 = [B : A] EB1 mi e1 e2 xmi e1 e2 = [B : A]

2



= [B : A]

2



i∈I i∈I

= [B : A]

 i∈I

i∈I

  mi e1 EB1 e2 xm∗i e1 e2 mi e1 EB (xm∗i e1 )EB1 (e2 )

  mi e1 EB xm∗i e1

= (J xJ )∗ = J x ∗ J, where we used extremality in the first equation and [B2, Theorem 2.6] in the last line. The second formula follows by a similar computation. We changed the notation in Proposition 3.4 slightly, denoting an inclusion of II1 factors by A ⊂ B. Clearly, if we let A = N and B = Mk , we get a statement about Jk · Jk : N  ∩ M2k+1 → N  ∩ M2k+1 . Note that the right-hand side of the formulas for J x ∗ J in Proposition 3.4 make sense even for a nonstandard system of finite-dimensional algebras, as in [Po3]. Hence multiplicativity of the formula is an obstruction for the system to be standard, that is, to come from a subfactor. More precisely, we must have [B : A]3 (EB1 (EB  ∩B2 (e1 e2 x) e1 e2 ))2 = EB1 (EB  ∩B2 (e1 e2 x 2 )e1 e2 ), x ∈ A ∩ B1 , in this case. We next mention a useful result regarding the normalizer of a subfactor, which can be found in [PiPo1]. Recall that if N ⊂ M is a subfactor, then the normalizer ᏺ(N ) is defined as ᏺ(N ) = {u ∈ ᐁ(M) | uNu∗ = N}. Clearly, all unitaries u ∈ N are contained in ᏺ(N ), and we call ᏺ(N) nontrivial if it contains a unitary in M not contained in N .

SINGLY GENERATED PLANAR ALGEBRAS

55

Proposition 3.5. Let N ⊂ M be an irreducible inclusion of II1 factors with finite index, and suppose there is a projection p ∈ N  ∩M1 that is equivalent to e1 (in M1 ), p  = e1 . Then N has a nontrivial normalizer. Proof. The proof is very simple. By assumption, there is a unitary u ∈ M1 such that ue1 u∗ = p. By [PiPo1], there is a unique v ∈ M with ue1 = ve1 . Thus ve1 v ∗ = ue1 u∗ = p, and hence [M : N ]−1 vv ∗ = EM (p) = [M : N]−1 · 1 (since N  ∩ M = C). Hence v ∈ M is a unitary. Clearly v ∈ / N , since otherwise p = ve1 v ∗ = e1 vv ∗ = e1 , which is a contradiction. Moreover, v normalizes N: Let x ∈ N, then v ∗ ve1 v ∗ xv = v ∗ (ve1 v ∗ )xv = v ∗ pxv = v ∗ xpv = v ∗ xvv ∗ pv = v ∗ xve1 . Thus v ∗ xv ∈ {e1 } ∩ M = N. Observe that the von Neumann algebra generated by ᏺ(N) is an intermediate subfactor of N ⊂ M (which is assumed to be irreducible). Proposition 3.5 can therefore be useful to show the existence of intermediate subfactors. We continue our collection of useful facts with a diagrammatic way of deciding whether a projection in N  ∩ M1 comes from an intermediate subfactor. The next proposition is a reformulation of [B1, Theorem 3.2]. (See also [La] for applications of similar types of relations as the ones in Proposition 3.6(iii) to planar algebras.) Proposition 3.6. Let N ⊂ M be an irreducible inclusion of II1 factors with finite index. Let p ∈ N  ∩ M1 be an operator depicted by P . Then p is the orthogonal projection onto an intermediate subfactor if and only if (i) p = p2 , p = p ∗ , or diagrammatically 

P =

P ,

∗

   P  =

P

Figure 13

(ii) e1 ≤ p, or diagrammatically

P

=

P

Figure 14

(iii) the following exchange relation holds

=

P

56

BISCH AND JONES

P

P P

=

P

Figure 15

Proof. We use the abstract characterization of intermediate subfactors in [B1]. Conditions (i) and (ii) are obviously necessary and are satisfied if p is the orthogonal projection onto an intermediate subfactor. p1 e1 Suppose N ⊂ M has an intermediate subfactor P , and let N ⊂ P ⊂ M ⊂ P1 ⊂ p2

e2

M1 ⊂ P2 ⊂ M2 be the first few steps in the tower of II1 factors associated to N ⊂ P ⊂ M (see [B1] and [BJ1]). In particular, p2 is the projection obtained from the basic construction for P1 ⊂ M1 , and p = p1 is the orthogonal projection ePM from M onto P . Let α = [P : N]−1 = tr(p2 ), β = [M : P ]−1 = tr(p1 ). Then p1 e2 e1 = βp2 e1 (see [B1]), and hence p2 = α −1 β −2 EM  ∩M2 (p1 e2 e1 ) and p1 = α −2 β −1 EM1 (p2 e1 e2 ). Thus p2 is depicted as

P Figure 16

(up to a scalar, i.e., the diagram in Figure 16 equals (β/α)1/2 p2 ), and (iii) is true if and only if p1 p2 = p2 p1 , which holds since p2 actually commutes with the II1 factor P1 . Conversely, if (iii) holds, then p1 EM  ∩M2 (p1 e2 e1 ) = EM  ∩M2 (p1 e2 e1 )p1 , and hence p1 EM  ∩M2 (p1 e2 e1 )e2 = EM  ∩M2 (p1 e2 e1 )p1 e2 , which implies EM  ∩M2 (p1 e2 e1 )e2 = αβEM  ∩M2 (p1 e2 ) = αβ 2 e2 (since EM  ∩M2 (p1 ) = β · 1). Thus we get  2      = EM  ∩M2 EM  ∩M2 p1 e2 e1 p1 e2 e1 EM  ∩M2 p1 e2 e1     = EM  ∩M2 p1 EM  ∩M2 p1 e2 e1 e2 e1   = αβ 2 EM  ∩M2 p1 e2 e1 . We next show that EM  ∩M2 (p1 e2 e1 ) is selfadjoint. Since e2 p1 EM  ∩M2 (p1 e2 e1 ) = e2 EM  ∩M2 (p1 e2 e1 )p1 and e2 p1 e2 = EM (p1 )e2 = βe2 , we get e2 p1 EM  ∩M2 (p1 e2 e1 ) = βe2 EM  ∩M2 (e1 )p1 = αβ 2 e2 p1 . Hence, multiplying the previous equation with e1 from the left and expecting onto M  ∩ M2 , we get EM  ∩M2 (e1 e2 p1 )EM  ∩M2 (p1 e2 e1 )

SINGLY GENERATED PLANAR ALGEBRAS

57

= αβ 2 EM  ∩M2 (e1 e2 p1 ). Note that the left-hand side is selfadjoint. Thus EM  ∩M2 (p1 e2 e1 ) is a multiple of a projection, which shows by [B1, Corollary 3.3] that N ⊂ M has an intermediate subfactor. Let us show how the proof proceeds, using diagrams. We multiply the equation in Proposition 3.6(iii) with the diagram E2 and get

P

= P

P

P Since P = β , P = β , we get condition (iii ) after expecting onto M1 ,

c P = P

P

Figure 17

for a nonzero scalar c. Thus, by applying the Fourier transform (Proposition 3.2), we get

P P

=c P Figure 18 P

which says that is a scalar multiple of an idempotent. We leave it to the reader to give the diagrammatic argument for selfadjointness of . Observe that we have actually shown that condition (iii) in Proposition 3.6 and condition (iii ) depicted in Figure 17 for some nonzero scalar c plus selfadjointness of are equivalent. Indeed, we showed that (iii) implies (iii ). Conversely, if (iii ) P

P

58

BISCH AND JONES P

holds, then is a multiple of a projection and hence, by [B1], is (up to a scalar) the projection p2 coming from an intermediate subfactor (notation is as in the first part of the proof of Proposition 3.6). But then p1 p2 = p2 p1 ; that is, condition (iii) holds. If a projection p ∈ N  ∩M1 satisfies (i), (ii), and (iii) (or (iii )), we call it a biprojection. Thus the biprojections are precisely those orthogonal projections in N  ∩M1 that project onto an intermediate subfactor N ⊂ P ⊂ M (assuming N ⊂ M is irreducible). The next part is probably the most important result in this section. Consider the maps ψk : N  ∩ Mk → M  ∩ Mk+1 , ψk (x) = [M : N](k+2)/2 EM  ∩Mk+1 (xek+1 ek · · · e2 e1 ), x ∈ N  ∩ Mk , and φk : M  ∩ Mk+1 → N  ∩ Mk , φk (x) = [M : N](k+2)/2 EMk (xe1 e2 · · · ek ek+1 ), as in Proposition 3.2 (k ≥ 1). Let φ˜ k : M  ∩ Mk+1 → N  ∩ Mk be defined by φ˜ k (x) = φk (x ∗ )∗ , x ∈ M  ∩ Mk+1 . Definition 3.7. The map rk : N  ∩Mk → N  ∩Mk defined by rk = φ˜ k ◦ψk is called the rotation of period k + 1 (or the rotation by 360/(k + 1) degrees) on the higher relative commutant N  ∩ Mk . We would like to point out that the fact that the map rk acts on the higher relative commutants of a subfactor is one of the key features of such a system of finitedimensional algebras. It is to a large extent “responsible” for the planar structure of the higher relative commutants (see [J2]). The next theorem justifies Definition 3.7. Theorem 3.8. Let N ⊂ M be an extremal inclusion of II1 factors with finite index, and let rk : N  ∩ Mk → N  ∩ Mk be defined by    rk (x) = [M : N]k+2 EMk ek+1 ek · · · e2 e1 EM  ∩Mk+1 xek+1 ek · · · e2 e1 , x ∈ N  ∩ Mk , as in Definition 3.7. Then rkk+1 = id. In this paper, we only use the maps r1 and r2 . Observe that the case k = 1 has already been shown in Proposition 3.3, and we give a proof in the case k = 2 below. For the general case, see [J2], where a different proof is presented. Observe that since Mk+1 ⊂ Mk ∪ ∪  ∩M  ⊂ N N ∩ Mk k+1 is a commuting square, we have rk (x) = [M : N]k+2 EN  ∩Mk (ek+1 ek · · · e2 e1 EM  ∩Mk+1 (xek+1 ek · · · e2 e1 )). We proceed with the proof of Theorem 3.8 for k = 2. ˜ Proof. Let ψ = ψ2 , φ = φ2 , φ(x) = φ(x ∗ )∗ , x ∈ M  ∩ M2 , and r2 = φ˜ ◦ ψ. Note ˜ that φ and ψ are invertible with inverses ψ −1 = φ and φ˜ −1 (x) = ψ(x ∗ )∗ , x ∈ N  ∩M2 (see Proposition 3.2). Thus r23 = id if and only if ψ ◦ φ˜ ◦ ψ = φ˜ −1 ◦ ψ −1 ◦ φ˜ −1 and if and only if   ∗    ∗ ∗ ψ φ ψ(x)∗ = ψ φ ψ(x ∗ )∗ , (∗)

59

SINGLY GENERATED PLANAR ALGEBRAS

for all x ∈ N  ∩ M2 . The equality (∗) holds if and only if      EM  ∩M3 EM2 e3 e2 e1 EM  ∩M3 xe3 e2 e1 e3 e2 e1    = EM  ∩M3 e1 e2 e3 EM2 EM  ∩M3 (e1 e2 e3 x)e1 e2 e3 ,

(∗∗) e13

for all x ∈ N  ∩ M2 . Consider next the 2-step basic construction N ⊂ M1 ⊂ M3 , where e13 = [M : N ]e2 e1 e3 e2 (see [PiPo2]). Furthermore, let {mi }i∈I ⊂ M be an orthonormal basis of M over N. Then {[M : N]1/2 mi e1 mj }i, j ∈I is a finite basis of  M1 (mj∗ e1 m∗i x), for all x ∈ M1 M1 over N such that x = [M : N] i, j ∈I mi e1 mj EN (compare with [B2, Proposition 2.10]). It is shown in [B2, Theorem 2.6] that one then has  M  J1 x ∗ J1 = [M : N ]3 EM13 e13 mi e1 mj x e13 mj∗ e1 m∗i i, j ∈I

= [M : N ]

5

= [M : N ]

5



i, j ∈I



i, j ∈I

 M  EM13 e2 e1 e3 e2 mi e1 mj x e2 e1 e3 e2 mj∗ e1 m∗i  M  EM13 e2 e1 e3 mi e2 e1 mj x e2 e1 mj∗ e3 e2 e1 m∗i ,

where J1 : L2 (M1 ) → L2 (M1 ) denotes, as usual, the modular conjugation. We compare this result the left-hand side of (∗∗) equals   with (∗∗). Since N ⊂ M is extremal, [M : N]−2 i,j ∈I mi EM2 e3 e2 e1 mj xe3 e2 e1 mj∗ e3 e2 e1 m∗i (see [B2, Proposition 2.7]). This in turn is equal to    EM2 e3 mi e2 e1 mj xe3 e2 e1 mj∗ e3 e2 e1 m∗i [M : N]−2 i, j ∈I

= [M : N]−2

 i, j ∈I

= [M : N]

−3



i, j ∈I

 M  M2 EM23 EM (mi e2 e1 mj x)e3 e2 e1 mj∗ e3 e2 e1 m∗i 1 EM1 (mi e2 e1 mj x)e2 e1 mj∗ e3 e2 e1 m∗i . M

Let y ∈ M1 be an arbitrary operator, then tr(yEM13 (e2 e1 e3 e2 mi e1 mj x)) = tr(ye2 e1 e3 e2 mi e1 mj x) = [M : N]−1 tr(ye2 e1 e2 mi e1 mj x) = [M : N]−2 tr(ymi e2 e1 M mj x). Thus EM13 (e2 e1 e3 e2 mi e1 mj x) = [M : N]−2 EM1 (mi e2 e1 mj x). This implies that the left-hand side of (∗∗) is equal to [M : N]−6 J1 x ∗ J1 . Since (J1 xJ1 )∗ = J1 x ∗ J1 , we have proved that the equality (∗∗) holds. We now discuss the diagrammatic interpretation of the rotations r1 and r2 . The map r1 is the rotation by 90 degrees, and the diagram for r1 (R), where R ∈ N  ∩ M1 is depicted by a 2-box, is given in Proposition 3.3 (case k = 0, see Figure 12). If an element R ∈ N  ∩ M2 is given by the 3-box R , then r2 (R) is given by

60

BISCH AND JONES

R

=

R

Figure 19

Note that all orientations of the vertical strings are up-down-up (from left to right), so that the right-hand side of Figure 19 is indeed an element of N  ∩M2 . It is now very easy to verify that r23 (R) = R, using the diagram for the rotation given by Figure 19. Let us also mention that the identity (∗∗) in the proof of Theorem 3.8 is depicted as

R

? =

R

Figure 20 R

This identity obviously holds true, since both sides are equal to (which is precisely J1 R ∗ J1 ). Note that the diagrams in Figure 20 depict elements in M  ∩ M3 ; the orientation of the vertical strings is down-up-down (from left to right). We next discuss another natural operation on the higher relative commutants, which is inspired by Proposition 3.6 and its reformulation given in Figure 19. Namely, given two elements of two higher relative commutants, there is a natural “comultiplication,” which in the depth-2 case is the actual comultiplication of Hopf algebras (using the canonical duality induced by the trace). For instance, given two 2-boxes R and Q in N  ∩ M1 , we can form another 2-box, the coproduct of R and Q as depicted in Figure 21.

SINGLY GENERATED PLANAR ALGEBRAS

R

61

Q

Figure 21

If R, Q ∈ N  ∩ M1 are represented by 2-boxes in the usual way, then the element in Figure 21 is given by the formula δ 5 EM1 (e2 REM  ∩M2 (e1 e2 Q)) = δ 5 EN  ∩M1 (e2 REM  ∩M2 (e1 e2 Q)). Observe that comultiplication is associative. To see this, we need to check that      EM1 e2 EM1 e2 REM  ∩M2 e1 e2 Q EM  ∩M2 (e1 e2 L)      , = EM1 e2 REM  ∩M2 e1 e2 EM1 e2 QEM  ∩M2 e1 e2 L

(∗ ∗ ∗)

for all R, Q, L ∈ N  ∩ M1 . By Lemma 3.1(i), the left-hand side of equation (∗ ∗ ∗) equals EM1 (e2 REM  ∩M2 (e1 e2 Q)EM  ∩M2 (e1 e2 L))=EM1 (e2 REM  ∩M2 (e1 e2 QEM  ∩M2 (e1 e2 L))), which is equal to the right-hand side of (∗ ∗ ∗), again by Lemma 3.1(i). Diagrammatically, associativity of the comultiplication is obvious; both sides of the identity (∗ ∗ ∗) are given by Figure 22.

R

Q

L

Figure 22

Clearly, the operation “comultiplication” generalizes in several ways to other higher relative commutants as indicated in Figure 23.

R Q

R Figure 23

Q

62

BISCH AND JONES

We will discuss these maps in another paper. 4. Proofs of the main results. We give in this section the proof of Theorem 2.1, as the proof of Theorem 4.1, and we present an example of a principal graph satisfying the conditions of the theorem, which is not covered by Theorem 2.2. Let us state again Theorem 2.1 for the convenience of the reader. Theorem 4.1. Let N ⊂ M be II1 factors with 3 < [M : N ] < ∞, dim N  ∩M1 = 3, and such that N  ∩ M2 is abelian modulo the basic construction ideal. Then there is an intermediate subfactor P of N ⊂ M, and hence we have J x ∗ J = x, for all x ∈ N  ∩ M1 . We first prove that under the above conditions, J · J has to be trivial on N  ∩ M1 . Since dim N  ∩M1 = 3, we can write N  ∩M1 = Ce1 ⊕Cq ⊕C(1−e1 −q), where e1 , q, and 1−e1 −q are the minimal projections of N  ∩M1 . We then have the following obvious lemma. Lemma 4.2. J x ∗ J = x, for all x ∈ N  ∩ M1 , if and only if J qJ = q. Thus J · J is nontrivial on N  ∩ M1 if and only if J qJ = 1 − e1 − q. Let us represent the minimal projection q by the 2-box Q . Nontriviality of J ·J is Q

=

Q

Q

then depicted by

Q

= 0 (see Proposition 3.3 for the diagram of J qJ ).

Proposition 4.3. Let N ⊂ M be II1 factors with 3 < [M : N ] < ∞, dim N  ∩ M1 = 3, and N  ∩ M2 abelian modulo the ideal generated by e2 . Then J · J is trivial on N  ∩ M1 ; that is, J x ∗ J = x, for all x ∈ N  ∩ M1 . The following lemma determines the orbits under the rotation r2 of special elements R

, which turn out to play an important role in the proof of

R

in N  ∩M2 of the form R

Proposition 4.3. A formula for this element is, for instance, δ 3 J R ∗ J EM  ∩M2 (e1 e2 R)R. Lemma 4.4. Let R ∈ N  ∩ M1 be an arbitrary element, depicted by R . We have the following orbits under the rotation r2 :

R

, R

,

R

R

R R

(1)

R

R

;

R

63

SINGLY GENERATED PLANAR ALGEBRAS

R R

,

,

R

R

;

R

R R

(2)

R

R

R

(3)

;

R

R

.

R R (4) R Furthermore, if R = R ∗ , then by taking adjoints, we map the orbit in (1) to the orbit in (2) and the orbit in (3) to the orbit in (4).

R

R

Proof. Apply the rotation given in Figure 20 to the elements in (1), (2), (3), and (4). The calculation of the orbits is then immediate. To get the second statement,  ∗  ∗ . Then the result follows easily. = observe that R = R implies Let us now give the proof of Proposition 4.3. Proof. Since N  ∩ M2 is abelian modulo the basic construction ideal, we have

Q

=

+α

Q

Q

Q

Q

+A

+B

+C

+D

Q (1)

+E

Q

+F

Q

+G

Figure 24

Q

+H

Q

64

BISCH AND JONES

for some scalars α, A, B, . . . , G, H ∈ C. This equations reads     A B C D α qEM  ∩M2 qe2 e1 = EM  ∩M2 qe2 e1 q + 2 qe2 q + 2 e1 + 2 e2 + e1 e2 + e2 e1 δ δ δ δ δ E F G H + 2 qe2 + 2 e2 q + qe2 e1 + e1 e2 q. (1 ) δ δ δ δ Orthogonality of the projections q and e1 is expressed diagrammatically by Q

=

Q

= 0. Furthermore, let β ∈ R be such that tr(q) = (β/δ 2 ) (tr denotes the

normalized trace as usual). Then e2 qe2 = EN  ∩M (q)e2 = (β/δ 2 )e2 , so that diagrammatically

Q

= βδ . Note that q is a projection, so that

Q Q

=

Q

.

Applying the conditional expectation EM1 to both sides of (1), multiplying (1) by the diagram E1 from above (i.e., multiplying (1 ) by the projection e1 from the right) and multiplying (1) by E1 from the bottom (i.e., multiplying (1 ) by e1 from the left), gives the following three equations: 0 = (α + E + F ) Q + (Aδ + C + D)

Q = (Aδ + C)

+ (B + Dδ)

+B

+ (E + Gδ) Q

(2)

(3)

Q Q

0=

+ (Aδ + D)

+ (B + Cδ)

+ (F + H δ) Q

(4)

Figure 25

The above equations, of course, can be rewritten by using formulas; we do this here. Note that EM1 (EM  ∩M2 (qe2 e1 )) = tr(qe2 e1 ) = tr(e1 qe2 ) = 0. Thus applying EM1 to equation (1 ) gives     B A C D α E F (2 ) 0 = q 4 + 4 + 4 + 4 · 1 + e1 2 + 3 + 3 , δ δ δ δ δ δ δ which is precisely equation (2) after multiplying by δ 4 . (Recall that tr(ei ) = [M : N]−1 = (1/δ 2 ).)

65

SINGLY GENERATED PLANAR ALGEBRAS

Observe that EM  ∩M2 (qe2 e1 )e1 = (1/δ 2 )qe2 e1 , so that multiplying (1 ) by e1 from the right yields       A C B D E G 1 e2 e1 + 2 + qe2 e1 , qe2 e1 = 2 + 3 e1 + 2 + (3 ) δ δ δ2 δ δ δ δ which is precisely equation (3) after multiplying by δ 4 . Similarly, one obtains equation (4) from (1 ). Since the diagrams occurring in equations (2) and (3) are linearly independent, we obtain A = B = C = D = 0. It follows from (4) that F +H δ = 0 if J ·J is nontrivial and F + H δ + 1 = 0 otherwise. Thus

Q

Q

Q

=

Q

Q

+α

+E Q

+F

Q

Q

+G Q +H

Q

(5)

We proceed by contradiction. Let us suppose that J · J is nontrivial, so that Q

= 0. We work with diagrams, leaving it to the reader to translate the dia-

Q

grams into formulas. Multiply (5) with This gives

Q

=

Q

Q

from below (i.e., with J qJ from the left).

Q

Q

0=

Q +F Q

Q

from above gives

Q

Q

Similarly, multiplying (5) with

Q

Q Q

(6)

=E

(7) Q

We have seen in Section 3, Figure 23, that we can comultiply a 3-box R in N  ∩M2 and a 2-box L in M  ∩ M2 to get a new 3-box R L in N  ∩ M2 . (Observe that all orientations of the vertical strings at the top and bottom of the boxes match.) Thus

66

BISCH AND JONES Q

and equation (7) with

Q

gives the following

(8) Q

F

Q

comultiplying equation (6) with two equations:

=0

Q Q

Q

(E − 1)

(9) =0

Q Q

 = 0. By taking adjoints,

Q

Q

Q

this implies

Q

We have to consider two cases. First, suppose that  = 0 (see Lemma 4.4).

Q

Thus we get F = 0 and E = 1, which in turn implies G = 0 by (3), H = 0 by (4), and α = −1 by (2). Therefore equation (5) reduces to

Q

−

Q

Q

= Q

Q

+

Q

Q

(10)

Applying the conditional expectation EM  ∩M2 to equation (10) and observing that β

Q = δ we obtain

β = δ

Q

Q

Q

β δ

− Q

Thus

Q Q

+

=

β δ

β δ

(11)

67

SINGLY GENERATED PLANAR ALGEBRAS

which implies Q =β Q   But Q = β , so that the left-hand side of the above equation equals βδ 2 δ

hence we obtain β = q  = 1.

δ2 .

Thus tr(q) =

(β/δ 2 )

and

= 1, which is a contradiction since Q

Q

Q

Q

Q

Q

of the rotation, as in Lemma 4.4, are zero. We next show that

 = 0, which is

Q

Q

Q

equivalent to

= 0. Then, by

= 0, and hence all elements in the first two orbits

Q

taking adjoints, we must have

Q

Let us now deal with the second case, that is, the case in which

 = 0 (by taking adjoints). Suppose that this is not the case. Then,

Q

Q

multiplying equation (5) by

from below, we obtain Q 0=F Q

which implies Q 0=F

=

β F δ Q

Q

so that F = 0. Then equation (7) implies E = 0, and hence α = 0 by (2), G = (1/δ) by (3), and H = 0 by (4). Thus equation (5) becomes

Q

+

Q

Q

= Q

1 δ

Q

(12)

68

BISCH AND JONES

Applying again the conditional expectation EM  ∩M2 to both sides of equation (12), we get +

1 δ

Q

β δ

Q

=

Q

β δ

Q

which is impossible since  = 0. Thus we have shown that the two above diagrams are indeed nonzero elements in the higher relative commutant N  ∩ M2 . Comultiplying equation (6) by gives Q

Q

Q

(F + 1)

=0

Q Q

which implies F = −1 by what we have just proved. Comultiplying equation (7) by implies E = 0. Therefore we obtain α = 1, G = (1/δ), and H = (1/δ). Thus equation (5) becomes

Q

Q

+

−

Q

+

1 δ Q

Q

Q =

Q

Q

+

Q

+

1 δ

Q

(13)

β δ

−

= β

β δ

+

1 δ

1 δ

(14)

from above gives

−

1 δ

−

Q

Q

Q Q

Multiplying equation (14) with Q

+

1 δ

(15) Q

=

Q

Q

β δ

Q

Applying again EM  ∩M2 to equation (13) leads to

so that (β/δ)2 = β − (2β/δ 2 ), that is, β(β + 2 − δ 2 ) = 0, and hence β = 0 or

69

SINGLY GENERATED PLANAR ALGEBRAS

β = δ 2 − 2. Since 0  = tr(q) = (β/δ 2 ), we get tr(q) = 1 − (2/δ 2 ). Since J qJ = 1−e1 −q by assumption (nontriviality of J ·J ), we√get 1−(2/δ 2 ) = tr(q) = tr(J qJ ) = 1 − (1/δ 2 ) − (1 − (2/δ 2 )) = (1/δ 2 ). Hence δ = 3, that is, [M : N] = 3, a contradiction. Thus, unless [M : N] = 3, we must have J x ∗ J = x, for all x ∈ N  ∩ M1 , as claimed. We can now proceed with the proof of the main theorem. Proof of Theorem 4.1. We use the same notation as in the proof of the previous proposition. We have shown in Proposition 4.3 that the map J ·J is necessarily trivial on N  ∩ M1 . Therefore Q = , and we just write for this minimal projection in Q

Q

=

Q

Q

= , and N  ∩ M1 (reducing the box Q to a point). Similarly, we have  for this operator in N ∩ M2 . In this notation, we then have, for we write briefly instance,

=

Q

=

+ 1δ

. The projection

=

where µ = δ tr(q), and hence

=

Q

P

We denote by p ∈ N  ∩ M1 the projection p = q + e1 , and we depict it as or, in short form, = + 1δ = + 1δ . Furthermore, we have satisfies

=

=0

= µ

satisfies

=

where ν = δ tr(p). Furthermore, we have

=0

=ν

70

BISCH AND JONES

=

=

and =ν

=µ

=

=0

In this simplified notation, a basis of the basic construction ideal (N  ∩M1 )e1 (N  ∩ M1 ) is given by

. To show that there is an interor

= −

is a biprojection,

that is, satisfies the conditions of Proposition 3.6. Since (N  ∩ M1 )e1 (N  ∩ M1 ) by hypothesis, we have

N  ∩ M2

is abelian modulo

mediate subfactor N ⊂ P ⊂ M, we show that either

=

+α

+A

+B

+C (16)

+D

+E

+F

+G

+H

As in the proof of Proposition 4.3, we apply the conditional expectation EM1 to both sides of equation (16) and multiply (16) with the diagram E1 from above and from below. We obtain the following three equations: =

+(α + E + F )

+(Aδ + C + D)

=

+(Aδ + C)

+(B + Dδ)

=

+(Aδ + D)

+(B + Cδ)

+B

+(E + Gδ)

+(F + H δ)

(17)

(18)

(19)

This implies A = B = C = D = 0, E + Gδ = 0, F + H δ = 0, and α + E + F = 0

71

SINGLY GENERATED PLANAR ALGEBRAS

by linear independence. Thus α = (G + H )δ = −(E + F ). We next apply the conditional expectation EM  ∩M2 to equation (16) and obtain

0=α

=ν

since

=

+(Eµ + F µ)

(G + H )

(20)

. (Note that equation (20) is written in N  ∩M2 ⊃ M  ∩M2 .)

Equation (20) is the same as −αµ

+

α δ

(21)

Q

0=α

Note that equation (21) is written in M  ∩ M2 . Suppose now that α  = 0, so that −µ

+

1 δ

(22)

Q

0=

Multiplying equation (22) with the diagram E2 from above yields 0 = µ2

−µδ

+

µ δ

(23)

is µ/δ; that is, tr(q) = so that µ = δ − (1/δ). But the normalized trace of Q = 1 − (1/δ 2 ) = tr(1 − e1 ). But q(1 − e1 ) = q, and hence q = 1 − e1 by nondegeneracy of the (positive) trace. This is a contradiction to dim N  ∩ M1 = 3. Thus we must have α = 0, and therefore G = −H and E = −F . Since G = −(E/δ), equation (16) simplifies to     1 − = +E − . + δ (24)

= =

. Furthermore, since + 1δ

, and thus

=

+ 1δ

P

We want to comultiply equation (24) with

=

. Observe that

, we have

=

+ 1δ

and , and

72

BISCH AND JONES

=

Q

P

Q

Since

=

−

1 δ

(25)

=

−

1 δ

(26)

, we have

=

=

P

Thus comultiplying equation (24) with

(27)

gives



 1 − δ

+E 

=

+ 1δ

. Thus

=

1 δ

+

−

1 δ

+ 1δ

and

+

1 δ

1 + δ

=

= −

Q

=

so that

P

Recall that P = Q + 1δ

−

1 + δ

+ 1δ 1 δ

1 − δ + 1δ



(28)

or, in short,

, which implies

=0

(29)

Thus  =

+E

 −

(30)

Q

If E  = 1, then = , so that P = is a biprojection, and hence N ⊂ M has an intermediate subfactor by Proposition 3.6. If E = 1, we show that e1 +(1−e1 −q) = 1−q is necessarily a biprojection. Recall  that we depict the projection 1 − q by = − . Hence = − recall that  = . We compute

73

SINGLY GENERATED PLANAR ALGEBRAS

=

−

=

−

=

−

−

+

=

−

−

(31)

+

(32)

Thus −

=

−

+

−

(33)

Next, we calculate 1 δ

=

+

1 δ

=

+

=

+

=

+

1 δ

+

1 δ

+

1 δ

+

1 δ

+

1 δ2

(34)

1 δ2

(35)

and

Therefore −

=

−

+

1 δ

−

1 δ

(36)

Hence, using equation (24) with E = 1, we obtain

−

=

−

(37)

. The last identity says that = 1−q is a biprojection, and hence = we are done by again applying Proposition 3.6. and thus

Examples of subfactors whose higher relative commutants satisfy the conditions of Theorem 4.1 can be constructed via free composition of two subfactors (see [BJ1] and [BJ2]; see also [Gn]). For instance, one can construct a spherical C ∗ -planar algebra (or a Popa system) as a free composition of two Popa systems associated to subfactors with principal graph E6 , and using Popa’s theorem (see [Po3]), one obtains in this way a subfactor N ⊂ M with principal graph given by Figure 26.

74

BISCH AND JONES

∗

Figure 26

Observe that dim N  ∩ M2 = 14 > 12 and that N  ∩ M2 modulo the basic construction ideal is (isomorphic to) C⊕C⊕C⊕C⊕C. Free compositions (or free products) of planar algebras will be discussed in [BJ2]; see also [BJ1]. References [BiWe] [B1] [B2]

[BH] [BJ1] [BJ2] [EK] [FRS]

[Gn] [GHaJ] [H] [Iz] [J1] [J2] [JSu]

J. Birman and H. Wenzl, Braids, link polynomials and a new algebra, Trans. Amer. Math. Soc. 313 (1989), 249–273. D. Bisch, A note on intermediate subfactors, Pacific J. Math. 163 (1994), 201–216. , “Bimodules, higher relative commutants and the fusion algebra associated to a subfactor” in Operator Algebras and Their Applications (Waterloo, Ontario, 1994/1995), Fields Inst. Commun. 13, Amer. Math. Soc., Providence, 1997, 13–63. D. Bisch and U. Haagerup, Composition of subfactors: New examples of infinite depth subfactors, Ann. Sci. École Norm. Sup. (4) 29 (1996), 329–383. D. Bisch and V. Jones, Algebras associated to intermediate subfactors, Invent. Math. 128 (1997), 89–157. , Planar algebras, II, in preparation. D. Evans and Y. Kawahigashi, Quantum Symmetries on Operator Algebras, Oxford Math. Monogr., Oxford Univ. Press, New York, 1998. K. Fredenhagen, K.-H. Rehren, and B. Schroer, Superselection sectors with braid group statistics and exchange algebras, I: General theory, Comm. Math. Phys. 125 (1989), 201–226. S. Gnerre, Free composition of paragroups, Ph.D. thesis, University of California at Berkeley, 1997. F. Goodman, P. de la Harpe, and V. Jones, Coxeter Graphs and Towers of Algebras, Math. Sci. Res. Inst. Publ. 14, Springer-Verlag, New York, 1989. √ U. Haagerup, “Principal graphs of subfactors in the index range 4 < [M : N] < 3 + 2” in Subfactors (Kyuzeso, 1993), World Scientific, River Edge, N.J., 1994, 1–38. M. Izumi, Application of fusion rules to classification of subfactors, Publ. Res. Inst. Math. Sci. 27 (1991), 953–994. V. Jones, Index for subfactors, Invent. Math. 72 (1983), 1–25. , Planar algebras, I, preprint, 1998. V. Jones and V. Sunder, Introduction to Subfactors, London Math. Soc. Lecture Note Ser. 234, Cambridge Univ. Press, Cambridge, 1997.

SINGLY GENERATED PLANAR ALGEBRAS [La] [Lo1] [Lo2] [Mu] [Oc1]

[Oc2]

[PiPo1] [PiPo2] [Po1] [Po2] [Po3] [Po4] [Sc] [Wa]

[We1] [We2]

75

Z. Landau, Ph.D. thesis, University of California at Berkeley, 1998. R. Longo, Index of subfactors and statistics of quantum fields, I, Comm. Math. Phys. 126 (1989), 217–247. , Index of subfactors and statistics of quantum fields, II: Correspondences, braid group statistics and Jones polynomial, Comm. Math. Phys. 130 (1990), 285–309. J. Murakami, The Kauffman polynomial of links and representation theory, Osaka J. Math. 24 (1987), 745–758. A. Ocneanu, “Quantized groups, string algebras and Galois theory for algebras” in Operator Algebras and Applications, Vol. 2, London Math. Soc. Lecture Note Ser. 136, Cambridge Univ. Press, Cambridge, 1988, 119–172. , Quantum symmetry, differential geometry of finite graphs and classification of subfactors, University of Tokyo Seminary Notes (notes recorded by Y. Kawahigashi), no. 45, 1991. M. Pimsner and S. Popa, Entropy and index for subfactors, Ann. Sci. École Norm. Sup. (4) 19 (1986), 57–106. , Iterating the basic construction, Trans. Amer. Math. Soc. 310 (1988), 127–133. S. Popa, Classification of subfactors: The reduction to commuting squares, Invent. Math. 101 (1990), 19–43. , Classification of amenable subfactors of type II, Acta Math. 172 (1994), 163–255. , An axiomatizaton of the lattice of higher relative commutants of a subfactor, Invent. Math. 120 (1995), 427–445. , Classification of Subfactors and Their Endomorphisms, CBMS Regional Conf. Ser. in Math. 86, Amer. Math. Soc., Providence, 1995. J. K. Schou, Commuting squares and index for subfactors, Ph.D. thesis, Odense Universitet, 1991. A. Wassermann, Operator algebras and conformal field theory, III: Fusion of positive energy representations of LSU(N) using bounded operators, Invent. Math. 133 (1998), 467–538. H. Wenzl, Hecke algebras of type An and subfactors, Invent. Math. 92 (1988), 349–383. , Quantum groups and subfactors of type B, C, and D, Comm. Math. Phys. 133 (1990), 383–432.

Bisch: Department of Mathematics, University of California at Santa Barbara, Santa Barbara, California 93106, U.S.A.; [email protected] Jones: Department of Mathematics, University of California at Berkeley, Berkeley, California 94720, U.S.A.; [email protected]

Vol. 101, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

A NEW FORMULA FOR WEIGHT MULTIPLICITIES AND CHARACTERS SIDDHARTHA SAHI

1. Introduction. The weight multiplicities of a representation of a simple Lie algebra g are the dimensions of eigenspaces with respect to a Cartan subalgebra h. In this paper, we give a new formula for these multiplicities. Our formula expresses the multiplicities as sums of positive rational numbers. Thus it is very different from the classical formulas of Freudenthal [F] and Kostant [Ks], which express them as sums of positive and negative integers. It is also quite different from recent formulas due to Lusztig [L1] and Littelmann [Li]. For example, for the multiplicity of the next-to-highest weight in the n-dimensional representation of sl2 , we get the following expression (which sums to 1): 1 1 1 1 + +···+ + . (1)(2) (2)(3) (n − 1)(n) n The key role in our formula is played by the dual affine Weyl group. Let V0 , ( , ) be the real Euclidean space spanned by the root system R0 of g, and let V be the space of affine linear functions on V0 . We identify V with Rδ ⊕ V0 via the pairing (rδ + x, y) = r + (x, y) for r ∈ R, x, y ∈ V0 . The dual affine root system is R = {mδ + α ∨ | m ∈ Z, α ∈ R0 } ⊆ V , where α ∨ means 2α/(α, α) as usual. Fix a positive subsystem R0+ ⊆ R0 with base {α1 , . . . , αn }, and let β be the highest short root. Then a base for R is given by a0 = δ − β ∨ , a1 = α1∨ , . . . , an = αn∨ , and we write si for the (affine) reflection about the hyperplane {x | (ai , x) = 0} ⊆ V0 . The dual affine Weyl group is the Coxeter group W generated by s0 , . . . , sn , and the finite Weyl group is the subgroup W0 generated by s1 , . . . , sn . For w ∈ W , its length is the length of a reduced (i.e., shortest) expression of w in terms of the si . The group W acts on the weight lattice P of g, and each orbit contains a unique (minuscule) weight from the set
   ᏻ := λ ∈ P | α ∨ , λ = 0 or 1, ∀α ∈ R0+ . Definition. For each λ in P , we define (1)  λ := λ + (1/2) α∈R + ε(α ∨ ,λ) α, where, for t ∈ R, εt is 1 if t > 0 and −1 if 0 t ≤ 0; Received 1 March 1999. 1991 Mathematics Subject Classification. Primary 17B10, 33C50; Secondary 20F55, 20E46. Author’s work supported by a National Science Foundation grant. 77

78

SIDDHARTHA SAHI

(2) wλ := unique shortest element in W such that λ := wλ · λ ∈ ᏻ. We fix a reduced expression si1 · · · sim for wλ , and, for each J ⊆ {1, . . . , m}, we define (3) wJ := the element of W obtained by deleting sij , j ∈ J , from the product si1 · · · sim ;  −1 and λ (4) cJ := j ∈J cj , where cj := (aij , λ (j ) ) (j ) := sij −1 · · · si1 · λ. + Let P ⊂ P be the cone of dominant weights; and, for λ ∈ P + , let Vλ be the irreducible representation of g with highest weight λ. Theorem 1.1. For λ in P + and  µ in P , the multiplicity mλ (µ) of µ in Vλ is given by mλ (µ) := (|W0 · λ|/|W0 · µ|) J cJ , where the summation is over all J such that wJ−1 · λ is in W0 · µ. (We prove in Corollary 6.2 that the cJ ’s are positive.) For µ in P , let eµ denote the function x  → e(µ,x) on V0 . Then W acts on the eµ ’s si ·µ , and Theorem 1.1 is equivalent to by virtue of its action on P , that is, si eµ = e the following formula for the character χλ := µ mλ (µ)eµ of Vλ .  Theorem 1.2. We have χλ = (|W0 · λ|/|W0 |) w∈W0 w(sim + cm ) · · · (si1 + c1 )eλ . We obtain Theorem 1.2 as a consequence of a more general result, namely, an analogous formula for the generalized Jacobi polynomial Pλ of Heckman and Opdam. For the definition and properties of Pλ , we refer the reader to [HSc] and [O]. We recall here that Pλ depends on certain parameters kα , α ∈ R0 , such that kw·α = kα for all w ∈ W0 . For special values of kα , Pλ can be interpreted as a spherical function on a compact symmetric space. In particular, in the limit as all kα → 1, we have Pλ → χλ . Definition. In the context of the previous definition, for λ in P , we redefine  λ := λ + (1/2) α∈R + kα ε(α ∨ ,λ) α; (1 )  0 −1 (4 ) cj = kij (aij , λ (j ) ) , where k0 = kβ and ki = kαi for i ≥ 1. Theorem 1.3. For λ in P + and for cj as above, the Heckman-Opdam polynomial Pλ is given by the same formula as in Theorem 1.2.  For λ in P + , define cλ := (|W0 |/|W0 ·λ|) j (aij , λ (j ) ), and let ᏼ := Z+ [kα ] be the set of polynomials in the parameters kα with nonnegative integral coefficients. Then we prove the following theorem. Theorem 1.4. We have that cλ is in ᏼ, as are all coefficients of cλ Pλ . Theorem 1.4 is a generalization of the main result of [KS] to arbitrary root systems. Our proof depends on three fundamental ideas in the “new” theory of special functions. The first idea, due to Macdonald, Heckman, Opdam, and others, is that one can treat root multiplicities on a symmetric space as parameters. The second idea, due to Dunkl and Cherednik, is that radial parts of invariant

79

A NEW FORMULA FOR WEIGHT MULTIPLICITIES

differential operators on symmetric spaces can be written as polynomials in certain commuting first-order differential-reflection operators, namely, the Cherednik operators. The third idea is the method of intertwiners for Cherednik operators. This was developed in [KS], [K], [S1], and [C2], and it can be regarded as the double affine analog of Lusztig’s fundamental relation [L2] in the affine Hecke algebra. Using the intertwiners of [C2] and [S2], our results can be extended to the context of Macdonald polynomials and to the 6-parameter Koornwinder polynomials. These intertwiners correspond to the affine Weyl group (rather than the dual affine Weyl group) and hence are not appropriate for the present context. We shall discuss them elsewhere in [S3]. 2. Preliminaries. The results of this section are due to Cherednik [C1], Heckman, and Opdam [O]. Let F = R(kα ) be the field of rational functions in the parameters kα , and let ᏾ be the F-span of {eλ | λ ∈ P } regarded as a W -module. Definition. For y ∈ V0 , the Cherednik operator Dy is defined by Dy = ∂y +



(y, α)kα

α∈R0+

1 (1 − sα ) − (y, ρ), 1 − e−α

where ρ :=

1 kα α. 2 + α∈R0

Here are some basic facts about Cherednik operators from [O, Section 2]. Proposition 2.1. We have the following. (1) The operators Dy act on ᏾ and commute pairwise. (2) For i = 1, . . . , n, we have si Dy − Dsi y si = −ki (y, αi ). (3) There is a basis {Eλ | λ ∈ P } of ᏾, characterized uniquely as follows: (a) the coefficient of eλ in Eλ is 1; (b) Dy Eλ = (y, λ)Eλ , where  λ is as in Definition (1’) of the introduction. (4)  For λ in P + , the Heckman-Opdam polynomial Pλ equals (|W0 · λ|/|W0 |) w∈W0 wEλ . λ. (5) For i = 1, . . . , n, if si · λ  = λ, then s i · λ = si · 3. The affine reflection. In this section, we prove some basic properties of the affine reflection s0 . Lemma 3.1. If α is a positive root different from β, then (α ∨ , β) equals zero or 1. Proof. Since β is in P + , (α ∨ , β) is a nonnegative integer. Also, since β is a short root, we have (α, α) ≥ (β, β). So, by the Cauchy-Schwartz inequality, we get 

 (α, β) (α, β) α∨, β = 2 ≤2 ≤ 2. (α, α) (α, α)1/2 (β, β)1/2

80

SIDDHARTHA SAHI

If α  = β, then α is not proportional to β and the last inequality is strict. For i = 0, 1, 2, define R0i = {α ∈ R0+ | (α ∨ , β) = i}, and, for α in R0+ , put

if α ∈ R00 , sβ · α  α = −sβ · α if α ∈ R01 ∪ R02 . Lemma 3.2. The involution α  → α  acts trivially on R00 and R02 , and permutes R01 . 

Proof. For α in R01 , we have (α ∨ , β) = (α ∨ , −sβ · β) = (α ∨ , β) = 1, which implies that α  is a (positive) root in R01 . The assertions about R00 and R02 = {β} are obvious. Lemma 3.3. For λ in P , if s0 · λ  = λ, then s λ. 0 · λ = s0 · λ = β + sβ λ using Lemma 3.2 and kα = kα  . This gives Proof. We compute s0 · s0 · λ = β + sβ · λ +

1 1 kα ε(α ∨ ,λ) α − 2 2 0 α∈R0



kα ε(α  ∨ ,λ) α.

α∈R01 ∪R02

Comparing this to the expression for  µ with µ = s0 · λ, it suffices to show that

if α ∈ R00 , ε(α ∨ ,λ) ε(α ∨ ,µ) = −ε(α  ∨ ,λ) if α ∈ R01 ∪ R02 . For α in R00 , we easily compute that (α ∨ , µ) = (α ∨ , λ).  For α in R01 , we get (α ∨ , µ) = (α ∨ , β + sβ · λ) = 1 − (α ∨ , λ). Being an integer, ∨ (α , λ) is either less than or equal to zero or greater than or equal to 1. In either case, we get ε(α ∨ ,µ) = −ε(α  ∨ ,λ) . Finally, for α in R02 , we have α = α  = β and (β ∨ , µ) = 2 − (β ∨ , λ). Now s0 λ  = λ implies that (β ∨ , λ)  = 1; thus we have either (β ∨ , λ) ≥ 2 or (β ∨ , λ) ≤ 0. In either case, we get ε(β ∨ ,λ) = ε(β ∨ ,λ) = −ε(β ∨ ,µ) . 4. The intertwining relation. Dualizing the action y  → w · y of W on V0 , we get a representation v  → wv of W on V satisfying (wv, y) = (v, w−1 · y). For y in V0 and w in W0 , we have wy = w · y. The affine reflection s0 acts on V by s0 (rδ + y) = (y, β)δ + rδ + sβ y. For v = rδ + y in V , we define the affine Cherednik operator simply by putting Dv = Dy +rI , where I is the identity operator. From Proposition 2.1(2), we know the intertwining relations between the (affine) Cherednik operators and s1 , . . . , sn . In this section, we prove the following intertwining relation between these operators and s0 . Proposition 4.1. For v = rδ + y in V , we have Dv s0 − s0 Ds0 v = kβ (y, β).

81

A NEW FORMULA FOR WEIGHT MULTIPLICITIES

Proof. Let us write Nα for 1/(1 − e−α )(1 − sα ), so that Dv = ∂y + kα (y, α)Nα − (y, ρ) + r. Since sβ Nα = Nsβ ·α sβ and sβ ∂y = ∂sβ y sβ , we get sβ Dv sβ = ∂sβ y + kα (y, α)Nsβ ·α − (y, ρ) + r. α∈R0+

Now, partitioning R0+ = R00 ∪ R01 ∪ R02 and using Lemma 3.2, we get     sβ Dv sβ = ∂sβ y + kα sβ y, α Nα − kα sβ y, α N−α − (y, ρ) + r. α∈R00

α∈R01 ∪R02

The following identities are easy to check: (1) eβ ∂sβ y e−β = ∂sβ y + (y, β); (2) eβ Nα e−β = Nα for α ∈ R00 ; (3) eβ N−α e−β = 1 − Nα for α ∈ R01 ; (4) eβ N−β e−β = 1 − Nβ + s0 . Using these, we get the following formula for s0 Dv s0 = eβ (sβ Dv sβ )e−β :       kα sβ y, α Nα − kα sβ y, α − kβ sβ y, β s0 − (y, ρ) + r. ∂sβ y + (y, β) + Since



α∈R0+

α∈R01 ∪R02

α∈R01 ∪R02 kα (sβ y, α) = (sβ y, ρ − sβ · ρ) = (sβ y, ρ) − (y, ρ),

we get

s0 Dv s0 = Dsβ y + (y, β) − kβ (sβ y, β)s0 + r = Ds0 v + kβ (y, β)s0 . The result follows. 5. The Heckman-Opdam polynomials. Let Eλ be as in Proposition 2.1. Proposition 5.1. The polynomials Eλ satisfy the following recursions: (1) Eλ = eλ for λ ∈ ᏻ; (2) if si · λ  = λ, then (si + (ki /(ai , λ)))Eλ is a multiple of Esi ·λ . Proof. For (1), we check simply that Dy eλ = (y, λ)eλ , using the identity

  eλ if α ∨ , λ = 1, λ   Nα e = 0 if α ∨ , λ = 0. For (2), we write F for (si + (ki /(ai , λ)))Eλ and first consider i  = 0. Then, for y in V0 , using Proposition 2.1(2), we get       y, λ ki  Dy F = si Dsi y − ki (y, αi ) +   Dy Eλ = si y, λ si + ki   − ki (y, αi ) Eλ . ai , λ λ ai ,

82

SIDDHARTHA SAHI

Since (y, λ) − (y, αi )(ai , λ) = (si y, λ), using Proposition 2.1(5), we get       λ F = y, si · λ F = y, s Dy F = si y, i · λ F. This proves (2) for i  = 0. For i = 0, we use Proposition 4.1 to get       y, λ kβ  Dy F = s0 Ds0 y + kβ (y, β) +   Dy Eλ = s0 y, λ s0 + kβ   + kβ (y, β) Eλ . λ λ a0 , a0 , This time, using (y, λ) + (a0 , λ)(y, β) = (s0 y, λ) and Lemma 3.3, we get       Dy F = s0 y, λ F = y, s0 · λ F = y, s 0 · λ F. This completes the proof of (2) for i = 0. Corollary 5.2. For λ in P , and ci as in Definition (4 ) of the introduction, we have     Eλ = sim + cm · · · si1 + c1 eλ . Proof. By the minimality of wλ , if w is a proper subexpression of wλ−1 = sim · · · si1 , then w · λ  = λ. This means that the coefficient of eλ in (sim + cm ) · · · (si1 + c1 )eλ is 1. The result now follows from Proposition 5.1. Proof of Theorem 1.3. This follows from Corollary 5.2 and Proposition 2.1(4). 6. Positivity. Let ᏼ1 ⊂ ᏼ be the set of polynomials of degree less than or equal to 1, with nonnegative integral coefficients and a positive constant term. For λ in P ,  let aij and λ (j ) be as in Definition (4 ) of the introduction. Proposition 6.1. For each j = 1, . . . , m, (aij , λ (j ) ) belongs to ᏼ1 . Proof. Fix j and write µ = λ(j ) , i = ij , and w = si1 · · · sij −1 . We need to show that (ai ,  µ) has a positive constant term and nonnegative integral coefficients. The lengths of w and wsi must be j − 1 and j , respectively, since otherwise we could shorten the expression si1 · · · sim for wλ . By a standard argument (see [Hu, Chapter 5]), this implies that w(ai ) is a positive (affine) coroot in R + . Since λ = µ is minuscule, we conclude that     0 ≤ w(ai ), µ = ai , w −1 · µ = (ai , µ). If (ai , µ) were zero, then λ(j +1) = si · µ = µ = λ(j ) and we could shorten the expression for wλ by dropping sij . This shows that (ai , µ), which is the constant term of (ai ,  µ), is positive. If i = 0, the nonconstant part of (a0 ,  µ) is   1 − kα ε(α ∨ ,µ) β ∨ , α , 2 + α∈R0

A NEW FORMULA FOR WEIGHT MULTIPLICITIES

83

and we consider separately the contributions of R00 , R01 , and R02 . For α in R00 , the contribution is zero. For α = β in R02 , we get the term −ε(β ∨ ,µ) kβ . By the first part, (a0 , µ) is a positive integer. Hence (β ∨ , µ) = 1 − (a0 , µ) ≤ 0, which implies that −ε(β ∨ ,µ) = 1. The roots in R01 can be grouped in pairs {α, −sβ · α}, and the contribution of such a pair is −kα

ε(α ∨ ,µ) + ε(−sβ α ∨ ,µ)  2

 β ∨, α .

Now (β ∨ , α) is positive, so the coefficient above is a nonnegative integer, unless (α ∨ , µ) and (−sβ α ∨ , µ) are both greater than zero. But in this case, we would get          0 < α ∨ , µ − sβ α ∨ , µ = α ∨ , µ − sβ · µ = β ∨ , µ α ∨ , β ≤ 0, which is a contradiction. µ) is The argument is similar if i > 0. The nonconstant part of (ai ,  1 kα ε(α ∨ ,µ) (ai , α). 2 + α∈R0

To compute this, we divide R0+ into three disjoint sets consisting of {αi }, {the roots orthogonal to αi }, and {the remaining positive roots}. For α = αi , we get the coefficient ε(ai ,µ) , which is 1 since (ai , µ) > 0 by the first part. If α is orthogonal to αi , then the coefficient is zero. Finally, the remaining positive roots can be grouped into pairs {α, si · α}, where we may assume that (α ∨ , αi ) > 0. The contribution of each such pair is kα

ε(α ∨ ,µ) − ε(si α ∨ ,µ) (ai , α). 2

Now (α ∨ , αi ) > 0 implies (ai , α) > 0. Therefore, this coefficient is a nonnegative integer, unless (α ∨ , µ) ≤ 0 and (si α ∨ , µ) > 0. But if this were the case, then we would have         0 > α ∨ , µ − si α ∨ , µ = α ∨ , µ − si · µ = (ai , µ) α ∨ , αi > 0, which is a contradiction. Proof of Theorem 1.4. This follows from Theorem 1.3 and Proposition 6.1. Setting all the kα ’s equal to 1 in Proposition 6.1, we deduce the following corollary. Corollary 6.2. The constants cj and cJ in Theorems 1.1 and 1.2 are positive.

84

SIDDHARTHA SAHI

References [C1] [C2] [F] [HSc] [Hu] [K] [KS] [Ko]

[Ks] [Li] [L1]

[L2] [M] [O] [S1] [S2] [S3]

I. Cherednik, Double affine Hecke algebras and Macdonald’s conjectures, Ann. of Math. (2) 141 (1997), 191–216. , Intertwining operators of double affine Hecke algebras, Selecta Math. (N.S.) 3 (1997), 459–495. H. Freudenthal, Zur Berechnung der Charaktere der halbeinfachen Lieschen Gruppen, I, Indag. Math. 16 (1954), 369–376. G. Heckman and H. Schlichtkrull, Harmonic Analysis and Special Functions on Symmetric Spaces, Perspect. Math. 16, Academic Press, San Diego, 1994. J. Humphreys, Reflection Groups and Coxeter Groups, Cambridge Stud. Adv. Math. 29, Cambridge Univ. Press, Cambridge, 1990. F. Knop, Integrality of two variable Kostka functions, J. Reine Angew. Math. 482 (1997), 177–189. F. Knop and S. Sahi, A recursion and a combinatorial formula for Jack polynomials, Invent. Math. 128 (1997), 9–22. T. Koornwinder, “Askey-Wilson polynomials for root systems of type BC” in Hypergeometric Functions on Domains of Positivity, Jack Polynomials, and Applications (Tampa, Fla., 1991), Contemp. Math. 138, Amer. Math. Soc., Providence, 1992, 189–204. B. Kostant, A formula for the multiplicity of a weight, Trans. Amer. Math. Soc. 93 (1959), 53–73. P. Littelmann, Paths and root operators in representation theory, Ann. of Math. (2) 142 (1995), 499–525. G. Lusztig, “Singularities, character formulas, and a q-analog of weight multiplicities” in Analysis and Topology on Singular Spaces, II, III (Luminy, 1981), Astérisque 101– 102, Soc. Math. France, Montrouge, 1983, 208–299. , Affine Hecke algebras and their graded version, J. Amer. Math. Soc. 2 (1989), 599–635. I. Macdonald, Affine Hecke algebras and orthogonal polynomials, Astérisque 237 (1996), 189–207, Séminaire Bourbaki 1994/95, exp. no. 797. E. Opdam, Harmonic analysis for certain representations of graded Hecke algebras, Acta Math. 175 (1995), 75–121. S. Sahi, Interpolation, integrality, and a generalization of Macdonald’s polynomials, Internat. Math. Res. Notices 1996, 457–471. , Nonsymmetric Koornwinder polynomials and duality, to appear in Ann. of Math. (2). , Some properties of Koornwinder polynomials, to appear in Contemp. Math.

Department of Mathematics, Rutgers University, New Brunswick, New Jersey 08903, USA; [email protected]

Vol. 101, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES ANVAR R. MAVLYUTOV

0. Introduction. While the geometry and cohomology of ample hypersurfaces in toric varieties have been studied (see [BC]), not much attention has been paid to semiample (i.e., “big” and “nef”) hypersurfaces defined by sections of line bundles generated by global sections with a positive self-intersection number. It turns out that mirror symmetric hypersurfaces in the Batyrev mirror construction [B2] are semiample, but often not ample. In this paper, we study semiample hypersurfaces. Such hypersurfaces bring a geometric construction that generalizes the way of construction in [B2]. The purpose of this paper is to present some approaches to studying the cohomology ring of semiample hypersurfaces in complete simplicial toric varieties. In particular, we explicitly describe the ring structure on the middle cohomology of regular semiample hypersurfaces, when the dimension of the ambient space is 4. Let us explain the main ideas of computing the topological cup product. The first step is to naturally relate the middle cohomology of the hypersurfaces to some graded ring; in our situation this is done using a Gysin spectral sequence. The origin of this idea is in [CarG] and [BC]. The second step is to use the multiplicative structure on the graded ring in order to compute the topological cup product on the middle cohomology. We remark that the cup product was computed on the middle cohomology of smooth hypersurfaces in a projective space (see [CarG]), and this paper generalizes some of the results in [CarG]. The following is a brief summary of the paper. In Section 1, we establish notation and then introduce a geometric construction associated with semiample divisors in complete toric varieties. At the end, we give a criterion for a divisor to be ample (generated by global sections) in terms of intersection numbers. This was known for simplicial toric varieties (the toric Nakai criterion), and we prove it for arbitrary complete toric varieties. Section 2 studies regular semiample hypersurfaces and describes a nice stratification of such hypersurfaces. These hypersurfaces generalize those in the Batyrev construction (see [B2]). Section 3 generalizes the results of [CarG] on an algebraic cup product formula for residues of rational differential forms (from here on, the toric variety is usually simplicial). It shows that there is a natural map from a graded ring (the Jacobian ring Received 15 December 1998. 1991 Mathematics Subject Classification. Primary 14M25. Author’s work partially supported by I. Mirkovic from National Science Foundation grant number DMS-9622863. 85

86

ANVAR R. MAVLYUTOV

R(f ) (see [BC])) to the middle cohomology of a quasismooth hypersurface such that the multiplicative structure on the ring is compatible with the topological cup product. In Section 4, we partially describe the middle cohomology of a regular semiample hypersurface X; in particular, we show that some graded pieces of the ring R1 (f ), considered in [BC], are imbedded into the middle cohomology of X. We explicitly compute the cup product on the part coming from the ring. We should point out that, when X is ample, the graded pieces of R1 (f ) fill up the middle cohomology of the hypersurface, but not so in the semiample case. Section 5 computes the middle cohomology and the cup product on it for regular semiample hypersurfaces in a 4-dimensional toric variety. This is the most interesting case for physics. We describe the whole middle cohomology in algebraic terms, even though R1 (f ) might fill up only part of the middle cohomology. In fact, the complement to the R1 (f ) part is a direct sum of the middle cohomologies of regular ample hypersurfaces in 2-dimensional toric varieties. Hence, this part can also be described in terms of rings similar to R1 (f ). In Section 6, we compute the Hodge numbers hp, 2 of a regular semiample hypersurface, and then apply the obtained formulas to the hypersurfaces in the Batyrev mirror construction (see [B2]) to verify that, in general, the duality predicted by physicists does not occur for the Hodge numbers of such hypersurfaces. Basic references on the theory of toric varieties are [F1], [Od], [D], and [C3]. Acknowledgments. I would like to thank David Cox for his advice and useful comments. I am grateful to David Cox and David Morrison for allowing me to use their unpublished notes for Theorems 3.3 and 3.5. I also thank the referee for pointing out that our notion of “semiample” is a little bit different from the common one (see Remark 1.1). 1. Semiample divisors. In this section, we first establish notation, review some basic facts from the toric geometry, and then discuss a geometric construction associated with semiample divisors in complete toric varieties. At the end of this section, we prove a generalization of the toric Nakai criterion for arbitrary complete toric varieties. As a consequence, we obtain a criterion for semiample divisors in terms of intersection numbers. In notation, we follow [BC] and [C3]. Let M be a lattice of rank d, and let N = Hom(M, Z) be the dual lattice; MR (resp., NR ) denotes the R-scalar extension of M (resp., of N). The symbol P stands for a complete toric variety associated with a finite complete fan in NR . Denote by (k) the set of all k-dimensional cones in ; in particular, (1) = {ρ1 , . . . , ρn } is the set of 1-dimensional cones in with the minimal integral generators e1 , . . . , en , respectively. Each 1-dimensional cone ρi
corresponds to a torus-invariant divisor Di in P . A torus-invariant Weil divisor D = ni=1 ai Di determines a convex polyhedron When D =


n

D = {m ∈ MR : m, ei  ≥ −ai for all i} ⊂ MR .

i=1 ai Di

is Cartier, there is a support function ψD : NR → R that is

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

87

linear on each cone σ ∈ and determined by some mσ ∈ M: ψD (ei ) = mσ , ei  = −ai

for all ei ∈ σ.

Since P is complete, a general fact is that a Cartier divisor D (i.e., the corresponding line bundle ᏻP (D)) is generated by global sections (resp., ample) if and only if ψD is convex (resp., strictly convex). A Cartier divisor D on P is called semiample if D is generated by global sections and the intersection number (D d ) > 0. In complete toric varieties, all ample divisors are semiample. From [F1, Section 5.3], it follows that (D d ) = d! vold (D ), where vold is the d-dimensional volume normalized with respect to the lattice M. So, the semiample torus-invariant divisors in complete toric varieties can be characterized by two conditions: The support function ψD is convex and the polyhedron D has maximal dimension d. Remark 1.1. We should mention here that our notion of “semiample” is a little bit different from the common one. In [EV], it is not assumed that semiample sheaves ᏸ have the additional property ᏸd > 0 (the Iitaka dimension is maximal). We believe that the results in this section can be easily generalized for all Cartier divisors generated by global sections. However, for the purpose of studying mirror symmetric hypersurfaces (see [BC]), we simply assume that semiample sheaves have the additional property. The same definition was used in the recent book [CKa]. Let us show how to construct a semiample (but not ample) divisor from an ample one. Consider a proper birational morphism π : P 1 → P 2 between two complete toric varieties corresponding to a subdivision 1 of a fan 2 with an ample torusinvariant divisor Y on P 2 . Then the pullback π ∗ (Y ) is a torus-invariant Cartier divisor with the same support function as the one for Y . Hence, π ∗ (Y ) is semiample, and it is not ample if 1 is different from 2 . We now show that all semiample divisors arise uniquely
this way, constructing a complete fan D for a semiample Cartier divisor D = ni=1 ai Di using our fan and the convex support function ψD . The value of the support function ψD on each d-dimensional cone σ ∈ is determined by a unique mσ ∈ M. We glue together those maximal dimensional cones in that have the same mσ . The glued set is again a convex rational polyhedral cone, and one can show that this cone is strongly convex using the fact that D has maximal dimension d. The set of these strongly convex cones with its faces comprise a new complete fan D in NR . This construction is independent of the equivalence relation on the divisors: If we change the divisor D to a linearly equivalent one, the fan D will remain the same. The fan D is exactly the normal fan of D . Indeed, by construction, ψD is strictly convex with respect to D . On the other hand, since D is generated by global sections, the support function ψD coincides with the function of D (see [F1, Section 3.4]): ψD (n) = min m, n. m∈D

88

ANVAR R. MAVLYUTOV

Theorem 2.22 in [Od] implies that D is the normal fan of D . Notice that is a refinement of D . So, the sets of 1-dimensional cones of the fans are related by D (1) ⊂ (1), and we have a proper birational morphism π : P → P D between the two toric varieties. Any proper morphism determines a pushforward map π∗ : Ad−1 (P ) → Ad−1 (P D ) on the Chow group, which takes the class of an irreducible divisor V to the class deg(V /π(V ))[π(V )] if π(V ) has the same dimension as V and to zero otherwise. Now apply the pushforward map to our semiample divisor:   n   π∗ [D] = ai π∗ [Di ] =  ai π(Di ), i=1

ρi ∈ D (1)

because Di maps birationally onto its image when ρi ∈ D , and dim π(Di ) < dim Di in all other cases. The divisors π(Di ) for ρi ∈ D (1) are torus-invariant corresponding to the 1-dimensional cones in D . The support function of the Weil divisor π∗ (D) :=
ρi ∈ D (1) ai π(Di ) coincides with ψD , which is strictly convex with respect to the fan D . Hence, the divisor class π∗ [D] is ample. For the birational map π : P → P D , we also have a commutative diagram (see [F1, Section 3.4] and [F2]): Ad−1 (P ) O Pic(P ) o

π∗

π∗

  / Ad−1 P D O   Pic P D ,

where the vertical maps are inclusions. Since the support functions for the Cartier divisors D and π∗ (D) coincide, the pullback π ∗ π∗ [D] is exactly the divisor class [D]. Thus, we have the following useful result. Proposition 1.2. Let P be a complete toric variety with a semiample divisor class [D] ∈ Ad−1 (P ). There exists a unique complete toric variety P D with a toric birational map π : P → P D , such that is a subdivision of D , π∗ [D] is ample,
n ∗ and π π∗ [D] = [D]. Moreover, if D = i=1 ai Di is torus-invariant, then D is the normal fan of D . Remark 1.3. Since the fan D is the normal fan of D , there is a one-to-one correspondence between the k-dimensional cones of D and (d − k)-dimensional faces of D . Note, however, that while D is canonical with respect to the equivalence relation on the divisors, the polyhedron D is only canonical up to translation. We next study the intersection theory for the semiample divisors in complete toric varieties. Any toric variety P is a disjoint union of its orbits by the action of the torus T = N ⊗ C∗ that sits naturally inside P . Each orbit Tσ is a torus corresponding to a cone σ ∈ . The closure of each orbit Tσ is again a toric variety denoted by V (σ ).

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

89

Lemma 1.4. If D is a semiample divisor on a complete toric variety P , then the intersection number (D k · V (σ )) > 0 for any σ ∈ (d − k) contained in a cone of D (d − k) and (D k · V (σ )) = 0 for all other σ ∈ (d − k).
Proof. We can assume that D = ni=1 ai Di , which gives a support function ψD determined on each cone by some mσ : ψD (n) = mσ , n for all n ∈ σ . Since D is generated by global sections, for a fixed σ ∈ (d −k) we have (see [F1, Section 5.3]): 



⊥

volk D ∩ σ + mσ



=

Dk · V (σ ) . k!

(1)

By Remark 1.3, there is a one-to-one correspondence between the cones of D and the faces of D . Let σ be the minimal cone in D , corresponding to a face of D and containing σ . We claim that   =  D ∩ σ ⊥ + mσ . Indeed, since ψD is strictly convex with respect to D , from [Od, Lemma 2.12] we have  = m ∈ D : m, n = ψD (n) for all n ∈ σ , whence m ∈ implies m − mσ , n = 0 for all n ∈ σ . Conversely, suppose m ∈ D and (m − mσ ) ∈ σ ⊥ . The first condition implies m, n ≥ ψD (n) for all n from the strongly convex cone σ , while the second one gives a point in the interior of σ (by the minimal choice of this cone) for which m and ψD have the same values. Hence, m and ψD have the same values on σ , and the claimed equality of the sets follows. Now, the lemma follows from the fact that volk ( ) > 0 if and only if dim σ = d −k. Remark 1.5. Lemma 1.4 provides another way of constructing the fan D , by gluing the d-dimensional cones in along those facets τ for which (D · V (τ )) = 0. We now give necessary and sufficient conditions for a Cartier divisor on a complete toric variety to be ample, generated by global sections or semiample. This is a generalization of the toric Nakai criterion proved for nonsingular toric varieties in [Od, Theorem 2.18]. Theorem 1.6. Let P be a d-dimensional complete toric variety, and let D be a Cartier divisor on P . Then (i) D is generated by global sections if and only if (D · V (τ )) ≥ 0 for any τ ∈ (d − 1); (ii) D is ample if and only if (D · V (τ )) > 0 for any τ ∈ (d − 1). Proof. Without loss of generality, we can assume that D is torus-invariant. (i) If D is generated by global sections, then the required condition follows from

90

ANVAR R. MAVLYUTOV

equation (1). Conversely, the torus-invariant divisor D has the support function ψD , and it suffices to show that ψD is convex. Here, we use a trick. Consider a nonsingular subdivision  of the fan and the corresponding toric morphism f : P  → P . Then the support function of the pullback divisor f ∗ (D) coincides with ψD . So, we just need to show that f ∗ (D) is generated by global sections. By [F2, Example 2.4.3], we have (f ∗ (D) · V (τ  )) = (D · f∗ (V (τ  ))), where V (τ  ) is the closure of the 1-dimensional orbit corresponding to τ  ∈  (d − 1). If the smallest cone in containing τ  is d-dimensional, then the image of V (τ  ) is a point, implying that the above intersection number vanishes. Otherwise, τ  is contained in some τ ∈ (d −1), in which case f∗ (V (τ  )) = V (τ ). So, in either case, by the given condition in (i), the intersection number (D ·f∗ (V (τ  ))) is nonnegative. Following the proof of [Od, Theorem 2.18], we get that (f ∗ (D) · V (τ  )) ≥ 0 for any τ  ∈  (d − 1) implies f ∗ (D) is generated by global sections. (ii) If D is ample, then the required condition follows from Lemma 1.4 or, more generally, from the Nakai criterion for arbitrary complete varieties (see [H, Chapter I, Theorem 5.1] and [K]). Conversely, by part (i), the divisor D is generated by global sections. We show that D is semiample and the fan is exactly the fan D associated with the semiample divisor. Then, by Proposition 1.2, the desired result follows. From equation (1) and the given condition, it follows that the polyhedron D intersects different lines, corresponding to τ ∈ (d − 1), in more than one point. These lines cannot lie in a hyperplane of MR , because is complete. Therefore, D is maximal dimensional, implying that D is semiample. By Remark 1.5 and the given condition, the fan coincides with D . Thus, Proposition 1.2 implies that D is ample. Corollary 1.7. Let P be a complete toric variety. Then a Cartier divisor D on P is semiample if and only if (D d ) > 0 and (D · V (τ )) ≥ 0 for any τ ∈ (d − 1). Remark 1.8. In Mori’s theory, Theorem 1.6(ii) and [R, Proposition 1.6] imply that D is ample if and only if (D ·(NE(P )\{0})) > 0, where NE(P ) is the cone coming from effective 1-cycles. Also, by Theorem 1.6(i), the pseudoample cone PA(P ) is spanned by the divisors generated by global sections. For details see [R] and [Od, Section 2.5]. 2. Regular semiample hypersurfaces. Next we apply results from the previous section to describe a stratification of regular semiample hypersurfaces in a complete toric variety P . The following definition has appeared in [B2]. Definition 2.1. A hypersurface X ⊂ P is called -regular if X ∩ Tσ is empty or a smooth subvariety of codimension 1 in Tσ for any σ ∈ . Remark 2.2. Proposition 6.8 in [D] says that a hypersurface X ⊂ P defined by a general section of a line bundle generated by global sections is -regular. When it is clear from the context, we simply say that a hypersurface is regular.

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

91

Lemma 2.3. Let X be a semiample hypersurface in a complete toric variety P , such that dim P ≥ 2. Then (i) X is connected, and (ii) X is irreducible if X is -regular. Proof. (i) Consider an effective torus-invariant divisor D equivalent to the divisor X. Since ᏻP (D) is generated by global sections, choosing a basis of the space H 0 (P , ᏻP (D)) gives a mapping ϕD : P −→ Pr−1 , where r = h0 (P , ᏻP (D)) = Card(D ∩M). By [F1, Section 3.4, exercise, page 73], the image of ϕD has dimension equal to dim D . Since D is semiample, we get that dim D = dim P ≥ 2. From [FLa, Theorem 2.1], it follows that every divisor in the linear system |D| is connected. In particular, X is connected. (ii) To prove that X is irreducible, we argue as follows. Consider a nonsingular subdivision  of the fan and the corresponding morphism p : P  → P . It follows from [B2, Proposition 3.2.1] that p −1 (X) is a  -regular hypersurface that supports a semiample divisor p ∗ (X). By the previous part, p−1 (X) is a smooth connected hypersurface that must be irreducible. Therefore, X is irreducible. Proposition 2.4. If X ⊂ P is a -regular semiample hypersurface with the associated morphism π : P → P X for the divisor class [X] ∈ Ad−1 (P ) from Proposition 1.2, then Y := π(X) is a X -regular ample hypersurface, and X = π −1 (Y ). Proof. From Lemma 2.3(ii), we know that X is irreducible. Since X is -regular, it maps birationally onto its image, implying π∗ [X] = [π(X)]. Therefore, by Proposition 1.2, the hypersurface Y = π(X) is ample. Let us now show that Y misses the zero-dimensional orbits in P X . Consider the 1-dimensional orbit closure V (τ0 ) ⊂ P X corresponding to a cone τ0 ∈ X (d − 1), and take a cone τ ∈ (d − 1) that lies in τ0 . Since X is -regular,       Card X ∩ Tτ = Card X ∩ V (τ ) = X · V (τ ) . We also know that the orbit Tτ maps onto Tτ0 ; hence,           Y · V (τ0 ) ≥ Card Y ∩ V (τ0 ) ≥ Card Y ∩ Tτ0 ≥ Card X ∩ Tτ = X · V (τ ) . By [F2, Example 2.4.3], we have (Y ·V (τ0 )) = (X ·V (τ )), whence the above inequalities are equalities. Therefore, the hypersurface Y intersects the orbit Tτ0 transversally and does not intersect the points in the complement V (τ0 )\Tτ0 , corresponding to the d-dimensional cones in X that contain τ0 . Thus, we have shown that Y misses all zero-dimensional orbits in P X . One can easily show X = π −1 (Y ) from the facts that X and Y = π(X) are irreducible and that Y misses the zero-dimensional orbits. Finally, for arbitrary σ0 ∈ X

92

ANVAR R. MAVLYUTOV

we take σ ∈ , contained in σ0 , of the same dimension. Then we have an isomorphism Tσ ∼ = Tσ0 inducing another isomorphism X ∩ Tσ = π −1 (Y ) ∩ Tσ ∼ = Y ∩ Tσ0 . So, the -regularity of X implies that π(X) is X -regular. Remark 2.5. By construction in [B2], the MPCP-desingularizations (maximal pro of regular projective hypersurfaces Z in jective crepant partial desingularizations) Z a toric Fano variety P are regular semiample hypersurfaces. Proposition 2.4 shows that if we start with an arbitrary regular semiample hypersurface, then we come up with a similar picture. Let us note that a regular ample hypersurface Y in a complete toric variety P intersects all orbits transversally, except for zero-dimensional orbits. Such a hypersurface is called nondegenerate in [BC] and [DKh]. Also, in this case, a hypersurface in the torus T isomorphic to the affine hypersurface Y ∩ T in T is called nondegenerate. Such a hypersurface satisfies the following property. Lemma 2.6 [DKh]. Let Z be a nondegenerate affine hypersurface in the torus T; then the natural map H i (T) → H i (Z), induced by the inclusion, is an isomorphism for i < dim(T) − 1 and an injection for i = dim(T) − 1. Like in [B2], by Proposition 2.4, we get a nice stratification of a semiample regular hypersurface X ⊂ P in terms of nondegenerate affine hypersurfaces. Let Y = π(X); then X = π −1 (Y ). Using the standard description of a toric blow-up, we obtain   (2) X ∩ Tσ ∼ = Y ∩ Tσ0 × (C∗ )dim σ0 −dim σ , where σ0 ∈ X is the smallest cone containing σ ∈ . 3. A cup product formula for quasismooth hypersurfaces. The purpose of this section is to give a generalization of the algebraic cup product formula for the residues of rational forms presented in [CarG]. In this section, we assume that P is a complete simplicial toric variety. Such a toric variety has a homogeneous coordinate ring S = C[x1 , . . . , xn ], with variables x1 , . . . , xn corresponding to the irreducible torusinvariant divisors (see [C1]). This ring is graded by the Chow group nD1 , .a.i . , Dn
Ad−1 (P) : deg( i=1 xi ) = [ ni=1 ai Di ]. Furthermore, if ᏸ is a line bundle on P, then for β = [ᏸ] ∈ Ad−1 (P) one has an isomorphism H 0 (P, ᏸ) ∼ = Sβ . So, the homogeneous polynomials in Sβ identified with the global sections of ᏸ determine hypersurfaces in the toric variety P. Definition 3.1 [BC]. A hypersurface X ⊂ P defined by a homogeneous polynomial f ∈ Sβ is called quasismooth if ∂f/∂xi , 1 ≤ i ≤ n, do not vanish simultaneously on P. Definition 3.2 [BC]. Fix an integer basis m1 , . . . , md for the lattice M. Then, given subset I = {i1 , . . . , id } ⊂ {1, . . . , n}, denote det(eI ) = det(mj , eik 1≤j, k≤d ),

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

dxI = dxi1 ∧ · · · ∧ dxid , and xˆI =



i ∈I / xi .



,=

93

Define the n-form , by the formula

det(eI )xˆI dxI ,

|I |=d

where the sum is over all d element subsets I ⊂ {1, . . . , n}. Let X ⊂ P
be a quasismooth hypersurface defined by f ∈ Sβ . For A ∈ S(a+1)β−β0 (here, β0 := ni=1 deg(xi )), consider a rational d-form ωA :=

   A, 0 d ∈ H P, , (a + 1)X . P f a+1

This form gives a class in H d (P \ X), and by the residue map Res : H d (P \ X) −→ H d−1 (X) we get Res(ωA ) ∈ H d−1 (X). We need an explicit algebraic formula for the Hodge ˇ cohomology. component Res(ωA )d−1−a, a in Cech Denote fi = ∂f/∂xi , and let Ui = {x ∈ P : fi (x)  = 0} for i = 1, . . . , n. If X is a quasismooth hypersurface, then ᐁ = {Ui }ni=1 is an open cover of P. The next two theorems with their proofs are corrected and generalized versions of unpublished results of D. Cox and D. Morrison. Theorem 3.3. Let
X ⊂ P be a quasismooth hypersurface defined by f ∈ Sβ and A ∈ S(a+1)β−β0 , β0 = ni=1 deg(xi ). Then, under the natural map   d−1−a  d−1−a  ∼ d−1−a, a Hˇ a ᐁ|X , ,X (X), −→ H a X, ,X =H ˇ cocycle ca {AKia · · · Ki0 ,/ the component Res(ωA )d−1−a, a corresponds to the Cech d−1+(a(a+1)/2) , and Ki is the contraction opfi0 · · · fia }i0 ...ia , where ca = (1/a!)(−1) erator (∂/∂xi )
. Proof. The residue map can be calculated in hypercohomology using the commutative diagram Res / H d−1 (X) H d (P \ X) O O   Hd ,•P (log X)

Res

  / Hd−1 ,• , X

ˇ where the vertical maps are isomorphisms. As in [CarG], we can work in the Cech–de Rham complex C • (ᐁ, ,• (∗X)) with arbitrary algebraic singularities along X, where ᐁ = {Ui }ni=1 . Then we can apply the arguments of [CarG, pages 58–62] almost without any change. We only need to check that df ∧ , ≡ 0

modulo multiples of f

(3)

94

ANVAR R. MAVLYUTOV

for part (i) of the lemma on page 60 in [CarG]. But df ∧ (dx1 ∧ · · · ∧ dxn ) = 0, and, by [C2, Lemma 6.2], , = θ1
· · ·
θn−d
(dx1 ∧ · · · ∧ dxn ) for some Euler vector fields θi . The equivalence (3) can be obtained by repeatedly applying the following argument. If df ∧ ω ≡ 0 (mod f ) for some form ω, and θ is a Euler vector field, then 0 ≡ θ
(df ∧ ω) = (θ
df )ω − df ∧ (θ
ω). Since θ
df = θ(f ) = θ(β)f (see [C2, proof of Proposition 5.3]), we get df ∧ (θ
ω) ≡ 0. Thus, the lemma on page 60 of [CarG] is true in our situation. The rest of the arguments apply without change, and the theorem is proved. Let us remark that the constructive proof of [CarG] implies ˇ cocycle. that ca {(AKia · · · Ki0 ,)/(fi0 · · · fia )}i0 ...ia is actually a Cech
n Definition 3.4. For β = [ i=1 bi Di ] ∈ Ad−1 (P) and a multi-index I = (i0 , . . . , id ) β viewed as an ordered subset of {1, . . . , n}, we introduce a constant cI that is the determinant of the (d + 1) × (d + 1) matrix obtained from (mj , eik 1≤j ≤d, ik ∈I ) by adding the first row (bi0 , . . . , bid ), where m1 , . . . , md is the fixed integer basis of the β lattice M as in Definition 3.2. One can easily check that cI is well defined. Theorem 3.5. Let X ⊂ P be a quasismooth hypersurface defined by f ∈ Sβ , and suppose a + b = d − 1, ωA = A,/f a+1 , and ωB = B,/f b+1 for A ∈ S(a+1)β−β0 and B ∈ S(b+1)β−β0 . Then, under the composition     ∪    d−1  δ H a X, ,bX ⊗ H b X, ,aX −−→ H d−1 X, ,X −→ H d P, ,dP (here, δ is the coboundary map in the Poincaré residue sequence), we have that ˇ δ(Res(ωA )ba ∪ Res(ωB )ab ) is represented by the Cech cocycle   β   ABcI xˆI , cab ∈ Hˇ d ᐁ, ,dP , fi0 · · · fid I

where I = (i0 , . . . , id ) and cab = (−1)(a(a+1)/2)+(b(b+1)/2)+a

2 +d−1

/a! b!.

Proof. As in [CarG, page 63], by Theorem 3.3 we see that the residue product Res(ωA )ba ∪ Res(ωB )ab is represented by the cocycle    AKia · · · Ki0 , BKid−1 · · · Kia , d−1  ψ = c˜ab ∧ ∈ C d−1 ᐁ|X , ,X , fi0 · · · fia fia · · · fid−1 i0 ...id−1 where c˜ab = (−1)a ca cb = (−1)(a(a+1)/2)+(b(b+1)/2)+a /a!b!. To calculate the coboundary of this cocycle, we use the following commutative diagram: 2

2

0

  / C d ᐁ , ,d P O

  / C d ᐁ, ,d (log X) PO

0

  / C d−1 ᐁ, ,d P

  / C d−1 ᐁ, ,d (log X) P

Res

Res

  / C d ᐁ|X , ,d−1 X O   / C d−1 ᐁ|X , ,d−1 . X

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

95

Lift the cocycle ψ to     AKia · · · Ki0 , BKid−1 · · · Kia , df ˜ ψ = c˜ab ∧ ∧ ∈ C d−1 ᐁ, ,dP (log X) . fi0 · · · fia fia · · · fid−1 f i0 ...id−1 From the diagram, we can see that changing of the numerator by a multiple of f will not affect the image of ψ in Hˇ d (ᐁ, ,dP ). Hence, we need to compute Kia · · · Ki0 , ∧ Kid−1 · · · Kia , ∧ df modulo multiples of f . First we show Kia · · · Ki0 , ∧ Kid−1 · · · Kia , ∧ df ≡ (some function) · , (mod f ).

(4)

As in the proof of Theorem 3.3, we can write , = E
dx, where E is a wedge of some Euler vector fields and dx = dx1 ∧ · · · ∧ dxn . Denote du = dxi0 ∧ · · · ∧ dxia−1 , dv = dxia+1 ∧ · · · ∧ dxid−1 , and dw = ∧i ∈I / 0 dxi , where I0 = (i0 , . . . , id−1 ). Then dx = ± du ∧ dxa ∧ dv ∧ dw. Now compute: Kia · · · Ki0 , = Kia · · · Ki0 (E
dx) = ±E
(Kia · · · Ki0 du ∧ dxa ∧ dv ∧ dw)   = ±E
(dv ∧ dw) = ± (E
dv) ∧ dw + (−1)(d−a−1)(n−d) dv(E
dw) . Similarly,   Kid−1 · · · Kia , = ± (E
du) ∧ dw + (−1)a(n−d) du(E
dw) . Since dw ∧ dw = 0, we get Kia · · · Ki0 , ∧ Kid−1 · · · Kia ,  = ±(E
dw) (E
dv) ∧ dw ∧ (−1)a(n−d) du + (−1)(d−a−1)(n−d) dv ∧ (E
du) ∧ dw  + (−1)(d−1)(n−d) dv ∧ du ∧ (E
dw)     = ±(E
dw) E
(dv ∧ du ∧ dw) = ±(E
dw) E
Kia
dx     = ±(E
dw) Kia
(E
dx) = ±(E
dw) Kia , . From equation (3), we know that ,∧df ≡ 0 modulo multiples of f . Applying the contraction operator Kia to this identity, we obtain (Kia ,) ∧ df ≡ ±fia ,, whence equation (4) follows:   Kia · · · Ki0 , ∧ Kid−1 · · · Kia , ∧ df = ±(E
dw) Kia , ∧ df ≡ ±(E
dw)fia ,. We next claim that   Kia · · · Ki0 , ∧ Kid−1 · · · Kia , ∧ df ≡ (−1)d−1 det eI0 xˆI0 fia , (mod f ).

(5)

96

ANVAR R. MAVLYUTOV

Examine the coefficient of dxI0 = dxi0 ∧ · · · ∧ dxid−1 in the left-hand side. The only place dxI0 can occur is in         Kia · · ·Ki0 det eI0 xˆI0 dxI0 ∧ Kid−1 · · · Kia det eI0 xˆI0 dxI0 ∧ fia dxia  2 = det eI0 xˆI20 dxia+1 ∧ · · · ∧ dxid−1 ∧ (−1)a(d−a) dxi0 ∧ · · · ∧ dxia−1 ∧ fia dxia       = (−1)d−1 det eI0 xˆI0 fia det eI0 xˆI0 dxI0 .
From here, equation (5) follows, because , = |I |=d det(eI )xˆI dxI , and the left-hand side of (5) is (some function) · , modulo multiples of f . Returning to the calculation of the coboundary δ(ψ), by equation (5) we have     (−1)d−1 c˜ab AB det eI0 xˆI0 , ψ˜ ≡ , f fi0 · · · fid−1 i0 ...id−1

so that  d    det eI \{ik } xik fik xˆI , (−1)d−1 c˜ab  k δ(ψ) ≡ (−1) AB , f fi0 · · · fid k=0

I


d

where I = (i0 , . . . , id ). But the identity k=0 (−1)k det(eI \{ik } )eik = 0 holds and gives
β an Euler formula cI f = dk=0 (−1)k det(eI \{ik } )xik fik (see [BC]). Thus,   β ABcI xˆI , δ(ψ) ≡ cab , fi0 · · · fid I

where cab = (−1)d−1 c˜ab . As in the classic case (see [PS]), we go further to relate the multiplicative structure on some quotient of the homogeneous ring S to the cup product on the middle cohomology of the quasismooth hypersurface X given by a homogeneous polynomial f ∈ Sβ . Definition 3.6 [BC]. For f ∈ Sβ , the Jacobian ideal J (f ) ⊂ S is the ideal generated by the partial derivatives ∂f/∂x1 , . . . , ∂f/∂xn . Also, the Jacobian ring R(f ) is the quotient ring S/J (f ) graded by the Chow group Ad−1 (P). To show a relation between the cup product and multiplication in R(f ), we need two lemmas. We have the natural map S(a+1)β−β0 → H d−1−a, a (X) that sends A to the corresponding component of Res(ωA ). The map Res( ω_ )d−1−a, a : R(f )(a+1)β−β0 −→ H d−1−a, a (X), induced by the above one is well defined because of the following statement.

97

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

Lemma 3.7. If A ∈ J (f )(a+1)β−β0 , then Res(ωA )d−1−a, a = 0. Proof. In the case a = 0, the statement is trivial because J (f )β−β0 = 0. Assume that a > 0. By Theorem 3.3, since A ∈ J (f ), it suffices to show that {(fj Kia · · · Ki0 ,)/ d−1−a ˇ (fi0 · · · fia )}i0 ...ia in C a (ᐁ|X , ,X ) is a Cech coboundary for one of the partial derivatives fj = ∂f/∂xj . We have Kj Kia · · · Ki0 (df ∧ ,) = (−1)a+2 df ∧ Kj Kia · · · Ki0 , + (−1)a+1 fj Kia · · · Ki0 , +

a   (−1)k fik Kj Kia · · · K ik · · · Ki0 ,. k=0

But df ∧ , ≡ 0 (mod f ) by equation (3), and df = 0 on the hypersurface X. Therefore, on X we have the identity a   fj Kia · · · Ki0 , = (−1)a+k fik Kj Kia · · · K ik · · · Ki0 ,. k=0

Hence, {(fj Kia · · · Ki0 ,)/(fi0 · · ·fia )}i0 ...ia is the image of (−1)a {(Kj Kja−1 · · · Kj0 ,)/ d−1−a ˇ (fj0 · · · fja−1 )}j0 ...ja−1 under the Cech coboundary map C a−1 (ᐁ|X , ,X ) → d−1−a a C (ᐁ|X , ,X ). Consider the map S(d+1)β−2β0 → H d, d (P) that sends a polynomial h to the class in H d, d (P) represented by the cocycle  β    hcI xˆI , ∈ Hˇ d ᐁ, ,dP fi0 · · · fid I

as in Theorem 3.5. This induces the map λ : R(f )(d+1)β−2β0 → H d,d (P), well defined by the following statement. β

ˇ Lemma 3.8. If h ∈ J (f ), then {hcI xˆI ,/(fi0 · · · fid )}I is a Cech coboundary. Proof. We can assume that h is one of the partial derivatives fj = ∂f/∂xj . Let I be the ordered subset {i0 , . . . , id } ⊂ {1, . . . , n}. Then the equality β

cI ej +

d  k=0

β

(−1)k+1 c{j }∪I \{ik } eik = 0

ik , . . . , id }) holds and gives the Euler (here, {j } ∪ I \ {ik } is the ordered set {j, i0 , . . . , formula (see [BC])   d d   β β β k+1 β cI bj + (−1) c{j }∪I \{ik } bik f = cI xj fj + (−1)k+1 c{j }∪I \{ik } xik fik , k=0


n

k=0

β

where the numbers bi are determined by β = [ i=1 bi Di ]. But the number cI bj +
d k+1 cβ k=0 (−1) {j }∪I \{ik } bik is the determinant of a matrix with the same two rows

98

ANVAR R. MAVLYUTOV

(bj , bi0 , . . . , bid ), so it vanishes. Using the above Euler formula, we see that under the ˇ Cech coboundary map C d−1 (ᐁ, ,dP ) → C d (ᐁ, ,dP ), the cocycle 

β

fj cI xˆI , fi0 · · · fid


d

 =

k=0 (−1)

k cβ {j }∪I \{ik } fik xˆ {j }∪(I \{ik }) ,

fi0 · · · fid

I

β is the image of {c{j }∪J xˆ{j }∪J ,/(fj0 · · · fjd−1 )}J , where J is the ordered set {j, j0 , . . . , jd−1 }.

 I

= {j0 , . . . , jd−1 } and {j }∪J

As a consequence of Theorem 3.5 and Lemmas 3.7 and 3.8, we have proved the following theorem. Theorem 3.9. Let X ⊂ P be a quasismooth hypersurface defined by f ∈ Sβ , and suppose a + b = d − 1. Then the diagram R(f )(a+1)β−β0 × R(f )(b+1)β−β0

cab ·multiplication

/ R(f )(d+1)β−2β0

Res( ω_ )ba ×Res( ω_ )ab

 H b, a (X) × H a, b (X)

λ

∪

/ H d−1, d−1 (X)

δ

 / H d, d (P)

commutes, where λ is as defined above and δ is the Gysin map. 4. Cohomology of regular hypersurfaces. In this section, we present an application of the Gysin spectral sequence for computing the cohomology of regular semiample hypersurfaces in a complete simplicial toric variety P. We obtain an explicit description of the cup product on some part of the middle cohomology of such hypersurfaces. Section 3 studied the relation between multiplication in R(f ) and the cup product, whereas this section studies such a relation of a smaller ring R1 (f ) and the cup product. The rings R(f ) and R1 (f ) were previously used in [BC] for studying the cohomology  of ample hypersurfaces. Let D = P \ T = ni=1 Di , and let X ⊂ P be a regular hypersurface. Then (X, X ∩ D) is a toroidal pair (see [D, Section 15]), and also X ∩ D consists of quasismooth components that intersect quasitransversally. Therefore, by the results from [D, Section 15] we have    • ∼ Gr W ,•−k k ,X log(X ∩ D) = X∩V (σ ) , dim σ =k

and the (Gysin) spectral sequence of this filtered complex (see [De, Section 3.2])    pq • E1 = Hp+q X, Gr W −p ,X log(X ∩ D)      ∼ H 2p+q X ∩ V (σ ) ⇒ H p+q X \ (X ∩ D) = dim σ =−p

99

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

degenerates at E2 and converges to the weight filtration W• on H p+q (X \ (X ∩ D)):   pq p+q E2 = Gr W X \ (X ∩ D) . q H In particular, note that 1 H 1 (X) ∼ = Gr W 1 H (X ∩ T).

(6)

Now assume that X ⊂ P is a regular semiample hypersurface. In this case X \ (X ∩ D) = X ∩ T is a nondegenerate affine hypersurface in T. Hence, by Lemma pq p+q (X ∩ T) vanishes unless p + q = d − 1 and q ≥ d − 1, or 2.6, E2 = Gr W q H p + q < d − 1 and q = −2p. Therefore, from the Gysin spectral sequence we obtain the following exact sequences. First, for s odd, s < d − 1, we have    0 −→ H 1 X ∩ V (σ ) dim σ =(s−1)/2

−→ · · · −→



  H s−2k X ∩ V (σ ) −→ · · · −→ H s (X) −→ 0,

dim σ =k

where the maps are alternating sums of the Gysin morphisms. Next, for s even, s < d − 1, we get    s/2 0 −→ Gr W (X ∩ T) −→ H 0 X ∩ V (σ ) −→ · · · −→ H s (X) −→ 0. s H dim σ =s/2

Finally, for s = d − 1,    · · · −→ H d−1−2k X ∩ V (σ ) dim σ =k d−1 (X ∩ T) −→ 0. (7) −→ · · · −→ H d−1 (X) −→ Gr W d−1 H

Similar sequences exist for s > d −1 that are exact except for one term. We are mainly concerned with the last exact sequence, which determines the middle cohomology group of X. The following fact, contained in [C2, Proposition 5.3], characterizes regular hypersurfaces. Lemma 4.1 [C2]. Let X ⊂ P be a hypersurface defined by a homogeneous polynomial f . Then X is regular if and only if xi (∂f /∂xi ), i = 1, . . . , n, do not vanish simultaneously on P. In this case, we call f nondegenerate. This lemma shows that, in complete simplicial toric varieties, regular hypersurfaces are quasismooth. In Theorem 4.5, we prove a stronger analog of Theorem 3.9 for regular semiample hypersurfaces. i = {x ∈ P : xi fi (x)  = 0} cover the In the case f is nondegenerate, the open sets U  i }n is a refinement of ᐁ defined toric variety P. In particular, the open cover ᐁ = {U i=1 in Section 3.

100

ANVAR R. MAVLYUTOV

Definition 4.2 [BC]. Given f ∈ Sβ , we get the ideal quotient   x1 ∂f xn ∂f ,..., : x 1 · · · xn J1 (f ) = ∂x1 ∂xn (see [CLO, page 193]) and the ring R1 (f ) = S/J1 (f ) graded by the Chow group Ad−1 (P). To show the relation between multiplication in R1 (f ) and the cup product on the hypersurface defined by f , we need some results similar to those in Section 3. Lemma 4.3. Let f ∈ Sβ be nondegenerate, and let h ∈ J1 (f ); then the cocycle β , ,d ). {hcI xˆI ,/(fi0 · · · fid )}I vanishes in Hˇ d (ᐁ P Proof. The proof of this is similar to the proof of Lemma 3.8. Theorem 4.4. Let X ⊂ P be a regular semiample hypersurface defined by f ∈ Sβ , and suppose a + b = d − 1. (i) If A ∈ J1 (f )(a+1)β−β0 , then Res(ωA )b, a = 0. (ii) The map Res( ω_ )b, a : R1 (f )(a+1)β−β0 → H b, a (X) is injective, and the natural composition   R1 (f )(a+1)β−β0 −→ H b, a (X) −→ H b, a H d−1 (X ∩ T) is an isomorphism, so that we have a natural imbedding Gr F Wd−1 H d−1 (X ∩ T) @→ Gr F H d−1 (X). Moreover, we have an isomorphism   n   b, a b−1, a−1 ∼ H (X) = R1 (f )(a+1)β−β ϕi ! H (X ∩ Di ) , 0

i=1

where ϕi ! are the Gysin maps for ϕi : X ∩ Di @→ X, and Res(ωA )b, a ∪ ϕi ! H a−1, b−1 (X ∩ Di ) = 0

for all A ∈ R1 (f )(a+1)β−β0 .

Proof. (i) We prove the statement using the Poincaré duality H b, a (X) ⊗ H a, b (X) −→ H d−1, d−1 (X), where b = d − 1 − a. Since the pairing is nondegenerate, it suffices to show for A ∈ J1 (f )(a+1)β−β0 that the cup product of Res(ωA )b, a with all elements in H a, b (X) vanishes. For this we need to find the elements that span the
group H a, b (X). Let X be linearly equivalent to a torus-invariant divisor ni=1 ai Di with ai ≥ 0, and let  be the corresponding polytope defined by the inequalities m, ei  ≥ −ai . As in [B1], S denotes the subring of C[t0 , t1±1 , . . . , td±1 ] spanned over C by all monomials of the form t0k t m = t0k t1m1 · · · t1md , where k ≥ 0 and m ∈ k. We have a

101

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

natural isomorphism of graded rings (see [BC, proof of Theorem 11.5]) ∞ 

S ∼ =

Skβ ⊂ S,

k=0

sending t0k t m to

n

kai +m, ei  . i=1 xi

Skβ−β0

This isomorphism induces the bijection

n

i=1 xi

  (1) / x1 · · · xn kβ ∼ = I , k

(1)

where I ⊂ S is the ideal spanned by all monomials t0k t m such that m is in the interior of k. As a consequence of the exact sequence (7) and [B1, Theorems 7.13 and 8.2], we have the diagram n  i=1

H a−1, b−1 (X ∩ Di )

⊕ϕi !

/ H a, b (X) O

  / H a, b H d−1 (X ∩ T) O

Res( ω_ )ab

Res(ω˜ _ )ab

S(b+1)β−β0

  / I (1) 

/0

(8) b+1

,

where the top row is exact and the right vertical map is defined by





dt1 dtd tm ∧···∧ ω˜ t b+1 t m = 0 t1 td f˜(t)b+1 (here, f˜(t) is the Laurent polynomial defining the affine hypersurface X ∩ T, so that t0 f˜(t) corresponds to f (x) under the isomorphism (S )1 ∼ = Sβ ) and by Resab induced by the Poincaré residue mapping [B1, Section 5]:   Res : H d T \ (X ∩ T) −→ H d−1 (X ∩ T). The diagram commutes because the restriction of the form ωB = B,/f b+1 , with  (b+1)ai −1+m, ei  , to the torus T coincides with (t m /f˜(t)b+1 )(dt1 /t1 ) ∧ B = ni=1 xi  m , e  · · · ∧ (dtd /td ). (Use the coordinates tj = ni=1 xi j i on the torus with the fixed integer basis m1 , . . . , md from Definition 3.2.) The first row in (8) is exact and the composition S(b+1)β−β0

  / I (1) 

Res( ω˜ _ )ab b+1

  / H a, b H d−1 (X ∩ T)

a, b is surjective by [B1, Theorem 8.2]. Therefore,  a−1, b−1the group H (X) is spanned by ab Res(ωB ) for B ∈ S(b+1)β−β0 and ϕi ! H (X ∩ Di ) for i = 1, . . . , n.

102

ANVAR R. MAVLYUTOV

From Theorem 3.5 and Lemma 4.3, it follows that Res(ωA )ba ∪ Res(ωB )ab = 0 for A ∈ J1 (f )(a+1)β−β0 and all B ∈ S(b+1)β−β0 . Also, for any A ∈ S(a+1)β−β0 and h ∈ H a−1, b−1 (X ∩ Di ) we have   (9) Res(ωA )ba ∪ ϕi ! h = ϕi ! ϕi∗ Res(ωA )ba ∪ h by the projection formula for Gysin homomorphisms. However, ϕi∗ Res(ωA )ba is repˇ resented by the restriction of the Cech cocycle ca {(AKia · · · Ki0 ,)/(fi0 · · · fia )}i0 ...ia ∈ b a |X , , ) from Theorem 3.3 to X ∩ Di . This restriction vanishes, because if Hˇ (ᐁ X i0 ∩ · · · ∩ U ia ∩ Di is empty, and if i ∈ i ∈ {i0 , . . . , id }, then U / {i0 , . . . , id }, then each term in the form Kia · · · Ki0 , contains xi or dxi . Thus, we have shown that the cup product of Res(ωA )ba for A ∈ J1 (f )(a+1)β−β0 , with all elements in H a, b (X), vanishes, and the result follows. (ii) From the diagram (8) and part (i), we get a natural map R1 (f )(a+1)β−β0 → H d−1−a, a (H d−1 (X ∩ T)). The fact that this map is an isomorphism follows from [BC, proof of Theorem 11.8]. Using the diagram (8), we can now see that the map Res( ω_ )d−1−a, a : R1 (f )(a+1)β−β0 → H d−1−a, a (X) is injective, and we get the desired description of the middle cohomology group H d−1 (X). By equation (9), Res(ωA )d−1−a, a ∪ ϕi ! H a−1, d−2−a (X ∩ Di ) = 0. Combining Theorem 3.9 with Lemma 4.3 and Theorem 4.4(i), we get the following result. Theorem 4.5. Let X ⊂ P be a regular semiample hypersurface defined by f ∈ Sβ , and suppose a + b = d − 1. Then the diagram R1 (f )(a+1)β−β0 × R1 (f )(b+1)β−β0

cab ·multiplication

/ R1 (f )(d+1)β−2β0

Res( ω_ )ba ×Res( ω_ )ab

 H b, a (X) × H a, b (X)

λ

∪

/ H d−1, d−1 (X)

commutes, where cab = (−1)(a(a+1)/2)+(b(b+1)/2)+a

2 +d−1

δ

 / H d, d (P)

/a!b!.

We finish this section with an explicit procedure of computing  Res(ωA )ba ∪ Res(ωB )ab . X

To have this, we need generalizations of some results in [C2]. Definition 4.6 [C2]. Assume F0 , . . . , Fd ∈ Sβ do not vanish simultaneously on a complete toric variety P. Then the toric residue map ResF :

Sρ −→ C, F0 , . . . , Fd ρ

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

103

ρ = (d + 1)β − β0 , is given by the formula ResF (H ) = Tr P ([ϕF (H )]), where Tr P : H d (P, ,dP ) → C is the trace map, and [ϕF (H )] is the class represented by the d-form ˇ H ,/F0 · · · Fd in Cech cohomology with respect to the open cover {x ∈ P : Fi (x)  = 0}. Proposition 4.7. If F0 , . . . , Fd ∈ Sβ , then there is JF ∈ S(d+1)β−β0 such that d 

j ∧ · · · ∧ dFd = JF ,. (−1)j Fj dF0 ∧ · · · ∧ dF

j =0 β

Furthermore, if I = {i0 , . . . , id } ⊂ {1, . . . , n} such that cI  = 0 (if β  = 0, there is at β least one such I ), then JF = det(∂Fj /∂xik )/cI xˆI . The polynomial JF is called the toric Jacobian of F0 , . . . , Fd . Proof. This is essentially Proposition 4.1 in [C2]. To show that JF coincides with the toric Jacobian in [C2], use the Euler formula β

cI g =

d    ∂g (−1)k det eI \{ik } xik ∂xik

for g ∈ Sβ

k=0

from the proof of Theorem 3.5. Theorem 4.8. Let P be a complete toric variety, and let β ∈ Ad−1 (P) be semiample. If F0 , . . . , Fd ∈ Sβ do not vanish simultaneously on P, then: (i) The toric residue map ResF : Sρ /F0 , . . . , Fd ρ → C, ρ = (d + 1)β − β0 , is an isomorphism. (ii) If JF ∈ S(d+1)β−β0 is the toric Jacobian of F0 , . . . , Fd , then ResF (JF ) = d! vol() = deg(F ), where  is the polyhedron associated to a torus-invariant divisor in the equivalence class of β and F : P → Pd is the map defined by F (x) = (F0 (x), . . . , Fd (x)). Proof. This statement was proved for ample β in [C2, Theorem 5.1], but the proof can be applied in our case almost without change. Indeed, consider the map F = (F0 , . . . , Fd ) : P → Pd given by the sections of a semiample line bundle ᏻP (D). Since (D d ) > 0 and F0 , . . . , Fd do not vanish simultaneously on P, it follows that F0 , . . . , Fd are linearly independent. We can extend F0 , . . . , Fd to a basis of H 0 (P, ᏻP (D)), which gives the associated map φ : P → PN , where N = h0 (P, ᏻP (D)) − 1. Then the map F factors through the map φ and a projection p : PN \ L −→ Pd ,

(y0 , . . . , yN ) ' −→ (y0 , . . . , yd ),

where L ⊂ PN is a projective subspace defined by y0 = · · · = yd = 0. By [F1, Section 3.4, exercise, page 73], the dimension of the image of φ is d. Using a dimension

104

ANVAR R. MAVLYUTOV

argument, one can show that p −1 (y0 , . . . , yd ) ∩ im(φ) is nonempty. Hence, F is surjective and, consequently, generically finite. Propositions 3.1, 3.2, and 3.3 in [C2] are still valid in the case when β is semiample, because the isomorphism S ∼ = S∗β holds. The rest of the arguments in [C2] apply without change. Definition 4.9 [BC]. Given f ∈ Sβ , let J0 (f ) ⊂ S denote the ideal generated by xi ∂f/∂xi , 1 ≤ i ≤ n, and put R0 (f ) = S/J0 (f ). β

Lemma 4.10. If I = {i0 , . . . , id } ⊂ {1, . . . , n} such that cI  = 0, then xi ∂f/∂xi , i ∈ I , do not vanish simultaneously on P, and J0 (f ) = xi0 ∂f/∂xi0 , . . . , xid ∂f/∂xid . β

β

Proof. If cI  = 0, then ei0 , . . . , eid span MR . From the Euler formula cI f =
d k k=0 (−1) det(eI \{ik } )xik (∂f /∂xik ) and [C2, Proposition 5.3] the lemma follows.  We now return to the calculation of X Res(ωA )ba ∪Res(ωB )ab , when X is a regular semiample hypersurface. Let Fj = xj (∂f /∂xj ), and let I = {i0 , . . . , id } ⊂ {1, . . . , n} β β be such that cI  = 0. Then denote J = det(∂Fj /∂xi )i, j ∈I /(cI )2 xˆI . One can show that J does not depend on the choice of I . By Lemma 4.10, the polynomials Fi , i ∈ I , do not vanish simultaneously on P and determine the toric residue map ResFI . From the definitions of λ, ResFI , and [C2, Proposition A.1], we obtain a commutative diagram R1 (f )(d+1)β−2β0 λ

 H d, d (P)



n

i=1 xi

/ R0 (f )(d+1)β−β0 β

√

−1/2π −1

c ResFI

d I

P

(10)

 / C,

where the arrow on the top is just the multiplication. Using Theorem 4.8, we get the following procedure. For given A ∈ R1 (f )(a+1)β−β0 and B ∈ R1 (f )(b+1)β−β0 , there is a unique constant c such that 

 x1 ∂f xn ∂f A · Bx1 · · · xn − cJ ∈ ,..., . ∂x1 ∂xn Then

 X

where D =

√ d  Res(ωA )ba ∪ Res(ωB )ab = c − 2π −1 cab d! vol(D ),


n

i=1 ai Di

such that [D] = β.

5. Cup product on regular semiample 3-folds. In this section, we completely describe the middle cohomology and the cup product on it for a regular semiample hypersurface X ⊂ P , when dim P = 4. It follows from (7) that the map

105

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

⊕ni=1 H 1 (X ∩ Di ) H

b, a

⊕ϕi !

/ H 3 (X) is injective. Hence, by Theorem 4.4,

(X) ∼ = R1 (f )(a+1)β−β0

 n  

ϕi ! H

b−1, a−1





X ∩ Di



,

(11)

i=1

where a + b = 3. We first determine which of the groups H b−1, a−1 (X ∩ Di ) vanish. Lemma 5.1. Let X ⊂ P , dim P = 4, be a -regular semiample hypersurface, and let π : P → P X be the morphism associated with X. Then: (i) H 1 (X ∩ Di ) = 0 unless ρi ⊂ σ for some 2-dimensional cone σ ∈ X , and ρi ∈ / X (1) (so ρi \ {0} lies in the relative interior of σ ). (ii) For ρi ⊂ σ , such that σ ∈ X (2) and ρi ∈ / X (1), we have     πi∗ : H 1 Y ∩ V (σ ) ∼ = H 1 X ∩ Di , where V (σ ) = π(Di ) is the orbit closure corresponding to σ ∈ X , Y := π(X), and πi : X ∩ Di → Y ∩ V (σ ) is the map induced by π. Proof. (i) Applying (6) to the regular hypersurface X ∩ Di in the toric variety Di , we have     1 H 1 X ∩ Di ∼ (12) = Gr W 1 H X ∩ Tρi . If ρi ∈ X (1), then X ∩ Tρi is a nondegenerate affine hypersurface in Tρi because of (2). Hence,     1 ∼ W 1 Gr W 1 H X ∩ Tρi = Gr 1 H Tρi = 0. If ρi does not lie in a cone σ ∈ X (2), then X ∩Tρi is empty or a disjoint finite union 1 of (C∗ )2 , by equation (2). In this case, Gr W 1 H (X ∩ Tρi ) also vanishes, and part (i) follows. (ii) Suppose ρi ∈ / X (1) is contained in a cone σ ∈ X (2), and let σ  ∈ (2) be the cone such that ρi ⊂ σ  ⊂ σ . Then we get a composition   H 1 Y ∩ V (σ )

πi∗

  / H 1 X ∩ Di

ϕi,∗ σ 

   / H1 X∩V σ ,

where ϕi,σ  : X ∩ V (σ  ) @→ X ∩ Di is the inclusion. To prove part (ii), it suffices to show that this composition is an isomorphism and all spaces in the composition are of the same dimension. Applying (6) to the regular hypersurfaces X ∩V (σ  ) in V (σ  ) and Y ∩ V (σ ) in V (σ ), we get a commutative diagram   1 ∼ H 1 Y ∩ V (σ ) = Gr W 1 H (Y ∩ Tσ )     H1 X∩V σ

∼ =

   1 X∩T  , H Gr W σ 1

106

ANVAR R. MAVLYUTOV

where the vertical arrow on the right is induced by the isomorphism Tσ  ∼ = Tσ . From the diagram, we see that the natural map H 1 (Y ∩ V (σ )) → H 1 (X ∩ V (σ  )) is an isomorphism. On the other hand, since X ∩ Tρi ∼ = (Y ∩ Tσ ) × C∗ because of (2), it follows from (12) that        1 1 H 1 X ∩ Di ∼ Y ∩ Tσ × C∗ ∼ = Gr W = Gr W 1 H 1 H Y ∩ Tσ by the Künneth isomorphism. Thus, the dimensions of spaces H 1 (X∩Di ) and H 1 (Y ∩ V (σ )) coincide. This finishes the proof of part (ii). Lemma 5.1 relates the nonvanishing groups H 1 (X ∩ Di ) to the middle cohomologies of regular ample hypersurfaces in 2-dimensional toric varieties. Using (11) and Theorem 4.4, we can now give a complete algebraic description of the middle cohomology group H 3 (X). Let S(V (σ )) = C[yγ : σ ⊂ γ ∈ X (3)] be the coordinate ring of the 2-dimensional complete toric variety V (σ ) ⊂ P X , and let fσ ∈ S(V (σ ))β σ denote the polynomial defining the hypersurface Y ∩V (σ ) in V (σ ). Then, as in Definition 4.2, we have the ideal J1 (fσ ) in S(V (σ )) and the quotient ring R1 (fσ ) = S(V (σ ))/J1 (fσ ). By Theorem 4.4(ii), we have an isomorphism     H 2−a, a−1 Y ∩ V (σ ) ∼ = R1 fσ aβ σ −β σ , where β0σ = deg( section.



γ

0

yγ ) ∈ A1 (V (σ )). We can now state our first main result of this

Theorem 5.2. Let X ⊂ P , dim P = 4, be a regular semiample hypersurface defined by f ∈ Sβ . Then there is a natural isomorphism   n(σ )      H 3−a, a (X) ∼ , R 1 fσ σ σ = R1 (f )(a+1)β−β 0

σ ∈ X (2)

aβ −β0

where n(σ ) is the number of cones ρi such that ρi ⊂ σ and ρi ∈ / X (1). Remark 5.3. As we mentioned in the introduction and Remark 2.5, in the Batyrev

of an ample Calabi-Yau mirror construction (see [B2]), an MPCP-desingularization Z hypersurface of a toric Fano variety P , corresponding to a reflexive polytope , is a regular semiample hypersurface. In [B2, Corollary 4.5.1], Batyrev calculated the Hodge number  

= l() − 5 − h2, 1 (Z) l ∗ (θ) + l ∗ (θ)l ∗ (θ ∗ ), codimθ=1

θ∗

codimθ=2

where θ is a face of , is the corresponding dual face of the dual reflexive ∗ polyhedron  , and l( ) (resp., l ∗ ( )) denotes the number of integer (resp., interior integer) points in . We can compare this number with the algebraic description

in Theorem 5.2. From Theorem 4.4, we know that dim R1 (f )2β−β0 = of H 2, 1 (Z)

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

107




∩ T), which is equal to l() − 5 − codimθ=1 l ∗ (θ) by [B2, Theorem 4.3.1]. h2, 1 (Z The number l ∗ (θ ∗ ) is equal to n(σ ) of Theorem 5.2 for the cone σ , corresponding to the face θ of . And finally, one can verify that dim R1 (fσ )β σ −β0σ corresponds to

is related to l ∗ (θ ). We can now see how the formula for the Hodge number h2, 1 (Z) our algebraic description. The next thing we want to do is to compute the cup product on H 3 (X) in terms of the algebraic description in Theorem 5.2. To compute this cup product, we need one topological result. Lemma 5.4. Let K and L be subvarieties of a compact V-manifold M, which intersect quasitransversally, and suppose that K, L, and K ∩ L are compact Vmanifolds. Then the diagram H • (K) i∗

 H • (K ∩ L)

i!

/ H • (M) j∗

α·j  !

 / H • (L)

commutes, where i, j, i  , and j  are inclusions, and the constant α satisfies [K]∪[L] = α[K ∩ L] for fundamental cohomology classes of K, L, and K ∩ L in M. Proof. The arguments are the same as in [Do, VIII, proof of Proposition 10.9]. The constant α in the above diagram is caused by the difference between [K] ∪ [L] and [K ∩ L]. (In the smooth case we do not see this difference.) Example 5.5. A simple nontrivial example of Lemma 5.4 occurs when M is a 2-dimensional toric variety and K and L are irreducible torus-invariant divisors, intersecting in a point. In this case, we have to compare the composition of maps i!

i∗

j∗

j !

H 0 (K) −→ H 2 (M) −→ H 2 (L) with H 0 (K) −→ H 0 (K ∩ L) −→ H 2 (L). Since i∗

j!

∼ = H 0 (K) and H 2 (L) ∼ = H 4 (M) are isomorphisms, it suffices to compare ∗ ∗ j! j i! i = [K] ∪ [L] ∪ _ with j! j  ! i  ∗ i ∗ = [K ∩ L] ∪ _ on H 0 (M). The difference between [K] ∪ [L] and [K ∩ L] can be easily determined by means of the ring isomorphism A• (M)⊗C ∼ = H 2• (M) (see [D, Section 10]), which sends a cycle class of a subvariety V to its fundamental cohomology class [V ] in M. H 0 (M)

Equation (11) provides a description of the middle cohomology group H 3 (X). We first show where the cup product on H 3 (X) vanishes. Lemma 5.6. Let ϕi ! H 1 (X ∩ Di ) ∪ ϕj ! H 1 (X ∩ Dj ) = 0, i  = j , unless ρi and ρj span a cone σ  ∈ contained in a 2-dimensional cone of X . Proof. By the projection formula for Gysin homomorphisms, we know that   ϕi ! _ ∪ ϕj ! _ = ϕj ! ϕj∗ ϕi ! _ ∪ _ .

108

ANVAR R. MAVLYUTOV

By Lemma 5.4, for i  = j we have a commutative diagram H 1 (X ∩ Di )

ϕi !

∗ ϕij

 H 1 (X ∩ Di ∩ Dj )

/ H 3 (X) ϕj∗

αij ϕj i !

 / H 3 (X ∩ Dj ),

(13)

where ϕij : X ∩ Di ∩ Dj @→ X ∩ Di is the inclusion map and αij is an appropriate constant. Hence, it suffices to show that H 1 (X ∩ Di ∩ Dj ) = 0. This is so, if ρi and ρj do not span a cone in , because Di ∩Dj is an empty set in this case. If ρi and ρj span a cone σ  ∈ , then Di ∩ Dj = V (σ  ). Applying (6) to the regular hypersurface X ∩ V (σ  ) in V (σ  ), we see      1 H1 X∩V σ ∼ = Gr W 1 H X ∩ Tσ  . On the other hand, if σ  is not contained in a 2-dimensional cone of X , then X ∩Tσ  1 is empty or a disjoint finite union of C∗ , by equation (2). In this case, Gr W 1 H (X ∩ Tσ  ) = 0, and the result follows. From the above result and Lemma 5.1, we can see that the cup product of two different spaces ϕi ! H 1 (X ∩ Di ) and ϕj ! H 1 (X ∩ Dj ) vanishes unless we assume that ρi \ {0} and ρj \ {0} lie in the relative interior of a 2-dimensional cone σ ∈ X and that ρi and ρj span a cone σ  ∈ : ✏ ✏✏ ✏✏ ✏ ✏ ✥✥ ✥✥✥ ✏✏ ✥ ✏ ρi ✥ ❤ ❤ P ❍P ❤❤❤σ  ❍❤ P ❤ ❤ ρ ❍P ❍PPP j ❍❍ σ ❍ ❍❍ .

In this case, by Lemma 5.1(ii) we have natural isomorphisms       ϕ i ! H 1 X ∩ Di ∼ = H 1 Y ∩ V (σ ) ∼ = ϕ j ! H 1 X ∩ Dj , which provide a natural way to compute the cup product on different spaces in the following lemma. Lemma 5.7. If ρi  = ρj , not belonging to X , span a cone σ  ∈ contained in a cone σ ∈ X (2), then   ϕi ! πi∗ l1 ∪ ϕj ! πj∗ l2 = mult σ  ϕσ  ! πσ∗ (l1 ∪ l2 ) for l1 , l2 ∈ H 1 (Y ∩ V (σ )), where πσ  : X ∩ V (σ  ) → Y ∩ V (σ ) is the projection and ϕσ  : X ∩ V (σ  ) @→ X is the inclusion.

109

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

Proof. Suppose that ρi and ρj span a cone σ  ∈ contained in σ ∈ X (2). Then, using (13) and the projection formula, for l1 , l2 ∈ H 1 (Y ∩ V (σ )) we compute     ϕi ! πi∗ l1 ∪ ϕj ! πj∗ l2 = ϕj ! ϕj∗ ϕi ! πi∗ l1 ∪ πj∗ l2 = ϕj ! αij ϕj i ! ϕij∗ πi∗ l1 ∪ πj∗ l2     = ϕj ! ϕj i ! αij ϕij∗ πi∗ l1 ∪ ϕj∗i πj∗ l2 = αij ϕj ! ϕj i ! ϕij∗ πi∗ l1 ∪ l2   = ϕj ! ϕj∗ ϕi ! πi∗ l1 ∪ l2 . We want to compare the map   ϕj ! ϕj∗ ϕi ! πi∗ : H 2 Y ∩ V (σ ) −→ H 6 (X) with the map

  ϕσ  ! πσ∗ : H 2 Y ∩ V (σ ) −→ H 6 (X).

These are the linear maps between 1-dimensional spaces, so they differ by a multiple of a constant. We determine this constant using the two commutative diagrams:   H 2 P X    H 2 V (σ )    H 2 Y ∩ V (σ )

π∗

/ H 2 (P ) ϕi∗

 / H 2 (D ) i

πi∗

πi∗

 / H 2 (X ∩ D ) i

ϕi !

/ H 4 (P )

ϕi !

 / H 4 (X)

ϕj∗

ϕj∗

ϕj !

/ H 4 (Dj )  / H 4 (X ∩ Dj )

/ H 6 (P ) i∗

ϕj !

 / H 6 (X) i!

 H 8 (P ),

  H 2 P X

π∗

   H 2 V (σ )

πσ∗

    / H2 V σ

   H 2 Y ∩ V (σ )

πσ∗

    / H2 X∩V σ

/ H 2 (P ) ϕσ∗ 

ϕσ  !

/ H 6 (P ) i∗

ϕσ  !

 / H 6 (X)

i!

/ H 8 (P ),

where the vertical maps are induced by the inclusions. By Lemma 5.4, we had to have some multiplicities in the above diagrams. These multiplicities are all 1 because for any γ ∈ we have X · V (γ ) = X ∩ V (γ ) in A• (P ). Indeed, consider a resolution p : P  → P , corresponding to a nonsingular subdivision  of . Then, by the

110

ANVAR R. MAVLYUTOV

proof of Lemma 2.3, p −1 (X) ⊂ P  is a regular semiample hypersurface. By the projection formula for cycles, for γ  ∈  (dim γ ), contained in γ , we have       X · V (γ ) = p∗ p ∗ (X) · V γ  = p∗ p −1 (X) · V γ     = p p −1 (X) ∩ V γ  = X ∩ V (γ ). We know a nonzero class [Y ] ∈ H 2 (P X ), the fundamental cohomology class of Y in P X . Mapping this class to H 8 (P ) in the above two diagrams, we get [X] ∪ [Di ] ∪ [Dj ] ∪ [X] and [X] ∪ [V (σ  )] ∪ [X], respectively. Using the ring isomorphism A• (P ) ⊗ C ∼ = H 2• (P ), from Lemma 1.4 we find that [X] ∪ [V (σ  )] ∪ [X] does not vanish, and, since Di · Dj = (1/mult(σ  ))V (σ  ) (see [F1, Section 5.1]), it follows that ϕj ! ϕj∗ ϕi ! πi∗ = mult(σ  )ϕσ  ! πσ∗ on H 2 (Y ∩ V (σ )). We have computed the cup product of any two different spaces in (11). Now we compute the cup product on ϕi ! H 1 (X ∩Di ), which does not vanish when ρi \{0} lies in the relative interior of a 2-dimensional cone σ ∈ X . In this case, there are exactly two cones in , contained in σ and containing ρi : ✏ ✏✏ ✏✏ ✏ ✏ ✥✥ ✥✥✥ σ  ρ ✏✏ ✥ ✏ ✥ i ❤ ❤ P ❍P ❤❤❤σ  ❍❤ P ❤ ❤ ❍P ❍PPP ❍❍ σ ❍❍ ❍.

In terms of this, we have the following lemma. / X be in some σ ∈ X (2), and let σ  , σ  ∈ (2) be the two Lemma 5.8. Let ρi ∈ cones containing ρi and contained in σ . Then     mult σ  + σ  ∗ ∗   ϕσ  ! πσ∗ l1 ∪ l2   ϕi ! πi l1 ∪ ϕi ! πi l2 = −   mult σ mult σ for l1 , l2 ∈ H 1 (Y ∩ V (σ )). Proof. By the projection formula, we have   ϕi ! πi∗ _ ∪ ϕi ! πi∗ _ = ϕi ! ϕi∗ ϕi ! πi∗ _ ∪ πi∗ _ = ϕi ! ϕi∗ ϕi ! πi∗ ( _ ∪ _ ). As in the proof of Lemma 5.7, we compare the maps ϕi ! ϕi∗ ϕi ! πi∗ and ϕσ  ! πσ∗ . Also, using the arguments of Lemma 5.7 we get  2      X · V σ ϕi ! ϕi∗ ϕi ! πi∗ = X 2 · Di2 ϕσ  ! πσ∗ (14) number (X 2 ·Di2 ). Take on H 2 (Y ∩V (σ )). All we need is to compute the intersection
n any m ∈ M, such that m, ei   = 0. The Weil divisor j =1 m, ej Dj is equivalent to

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

zero, whence  2 2 X · Di =

111

    1 X 2 · Di · −m, ej Dj . m, ei  j =i

However, Di ·Dj = (1/mult(γ ))V (γ ), if ρi and ρj span a cone γ ∈ , or Di ·Dj = 0 otherwise. On the other hand, by Lemma 1.4, (X 2 · V (γ )) = 0 unless γ is contained in σ . There are exactly two such cones σ  and σ  , contained in σ and containing ρi . Suppose that e and e are the primitive generators of the cones σ  and σ  , not lying in ρi . Then 

 X2 · Di2 = −

      m, e  m, e    X2 · V σ  −   X 2 · V σ  .   m, ei  mult σ m, ei  mult σ

Since σ  ⊥ = σ  ⊥ , equation (1) shows that (X 2 ·V (σ  )) = (X 2 ·V (σ  )). Also, from [D, Section 8.2] it follows that mult(σ  + σ  )ei = mult(σ  )e + mult(σ  )e . Therefore,       2 2 mult σ  + σ      X2 · V σ  , X · Di = − mult σ  mult σ  and the result follows from equation (14). We have finished the calculation of the cup product on H 3 (X). To state a theorem in a nice form we need to define a couple of maps. The map η : R1 (f ) → C is defined as P λ on R1 (f )5β−2β0 (different by a multiple from the map in (10)), and zero in all other degrees. Similarly, replacing P with V (σ ) and f with fσ , we have the map ησ : R1 (fσ ) → C equal to zero in all degrees except for 3β σ − 2β0σ . Recall also that Theorem 5.2 gives the isomorphism   n(σ )      H 3−a, a (X) ∼ R 1 fσ σ σ . = R1 (f )(a+1)β−β aβ −β0

0

σ ∈ X (2)

The following is the description of the cup product on the middle cohomology of the hypersurface. Theorem 5.9. Let X ⊂ P , dim P = 4, be a regular semiample hypersurface defined by f ∈ Sβ . If A ∈ R1 (f )(a+1)β−β0 and B ∈ R1 (f )(b+1)β−β0 are identified with elements of Gr F H 3 (X) by means of the isomorphism in Theorem 5.2, then  (a(a+1)/2)+(b(b+1)/2)+a 2 +3 /a!b!. If we write X A∪B = cab η(A·B), where cab = (−1) n(σ )   i = Lσ, R1 (f )aβ σ −β0σ a , σ ⊃ρi ∈ / X

i where Lσ, a = R1 (fσ )aβ σ −β0σ correspond to the cones ρi lying in a 2-dimensional σ, j

i  σ, i cone σ ∈ , then for li ∈ Lσ, a , li ∈ Lb , and lj ∈ Lb

(identified with elements of

112

ANVAR R. MAVLYUTOV

Gr F H 3 (X)) we have

 X

    li ∪ lj = mult σ  (−1)a−1 ησ li · lj

in the case when ρi and ρj span a cone σ  ∈ (2), and      mult σ  + σ       (−1)a−1 ησ li · li , li ∪ l i = −   mult σ mult σ X where σ  , σ  ∈ (2) are the two cones contained in σ and containing ρi . The cup product vanishes in all other cases.

of an ample Calabi-Yau hyRemark 5.10. If X is an MPCP-desingularization Z  persurface as in [B2], then the multiplicity mult(σ ) is 1 for all 2-dimensional cones σ  , by the properties of a reflexive polytope. Also, in this case mult(σ  + σ  ) = 2 in Theorem 5.9. 6. Hodge numbers and a “counterexample” in mirror symmetry. In this section, we discuss what kind of mirror symmetry has to be studied. Mirror symmetry proposes that if two smooth m-dimensional Calabi-Yau varieties V and V ∗ form a mirror pair, then their Hodge numbers must satisfy the relations hp, q (V ) = hm−p, q (V ∗ ),

0 ≤ p, q ≤ m.

(15)

A construction in [B2], associated with a pair of reflexive polytopes, satisfies the above equalities for q = 0, 1 (see [BDa]), even if V and V ∗ are compact orbifolds (i.e., V -manifolds). We compute the Hodge numbers hp, 2 of a regular semiample hypersurface in a complete simplicial toric variety P . Then we apply our formula to the Batyrev mirror construction [B2], and we check that there is no symmetry for the

of ample Calabi-Yau hypersurfaces Z Hodge numbers of MPCP-desingularizations Z coming from a pair of reflexive polytopes  and ∗ . However, [BBo, Theorem 4.15]

are smooth, and [BDa, Theorem 6.9] show that if these MPCP-desingularizations Z then the duality (15) holds. On the other hand, [BBo, Theorem 4.15] shows that (15) p, q holds for the string-theoretic Hodge numbers hst of the singular ample Calabi-Yau hypersurfaces Z. This confirms the idea that mirror symmetry has to be studied for smooth varieties with usual Hodge numbers or for singular varieties with stringtheoretic Hodge numbers. In order to compute the Hodge numbers, we use the ep, q numbers introduced in [DKh]:    (−1)k hp, q Hck (V ) , ep, q (V ) = k

defined for arbitrary algebraic variety V . These numbers satisfy the property ep, q(V )= (−1)p+q hp, q (V ) if V is a compact orbifold.

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

113

Let X ⊂ P , dim P = d, be a -regular semiample hypersurface with the associated map π : P → P X , Y = π(X), as in Proposition 2.4. Using the properties of ep, q numbers (see [DKh]) and equation (2), for p + q > d − 1, p  = q, we compute hp, q (X) = (−1)p+q ep, q (X) = (−1)p+q = (−1)



p+q

e

p−i, q−i



ep, q (X ∩ Tσ )

σ ∈



   Y ∩ Tγ · ei, i (C∗ )dim γ −dim σ ,

γ ∈ X γ ⊃σ ∈

where the sum is by all σ ∈ such that γ ∈ X is the smallest cone containing σ . Hence, we get h

p, q

(X) = (−1)



p+q

ak (γ )(−1)

dim(γ )−k+i

γ ∈ X 0


 dim(γ ) − k p−i, q−i  Y ∩ Tγ , e i

of cones σ ∈ (k) such that γ ∈ X is the smallest where ak (γ ) denotes the number  cone containing σ , and si is the usual binomial coefficient. It follows from a formula in [DKh, Section 3.11] that in the last sum ep−i, q−i (Y ∩ Tγ ) = 0 unless (p − i) + (q − i) ≤ dim(Y ∩ Tγ ) (equivalently, dim γ ≤ d − 1 − p − q + 2i). We now assume q = d − 3. Then, for p > 2, p  = d − 3, we have hp, d−3 (X) = (−1)



p+d−3

ak (γ )(−1)

l−k+i



γ ∈ X (l) 0


= (−1)p+d−3

 l − k p−i, d−3−i  Y ∩ Tγ e i

ak (γ )(−1)l−k+i

γ ∈ X (l) 0
 = (−1)p+d−3



 l − k d−3−i, p−i  e Y ∩ Tγ i

   a1 (γ )ed−2−p, 1 Y ∩ Tγ +

γ ∈ X (p)

+



 γ ∈ X (p+2) 1≤k≤2

ak (γ )(−1)

k

  a1 (γ )ed−3−p, 0 Y ∩ Tγ

γ ∈ X (p+1)



 p + 2 − k d−3−p, 0  Y ∩ Tγ . e p




Let X be linearly equivalent to a torus-invariant divisor ni=1 bi Di , which gives a polytope . By Remark 1.3, a cone γ ∈ X corresponds to a face γ of . Applying Corollary 5.9 and Proposition 5.8 in [DKh], we get

114

ANVAR R. MAVLYUTOV





 ∗   ed−2−p, 1 Y ∩ Tγ = (−1)d−p−1  l 2

γ



− (d − p + 1)l ∗





γ



−

⊂ γ codim =1

 l ∗ ( ) 

if dim γ = p. (Here, l ∗ ( ) is the number of interior integral points in .) Furthermore,    l ∗ ( ) if dim γ = p + 1, ed−3−p, 0 Y ∩ Tγ = (−1)d−p−2  ed−3−p, 0 Y

⊂ γ codim =1    ∩ Tγ = (−1)d−p−3 l ∗ γ

if dim γ = p + 2.

Substituting these numbers in the above formula, we obtain hp, d−3 (X) 

=

γ ∈ X (p)





 ∗ a1 (γ )  l 2

γ



− (d − p + 1)l ∗

 −



γ



⊂ γ codim =1



−

⊂ γ codim =1



  a1 (γ )  

γ ∈ X (p+1)



 l ∗ ( ) +



 l ∗ ( ) 

  a2 (γ ) − a1 (γ )(p + 1) l ∗



 γ ∈ X (p)

 ∗ a1 (γ )  l 2

γ



− (d − p + 1)l ∗



γ



−

 +

 γ ∈ X (p+2)

l∗





.

γ ∈ X (p+2)

Simplifying and using Poincaré duality, we find for p > 2, p  = d − 3,  hd−1−p, 2 (X) =

γ

γ

 a2 (γ ) − (p + 1)a1 (γ ) −



 ⊂ γ codim =1

 τ ⊂γ codimτ =1

 l ∗ ( )  

 a1 (τ ) .

We now can apply this formula to the mirror construction in [B2]. We recall that an MPCP-desingularization of a toric variety P , associated with a reflexive polytope , is a complete simplicial toric variety, corresponding to a refinement of the normal fan of  such that the cone generators in the fan are exactly N ∩ ∗ − {0}. (Here,

 is an MPCP-desingularization ∗ is the dual reflexive polytope.) Notice that if Z of a regular ample hypersurface Z  ⊂ P , then the toric variety P coincides with P Zˆ . We also note that in the above formula the number a1 (γ ) for γ ∈ Zˆ  is equal  to l ∗ ( γ∗ ), where γ∗ is the dual face of ∗ .

SEMIAMPLE HYPERSURFACES IN TORIC VARIETIES

115

The example we use comes from the reflexive polytope  of dimension 7 in MR = R7 , given by the equations zi ≥ −1,

i = 1, . . . , 7,

−2z1 − 2z2 − 2z3 − 2z4 − 3z5 − 3z6 − 3z7 ≥ −1.

The dual reflexive polytope ∗ has vertices at n0 = (−2, −2, −2, −2, −3, −3, −3),

n 1 , . . . , n7 ,

where n1 , . . . , n7 are the standard basis of the lattice N = Z7 . Notice that the toric variety P , corresponding to the polytope , is the weighted projective space P(1, 2, 2, 2, 2, 3, 3, 3). The only integral points in ∗ are the vertices and the origin, implying that

 ) = 0 because no subdivision occurs for the normal fan of ; consequently, h3, 2 (Z 3, 2 in the above formula for h all the numbers a1 and a2 vanish. On the other hand,

∗ of a regular ample hypersurface in P∗ , the for a dual MPCP-desingularization Z d−1−p, 2 above formula for h (X) with d = 7 and p = 3 simplifies to   

∗ = l ∗ ( ) · l ∗ (2 ∗ ), h3, 2 Z ∗ ⊂∗ dim( ∗ )=4

∗ ) is because l ∗ ( ∗ ) = 0 for all faces ∗ of ∗ . We want to show that h3, 2 (Z

positive, which would imply that the duality (15) fails for the pair (Z , Z∗ ). Indeed, consider the 4-dimensional face ∗ of ∗ with vertices at n0 , n1 , n2 , n3 , and n4 . Then (−1, −1, −1, −1, −2, −2, −2) is the interior integral point of 2 ∗ . The dual 2-dimensional face , which has vertices at (−1, −1, −1, −1, 5, −1, −1), (−1, −1, −1, −1, −1, 5, −1), and (−1, −1, −1, −1, −1, −1, 5), contains the integral point (−1, −1, −1, −1, 1, 1, 1) in its relative interior. Thus, we have shown that

 )  = h3, 2 (Z

∗ ). This happened because the hypersurface Z

∗ is singular. h3, 2 (Z Other “counterexamples” can be easily found in higher dimensions, showing that, in general, the duality (15) fails for the MPCP-desingularizations of the construction in [B2]. We should also point out that there is such a “counterexample” for 4-folds (see [B3, Example 1.2]). References [B1] [B2] [B3]

[BBo] [BC]

V. V. Batyrev, Variations of the mixed Hodge structure of affine hypersurfaces in algebraic tori, Duke Math. J. 69 (1993), 349–409. , Dual polyhedra and mirror symmetry for Calabi-Yau hypersurfaces in toric varieties, J. Algebraic Geom. 3 (1994), 493–535. , “Stringy Hodge numbers of varieties with Gorenstein canonical singularities” in Integrable Systems and Algebraic Geometry (Kobe/Kyoto, 1997), World Scientific, River Edge, N.J., 1998, 1–32. V. V. Batyrev and L. A. Borisov, Mirror duality and string-theoretic Hodge numbers, Invent. Math. 126 (1996), 183–203. V. V. Batyrev and D. A. Cox, On the Hodge structure of projective hypersurfaces in toric varieties, Duke Math. J. 75 (1994), 293–338.

116 [BDa] [CarG]

[CaCDi] [C1] [C2] [C3]

[CKa] [CLO] [D] [DKh] [De] [Do] [EV] [F1] [F2] [FLa]

[H] [K] [Od] [PS]

[R]

ANVAR R. MAVLYUTOV V. V. Batyrev and D. Dais, Strong McKay correspondence, string-theoretic Hodge numbers and mirror symmetry, Topology 35 (1996), 901–929. J. Carlson and P. Griffiths, “Infinitesimal variations of Hodge structure and the global Torelli problem” in Journées de géométrie algébrique d’Angers, juillet 1979, Sijthoff and Noordhoff, Alphen aan den Rijn, 1980, 51–76. E. Cattani, D. Cox, and A. Dickenstein, Residues in toric varieties, Compositio Math. 108 (1997), 35–76. D. Cox, The homogeneous coordinate ring of a toric variety, J. Algebraic Geom. 4 (1995), 17–50. , Toric residues, Ark. Mat. 34 (1996), 73–96. , “Recent developments in toric geometry” in Algebraic Geometry (Santa Cruz, 1995), Proc. Sympos. Pure Math. 62, Part 2, Amer. Math. Soc., Providence, 1997, 389–436. D. Cox and S. Katz, Mirror Symmetry and Algebraic Geometry, Math. Surveys Monogr. 68, Amer. Math. Soc., Providence, 1999. D. Cox, J. Little, and D. O’Shea, Ideals, Varieties, and Algorithms, Undergrad. Texts Math., Springer-Verlag, New York, 1992. V. Danilov, The geometry of toric varieties, Russian Math. Surveys 33, no. 2 (1978), 97–154. V. Danilov and A. Khovanskii, Newton polyhedra and an algorithm for computing Hodge-Deligne numbers, Math. USSR-Izv. 29 (1987), 279–298. P. Deligne, Théorie de Hodge, II, Inst. Hautes Études Sci. Publ. Math. 40 (1971), 5–57; III, 44 (1974), 5–77. A. Dold, Lectures on Algebraic Topology (in German), Grundlehren Math. Wiss. 200, Springer-Verlag, New York, 1972. H. Esnault and E. Viehweg, Lectures on Vanishing Theorems, DMV Sem. 20, Birkhäuser, Basel, 1992. W. Fulton, Introduction to Toric Varieties, Ann. of Math. Stud. 131, Princeton Univ. Press, Princeton, 1993. , Intersection Theory, 2d ed., Ergeb. Math. Grenzgeb. (3) 2, Springer-Verlag, Berlin, 1998. W. Fulton and R. Lazarsfeld, “Connectivity and its applications in algebraic geometry” in Algebraic Geometry (Chicago, Ill., 1980), Lecture Notes in Math. 862, SpringerVerlag, Berlin, 1981, 26–92. R. Hartshorne, Ample Subvarieties of Algebraic Varieties, Lecture Notes in Math. 156, Springer-Verlag, Berlin, 1970. S. L. Kleiman, A note on the Nakai-Moisezon test for ampleness of a divisor, Amer. J. Math. 87 (1965), 221–226. T. Oda, Convex Bodies and Algebraic Geometry: An Introduction to the Theory of Toric Varieties, Ergeb. Math. Grenzgeb. (3) 15, Springer-Verlag, Berlin, 1988. C. Peters and J. Steenbrink, “Infinitesimal variations of Hodge structure and the generic Torelli problem for projective hypersurfaces (after Carlson, Donagi, Green, Griffiths, Harris)” in Classification of Algebraic and Analytic Manifolds (Katata, 1982), Progr. Math. 39, Birkhäuser, Boston, 1983, 399–463. M. Reid, “Decomposition of toric morphisms” in Arithmetic and Geometry, Vol. 2, Progr. Math. 36, Birkhäuser, Boston, 1983, 395–418.

Department of Mathematics and Statistics, University of Massachusetts, Amherst, Massachusetts 01003, USA; [email protected]

Vol. 101, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

DEFINING INTEGRALITY AT PRIME SETS OF HIGH DENSITY IN NUMBER FIELDS ALEXANDRA SHLAPENTOKH

1. Introduction. Interest in the questions of Diophantine definability and decidability goes back to a question that was posed by Hilbert: Given an arbitrary polynomial equation in several variables over Z, is there a uniform algorithm to determine whether such an equation has solutions in Z? This question, otherwise known as Hilbert’s tenth problem, has been answered negatively in the work of M. Davis, H. Putnam, J. Robinson, and Yu. Matijasevich (see [2] and [3]). Since the time when this result was obtained, similar questions have been raised for other fields and rings. Arguably the two most interesting and difficult problems in the area are the questions of Diophantine decidability of Q and the rings of algebraic integers of arbitrary number fields. One way to resolve the question of Diophantine decidability negatively over a ring of characteristic zero is to construct a Diophantine definition of Z over such a ring. This notion is defined below. Definition 1.1. Let R be a ring and let A ⊂ R. Then we say that A has a Diophantine definition over R if there exists a polynomial f (t, x1 , . . . , xn ) ∈ R[t, x1 , . . . , xn ] such that for any t ∈ R, ∃ x1 , . . . , xn ∈ R,

f (t, x1 , . . . , xn ) = 0 ⇐⇒ t ∈ A.

If the quotient field of R is not algebraically closed, it can be shown that we can allow the Diophantine definition to consist of several polynomials without changing the nature of the relationship (for more details see [3]). Such Diophantine definitions have been obtained for Z over the rings of algebraic integers of the following fields: totally real extensions of Q, their extensions of degree 2, fields with exactly one pair of complex conjugate embeddings, and some fields of degree 4. For more details concerning these results see [4], [5], [6], [13], [15], and [16]. However, not much progress has been made towards resolving the Diophantine problem of Q. Furthermore, one of the consequences of a series of conjectures by Mazur and Colliot-Thélène, SwinnertonDyer, and Skorobogatov is that Z does not have a Diophantine definition over Q, and thus one would have to look to some other method for resolving the Diophantine problem of Q. (Mazur’s conjectures can be found in [10] and [11]. However, Colliot-Thélène, Swinnerton-Dyer, and Skorobogatov gave a counterexample to the strongest of the conjectures in the papers cited above. Their modification of Mazur’s Received 15 September 1998. Revision received 12 March 1999. 1991 Mathematics Subject Classification. Primary 11U05, 11R80; Secondary 11R04, 03D25. Author’s work partially supported by National Security Agency grant number MDA904-98-1-0510. 117

118

ALEXANDRA SHLAPENTOKH

conjecture in view of the counterexample can be found in [1].) Given the difficulty of the Diophantine problem for Q (and number fields in general), one might adopt a gradual approach, that is, one might consider the Diophantine problem of a recursive ring of W -integers, which is defined below for any product formula field. Definition 1.2. Let M be a product formula field (i.e., an algebraic function field in one variable or a number field) and let W be a set of its non-Archimedean primes. Then a ring
 OM,W = x ∈ M | ordp x ≥ 0 ∀p ∈ W is called a holomorphy ring of M if M is a function field and a ring of W -integers in the case of number fields. (The term W -integers usually presupposes that W is finite, but we also use this term for infinite W .) Furthermore, we might wonder if rational (algebraic) integers are existentially definable in any such subring of a number field, where W is infinite. (This result is known for finite W ; see Theorem 1.3 below.) In [18], we established the following result. Theorem (Theorem 4.5 in [18]). Let M be a totally real nontrivial cyclic extension of Q. Then there exists an infinite recursively definable set W of finite primes of M such that Z and the ring of algebraic integers of M have a Diophantine definition over OM,W . (Thus, the Diophantine problem of OM,W is undecidable.) In this paper we improve this result in two ways. First of all, we show that this result is true for all nontrivial totally real extensions of Q. Second, we give some estimates on the density of the prime sets involved. In particular, we prove the following theorem. Theorem 4.6. If M is a totally real number field with a cyclic subextension F of degree m, then for any ε > 0, there exists a set of primes WM of M of Dirichlet density greater than ((m−1)/m)−ε such that OM has a Diophantine definition over OM,WM . Furthermore, if WF is the set of F -primes below WM , then the Dirichlet density of F can be arranged to be greater than 1 − φ(m)/m − ε. As a corollary of this theorem, we obtain the following result. Corollary 4.7. For every δ > 0, there exist a number field M and a set of Mprimes W such that the following statements are true. • OM is definable over OM,W . • The density of W is greater than 1 − δ. • The density of the set of rational primes below W is greater than 1 − δ. Before we proceed with the details of the proof, we need to discuss some of the known facts concerning Diophantine definability over number fields. First of all, we have the following theorem, whose proof can be found in [17].

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

119

Theorem 1.3. Let M be a number field. Let Q be a non-Archimedean prime of M. Then the set of elements of M integral at Q is Diophantine over M. Second, we have the following proposition, whose proof can also be found in [17]. Proposition 1.4. Let M be a number field. Let WM be a set of non-Archimedean primes of M. Then the set of nonzero elements of OM,WM is Diophantine over OM,WM . This proposition implies the following corollary. Corollary 1.5. Let M be a number field and let A ⊂ M have a Diophantine definition over M. Let WM be any set of non-Archimedean primes of M. Then A ∩ OM,WM has a Diophantine definition over OM,WM . In particular, we can conclude that for any prime p ∈ WM , the set of elements of OM,WM integral at p has a Diophantine definition over OM,WM . Finally, we need to state a few simple “rewriting” results, that is, propositions dealing with replacing polynomial equations over bigger fields with equivalent polynomial equations over smaller fields. Proposition 1.6. Let M/G be a finite extension of number fields. Let OG,WG be a ring of WG -integers of G. Let P (x1 , . . . , xs ) ∈ M[x1 , . . . , xs ]. Then there exists Q(x1 , . . . , xs ) ∈ G[x1 , . . . , xs ] such that for all (a1 , . . . , as ) ∈ (OG,WG )s , P (a1 , . . . , as ) = 0 ⇐⇒ Q(a1 , . . . , as ) = 0. Proposition 1.7. Let M/G be a finite extension of number fields of degree l. Let OG,WG be a ring of WG -integers of G and let OM,WM be the integral closure of OG,WG in M. Let P (t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ) ∈ M[t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ]. Then there exists Q(t1 , . . . , tr , u1 , . . . , uw , x1,1 , . . . , xs,l ) ∈ G[t1 , . . . , tr , u1 , . . . , uw , x1,1 , . . . , xs,l ] such that ∀t1 , . . . , tr ∈ OG,WG , ∃ u1 , . . . , uw ∈ OG,WG , ∃ X1 , . . . , Xs ∈ OM,WM , P (t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs)= 0 ⇐⇒ ∃ u1 , . . . , uw , x1,1 , . . . , xs,l ∈ OG,WG , Q(t1 , . . . , tr , u1 , . . . , uw , x1,1 , . . . , xs,l ) = 0. Proof. Let  = {ω1 , . . . , ωl } ⊂ OM be a basis of M/G. Let D = DM/G be the discriminant of the basis. Then it is well known that for all X ∈ OM,WM , X =  (xi /D)ωi , where xi ∈ OG,WG . Let A1 (x1 /D, . . . , xl /D) = T rM/G (X), . . . , Al (x1 /D, . . . , xl /D) = NM/G (X) be all the coefficients the monic irreducible

120

ALEXANDRA SHLAPENTOKH

polynomial of X over G. Note that all these coefficients are fixed polynomials in x1 , . . . , xl whose  coefficients depend on the choice of  only. Then for any x1 , . . . , xl ∈ OG,WG , (xi /D)ωi ∈ OM,WM if and only if x x xl  xl  1 1 A1 ,..., ∈ OG,WG , . . . , Al ,..., ∈ OG,WG . D D D D Then ∀t1 , . . . , tr ∈ OG,WG , ∃ u1 , . . . , uw ∈OG,WG , ∃ X1 , . . . , Xs ∈OM,WM , P (t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ) = 0 ⇐⇒ ∃ u1 , . . . , uw , x1,1 , . . . , xs,l , z1,1 , . . . , zs,l ∈ OG,WG , 

l l    x1,i xs,i    P t 1 , . . . , t r , u1 , . . . , u w , ωi , . . . , ωi = 0,   D D   i=1 i=1 x   x1,l  1,1   , . . . , = A , z  1,1 1  D D       x···  x1,l  1,1 z1,l = Al ,..., , D D        x· · ·  xs,l  s,1   , . . . , , = A z  s,1 1   D D       x···   zs,l = Al s,1 , . . . , xs,l . D D By Proposition 1.6, we know that this system of equations can be replaced by an equivalent system of equations with coefficients in G. The system over G can, in turn, be replaced by a single equivalent polynomial equation over G. Corollary 1.8. Let M/G be a finite extension of number fields of degree l. Let OG,WG be a ring of WG -integers of G and let OM,WM be the integral closure of OG,WG in M. Let p1 , . . . , ps ∈ WM,WM . Let UM = WM \ {p1 , . . . , ps }. Let P (t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ) ∈ M[t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ]. Then there exists Q(t1 , . . . , tr , u1 , . . . , uw , x1,1 , . . . , xs,l ) ∈ G[t1 , . . . , tr , u1 , . . . , uw , x1,1 , . . . , xs,l ] such that ∀t1 , . . . , tr ∈ OG,WG , ∃ u1 , . . . , uw ∈ OG,WG , X1 , . . . , Xs ∈ OM,UM , P (t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ) = 0 ⇐⇒ ∃ u1 , . . . , uw , x1,1 , . . . , xs,l ∈OG,WG , Q(t1 , . . . , tr , u1 , . . . , uw , x1,1 , . . . , xs,l ) = 0. Note that OM,UM is not necessarily the integral closure in M of any ring of WG integers.

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

121

Proof. By Theorem 1.3 and by Corollary 1.5, there exists a polynomial S(X, Y1 , . . . , Yl ) ∈ M[X, Y1 , . . . , Yl ] such that for any X ∈ OM,WM , X ∈ OM,UM ⇐⇒ ∃ Y1 , . . . , Yl ∈ OM,WM ,

S(X, Y1 , . . . , Yl ) = 0.

Thus, ∀t1 , . . . , tr ∈ OG,WG , ∃ u1 , . . . , uw ∈OG,WG , ∃ X1 , . . . , Xs ∈OM,UM , P (t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ) = 0 if and only if ∃ u1 , . . . , uw ∈ OG,WG , ∃ X1 , . . . , Xs , Y1,1 , . . . , Ys,l ∈ OM,WM ,  P (t1 , . . . , tr , u1 , . . . , uw , X1 , . . . , Xs ) = 0,     S(X1 , Y1,1 , . . . , Y1,l ) = 0,  ...    S(Xs , Ys,1 , . . . , Ys,l ) = 0. Now the rest of the proof follows by Proposition 1.7. 2. Notation and assumptions. In this section, we describe the notation that is used for the remainder of the paper. We also prove some technical results concerning units in number fields. The properties of units described below are the foundations of our Diophantine definitions. Notation 2.1. • Let M/F denote a nontrivial finite extension of totally real fields of degree m. • Let M G denote the Galois closure of M over F . • Let [F : Q] = n. • Let PF denote a prime of F that does not split in M. Let PM denote the sole factor of PF in M. • Let L denote a totally complex extension of F of degree 2. • Let K denote a totally real cyclic extension of F of degree p > max(n+1, m+1). • The product of any two fields from the triple of fields (M G , K, L) is linearly disjoint from the third one over F and over M. • PF splits completely in the extension KL/F . • Let δ ∈ OKL denote a generator of MKL over M and a generator of KL over F . • Let φ ∈ OM denote a generator of M over F . • Let VM denote the set of primes of M dividing the discriminant of φ, and let D denote a natural number divisible by all the primes of VM . • Let δG denote an integral generator of M G K over M such that NM G K/M (δG ) ∼ =0 modulo PM . • Let VKM G denote the set of all the primes of M either dividing the discriminant of δG or one of the coefficients of the monic irreducible polynomial of δG over M.

122 •

• • • • •

ALEXANDRA SHLAPENTOKH

Let H G (T ) ∈ OM [T ] denote the monic irreducible polynomial of δG over M. K denote a set of primes in M not splitting in the extension MK/M and Let WM K does not contain any primes not contained in VKM G . Furthermore, assume WM with F -conjugates in VKM G . Let WFK denote the set of primes of F lying below K. the primes of WM Let k ∈ N denote a constant defined as in Lemma 2.4. Let hM , hF denote the class numbers of M and F , respectively. Let c ∈ N denote the constant defined in Lemma 3.1. Let aPF denote a fixed element of N, such that aPF ∼ = 0 modulo PF . Let l0 = 0, l1 , . . . , l[KM G :Q] denote distinct natural numbers equivalent to zero modulo PM .

Proposition 2.2. There exists α ∈ KL satisfying the following requirements. (1) NKL/K (α) = 1. (2) NKL/L (α) = 1. (3) The divisor of α is composed of the factors of PF only. (4) α is not a root of unity. Proof. Let UKL,PF be the multiplicative group of PF -units of KL, that is, the multiplicative group of elements of KL whose divisors consist of factors of PF in KL only. By generalized Dirichlet’s unit theorem (see [12, p. 77]), the rank of this group is equal to np +2p −1 = (n+2)p −1, where np is the number of independent Archimedean valuations on KL, and 2p is the number of non-Archimedean valuations on KL restricting to the PF -adic valuation on F . (We remind the reader that, by assumption, PF splits completely in KL.) Let UK,PF be the group of PF -units of K defined analogously to UKL,PF . The rank of that unit group is np + p − 1 = (n + 1)p − 1. Since the norm map from KL to K sends UKL,PF to UK,PF , we must conclude that the rank of the kernel is equal to p > n + 1. Thus there exists β ∈ KL whose divisor consists of factors of PF only such that NKL/K (β) = 1 and β is not a root of unity. Let β1 = β, . . . , βp be all the conjugates of β over L. If for all i = 2, . . . , p, β = ξi βi , where ξi is a root of unity, then some power of β is in UL,PF , where UL,PF is the multiplicative group of PF units in L. The rank of UL,PF is equal to n + 2 − 1 = n + 1 < p. Thus, there exists β ∈ ULK,PF such that NKL/K (β) = 1 and for some 2 ≤ i ≤ p, β/βi is not a root √ of unity. Further, for all i = 1, . . . , p, NKL/K (βi ) = 1. Indeed, assume L = F ( d), where d is a totally negative element √ of F . (In other words, all conjugates √ of d over Q are negative.) Then β = x − dy, where x, y ∈ K, and βi = xi − dyi , where xi and yi are conjugates of x and y, respectively, over L and over F . (This is so because K and L are linearly disjoint over F .) Furthermore, NKL/K (β) = 1 translates into x 2 − dy 2 = 1. The last equation implies xi2 − dyi2 = 1. Let α = β/βi . Then NKL/K (α) = (NKL/K (β))/(NKL/K (βi )) = 1/1 = 1. At the same time, NKL/L (α) = (NKL/L (β))/(NKL/L (βi )) = 1 because the numerator and the denominator are equal.

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

123

Lemma 2.3. Let ᏿M be the set of all the primes of M splitting completely in the extension MKL/M, and let ᏿MK , ᏿ML , and ᏿MKL be the sets of primes above ᏿M in MK, ML, and MKL, respectively. Let x ∈ MKL be such that NMKL/ML (x) = 1,

(2.1)

NMKL/MK (x) = 1.

(2.2)

Then x is an ᏿MKL -unit; that is, the divisor of x is composed of primes from ᏿MKL only. Proof. Let x be as described in the statement of the lemma. Let QMKL be a prime factor of the divisor of x. Then x must have a nonzero order at a conjugate of QMKL over MK and ML distinct from QMKL . Denote by QMK and QML the primes below QMKL in MK and ML, respectively. Since extensions MKL/MK and MKL/ML are cyclic of prime degrees, we must conclude that QMK and QML split completely in the respective extensions. Let QM be the prime below QMKL in M. Then the number of factors of QM is divisible by 2 and p. Hence this prime will split completely in the extension MKL/M. ns ⊂ ᏿ contain primes of ᏿ without other conjugates over Lemma 2.4. Let ᏿M M M ns ns . Assume further that all the primes F . Let ᏿F be the set of F -primes below ᏿M ns in ᏿Fns split completely in the extension KL/F . Let x ∈ MKL be an ᏿MKL unit, ns ns where ᏿MKL is the set of MKL primes above ᏿M primes, such that x satisfies (2.1) and (2.2). Then there exists a positive integer k depending on M, K, L only such that x k ∈ KL. ns , let PKL be the KL-prime below PMKL , and let PF Proof. Let PMKL ∈ ᏿MKL be the F -prime below PMKL . In MKL, PF has [MLK : M] = [LK : F ] factors. Therefore, PKL does not split in the extension MKL/KL. Let y = NMLK/LK (x). Then the divisors of y and x [MLK:LK] are equal as divisors in MKL. On the other hand, consider the following: 

NMLK/KM (y) = NLK/K (y) = NLK/K NMLK/LK (x) 

= NMLK/K (x) = NMK/K NMLK/MK (x) = 1.

Therefore,

 NMLK/KM

y x [MLK:LK]



= 1,

and y/x [MLK:LK] is an integral unit. But the only integral units of MLK with KMnorm 1 are roots of unity. Thus, for some positive natural number k depending on M, L, and K only, x k ∈ LK.  Corollary 2.5. Let x be as in Lemma 2.4, and let x k = [MLK:M]−1 ai δ i , a i ∈ i=0 M. Then ai ∈ F .

124

ALEXANDRA SHLAPENTOKH

3. Bounds. This section is devoted to establishing polynomial ways to bound the heights. Lemma 3.1. Let (α/β) ∈ M, with α, β ∈ OM and relatively prime to each other. Let y ∈ OF be such that   y G , i = 0, . . . , KM : Q . (3.1) ∈ O M H G (α/β − li ) Assume also that for every embedding σ : M → C and for every root γ of H G (T ),      σ α  > |σ (γ )| + 1. (3.2)  β  Let NM/Q (β)α = d0 + d1 φ + · · · + dm−1 φ m−1 ,

d0 , . . . , dm−1 ∈ F.

(3.3)

Then there exists a constant c > 0 depending on D, l0 , . . . , l[KM G :Q] , M, H G (T ), and F only such that     NF /Q (Ddi ) < NF /Q (y)c . (3.4) Proof. In KM G , G



yβ deg(H ) y = ∈ OKM G , γ (α/β − li − γ ) γ (α − li β − γβ)

(3.5)

where the product is taken over all roots γ of H G . Since (α, β) = 1 in OKM G , (β, α − li β − γβ) = 1. Thus, for each γ , (y/(α − li β − γβ)) ∈ OKM G . Therefore,     N G (α − li β − γβ) ≤ NF /Q (y)[KM G :F ] , KM /Q         α N G (β) N G  ≤ NF /Q (y)[KM G :F ] . − l − γ KM /Q  KM /Q β i  Using the fact that |NKM G /Q (β)| ≥ 1, we can conclude that        α [KM G :F ]    N G .  KM /Q β − li − γ  ≤ NF /Q (y) On the other hand, using i = 0 and the assumption (3.2), we can conclude that     N G (β) ≤ NF /Q (y)[KM G :F ] , KM /Q and also       NM/Q (β) ≤ NF /Q (y)[M:F ]  ≤ NF /Q (y)[KM G :F ] .

(3.6)

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

125

From (3.6) and an argument similar to the one used to prove Lemma 3.2 of [18], we can conclude that there exists a positive constant C depending on l0 , . . . , l[KM G :Q] , M, F , and H G only such that        σ α  ≤ C NF /Q (y)[KM G :F ]  (3.7)   β for all σ -embeddings of M into C. Using (3.7) and Lemma A.1, we can now conclude that m−1    NM/Q (β) α = di φ i , β

(3.8)

i=0

where di ∈ F , and for all embeddings σ of F into C,   G |σ (di )| ≤ C¯ NF /Q (y)2[KM :F ] ,

(3.9)

where C¯ is a positive constant depending on l0 , . . . , l[KM G :Q] , M, F , and H G only. Thus, for some positive constant c depending on D, l0 , . . . , l[KM G :Q] , M, F, H G (T ) and D only,     NF /Q (Ddi ) < NF /Q (y)c . (3.10) 4. Main theorems Lemma 4.1. Let QM be a prime of M not splitting in the extension MK/M. Let TM be an F -conjugate of QM . Then TM does not split completely in the extension M G K/M. Proof. Let QF = TF be the F -prime below QM and TM . Let QM G K and TM G K be factors of QM and TM , respectively, in M G K. Since M G K/F is Galois,





 f Q M G K / Q F = f T M G K /T F = f T M G K /T M f ( T M / T F ) . Since QM does not split in the extension MK/M, f (QM G K /QF ) ∼ = 0 modulo p > m + 1 = [M : F ] + 1 > f (TM /TF ). Thus, p | f (TM G K /TM ). Lemma 4.2. Suppose the following equations and conditions are satisfied in variables a0,j , . . . , a2p−1,j , b0,j , . . . , b2p−1,j , x, xr , wr,0 , . . . , wr,[KM G :Q] , U0,r , U2p−1,r , v0,r , . . . , v2p−1,r over OM,W K ∪{PM } for all r = 1, . . . , hM , j = 1, . . . , hM + 2: M

αj =

2p−1  i=0

ai,j δ i ,

βj =

2p−1  i=0

bi,j δ i ;

(4.1)

126

ALEXANDRA SHLAPENTOKH

 NMKL/KM βj = 1;

 NMKL/LM βj = 1;

(4.2) (4.3)

αj = βjk ; xr = aPF (x + sr )

hM

(4.4)

, x is integral at all the primes of M dividing aPF ,

(4.5)

where s1 = 0, . . . , shM are distinct natural numbers; wr,i =

hF a1,1

H G (x

r − li )

,

  i = 0, . . . , KM G : Q ;

(4.6)

αr+2 − 1 ; α2 − 1

(4.7)

U0,r + U1,r δ + · · · + U2p−1,r δ 2p−1 =

xr − U0,r + U1,r δ + · · · + U2p−1,r δ 2p−1  chF = a1,1 v0,r + v1,r δ + · · · + v2p−1,r δ 2p−1 .

(4.8)

Assume also that for every embedding σ : KM G → C and for every root γ of H G (T ), |σ (xr )| > |σ (γ )| + 1.

(4.9)

Then x ∈ F . Proof. First of all, we observe that ordPM H G (xr − li ) = 0. Since the M-norm of δG is not divisible by PM , the free term of H G (T ) is not zero modulo PM . From (4.5) we deduce that for all r, xr ∼ = 0 modulo PM , and by assumption, for K all i, li is equivalent to zero modulo PM . Thus, our assertion is true. Since WM contains only primes not splitting in the extension MK/M and does not contain any primes dividing the discriminant of H G (T ) or its coefficients or any primes whose F conjugates divide the discriminant of H G (T ) or its coefficients, by Lemmas A.2 and K or their conjugates 4.1, H G (xr −li ) does not have positive order at the primes of WM G over F . Hence, we can conclude that H (xr −li ) does not have positive order at any primes where elements of the ring OM,W K ∪{PM } are allowed negative orders or any M

hF /H G (xr − li )) can have negative order only at primes primes above WFK . Thus, (a1,1 where a1,1 has negative order. By Lemma 2.4, αj ∈ KL for all j = 1, . . . , hM + 2. Since δ generates KL over F , we have a1,1 ∈ F ∩ OM,W K ∪{PM } ⊂ OF,W K ∪{PF } . Let A be the divisor of a1,1 . M F Then A = BC, where B has a product of primes from WFK and powers of PF in the denominator and numerator, while C is an integral divisor composed of primes hF = zy, z ∈ F, y ∈ OF , such that outside WFK ∪ {PF }. Thus, a1,1

y ∈ OM , HG (xr − li )

  i = 0, . . . , M G K : Q .

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

127

Therefore, by Lemma 3.1 we can conclude the following: xr =

αr , βr

αr , βr ∈ O M ; (4.10)     NF /Q (Ddj,r ) < NF /Q (y c ). NM/Q (βr )xr = d0,r + d1,r φ + · · · + dm−1,r φ m−1 ; (4.11) chF By (4.7), xr − U0,r = a1,1 v0,r , where U0,r ∈ F . Further, chF NF /Q (βr )xr − NF /Q (βr )U0,r = NF /Q (βr )a1,1 v0,r .

Hence, chF Cr , NF /Q (βr )xr − Br = a1,1

where a1,1 , Br ∈ OF,W K ∪{PF } , NF /Q (βr )xr ∈ OM . From the discussion above, F chF = y c y, ¯ where y c ∈ OF , y¯ ∈ OF,W K ∪{PF } and the divisor of y c has no faca1,1 F tors at which elements of the ring OM,W K ∪{PM } are allowed to have negative orders. M Thus, NF /Q (βr )xr − Br = y c Zr , where Zr ∈ OM,W K ∪{PM } . Further, by the strong approximation theorem, there exists M Ar ∈ OF such that ((Br − Ar )/y c ) ∈ OF,W K ∪{PF } . Thus, we have F

NF /Q (βr )xr − Ar = y c Er ,

(4.12)

where Er ∈ OM . The left-hand side of (4.12) can be rewritten as d0,r − Ar + d1,r φ + · · · + dm−1,r φ m−1 = y c Er . Hence, we can conclude that d0,r − Ar d1,r dm−1,r m−1 + c φ +···+ φ c y y yc is an algebraic integer. Then, by a well-known number-theoretic result, for each s > 0, Dds,r /y c is an algebraic integer. However, this implies that NF /Q (Dds,r ) = 0,

s = 1, . . . , m − 1,

or NF /Q (Dds,r ) ≥ NF /Q (y c ),

s = 1, . . . , m − 1.

The latter case contradicts (4.11). Thus, for all s > 0, ds,r = 0, and consequently, xr ∈ F . By [18, Lemma 5.2], having xr ∈ F for r = 1, . . . , hM implies x ∈ F .

128

ALEXANDRA SHLAPENTOKH

Lemma 4.3. Let x ∈ N be large enough in absolute value so that (4.9) is satisfied. Then all the equations (4.1)–(4.8) can be satisfied in all the other variables over OM,W K ∪{PM } . M

Proof. By Lemma 2.2, there exist α, β in the integral closure of OM,{PM } in MKL satisfying equations (4.2)–(4.4). Furthermore, by Lemma 2.3, the divisors of α and β are composed of factors of PM = PF . Using an argument similar to the one used in [18, Lemma 2.6 and Corollary 2.7], one can show that for any integral divisor A of F relatively prime to PF , there exists a natural number s such that the following conditions are satisfied. • βs ∼ = 1 mod A in the integral closure of OM,PM in MKL. • The coordinates of β s with respect to the power basis of δ are in OM,PM . • The first coordinate of β s with respect to the power basis of δ is equivalent to 1 modulo A in OM,PM . • All the other coordinates of β s are equivalent to 0 modulo A in OM,PM . If x ∈ Z, then (4.5) is satisfied. Let α1 = α s and β1 = β s where s corresponds to A equal to the product of numerators of the divisors of H G (xr − li ) for all i and r that we know by Lemma 4.2 to be prime to PM . Then (4.1) and (4.6) can also be satisfied. Similarly, let α2 be a sufficiently high power of α1 so that chF | α2 − 1 a1,1

in OM,PM [δ]. To satisfy (4.7) and (4.8), set αr+2 = α2xr , βr+2 = β2xr . Observe that αr+2 − 1 ∼ = xr α2 − 1 in Z[α2 ] ⊂ OM,PM [δ]. Thus,

modulo α2 − 1

αr+2 − 1 ∼ ch = xr modulo a1,1F α2 − 1 in OM,PM [δ]. Note that α2 , . . . , αhM +2 , β2 , . . . , βhM +2 ∈ OM,PM [δ], chosen in such a manner, also satisfy (4.1)–(4.4). Now we are ready to state the first of our main theorems. Theorem 4.4. Let F, OM,W K ∪{PM } be as in Notation 2.1. Then F ∩OM,W K ∪{PM } M M has a Diophantine definition over OM,W K ∪{PM } . M

Proof. We only need to note two things. First, every element of F ∩OM,W K ∪{PM } M  :Q]−1 can be written in the form [F ±(ui /v)ψ i , where ui , v ∈ N and ψ is an integral i=0 generator of F over Q. Second, (4.9) can be rewritten as a polynomial equation using an argument similar to the one used in [18, Corollary 4.4]. Theorem 4.5. Let G be any totally real nontrivial extension of Q. Then there exists an infinite set WG of primes of G such that OG has a Diophantine definition over OG,WG .

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

129

Proof. Let M be the Galois closure of G over Q. Let F be a cyclic subextension of M such that G = G ∩ F . Such an F always exists. Since G = Q, there exists σ ∈ Gal(M/Q) such that G is not fixed by σ . Then let F be the fixed field of σ . Clearly G ⊂ F , and thus R = G∩F is a proper subfield of G. Let LQ , KQ be cyclic, prime-degree extensions of Q defined so that the fields L = F LQ and K = F KQ satisfy the requirements of Notation 2.1 with respect to M and F , and KQ R is linearly disjoint from M over R. (It is clear that such a pair of fields LQ , KQ always exists for any totally real pair of fields M and F .) By Lemma A.3, there exist infinitely many primes of R splitting completely in the extension M/R, such that their factors remain prime in the extension MK/M. Let WGK be an infinite collection of factors of these primes in G such that every prime in WGK has a conjugate over R that is not in K be the set of all the factors of primes in W K in M with possibly finitely WGK . Let WM G K . (The many primes removed to satisfy the assumptions listed in Notation 2.1 for WM G reader should note here that, in this case, M = M.) Applying Lemma A.3 again, we conclude that there exist infinitely many primes of F splitting completely in KL and not splitting in the extension M/F . (Note that factors of these primes in M split completely in the extensions MKL/M.) Let PF be one of these primes. Let PM be its sole factor in M and let PG be the G-prime below PM . By Theorem 4.4, there exists a polynomial P (t, X1 , . . . , Xr ) ∈ M[t, X1 , . . . , Xr ] such that ∀t ∈ OM,W K ∪{PM } , M

∃ X1 , . . . , Xr ∈ OM,W K ∪{PM } , M

P (t, X1 , . . . , Xr ) = 0

if and only if t ∈ OM,W K ∪{PM } ∩F . Therefore, there exists a polynomial P (t, X1 , . . . , M Xr ) ∈ M[t, X1 , . . . , xr ] such that ∀t ∈ OM,W K ∪{PM } ∩ G, M

∃ X1 , . . . , Xr ∈ OM,W K ∪{PM } , M

P (t, X1 , . . . , Xr ) = 0

if and only if t ∈ OM,W K ∪{PM } ∩ F ∩ G = OM,W K ∪{PM } ∩ R. M

M

Let us examine OM,W K ∪{PM } ∩G more closely. If PG has only one factor in M, then M OM,W K ∪{PM } ∩ G = OG,W K ∪{PG } . However, if PG has more than one factor in M, M G then OM,W K ∪{PM } ∩ G = OG,W K . Without loss of generality, assume we are in the M G second case. Then by Theorem 1.3, there exists a polynomial S(t, z1 , . . . , zs ) such that for every t ∈ OG,W K ∪{PG } , ∃ z1 , . . . , zs ∈ OG,W K ∪{PG } , S(t, z1 , . . . , zs ) = 0 ⇔ G G t ∈ OG,W K . Thus, for every t ∈ OG,W K ∪{PG } , G

G

∃ z1 , . . . , zs ∈ OG,W K ∪{PG } , G

X1 , . . . , Xr ∈ OM,W K ∪{PM } , M

(4.13)

130

ALEXANDRA SHLAPENTOKH



S(t, z1 , . . . , zs ) = 0, P (t, X1 , . . . , Xr ) = 0

if and only if t ∈ OM,W K ∪{PM } ∩ R. M

By Corollary 1.8, we can find Q(t, z1 , . . . , zs , x1,1 , . . . , xr,[M:G] ) ∈ G[t, z1 , . . . , zs , x1,1 , . . . , xr,[M:G] ] such that for every t ∈ OG,W K ∪{PG } , G



∃ z1 , . . . , zs , x1,1 , . . . , xr,[M:G] ∈ OG,W K ∪{PG } , Q t, z1 , . . . , zs , x1,1 , . . . , xr,[M:G] = 0 G

if and only if t ∈ OM,W K ∪{PM } ∩ R. M

Next consider the following: Under the assumption that either PG had more than one factor in M or more than one conjugate over R,

 OM,W K ∪{PM } ∩ R = OM,W K ∪{PM } ∩ G ∩ R = OG,W K ∩ R = OR . M

M

G

In the case PG has just one factor in M and has no other conjugates over R, OM,W K ∪{PM } ∩ R = OR,{PR } , M

where PR is the R prime below PG . Since we know how to define integrality at finitely many primes using polynomial equations, again, without loss of generality, we can assume that the intersection is equal to OR . Thus we conclude that OR has a Diophantine definition over OG,W K ∪{PG } . Next we note that rational integers have a G Diophantine definition over OR , and we can use that fact, together with an integral basis of OG over Z, to complete a Diophantine definition of OG over OG,W K ∪{PG } . G

Theorem 4.6. Let M be a totally real number field with a cyclic subextension F of degree m. Then for any ε > 0, there exists a set of primes WM of M of Dirichlet density greater than ((m−1)/m)−ε such that OM has a Diophantine definition over OM,WM . Furthermore, if WF is the set of F -primes below the primes of WM , then the Dirichlet density of WF can be arranged to be greater than 1 − (φ(m)/m) − ε, where φ is the Euler function. Proof. First of all, note the existence of extensions L and K of F satisfying the conditions of Notation 2.1. Next let WFK be the set of all the primes in F splitting in K be the largest possible M with factors not splitting in the extension MK/M. Let WM K set of M-primes above the primes of WF satisfying the following condition. Every K has exactly one F -conjugate not in W K . (As above, we might have to prime in WM M

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

131

K and W K to make sure all the requirements remove finitely many primes from WM F described in Notation 2.1 are satisfied. Here again, M G = M.) Using Lemma A.3, again we ascertain existence of an M-prime PM such that PM splits completely in MKL/M but has no other conjugates over F . By Theorem 4.4, we conclude that F ∩ OM,W M ∪{PM } = OF,{PF } has a Diophantine definition over OM,W M ∪{PM } . Since we K K can define integrality at one prime, we can now conclude that OF has a Diophantine definition over OM,W M ∪{PM } . Using the same argument as in Theorem 4.5, we deduce K that OM has a Diophantine definition over OM,W M ∪{PM } . It remains to calculate the K

K and W K . We start with the latter. density of WM F By the Chebotarev density theorem (see [8, Theorem 10.4]), the density of WFK is ((p −1)/pm)(m−φ(m)). Indeed, consider Gal(MK/F ) = Gal(M/F )×Gal(K/F ). If p ∈ WFK , then, except for finitely many primes, either p does not split in M or some factors of p split in K. Let σM , σK be generators of Gal(M/F ), Gal(K/F ), respectively. Then the Frobenius automorphism of factors of p in Gal(MK/F ) must i , σ j ), where either (i, m) = 1 and j = 0, 1, . . . , p −1, or (i, m) > 1 be of the form (σM K and j = 0. Thus, the total number of elements of the Galois group of MK over F that can be Frobenius automorphisms of the primes not above WFK is φ(m)p+(m−φ(m)). Therefore, the number of elements in the Galois group of MK over F that can be Frobenius automorphisms of primes above WFK is mp − φ(m)p − m + φ(m) = (m−φ(m))(p−1). Hence, it follows by Chebotarev’s density theorem that the density of WFK is ((p − 1)/pm)(m − φ(m)). By letting p → ∞, we get the desired result for the density of the prime set in F . K , we first note that we need to consider primes of To calculate the density of WM K WM with m conjugates over F only, since the density of the set of M-primes lying above totally splitting primes of F is 1. (See [8, Theorem 4.6].) By assumption, K except possibly for finitely many primes, for every totally splitting prime of F , WM contains m − 1 of its factors if they do not split in K. Let WM be a set of M primes containing m−1 factors of every totally splitting prime of F . Then by the Chebotarev K and W is less than 1/p and thus density theorem, the difference in densities of WM M can be made arbitrarily small by letting p be arbitrarily large. Thus it is sufficient to compute the density of WM . Let UM be the set of all the M-primes above totally splitting primes of F . Then the density of UM is 1. We can obtain the density of WM from the density of UM by subtracting the density of the set consisting of single factors of every totally splitting prime of F . The density of that set is exactly the same as the density of the set of all the primes in F totally splitting in M, since their Q-norms are the same. This density is equal to 1/m. Therefore, the density of WM is 1 − (1/m) as required.

Corollary 4.7. For every δ > 0, there exist a number field M and a set of Mprimes W such that the following statements are true. • OM is definable over OM,W . • The density of W is greater than 1 − δ.

132

ALEXANDRA SHLAPENTOKH

• The density of the set of rational primes below W is greater than 1 − δ. Proof. Let M be a totally real cyclic extension of Q of degree m = p1 · · · pk , where pi is the ith rational prime under the standard enumeration of primes. Then K of M primes such that the by Theorem 4.6, for every ε > 0, there exists a set WM following are true: K is greater than ((m − 1)/m) − ε; (1) the density of WM K is greater than (2) the density of the set of rational primes below WM k

1− Since limk→∞ true.

k

 pi − 1 φ(m) − ε. −ε = 1− m pi i=1

i=1 ((pi

− 1)/pi ) = 0 (see [14, (9.1.14)]), the corollary is clearly Appendix

Lemma A.1. Let M/F be a finite extension of number fields of degree m. Let δ be  i a generator of M/F . Let x ∈ M, x = m−1 i=0 ai δ , ai ∈ F . Assume for some positive constant C, for every σ -embedding of M into C, |σ (x)| < C. Then for every σ ˜ where C˜ depends embedding of M into C, and every i = 0, . . . , m−1, |σ (ai )| < CC, on δ only. Proof. Consider the following linear system: m−1 

 aj σj δ i = σj (x),

j = 1, . . . , m,

i=0

where σ1 = identity, . . . , σm are all the embeddings of M into C leaving F fixed. Solving this system via Cramer’s rule, we can deduce that for each i = 0, . . . , m − 1, ai = Pi (δ, σ1 (x), . . . , σm (x)), where for each i, Pi is a fixed polynomial, linear in σj (x), depending only on m. Next let τ be an embedding of M into C that does not fix F . Repeating the argument above over τ (F ), we obtain a similar bound for all τ (ai ). This way we can obtain a bound as described in the statement of the lemma for all the conjugates of ai over Q. Lemma A.2. Let E/M be a finite extension of algebraic number fields. Let γ ∈ OE generate E over M and let H (T ) be the monic irreducible polynomial of γ over M. Let WM be a set of primes of M without relative degree 1 factors in E. Then for every Q ∈ WM such that Q does not divide the discriminant of γ and no coefficient of H (T ) has a positive order Q, for every x ∈ M, ordQ H (x) ≤ 0. Proof. Let γ and Q be as in the statement of the lemma. Then powers of γ constitute a local integral basis of E over M with respect to Q. Thus the factorization of the minimal polynomial of γ modulo Q corresponds to the factorization of Q in

INTEGRALITY AT PRIME SETS OF HIGH DENSITY

133

E (see [9, Proposition 25].) Let x ∈ M and assume x has a negative order at Q. Then H (x) has a negative order at Q. On the other hand, suppose x is integral at Q and H (x) has positive order at Q. Then H (T ) has a root modulo Q and thus a linear factor modulo Q. This implies Q has a factor of relative degree 1 in E in contradiction of our assumption. Lemma A.3. Let F be a number field. Let N1 be a cyclic extension of F , and let N2 be a Galois extension of F linearly disjoint from N1 over F . Then there are infinitely many primes of p of F such that p does not split in N1 ; its unique factor in N1 splits completely in the extension N1 N2 /N1 , and p splits completely in N2 . If N2 is also cyclic, then there are infinitely many primes p of F such that p splits completely in N1 but none of its factors split or ramify in the extension N1 N2 /N1 . Proof. For the first and the second assertions of the lemma, apply parts 2 and 3 of [18, Lemma 2.1], respectively, with L = F , K = N1 , and E = N2 . References [1] [2] [3]

[4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]

J.-L. Colliot-Thélène, A. N. Skorobogatov, and P. Swinnerton-Dyer, Double fibres and double covers: Paucity of rational points, Acta. Arith. 79 (1997), 113–135. M. Davis, Hilbert’s tenth problem is unsolvable, Amer. Math. Monthly 80 (1973), 233–269. M. Davis, Y. Matijasevich, and J. Robinson, “Hilbert’s tenth problem: Diophantine equations: Positive aspects of a negative solution” in Mathematical Developments Arising from Hilbert Problems (Northern Illinois University, Dekalb, Ill., 1974), Proc. Sympos. Pure Math. 28, Amer. Math. Soc., Providence, 1976, 323–378. J. Denef, Hilbert’s tenth problem for quadratic rings, Proc. Amer. Math. Soc. 48 (1975), 214–220. , Diophantine sets over algebraic integer rings, II, Trans. Amer. Math. Soc. 257 (1980), 227–236. J. Denef and L. Lipschitz, Diophantine sets over some rings of algebraic integers, J. London Math. Soc. (2) 18 (1978), 385–391. M. Fried and M. Jarden, Field Arithmetic, Ergeb. Math. Grenzgeb. (3) 11, Springer-Verlag, Berlin, 1986. G. Janusz, Algebraic Number Fields, Pure Appl. Math. 55, Academic Press, New York, 1973. S. Lang, Algebraic Number Theory, Addison-Wesley, Reading, Mass., 1970. B. Mazur, The topology of rational points, Experiment. Math. 1 (1992), 35–45. , Questions of decidability and undecidability in number theory, J. Symbolic Logic 59 (1994), 353–371. O. T. O’Meara, Introduction to Quadratic Forms, Grundlehren Math. Wiss. 117, SpringerVerlag, Berlin, 1963. T. Pheidas, Hilbert’s tenth problem for a class of rings of algebraic integers, Proc. Amer. Math. Soc. 104 (1988), 611–620. H. N. Shapiro, Introduction to the Theory of Numbers, Pure Appl. Math., Wiley, New York, 1983. H. N. Shapiro and A. Shlapentokh, Diophantine relationships between algebraic number fields, Comm. Pure Appl. Math. 42 (1989), 1113–1122. A. Shlapentokh, Extension of Hilbert’s tenth problem to some algebraic number fields, Comm. Pure Appl. Math. 42 (1989), 939–962.

134 [17] [18]

ALEXANDRA SHLAPENTOKH , Diophantine classes of holomorphy rings of global fields, J. Algebra 169 (1994), 139–175. , Diophantine definability over some rings of algebraic numbers with infinite number of primes allowed in the denominator, Invent. Math. 129 (1997), 489–507.

Department of Mathematics, East Carolina University, Greenville, North Carolina 27858, USA; [email protected]

Vol. 101, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

ALMOST COMPLEX STRUCTURES ON S 2 × S 2 DUSA MCDUFF

Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 2. Main ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 2.1. The effect of increasing λ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 2.2. Stable maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 2.3. Gluing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 2.4. Moduli spaces and the stratification of ᏶ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 3. The link ᏸ2,0 of ᏶2 in ᏶0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 3.1. Some topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 3.2. Structure of the pair (ᐂJ , ᐆJ ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 3.3. The projection ᐂJ → ᏶ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 4. Analytic arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 4.1. Regularity in dimension 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 4.2. Gluing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 1. Introduction. It is well known that every symplectic form on X = S 2 × S 2 is, after multiplication by a suitable constant, symplectomorphic to a product form ωλ = (1 + λ)σ1 + σ2 for some λ ≥ 0, where the 2-form σi has total area 1 on the ith factor. We are interested in the structure of the space ᏶λ of all C ∞ ωλ -compatible, almost complex structures on X. Observe that ᏶λ itself is always contractible. However, it has a natural stratification that changes as λ passes each integer. The reason for this is that as λ grows, the set of homology classes that can be represented by an ωλ -symplectically embedded 2-sphere changes. Since each such 2-sphere can be parametrized to be J -holomorphic for some J ∈ ᏶λ , there is a corresponding change in the structure of ᏶λ . To explain this in more detail, let A ∈ H2 (X, Z) be the homology class [S 2 × pt] and let F = [pt ×S 2 ]. (The reason for this notation is that we are thinking of X as a fibered space over the first S 2 -factor, so that the smaller sphere F is the fiber.) When  − 1 < λ ≤ , ωλ (A − kF ) > 0 for 0 ≤ k ≤ . Moreover, it is not hard to see that for each such k, there is a map ρk : S 2 → S 2 of Received 13 October 1998. 1991 Mathematics Subject Classification. Primary 53C15. Author partially supported by National Science Foundation grant number DMS 9704825. 135

136

DUSA MCDUFF

degree −k whose graph


 z −→ z, ρk (z)

is an ωλ -symplectically embedded sphere in X. It follows easily that the space   ᏶λk = J ∈ ᏶λ : there is a J -hol curve in class A − kF is nonempty whenever k < λ + 1. Let λ

᏶k = ∪m≥k ᏶λm .

Because (A−kF )·(A−mF ) < 0 when k  = m > 0, positivity of intersections implies that there is exactly one J -holomorphic curve in class A − kF for each J ∈ ᏶k . We denote this curve by J . λ

Lemma 1.1. The spaces ᏶λk , 0 ≤ k ≤ , are disjoint, and ᏶k is the closure of ᏶λk λ

in ᏶λ . Further, ᏶λ = ᏶0 .

Proof. It is well known that for every J ∈ ᏶, the set of J -holomorphic curves in class F form the fibers of a fibration πJ : X → S 2 . Moreover, the class A is represented by either a curve or a cusp-curve (i.e., a stable map).1 Since the class F is always represented and (mA+pF )·F = m, it follows from positivity of intersections that m ≥ 0 whenever mA + pF is represented by a curve. Hence any cusp-curve in class A has one component in some class A−kF for k ≥ 0, and all others represent a multiple of F . In particular, each J ∈ ᏶λ belongs to some set ᏶λk . Moreover, because (A − kF ) · (A − mF ) < 0 when k  = m and k, m ≥ 0, the different ᏶λk are disjoint. The second statement holds because if Jn is a sequence of elements in ᏶λk , then the corresponding sequence of Jn -holomorphic curves in class A − kF has a convergent subsequence whose limit is a cusp-curve in class A − kF . This limit has to have a component in some class A − mF , for m ≥ k, and so J ∈ ᏶λm for some m ≥ k. For further details, see Lalonde and McDuff [LM]. Here is our main result. Throughout we work with C ∞ -maps and almost complex structures, and so by manifold we mean a Fréchet manifold. By a stratified space ᐄ we mean a topological space that is a union of a finite number of disjoint manifolds that are called strata. Each stratum ᏿ has a neighborhood ᏺ᏿ that projects to ᏿ by a map ᏺ᏿ → ᏿. When ᏺ᏿ is given the induced stratification, this map is a locally trivial fiber bundle whose fiber has the form of a cone C(ᏸ) over a finite-dimensional stratified space ᏸ that is called the link of ᏿ in ᐄ. Moreover, ᏿ sits inside ᏺ᏿ as the set of vertices of all these cones. Theorem 1.2. (i) For each 1 ≤ k ≤ , ᏶λk is a submanifold of ᏶λ of codimension 4k − 2. λ λ (ii) For each m > k ≥ 1, the normal link ᏸm,k of ᏶λm in ᏶k is a stratified space 1 We

follow the convention of [LM] by defining a “curve” to be the image of a single sphere, while a “cusp-curve” is either multiply covered or has domain equal to a union of two or more spheres.

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

137 λ

of dimension 4(m − k) − 1. Thus, there is a neighborhood of ᏶λm in ᏶k that is fibered λ . over ᏶λm with fiber equal to the cone on ᏸm,k λ is independent of λ (provided that λ > m − 1). (iii) The structure of the link ᏸm,k The first part of this theorem was proved by Abreu in [A], at least in the C s -case where s < ∞. (Details are given in §4.1 below.) The second and third parts follow by globalizing recent work by Fukaya and Ono [FO], Li and Tian [LiT], Liu and Tian [LiuT1], Ruan [R], and others on the structure of the compactification of moduli spaces of J -holomorphic spheres via stable maps. We extend current gluing methods by showing that it is possible to deal with obstruction bundles whose elements do not vanish at the gluing point (see §4.2.3). Another essential point is that we use Fukaya and Ono’s method of dealing with the ambiguity in the parametrization of a stable map, since this involves the least number of choices and allows us to globalize by constructing a gluing map that is equivariant with respect to suitable local torus actions (see §4.2.4 and §4.2.5). The above theorem is the main tool used in [AM] to calculate the rational cohomology ring of the group Gλ of symplectomorphisms of (X, ωλ ). The methods of proof may also prove useful in situations in which one wants to understand properties of families of almost complex structures. These arise when one is considering family Gromov-Witten invariants, as in the work of Bryan and Leung [BL] on the Yau-Zaslow conjecture [YZ], and that of Li and Liu [LL1], [LL2] on symplectic 4-manifolds with torsion canonical class. Observe that part (iii) states that the normal structure of the stratum ᏶λk does not change with λ. On the other hand, it follows from the results of [AM] that the cohomology of ᏶λk definitely does change as λ passes each integer. Obviously, it would be interesting to know if the topology of ᏶λk is otherwise fixed. For example, one could try to construct maps ᏶λ → ᏶µ for λ < µ that preserve the stratification, µ and then try to prove that they induce homotopy equivalences ᏶λk → ᏶k whenever  − 1 < λ ≤ µ ≤ . So far, the most we have managed to do in this direction is to prove the following lemma, which, in essence, constructs maps ᏶λ → ᏶µ for λ < µ. It is not clear whether these are homotopy equivalences for λ, µ ∈ ( − 1, ]. It is convenient to fix a fiber F0 = pt ×S 2 and define  
 ᏶λk ᏺ(F0 ) = J ∈ ᏶λk : J = Jsplit near F0 , where Jsplit is the standard product almost complex structure. Lemma 1.3. (i) The inclusion ᏶λ (ᏺ(F0 )) → ᏶λ induces a homotopy equivalence for all k < λ + 1. (ii) Given any compact subset C ⊂ ᏶λ (ᏺ(F0 )) and any µ > λ, there is a map 
ιµ,λ : C −→ ᏶µ ᏺ(F0 )

 ᏶λk (ᏺ(F0 )) −−→ ᏶λk

µ

that takes C ∩ ᏶λk (ᏺ(F0 )) into ᏶k (ᏺ(F0 )) for all k.

138

DUSA MCDUFF

λ . So far this This lemma is proved in §2. The next task is to calculate the links ᏸm,k has been done for the easiest case as follows. λ Proposition 1.4. For each k ≥ 1 and λ, the link ᏸk+1, k is the 3-dimensional lens space L(2k, 1).

Finally, we illustrate our methods by using the stable map approach to confirm that the link of ᏶λ2 in ᏶λ is S 5 , as predicted by part (i) of Theorem 1.2. Our method first calculates an auxiliary link ᏸᐆ from which the desired link is obtained by collapsing certain strata. The S 5 appears in a surprisingly interesting way, which can be briefly described as follows. Let ᏻ(k) denote the complex line bundle over S 2 with Euler number k, where we write C instead of ᏻ(0). Given a vector bundle E → B, we write S(E) → B for its unit sphere bundle. Note that the unit 3-sphere bundle
 S ᏻ(k) ⊕ ᏻ(m) −→ S 2 decomposes as the composite 

S LP (k,m) −→ ᏼ ᏻ(k) ⊕ ᏻ(m) −→ S 2 , where LP (k,m) → ᏼ(ᏻ(k) ⊕ ᏻ(m)) is the canonical line bundle over the projectivization of ᏻ(k) ⊕ ᏻ(m). In particular, the space S(ᏻ(−1) ⊕ C) can be identified with S(LP (−1,0) ). But ᏼ(ᏻ(−1) ⊕ C) is simply the blow-up CP 2 #CP 2 , and its canonical bundle is the pullback of the canonical bundle over CP 2 . We are interested in the singular line bundle (or orbibundle) LY → Y whose associated unit sphere bundle has total space S(LY ) = S 5 and fibers equal to the orbits of the following S 1 -action on S 5 : 
θ · (x, y, z) = eiθ x, eiθ y, e2iθ z , x, y, z ∈ C. The plumbing described in (i) below can be considered as an orbifold analog of the process of blowing up a point lying on the zero section of a bundle (see §3.1 below and [G]). Theorem 1.5. (i) The space ᏸᐆ obtained by plumbing the unit sphere bundle of ᏻ(−3) ⊕ ᏻ(−1) with the singular circle bundle S(LY ) → Y may be identified with the unit circle bundle of the canonical bundle over ᏼ(ᏻ(−1) ⊕ C) = CP 2 #CP 2 . λ is obtained from ᏸ by collapsing the fibers over the exceptional (ii) The link ᏸ2,0 ᐆ divisor to a single fiber, and hence may be identified with S 5 . Under this identification, λ = RP 3 corresponds to the inverse image of a conic in CP 2 . the link ᏸ2,1 In his recent paper [K], Kronheimer shows that the universal deformation of the quotient singularity C2 /(Z/mZ) is transverse to all the submanifolds ᏶k and so is an explicit model for the normal slice of ᏶m in ᏶. Hence one can investigate the structure λ using tools from algebraic geometry. It is very possible of the intermediate links ᏸm,k

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

139

that it would be easier to calculate these links this way. However, it is still interesting to try to understand these links from the point of view of stable maps, since this is more closely connected to the symplectic geometry of the manifold X. Another point is that throughout we consider ωλ -compatible, almost complex structures rather than ωλ -tame ones. However, it is easy to see that all our results hold in the latter case. Other ruled surfaces. All the above results have analogs for other ruled surfaces Y → ). If Y is diffeomorphic to the product ) × S 2 , we can define ωλ , ᏶λk as above, though now we should allow λ to be any number greater than −1 since there is no symmetry between the class A = [) ×pt] and F = [pt ×S 2 ]. In this case, Theorem 1.2 still holds. The reason for this is that if u : ) → Y is an injective J -holomorphic map in class A−kF where k ≥ 1, then the normal bundle E to the image u()) has negative first Chern class so that the linearization Du of u has kernel and cokernel of constant dimension. (In fact, the normal part of Du with image in E is injective in this case; see [HLS, Theorem 1 ].) However, Lemma 1.1 fails unless ) is a torus since there are tame, almost complex structures on Y with no curve in class A. One might think to remedy this by adding other strata ᏶λ−k consisting of all J such that the class A + kF is represented by a J -holomorphic curve u : (), j ) → Y for some complex structure j on ). However, although the universal moduli space ᏹ(A + kF, ᏶λ ) of all such pairs (u, J ) is a manifold, the map (u, J ) → J is no longer injective: even if one cuts down the dimension by fixing a suitable number of points, each J , in general, admits several curves through these points. Moreover, as (uJ ) varies over ᏹ(A + kF, ᏶λ ), the dimension of the kernel and cokernel of Du can jump. Hence the argument given in §4.1 that the strata ᏶λk are submanifolds of ᏶λ fails on several counts. In the case of the torus, ᏶λ0 is open, and so Lemma 1.1 does hold. However, it is not clear whether this is enough for the main application, which is to further our understanding of the groups Gλ of symplectomorphisms of (Y, ωλ ). One crucial ingredient of the argument in [AM] is that the action of this group on each stratum ᏶λk is essentially transitive. More precisely, we show that the action of Gλ on ᏶λk induces a homotopy equivalence Gλ / Aut(Jk ) → ᏶λk , where Jk is an integrable element of ᏶λk and Aut(Jk ) is its stabilizer. It is not clear whether this would hold for the stratum ᏶λ0 when ) = T 2 . One might have to take into account the finer stratification considered by Lorek in [Lo]. He points out that the space ᏶λ0 of all J that admit a curve in class A is not homogeneous. A generic element admits a finite number of such curves that are regular (that is, curves u with Du surjective), but since this number can vary, the set of regular elements in ᏶λ0 has an infinite number of components. Lorek also characterises the other strata that occur. For example, the codimension 1 stratum consists of J such that all J -holomorphic A curves are isolated, but there is at least one where the kernel of Du has dimension 3 instead of 2. (Note that these two dimensions correspond to the reparametrization group, since Du is the full linearization, not just the normal component.)

140

DUSA MCDUFF

Similar remarks can be made about the case when Y → ) is a nontrivial bundle. In this case we can label the strata ᏶λk so that the J ∈ ᏶λk admit sections with selfintersection −2k + 1. Again, Theorem 1.2 holds, but Lemma 1.1 may not. When ) = S 2 , the homology class of the exceptional divisor is always represented so that λ ᏶λ = ᏶1 . When ) = T 2 , the homology class of the section of self-intersection +1 is λ always represented. Thus ᏶λ = ᏶−1 . Hence the analog of Lemma 1.1 holds in these two cases. Moreover, all embedded tori of self-intersection +1 are regular (by the same result in [HLS]), which may help in the application to Symp(Y ). We now state in detail the result for the nontrivial bundle Y → S 2 since this is used in [AM]. Here Y = CP 2 #CP 2 , and so every symplectic form on Y can be obtained from an annulus Ar,s = {z ∈ C2 : r ≤ |z| ≤ s} by collapsing the boundary spheres to S 2 along the characteristic orbits. This gives rise to a form ωr,s that takes the value π s 2 on the class L of a line and πr 2 on the exceptional divisor E. Let us write ωλ for the form ωr,s where π s 2 = 1 + λ, πr 2 = λ > 0. Then the class F = L − E of the fiber has size 1 as before, and ᏶λk , k ≥ 1 is the set of ωλ -compatible J for which the class E − (k − 1)F is represented. Theorem 1.6. When Y = CP 2 #CP 2 , the spaces ᏶λk are Fréchet submanifolds of of codimension 4k, and form the strata of a stratification of ᏶λ whose normal structure is independent of λ. Moreover, the normal link of ᏶λk+1 in ᏶λk is the lens space L(4k + 1, 1), k ≥ 1.

᏶λ

This paper is organized as follows. §2 describes the main ideas in the proof of Theorem 1.2. This relies heavily on the theory of stable maps, and for the convenience of the reader, we outline its main points. References for the basic theory are, for example, [FO], [LiT], and [LiuT1]. §3 contains a detailed calculation of the link of ᏶λ2 in ᏶λ . In particular, we discuss the topological structure of the space of degree 2 holomorphic self-maps of S 2 with up to two marked points, and of the canonical line bundle that it carries. Plumbing with the orbibundle LY → Y turns out to be a kind of orbifold blowing-up process (see §3.1). Finally, in §4 we work out the technical details of gluing that are needed to establish that the submanifolds ᏶λk do have a good normal structure. The basic method here is taken from McDuff and Salamon [MS] and Fukaya and Ono [FO]. Acknowledgements. I wish to thank Dan Freed, Eleni Ionel, and, particularly, John Milnor for useful discussions on various aspects of the calculation in §3, and Fukaya and Ono for explaining to me various details of their arguments. 2. Main ideas. We begin by proving Lemma 1.3 since this is elementary; then we describe the main points in the proof of Proposition 1.2. 2.1. The effect of increasing λ Proof of Lemma 1.3. Recall that F0 is a fixed fiber pt ×S 2 and that

ALMOST COMPLEX STRUCTURES ON S 2 × S 2








141



᏶λk ᏺ(F0 ) = J ∈ ᏶λk : J = Jsplit near F0 .

We also use the space 



᏶λk (F0 ) = J ∈ ᏶λk : J = Jsplit on T F0 .

Let Ᏺλ be the space of ωλ -symplectically embedded curves in the class F through a fixed point x0 . Because there is a unique J -holomorphic F -curve through x0 for each J ∈ ᏶ (see Lemma 1.1), there is a fibration ᏶λ (F0 ) −→ ᏶λ −→ Ᏺλ .

Since the elements of ᏶λ (F0 ) are sections of a bundle with contractible fibers, ᏶λ (F0 ) is contractible. Hence Ᏺλ is also contractible. By using the methods of Abreu [A], it is not hard to show that the symplectomorphism group Ᏻλ = Symp0 (X, ωλ ) of (X, ωλ ) acts transitively on Ᏺλ . Since the action of Ᏻλ on ᏶λ preserves the strata ᏶λk , it follows that the projection ᏶λk → Ᏺλ is surjective. Hence there are induced fibrations ᏶λk (F0 ) −→ ᏶λk −→ Ᏺλ .

This implies that the inclusion ᏶λk (F0 ) → ᏶λk is a weak homotopy equivalence. We now claim that the inclusion ᏶λk (ᏺ(F0 )) → ᏶λk (F0 ) is also a weak homotopy equivalence. To prove this, we need to show that the elements of any compact set ᏷ ⊂ ᏶λk (F0 ) can be altered near F0 by a homotopy so as to make them coincide with Jsplit . Since the set of tame, almost complex structures at a point is contractible, this is always possible in ᏶λ ; the difficulty here is to ensure that ᏷ remains in ᏶λk throughout the homotopy. To deal with this, we argue as follows. For each J ∈ ᏶λk (F0 ), let J denote the unique J -holomorphic curve in class A − kF . Then J meets F0 transversally at one point; call it qJ . For each J ∈ ᏷, move the curve J by a symplectic isotopy that fixes qJ to make it coincide in a small neighborhood of qJ with the flat section S 2 ×pt that contains qJ . (Details of a very similar construction can be found in [MP, Proposition 4.1.C].) Now lift this isotopy to ᏶λk . Finally, adjust the family of almost complex structures near F0 , keeping J holomorphic throughout. This proves (i). Statement (ii) is now easy. For any compact subset C of ᏶λ (ᏺ(F0 )), there is ε > 0 such that J = Jsplit on the ε-neighborhood ᏺε (F0 ) of F0 . Let ρ be a nonnegative 2-form supported inside the 2-disc of radius ε that vanishes near 0, and let π ∗ (ρ) denote its pullback to ᏺε (F0 ) by the obvious projection. Then every J that equals Jsplit on ᏺε (F0 ) is compatible with the form ωλ + κπ ∗ (ρ) for all κ > 0. Since ωλ + κπ ∗ (ρ) is isotopic to ωµ for some µ, there is a diffeomorphism φ of X that is isotopic to the identity and is such that φ ∗ (ωλ + κπ ∗ (ρ)) = ωµ . Moreover, because, by construction, π ∗ (ρ) = 0 near F0 , we can choose φ = Id near F0 . Hence the map J → φ ∗ (J ) takes ᏶λ (ᏺ(F0 )) to ᏶µ (ᏺ(F0 )). Clearly it preserves the strata ᏶k . 2.2. Stable maps. From now on, we drop λ from the notation, assuming that k < λ+1 as before. We study the spaces ᏶k and ᏶k by exploiting their relation to the

142

DUSA MCDUFF

corresponding moduli spaces of J -holomorphic curves in X. Definition 2.1. When k ≥ 1, ᏹk = ᏹ(A − kF, ᏶) is the universal moduli space of all unparametrized J -holomorphic curves in class A − kF . Thus its elements are equivalence classes [h, J ] of pairs (h, J ), where J ∈ ᏶ = ᏶λ , h is a J -holomorphic map S 2 → X in class A − kF , and where (h, J ) ≡ (h ◦ γ , J ) when γ : S 2 → S 2 is a holomorphic reparametrization of S 2 . Similarly, we write ᏹ0 = ᏹ(A, x0 , ᏶) for the universal moduli space of all unparametrized J -holomorphic curves in class A that go through a fixed point x0 ∈ X. Thus its elements are equivalence classes of triples [h, z, J ] with z ∈ S 2 , (h, J ) as before, h(z) = x0 , and where (h, z, J ) ∼ (h ◦ γ , γ −1 (z), J ) when γ : S 2 → S 2 is a holomorphic reparametrization of S 2 . The next lemma restates part (i) of Theorem 1.2. The proof uses standard Fredholm theory for J -holomorphic curves and is given in §4.1. The only noteworthy point is that when k > 0, the almost complex structures in ᏶k are not regular. In fact, the index of the relevant Fredholm operator is −(4k − 2). However, because we are in 4 dimensions, the Fredholm operator has no kernel, which is the basic reason why the space of J for which it has a solution is a submanifold of codimension 4k − 2. Lemma 2.2. For all k ≥ 0, the projection πk : ᏹk −→ ᏶k : [h, J ] −→ J is a diffeomorphism of the Fréchet manifold ᏹk onto the submanifold ᏶k of ᏶. This submanifold is an open subset of ᏶ when k = 0 and has codimension 4k −2 otherwise. Our tool for understanding the stratification of ᏶ by the ᏶k is the compactification ᏹ(A − kF, ᏶) of ᏹ(A − kF, ᏶) that is formed by J -holomorphic stable maps. For the convenience of the reader, we recall the definition of stable maps with p marked points. We always assume the domain ) to have genus 0. Therefore it is a connected union ∪m i=0 )i of Riemann surfaces, each of which has a given identification with the standard sphere (S 2 , j0 ). (Note that we consider ) to be a topological space: the labelling of its components is a convenience and not part of the data.) The intersection pattern of the components can be described by a tree graph with m+1 vertices (one for each component of )) that are connected by an edge if and only if the corresponding components intersect. No more than two components meet at any point. Also, there are p-marked points z1 , . . . , zp placed anywhere on ) except at an intersection point of two components. (Such pairs (), z1 , . . . , zp ) = (), z) are called semistable curves.) Now consider a triple (), h, z) where h : ) → X is such that h∗ ([)]) = B and where the following stability condition is satisfied: the restriction hi of the map h to )i is nonconstant unless )i contains at least three special points. (By definition, special points are either points of intersection with other components or marked points.) A stable map σ = [), h, z] in class B ∈ H2 (X, Z) is an equivalence class of such triples, where (), h, z ) ≡ (), h ◦ γ , z) if there is an element γ of the group Aut()) of all holomorphic self-maps of ) such that γ (zi ) = zi for all i. For example, if ) has

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

143

only one component and there are no marked points, then (), h) ≡ (), h ◦ γ ) for all γ ∈ Aut(S 2 ) = PSL(2, C). Thus stable maps are unparametrized. We may think of the triple (), h, z) as a parametrized stable map. We almost always consider only stable maps that are J -holomorphic for some J . If necessary, we include J in the notation, writing elements as σ = [), h, z, J ], but often J is understood. Note that some stable maps σ = [), h, z, J ] have a nontrivial reparametrization group 5σ . Given a representative (), h, z, J ) of σ , this group may be defined as   5σ = γ ∈ Aut()) : h ◦ γ = h, γ (zi ) = zi , 1 ≤ i ≤ p . It is finite because of the stability condition. The points where this reparametrization group 5σ is nontrivial are singular or orbifold points of the moduli space. Here is an example where it is nontrivial. Example 2.3. Let ) have three components, with )2 and )3 both intersecting )1 , and let z1 be a marked point on )1 . Then we can allow h1 to be constant without violating stability. If, in addition, h2 and h3 have the same image curve, there is an automorphism that interchanges )2 and )3 . Since nearby stable maps do not have this extra symmetry, [), h, z1 ] is a singular point in its moduli space. However, because marked points are labelled, there is no such automorphism if we put one marked point z2 on )2 and another z3 at the corresponding point on )3 , that is, so that h2 (z2 ) = h3 (z3 ). One can also destroy this automorphism by adding just one marked point z0 to [), h, z1 ] anywhere on )2 or )3 . Definition 2.4. For k ≥ 0, we define ᏹ(A − kF, J ) to be the space of all J holomorphic stable maps σ = [), h, J ] in class A − kF . Further, given any subset ᏷ of ᏶, we write ᏹ(A − kF, ᏷) = ∪J ∈᏷ ᏹ(A − kF, J ). p

It follows from the proof of Lemma 1.1 that the domain ) = ∪i=0 )i of σ ∈ ᏹ(A − kF, ᏶) contains a unique component that is mapped to a curve in some class A−mF , where m ≥ k. We call this component the stem of ) and label it )0 . Thus ᏹ(A − kF, ᏶m ) is the moduli space of all curves whose stems lie in class A − mF . Note that ) − )0 has a finite number of connected components called branches. If h0 is parametrized as a section, a branch Bw that is attached to )0 = S 2 at the point w is mapped into the fiber πJ−1 (w). In particular, distinct branches are mapped to distinct fibers. The moduli spaces ᏹ(A − kF, J ) and ᏹ(A − kF, ᏶) have natural stratifications in which each stratum is defined by fixing the topological type of the pair (), z) and the homology classes [h∗ ()i )] of the components. Observe that the class A − mF of the stem is fixed on each stratum ᏿ in ᏹ(A − kF, ᏶). Hence there is a projection ᏿ −→ ᏶m ,

whose fiber at J ∈ ᏶m is some stratum of ᏹ(A − kF, J ). Usually, in order to have a moduli space with a nice structure, one needs to consider perturbed J -holomorphic

144

DUSA MCDUFF

curves. But, because we are working with genus 0 curves in dimension 4, the work of Hofer-Lizan-Sikorav [HLS] shows that all J -holomorphic curves are essentially regular. In particular, all curves representing some multiple mF of the fiber class are regular. Therefore, each stratum of ᏹ(A − kF, J ) is a (finite-dimensional) manifold. The following result is an immediate consequence of Lemma 2.2. Lemma 2.5. Each stratum ᏿ of ᏹ(A − kF, ᏶) is a manifold, and the projection ᏿ → ᏶m is a locally trivial fibration. Definition 2.6. When k ≥ 1, we set ᏹk = ᏹ(A−kF, ᏶). Further, ᏹ0 = ᏹ(A, x0 , ᏶) is the space of all stable maps [), h, z, J ] where [), h, z] is a J -holomorphic stable map in class A with one marked point z such that h(z) = x0 . In the next section we show how to fit the strata of ᏹk together by gluing to form an orbifold structure on ᏹk itself. 2.3. Gluing. In this section we describe the structure of a neighborhood ᏺ(σ ) ⊂ ᏹk of a single point σ ∈ ᏹ(A − kF, ᏶m ). Suppose that σ = [), h, J ], and order the

components )i of ) so that )0 is the stem and so that the union ∪i≤ )i is connected for all . Then each )i , i > 0, is attached to a unique component )ji , ji < i by identifying some point wi ∈ )i with a point zi ∈ )ji . At each such intersection point, consider the gluing parameter: ai ∈ Twi )i ⊗C Tzi )ji . The basic process of gluing allows one to resolve the singularity of ) at the node wi = zi by replacing the component )i by a disc attached to )ji − nbhd(zi ) and suitably altering the map h. As we now explain, there is a two-dimensional family of ways of doing this that is parametrized by (small) ai . Proposition 2.7. Each σ ∈ ᏹ(A−kF, ᏶m ) has a neighborhood ᏺ(σ ) in ᏹk that is a product ᐁ᏿ (σ )×(ᏺ(Vσ )/ 5σ ) where ᐁ᏿ (σ ) ⊂ ᏹ(A−kF, ᏶m ) is a small neighborhood of σ in its stratum ᏿ and where ᏺ(Vσ ) is a small 5σ -invariant neighborhood of 0 in the space of gluing parameters,  Vσ = Twi )i ⊗C Tzi )ji . i>0

Proof. The proof is an adaptation of standard arguments in the theory of stable maps. The only new point is that the stem components are not regular, so that when one does any gluing that involves this component, one has to allow J to vary in a normal slice ᏷J to the submanifold ᏶m at J . This analytic detail is explained in §4.2. What we do here is describe the topological aspect of the proof. First of all, let us describe the process of gluing. Given a ∈ Vσ , the idea is first to construct a (parametrized) approximately J -holomorphic stable map ()a , ha , J ) on a glued domain )a and then to perturb ha and J , using a Newton process, to a Ja holomorphic map ha : )a → X in ᏹ(A−kF, ᏷J ). We describe the first step in some

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

145

detail here since it is used in §3. The analytic arguments needed for the second step are postponed to §4. It is important to note that one must choose a parametrization of the stable map before one can glue. We denote gluing on the parametrized level by Ᏻ˜ , and we write Ᏻ for the appropriate induced map on the unparametrized level. Thus
   ˜ (), h, J ; a) = )a , ha , Ja , Ᏻ Ᏻ(), h, J ; a) = )a , ha , Ja . The glued domain )a is constructed as follows. For each i such that ai  = 0, cut out a small open disc IntDwi (ri ) in )i centered at wi and a similar disc IntDzi (ri ) in Bji where ri2 = |ai |, and then glue the boundaries of these discs together with a twist prescribed by the argument of ai . The Riemann surface )a is the result of performing this operation for each i with ai  = 0. (When ai = 0, one simply leaves the component )i alone.) To be more precise, consider gluing w ∈ )0 to z ∈ )1 . Take a Kähler metric on )0 that is flat near w and identify the disc Dw (r) isometrically with the disc of radius r in the tangent space Tw = Tw ()0 ) via the exponential map. Take a similar metric on ()1 , z). Then the gluing ∂Dw (r) → ∂Dz (r) may be considered as the restriction of the map :a : Tw − {0} −→ Tz − {0} that is defined for x ∈ Tw by the requirement x ⊗ :a (x) = a,

x ∈ Tw .

Tz with C, :a is given by the Thus, with respect to chosen identifications of Tw and√ formula x → a/x and so takes the circle of radius r = |a| into itself. This describes the glued domain )a as a point set. It remains to put a metric on )a in order to make it a Riemann surface. By hypothesis, the original metrics on )0 , )1 are flat near w and z and so may be identified with the flat metric |dx|2 on C. Since
 |a|2 :a∗ |dx|2 = 4 |dx|2 , |x| :a (|dx|2 ) = |dx|2 on the circle |x| = r. Hence we may choose a function χr : (0, ∞) → (0, ∞) so that the metric χr (|x|)|dx|2 is invariant by :a and so that χr (s) = 1 when s > (1 + ε)r. Then patch together the given metrics on )0 − Dz (2r) and )1 − Dz (2r) via χr (|x|)|dx|2 . In §3 we need to understand what happens as a rotates around the origin. It is not hard to check that if we write aθ = eiθ aw ⊗ az , where aw ∈ Tw , az ∈ Tz are fixed, then
 :aθ eiθ pw = pz for all θ, where pw ∈ ∂Dw (r), pz ∈ ∂Dz (r). The next step is to define the approximately holomorphic map (or pregluing) ha : )a → X for sufficiently small |a|. The map ha equals h away from the discs

146

DUSA MCDUFF

Dwi (ri ), Dzi (ri ), and elsewhere it is defined by using cutoff functions that depend only on |a|. To describe the deformation of ha to a holomorphic map, one needs to use analytical arguments. Hence further details are postponed until §4. We are now in a position to describe a neighborhood of σ . It is convenient to think of Vσ as the direct sum Vσ ⊕Vσ where Vσ consists of the summands Twi )i ⊗C Tzi )ji with ji = 0 and Vσ of the rest. Note that the obvious action of 5σ on Vσ preserves this splitting. (It is tempting to think that the induced action on Vσ is trivial since the elements of 5σ act trivially on the stem. However, this need not be so since they may rotate branch components that are attached to the stem.) If we glue at points parametrized by a  ∈ Vσ , then the corresponding curves lie in some branch and are regular. Hence the result of gluing is a J -holomorphic curve (i.e., there is no need to perturb J ). Further, because the gluing map Ᏻ˜ is 5σ -equivariant, a normal slice in ᏹ(A − kF, ᏶m ) to the stratum ᏿ at σ has the form 
 ᏺ Vσ  ᐁ (σ ) = ᐁ᏿ (σ ) × , 5σ where ᏺ(V ) denotes a neighborhood of 0 in the vector space V . When we glue with elements from Vσ , the homology class of the stem changes, and so the result cannot be J -holomorphic since J ∈ ᏶m . We show in Proposition 4.4 that if ᏷J is a normal slice to the submanifold ᏶m at J , then for sufficiently small a ∈ Vσ , the approximately holomorphic map ha : )a → X deforms to a unique Ja -holomorphic map Ᏻ˜ (hσ , a) with Ja ∈ ᏷J . Therefore, for each element σ1 = [), h1 , J1 ] ∈ ᐁ (σ ), there is a homeomorphism from some neighborhood ᏺ(Vσ1 ) onto a slice in ᏹ(A−kF, ᏷J1 ) that meets ᏹ(A−kF, ᏶m ) transversally at σ1 . Moreover, if ᐁ (σ ) is sufficiently small, the spaces Vσ1 can all be identified with Vσ , and it follows from the proof of Proposition 4.4 that the neighborhoods ᏺ(Vσ1 ) can be taken to have uniform size and so may all be identified. Hence the neighborhood ᏺ(σ ) projects to ᐁ (σ ) with fiber at σ1 equal to ᏺ(Vσ )/ 5σ1 . In general, the groups 5σ1 are subgroups of 5σ that vary with σ1 ; in fact, they equal the stabilizer of the corresponding gluing parameter a  ∈ Vσ . However, since elements of ᐁ᏿ (σ ) lie in the same stratum, the group 5σ itself does not vary as σ varies in ᐁ᏿ (σ ). It is now easy to check that the composite map ᏺ(σ ) −→ ᐁ (σ ) −→ ᐁ᏿ (σ )

has fiber ᏺ(Vσ )/ 5σ as claimed. 2.4. Moduli spaces and the stratification of ᏶. Since each stable J -curve in class A−kF has exactly one component in some class A−mF with m ≥ k, the projection πk : ᏹ(A − kF, ᏶) → ᏶ has image ᏶k . Consider the inverse image

 ᏹ A − kF, ᏶m = πk−1 ᏶m . The next result shows that we can get a handle on the structure of ᏶k by looking at

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

147

the spaces ᏹ(A − kF, ᏶m ). Proposition 2.8. When k > 0, the projection
 πk : ᏹ A − kF, ᏶m −→ ᏶m is a locally trivial fibration whose fiber ᏲJ (m − k) at J is the space of all stable J curves [), h] in class A−kF that have as one component the unique J -holomorphic curve J in class A − mF . In particular, ᏲJ (m − k) is a stratified space with strata that are manifolds of (real) dimension less than or equal to 4(m − k). Its diffeomorphism type depends only on m − k. Proof. Let us look at the structure of ᏲJ (m − k) = πk−1 (J ). The stem of each element [), h, J ] ∈ ᏲJ (m−k) is mapped to the unique J -curve J in class A−mF . Fix this component further by supposing that it is parametrized as a section of the fibration πJ : X → S 2 (where πJ is as in Lemma 1.1). We may divide the fiber ᏲJ (m − k) into disjoint sets ᐆᏰ,J , each parametrized by a fixed decomposition Ᏸ of m − k into a sum d1 + · · · + dp of unordered positive numbers. The elements of ᐆᏰ,J are those with p branches Bw1 , . . . , Bwp where h∗ [Bwi ] = di [F ]. Thus ᐆᏰ,J maps onto the configuration space of p distinct (unordered) points in S 2 labelled by the positive integers d1 , . . . , dp with sum m − k. Moreover, this map is a fibration with fiber equal to the product P


 ᏹ0,1 S 2 , q, di , i=1

where ᏹ0,1 (S 2 , q, d) is the space of J -holomorphic stable maps into S 2 of degree d and with one marked point z such that h(z) = q. (This point q is where the branch is attached to J .) According to the general theory, ᏹ0,1 (S 2 , q, d) is an orbifold of real dimension 4(d − 1). It follows easily that ᐆᏰ,J is an orbifold of real dimension 4(m − k) − 2p. It remains to be understood how the different sets ᐆᏰ,J fit together, that is, what happens when two or more of the points wi come together. This may be described by suitable gluing parameters as in Proposition 2.7. The result follows. (For more details, see any reference on stable maps, e.g., [FO], [LiT], and [LiuT1]. An example is worked out in §3.2.4 below.) Note. For an analogous statement when k = 0, see Proposition 3.5. Next, we describe the structure of a neighborhood of ᏹ(A − kF, ᏶m ) in ᏹk = ᏹ(A − kF, ᏶). We write ᐆJ for the fiber ᏲJ (m − k) of πk that was considered above and set ᐆ=

 J ∈᏶m

ᐆJ ,

ᐆᏰ =



ᐆᏰ,J .

J ∈᏶m

(The letter ᐆ is used here because ᐆ is the zero section of the space of gluing parameters ᐂ constructed below.) Consider an element σ = [), h, J ] that lies in a

148

DUSA MCDUFF

substratum ᐆ᏿ of ᐆᏰ where Ᏸ = d1 + · · · + dp . Then ) has p branches B1 , . . . , Bp that are attached at the distinct points w1 , . . . , wp ∈ )0 . Let zi be the point in Bi that is identified with wi ∈ )0 , and define Vσ =

p 

Tzi Bi ⊗C Twi )0 .

i=1

As explained in Proposition 2.7, the gluing parameters a ∈ Vσ (when quotiented out by 5σ ) parametrize a normal slice to ᐆᏰ at σ . (Note that previously Vσ was called Vσ .) We now want to show how to fit these vector spaces together to form the fibers of an orbibundle2 over ᐆᏰ . Here we must incorporate twisting that arises from the fact that gluing takes place on the space of parametrized stable maps. Since this is an important point, we dwell on it at some length. For the sake of clarity, in the next few paragraphs we denote parametrized stable maps by σ˜ = (), h) and the usual (unparametrized) maps by σ = [), h]. Further, 5σ˜ denotes the corresponding realization of the group 5σ as a subgroup of Aut()). Recall that X is identified with S 2 ×S 2 in such a way that the fibration πJ : X → S 2 , whose fibers are the J -holomorphic F -curves, is simply given by projection onto the first factor. Hence each such fiber has a given identification with S 2 . Further, we assume that the stem hσ,0 : )σ,0 → J is parametrized as a section z → (z, ρ(z)). Hence we only have to choose parametrizations of each branch. Since each branch component has at least one special point, its automorphism group either is trivial or has the homotopy type of S 1 . Let Aut ()) be the subgroup of Aut()) consisting of automorphisms that are the identity on the stem. Then the identity component of Aut ()) is homotopy equivalent to a torus T k(᏿) . (Here ᏿ is the label for the stratum containing σ .) Let g be a 5σ˜ -invariant metric on the domain ) that is also invariant under some action of the torus T k(᏿) . Definition 2.9. The group AutK ()) is defined to be the (compact) subgroup of the isometry group of (), g) generated by 5σ˜ and T k(᏿) . Note that 5σ˜ is the semidirect product of a subgroup 5σ˜ of T k(᏿) with a subgroup 5σ˜ that permutes the components of each branch. Further, Aut K ()) is a deformation retract of the subgroup p−1 (5σ˜ ) of Aut()), where we consider 5σ˜ as a subgroup of π0 (Aut())), and
 p : Aut()) −→ π0 Aut()) is the projection. For a further discussion, see §4.2.4. Let us first consider a fixed J ∈ ᏶k . It follows from the above discussion that on each stratum ᐆ᏿,J , there is a principal bundle rank k orbibundle π : E → Y over an orbifold Y has the following local structure. Suppose that σ ∈ Y has local chart U ⊂ U˜ / 5σ where the uniformizer U˜ is a subset of Rn . Then π −1 (U ) has the form U˜ × Rk / 5σ where the action of 5σ on Rn × Rk lifts that on Rn and is linear on Rk . There is an obvious compatibility condition between charts: see [FO, §2]. 2A

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

149

para

ᐆ᏿,J −→ ᐆ᏿,J

with fiber Aut K ()) such that the elements of ᐆ᏿,J are parametrized stable maps σ˜ = (), h). Since the space Vσ˜ of gluing parameters at σ˜ is made from tangent spaces to ), there is a well-defined bundle para

para

para

ᐂ᏿,J −→ ᐆ᏿,J

with fiber Vσ˜ . Further, the action of the reparametrization group Aut K ()) lifts to para para ᐂ᏿,J , and we define ᐂ᏿,J to be the quotient ᐂ᏿,J / Aut K ()). Thus there is a commutative diagram para / ᐂ᏿,J ᐂ ᏿,J

 para ᐆ᏿,J

 / ᐆ᏿,J,

where the right-hand vertical map is an orbibundle with fiber Vσ˜ / 5σ˜ . Now consider the space ᐆᏰ,J = ∪᏿⊂Ᏸ ᐆ᏿,J . The local topological structure of ᐆᏰ,J is given by gluing parameters as in Proposition 2.7. Observe that every J is regular for the branch components so that the necessary gluing operations can be performed para para keeping J fixed. The spaces ᐆᏰ,J and ᐂᏰ,J are defined similarly, and clearly there para para is a vector bundle ᐂᏰ,J → ᐆᏰ,J . We want to see that the union  ᐂᏰ,J = ᐂ᏿,J ᏿∈Ᏸ

has the structure of an orbibundle over ᐆᏰ,J . The point here is that the groups AutK ()) change dimension as σ˜ moves from stratum to stratum. Hence we need to see that the local gluing construction that fits the different strata in ᐂᏰ,J together is compatible with the group actions. We show in §4.2.4 that the gluing map Ᏻ can be defined at the point σ˜ to be AutK ())-invariant. More precisely,    
)a , Ᏻ˜ (hσ , a) = )θ·a , Ᏻ˜ hσ ◦ θ −1 , θ · a , where Ᏻ˜ (hσ , a) is the result of gluing the map hσ with parameters a. In the situation considered here, we divide the set of gluing parameters at σ˜ into two, and we write a = (ab , as ) where ab are the gluing parameters at intersections of branch components para and as are those involving the stem component. As hσ moves within ᐆᏰ,J , we glue along ab , considering as to be part of the p-dimensional fiber Vσ . (Recall that p is the number of elements of the decomposition Ᏸ = d1 + · · · + dp .) Moreover, if σ˜  = ()a , Ᏻ˜ (hσ , ab )), Lemma 4.9(ii) shows that the parametrized gluing map Ᏻ˜ b

can be constructed to be compatible with the actions of the groups Aut K ()) and

150

DUSA MCDUFF

AutK ()  ) on the fibers Vσ and Vσ  of ᐂᏰ,J at σ˜ , σ˜  . It follows without difficulty that the quotient ᐂᏰ,J −→ ᐆᏰ,J para

is an orbibundle with generic fiber Cp . Finally, one forms spaces  ᐂJ = ᐂᏰ,J ,

ᐂ=



ᐂJ ,

J ∈᏶k

Ᏸ

whose local structure is also described by appropriate gluing parameters as above. Forgetting the gluing parameters gives projections ᐂJ −→ ᐆJ = ᏲJ (m − k),

ᐂ −→ ᐆ = Ᏺ(m − k),

and ᐆJ , ᐆ embed in ᐂJ and ᐂ as the zero sections. The map ᐂJ → ᐆJ preserves the stratifications of both spaces. However, it is no longer an orbibundle since the dimension of the fiber Vσ depends on Ᏸ. In fact, the way that the different sets ᐂᏰ,J are fitted together is best thought of as a kind of plumbing; see §3.2.4. Example 2.10. Everything is greatly simplified when m−k = 1. Here there is only one decomposition Ᏸ and the space ᐆᏰ,J consists of just one stratum diffeomorphic to para S 2 . Moreover, the bundle ᐆᏰ,J → ᐆᏰ,J has a section with the following description. Choose J ∈ ᏶k+1 so that πJ : X → S 2 is the standard projection onto the first factor and so that the graph h0 of the map ρk : S 2 → S 2 of degree −(k +1) is J -holomorphic. para Let )0 , )1 be two copies of S 2 , and for each w ∈ S 2 , define ()w , hw ) ∈ ᐆᏰ,J by )w = )0 ∪w=ρk (w) )1 , hw |)0 = h0 ,

hw |)1 : z −→ (w, z).

Hence, in this case, ᐂJ is a complex line bundle over ᐆJ = S 2 . To calculate its Chern class, observe that ᐂJ can be identified with the space 
−k−1 Tρk (w) )1 ⊗ Tw ()0 ) = T S 2 ⊗ T S2, w∈S 2

and so ᐂJ has Chern class −2k. The following result is proved in §4. Proposition 2.11. There is a neighborhood ᏺᐂ (ᐆ) of ᐆ in ᐂ and a gluing map Ᏻ : ᏺᐂ (ᐆ) −→ ᏹ(A − kF, ᏶)

that maps ᏺᐂ (ᐆ) homeomorphically onto a neighborhood of ᏹ(A − kF, ᏶m ) in ᏹ(A − kF, ᏶). It follows from the construction of Ᏻ : ᏺᐂ (ᐆ) → ᏹ(A − kF, ᏶m ) outlined in Proposition 2.7 that the stem of the glued map Ᏻ(σ, a) lies in the class

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

151



 A− m− ni F, i

where the indices i label the branches Bi of σ and ni is defined as follows. If ai = 0, then ni = 0. Otherwise, if )ji is the component of Bi that meets )0 , then ni is the multiplicity of hji , that is, [h()ji )] = ni F . Let ᏺp denote the set of all elements (σ, a) ∈ ᏺᐂ (ᐆ) such that the stem of the glued map lies in class A − pF . In other words,
−1 ᏺp = πk ◦ Ᏻ ᏶p . Clearly, ᏺp is a union of strata in the stratified space ᏺᐂ (ᐆ). Further, when k > 0, the map πk : Ᏻ(ᏺp ) → ᏶p is a fibration with fiber Ᏺ(p − k). The next proposition follows immediately from Proposition 2.8. Proposition 2.12. The link ᏸm,k is the finite-dimensional stratified space obtained from the link of ᐆJ in ᐂJ by collapsing the fibers of the projections ᐂJ ∩ ᏺp → ᏶p to single points. Proof of Proposition 1.4. We have to show that the link ᏸk+1, k is the lens space L(2k, 1). We saw in Example 2.10 that ᐂJ is a line bundle with Chern class −2k. In this case, there is only one nontrivial stratum in ᏺᐂ (ᐆ), namely, ᏺk , which is the complement of the zero section. Moreover, the map πk ◦ Ᏻ is clearly injective. Hence, by the above lemma, ᏸk+1, k is simply the unit sphere bundle of ᐂJ and so is a lens space as claimed. 3. The link ᏸ2,0 of ᏶2 in ᏶0 . In this section, we illustrate Proposition 2.12 by calculating the link ᏸ2,0 . We know from Lemma 2.2 that ᏸ2,0 = S 5 . The general theory of §2 implies that ᏸ2,0 can be obtained from the link ᏸᐆ of the zero section ᐆJ in the stratified space ᐂJ of gluing data by collapsing certain strata. When looked at from this point of view, the S 5 appears in quite a complicated way, which was described in Theorem 1.5. We begin here by explaining the plumbing construction and then discussing how this relates to ᏸᐆ . 3.1. Some topology. Recall that S(LP ) → ᏼ(ᏻ(k)⊕ ᏻ(m)) is the unit circle bundle of the canonical line bundle LP over the projectivization ᏼ(ᏻ(k) ⊕ ᏻ(m)). Lemma 3.1. The bundle S(LP ) → ᏼ(ᏻ(−1) ⊕ C) can be identified with the pullback of the canonical circle bundle S(Lcan ) → CP 2 over the blow-down map CP 2 #CP 2 → CP 2 . Proof. It is well known that ᏼ(ᏻ(−1) ⊕ C) can be identified with CP 2 #CP 2 → CP 2 . Indeed, the section S− = ᏼ({0} ⊕ C) has self-intersection −1, while S+ = ᏼ(ᏻ(−1) ⊕ {0}) has self-intersection 1. Further, the circle bundle S(LP ) is trivial over S− and has Euler class −1 over S+ and over the fiber class. The result follows.

152

DUSA MCDUFF

The space we are interested in is formed by plumbing a rank 2 bundle E → S 2 to a line bundle L → Y where dim(Y ) = 4. This plumbing E ≡ ! L is the space obtained from the unit disc bundles D(E) → S 2 and D(L) → Y by identifying the inverse images of discs D 2 , D 4 on the two bases in the obvious way: The disc fibers of D(E) → S 2 are identified with flat sections of D(L) over D 4 , and flat sections of D(E) over D 2 are identified with fibers of D(L). There is a corresponding plumbing S(E) ≡ ! S(L) of the two sphere bundles S 3 → S(E) → S 2 and S 1 → S(L) → Y , obtained by cutting out the inverse images of open discs in the two bases and appropriately gluing the boundaries. The resulting space S(E) ≡ ! S(L) is the link of the core S 2 ∪ Y in the plumbed bundle E ≡ ! L. Lemma 3.2. Let Lcan → CP 2 be the canonical line bundle, and let E = ᏻ(k) ⊕ ᏻ(m). Then E ≡ ! Lcan may be identified with the blow-up of ᏻ(k + 1) ⊕ ᏻ(m + 1) at a point on its zero section. Hence
 S(E) ≡ ! S(Lcan ) = S ᏻ(k + 1) ⊕ ᏻ(m + 1) . Proof. First consider the structure of the blow-up  C3 of C3 = C×C2 at the origin. 2 The fibration π : C × C → C induces a fibration  π : C3 −→ C. π −1 (0) Clearly, the inverse image  π −1 (z) of each point z  = 0 is a copy of C2 , while  is the union of the exceptional divisor together with the set of lines in the original fiber π −1 (0). Let λ be the line in C × C2 through the origin and the point (1, a, b). Lift λ to the blow-up and consider its intersection with

  π −1 S 1 = π −1 S 1 ⊂ C × C2 , where S 1 is the unit circle in C. This intersection consists of the points (eit , eit a, eit b); hence it is these circles (rather than the circles (eit , a, b)) that bound discs in the blow-up. Therefore, if we think of the blow-up  C3 as the plumbing of the bundle π : 2 C × C → C with Lcan , the original trivialization of π differs from the trivialization (or product structure) near π −1 (0) that is used to construct the plumbing. Now recall that ᏻ(k) = D + × C ∪α D − × C, where D + , D − are 2-discs, with D + oriented positively and D − oriented negatively, and where the gluing map α is given by

 α : ∂D + × C −→ ∂D − × C : eit , w −→ eit , e− ikt w . It follows easily that the blow-up of D(ᏻ(k + 1) ⊕ ᏻ(m + 1)) at a point on its zero section is obtained by plumbing the disc bundle D(ᏻ(k) ⊕ ᏻ(m)) with D(Lcan ). This proves the first statement. The second statement is then immediate. We are interested in plumbing not with Lcan → CP 2 but with a particular singular

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

153

line bundle (or orbibundle) LY → Y . This means that the unit circle bundle S(LY ) → Y is a Seifert fibration with a finite number of singular (or multiple) fibers. In our case, there is an S 1 -action on S(LY ) such that the fibers of the map S(LY ) → Y are the S 1 -orbits. In fact, we can identify S(LY ) with S 5 in such a way that the S 1 action is
 θ · (x, y, z) = eiθ x, eiθ y, e2iθ z , x, y, z ∈ C. Thus there is one singular fiber that goes through the point (0, 0, 1). All other fibers F are regular. For each such F , there is a diffeomorphism of S 5 that takes F to the circle γ0 = (eiθ , eiθ , 0). Identify a neighborhood of γ0 with S 1 × D 4 in such a way that S 5 = S 1 × D4 ∪ D2 × S 3, with the identity map of S 1 × S 3 as gluing map. Then, in these coordinates near γ0 , the fibers of S(LY ) are (diffeomorphic to) the circles  iθ 
 e 0 1 4 . γx = θ, Aθ (x) ∈ S × D : Aθ = 0 e2iθ By way of contrast, the fibers of S 5 with the Hopf fibration have neighborhoods fibered by the circles  iθ 
 e 0   1 4  γx = θ, Aθ (x) ∈ S × D : Aθ = . 0 eiθ The next result shows that plumbing with S(L) is a kind of twisted blow-up. Proposition 3.3. Let LY → Y be the orbibundle described in the previous paragraph. Then the manifold obtained by plumbing S(ᏻ(k) ⊕ ᏻ(m)) with a regular fiber of S(LY ) is diffeomorphic to S(ᏻ(k + 2) ⊕ ᏻ(m + 1)). Proof. We may think of plumbing as the result of a surgery that matches the flat circles S 1 × pt in the copy of S 1 × S 3 in S(ᏻ(k) ⊕ ᏻ(m)) with the circles γx in the neighborhood of a regular fiber γ0 of S(LY ). We would get the same result if we matched the circles  −iθ 
 e 0 δx = θ, Aθ (x) ∈ S 1 × S 3 : Aθ = 0 1 in S 1 × S 3 ⊂ S(ᏻ(k) ⊕ ᏻ(m)) with the circles γx in the standard (Hopf) S 5 . But if we trivialize the boundary of S(ᏻ(k) ⊕ ᏻ(m)) − D 2 × S 3 by the circles δx , we get the same result as if we trivialized the boundary of S(ᏻ(k + 1) ⊕ ᏻ(m)) − D 2 × S 3 in the usual way by flat circles. Thus

 S ᏻ(k) ⊕ ᏻ(m) ≡ ! S(LY ) = S ᏻ(k + 1) ⊕ ᏻ(m) ≡ ! S(Lcan ) 
= S ᏻ(k + 2) ⊕ ᏻ(m + 1) .

154

DUSA MCDUFF

There is a question of orientation here: Do we have to add or subtract 1 from k to compensate for the extra twisting in S(LY )? We can check that it is correct to add 1 by using the present approach to give an alternate proof of the previous lemma. For if we completely untwisted the circles in the neighborhood of γ0 (thereby increasing the twisting of the other side by an additional 2), we would be doing the trivial surgery in which the attaching map is the identity. Note also that because the sum ᏻ(k) ⊕ ᏻ(m) depends only on k + m, we could equally well have put the extra twist on the other factor. 3.2. Structure of the pair (ᐂJ , ᐆJ ). Our aim is to prove the following proposition where ᐂJ is the space of gluing parameters for a fixed J ∈ ᏶2 that describes the link of the space of (A − 2F )-curves in the space of (pointed) A-curves. Proposition 3.4. The link ᏸᐆ of the zero section ᐆJ in the stratified space ᐂJ is constructed by plumbing S(ᏻ(−3) ⊕ ᏻ(−1)) to S(LY ). Hence
 ᏸᐆ = S ᏻ(−1) ⊕ C . We are now not quite in the situation described in Proposition 2.8 because we are including the open stratum ᏶0 of ᏶. This means that we have to replace the space ᏹk = ᏹ(A − kF, ᏶) by a space ᏹ0 of curves of class A that go through the fixed point x0 . Since we are interested in working out the structure of the fiber of the projection ᏹ0 → ᏶ at a point J ∈ ᏶2 , we choose x0 so that it does not lie on the unique J -holomorphic (A − 2F )-curve J , and then we define ᏹ0 to be the space ᏹ(A, x0 , ᏶) in Definition 2.6. Let π0 denote the projection π0 : ᏹ0 −→ ᏶ and set ᏹ0 (᏶m ) = π0−1 (᏶m ) as before. It is not hard to see that the following analog of Proposition 2.8 holds. Proposition 3.5. (i) Let J ∈ ᏶m be any almost complex structure such that the unique J -holomorphic (A−mF )-curve J does not go through x0 . Then the projection πk : ᏹ0 (᏶m ) −→ ᏶m is a locally trivial fibration near J whose fiber ᏲJ (0, m) is the space of all stable J -curves [), u] in class A that have J as one component and go through x0 . In particular, ᏲJ (0, m) is a stratified space whose strata are orbifolds of (real) dimension less than or equal to 4m − 2. (ii) The singular fibers of πk : ᏹ0 (᏶m ) → ᏶m occur at points J for which x0 ∈ J . For such J , πk−1 (J ) can be identified with the space ᏲJ (m) described in Proposition 2.8. As before, we now construct a pair (ᐂJ , ᐆJ ) that describes a neighborhood of ᏹ0 (᏶m ) in ᏹ0 . We concentrate on the case m = 2 and suppose that x0 ∈ / J . We

further normalize J by requiring that the projection πJ along the J -holomorphic

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

)3

)2 )3

155 )2

z0

z0

a0 b

b

)1

a1

a w )0

σ∗ in ᐆ1,J

)0

σ in ᐆ2,J Figure 1

F -curves is simply the projection onto the first factor S 2 . We write q0 = πJ (x0 ). 3.2.1. The bundle ᐂ2,J → ᐆ2,J . Observe first that ᐆJ has two subsets: ᐆ1,J consisting of all stable A-maps [), z0 , h] that are the union of the (A − 2F )-curve J with a double covering of the fiber F0 through x0 , and ᐆ2,J consisting of all stable A-maps [), z0 , h] that are the union of J with two distinct fibers. We call the ᐆi,J strata. This is accurate as far as ᐆ2,J is concerned, but strictly speaking, ᐆ1,J is a union of strata. (Recall that the strata are determined by the topological type of the marked domain [), z0 ], and the homology class of the images of its components under h.) Let us first consider ᐆ2,J . Since h(z0 ) = x0 always, one of the two fibers has to be F0 , and the other moves. Therefore, the stratum ᐆ2,J maps onto S 2 − {q0 }. It is convenient to compactify ᐆ2,J by adding a point σ∗ that projects to q0 . The domain ) of σ∗ has four components with )0 , )2 , )3 , all meeting )1 and a marked point z0 ∈ )3 (see Figure 1). The map h0 : )0 → J parametrizes J as a section, h1 takes )1 onto the point F0 ∩ J , and h2 , h3 have image F0 with h3 (z0 ) = x0 . The argument of Example 2.10 gives the following result, where ᐂ2,J is extended over σ∗ using the trivialization given by the gluing coordinates a, b of Figure 1. Lemma 3.6. The space ᐂ2,J of gluing parameters over ᐆ2,J ∪ {σ∗ } = S 2 is the bundle ᏻ(−2) ⊕ C. The structure of the bundle ᐂ1,J → ᐆ1,J is more complicated, and we start by looking at its base. 3.2.2. The stratum ᐆ1,J . Let p0 , p1 be two distinct points on F0 ≡ S 2 , with p0 = x0 and p1 = J ∩ F0 . Then ᐆ1,J is the orbifold 
ᐆ1,J = Y = ᏹ0,2 S 2 , p0 , p1 , 2 of all stable maps to S 2 with two marked points z0 , z1 that are in the class 2[S 2 ] and

156

DUSA MCDUFF

are such that h(z0 ) = p0 , h(z1 ) = p1 . We also need to consider the space ᏹ0,0 (S 2 , 2) of genus 0 stable maps of degree 2 into S 2 that have no marked points, and the space 
Y˜ = ᏹ0,3 S 2 , p0 , p1 , p2 , 2 of all degree 2 stable maps to S 2 with three marked points z0 , z1 , z2 such that h(z0 ) = p0 , h(z1 ) = p1 , h(z2 ) = p2 . Lemma 3.7. (i) ᏹ0,0 (S 2 , 2) is a smooth manifold diffeomorphic to CP 2 . (ii) Y˜ = ᏹ0,3 (S 2 , p0 , p1 , p2 , 2) is a smooth manifold diffeomorphic to CP 2 . (iii) The forgetful map f : Y˜ → Y may be identified with the 2-fold cover that quotients CP 2 by the involution τ : [x : y : z] → [x : y : −z]. In particular, Y is smooth except at the point σ01 = f ([0 : 0 : 1]) that has the local chart C2 /(x, y) = (−x, −y). This point σ01 is the stable map [S 2 , h, z0 , z1 ] where the critical values of h are at p0 and p1 . Proof. (i) The space ᏹ0,0 (S 2 , 2) has two strata. The first, ᏿1 , consists of selfmaps of S 2 of degree 2, and the second, ᏿2 , consists of maps whose domain has two components, each taken into S 2 by a map of degree 1. The equivalence relation on each stratum is given by precomposition with a holomorphic self-map of the domain. It is not hard to check that each equivalence class of maps in ᏿1 is uniquely determined by its two critical values (or branch points). Since these can be any pair of distinct points, ᏿1 is diffeomorphic to the set of unordered pairs of distinct points in S 2 . On the other hand, there is one element σw of ᏿2 for each point w ∈ S 2 , the correspondence being given by taking w to be the image under h of the point of intersection of the two components. If σ{x,y} denotes the element of ᏿1 with critical values {x, y}, we claim that σ{x,y} → σw when x, y both converge to w. To see this, let h{x,y} : S 2 → S 2 be a representative of σ{x,y} and let α{x,y} be the shortest geodesic from x to y. (We assume that x, y are close to w.) Then h−1 {x,y} (α{x,y} ) is a circle γ{x,y} through the critical points of h{x,y} . This is obvious if h{x,y} is chosen to have critical points at 0, ∞, and if x = 0, y = ∞ since h{x,y} is then a map of the form z → az2 . It follows in the general case because Mobius transformations take circles to circles. Hence h{x,y} takes each component of S 2 − γ{x,y} onto S 2 − α{x,y} . If we now let x, y converge to w, we see that σ{x,y} converges to σw . The above argument shows that ᏹ0,0 (S 2 , 2) is the quotient of S 2 × S 2 by the involution (x, y) → (y, x). This is well known to be CP 2 . In fact, it is easy to check that the map
   H : [x0 : x1 ], [y0 : y1 ] −→ x0 y0 : x1 y1 : x0 y1 + x1 y0 − x0 y0 − x1 y1 induces a diffeomorphism from the quotient to CP 2 . Under this identification, the stratum ᏿2 = H (diag) is the quadric (u + v + w)2 = 4uv (where we use coordinates

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

157

[u : v : w] on CP 2 ). Further, if we put p0 = [0 : 1],

p1 = [1 : 0],

p2 = [1, 1],

the set of points in ᏹ0,0 (S 2 , 2) = CP 2 consisting of maps that branch over pi is a line i , the image by H of (S 2 × pi ) ∪ pi × S 2 . Thus 0 = {u = 0},

1 = {v = 0},

2 = {w = 0}.

Note finally that all stable maps in ᏹ0,0 (S 2 , 2) are invariant by an involution; for example, the map z → z2 is invariant under the reparametrization z → −z. Since all elements have the same reparametrization group, ᏹ0,0 (S 2 , 2) is smooth. However, this is no longer the case when we add two marked points. (ii) Now consider the forgetful map

  φ30 : ᏹ0,3 S 2 , p0 , p1 , p2 , 2 −→ ᏹ0,0 S 2 , 2 . For a general point of ᏹ0,0 (S 2 , 2), that is, a point where neither branching point is at p0 , p1 or p2 , φ30 is 4-to-1. To see this, note that for i = 0, 1, 2, zi can be either of the points that get mapped to pi , which seems to give an 8-fold cover. However, because h has degree 2, h is invariant under an involution γh of S 2 that interchanges the two inverse images of a generic point. Hence the cover is 4-to-1, and the covering group is Z/2Z ⊕ Z/2Z. When just one branching point is at some pi , φ30 is 2-to-1, and when both branching points are at some pi , it is 1-to-1. This determines φ30 . In fact, with the above identification for ᏹ0,0 (S 2 , 2) = CP 2 , φ30 is the map   φ30 : CP 2 −→ CP 2 : [x : y : z] −→ x 2 : y 2 : z2 . Note that the inverse image of ᏿2 = {4uv = (u + v + w)2 } consists of the four lines x ± y ± iz = 0. These components correspond to the four different ways of arranging three points on the two components of the stable maps in ᏿2 . Note further that none of the points in ᏹ0,3 (S 2 , p0 , p1 , p2 , 2) is invariant by any reparametrization of their domains. Hence all points of this moduli space are smooth. (iii) Similar reasoning shows that the forgetful map  

φ20 : Y = ᏹ0,2 S 2 , p0 , p1 , 2 −→ ᏹ0,0 S 2 , 2 is a 2-fold cover branched over 0 ∪ 1 . Hence we may identify Y as   Y = [u : v : w : t] ∈ CP 3 : t 2 = uv ,

158

DUSA MCDUFF

where the cover φ20 : Y → CP 2 forgets t. There is one point in Y that is invariant under a reparametrization of its domain, namely, the point σ01 corresponding to the map h : S 2 → S 2 that branches at p0 and p1 . In the above coordinates on Y ,
 σ01 = φ2−1 0 ∩ 1 = [0 : 0 : 1 : 0]. It is also easy to check that 
φ32 : ᏹ0,3 S 2 , p0 , p1 , p2 , 2 = CP 2 −→ Y has the formula


   φ32 [x : y : z] = x 2 : y 2 : z2 : xy .

Since φ32 ◦ τ ([x : y : z]) = φ32 ([x : y : −z]) = φ32 ([x : y : z]), φ32 is equivalent to quotienting out by τ as claimed. 3.2.3. The bundle ᐂ1,J → ᐆ1,J . Now we consider the structure of the orbibundles of gluing parameters over Y˜ = ᏹ0,3 (S 2 , p0 , p1 , p3 , 2) and ᐆ1,J = Y = ˜ → Y˜ and the second LY → Y . In both cases, ᏹ0,2 (S 2 , p0 , p1 , 2). We call the first L the fiber at the stable map [), h, zi ] is the tangent space Tz1 ). Lemma 3.8. (i) The orbibundle L˜ → Y˜ is smooth and may be identified with the canonical line bundle Lcan over Y˜ = CP 2 . (ii) The orbibundle LY → Y is smooth except at the point σ01 . It can be identified with the quotient of Lcan by the obvious lift τ˜ of τ . (iii) The set S(LY ) of unit vectors in LY is smooth and diffeomorphic to S 5 . The orbibundle S(LY ) → Y can be identified with the quotient of S 5 by the circle action 
θ · (x, y, z) = eiθ x, eiθ y, e2iθ z . Proof. (i) Since Y˜ is smooth, the general theory implies that L˜ is smooth. Therefore, it is a line bundle over CP 2 , and to understand its structure we just have to figure out its restriction to one line. It is easiest to consider one of the lines x ± y ± iz = 0 that lie over ᏿2 . Recall that σw ∈ ᏿2 is the stable map [)w , hw ] with domain )w = S 2 ∪w=w S 2 and where h is the identity map on each component. Suppose we look at the line in Y˜ whose generic point has z1 on one component of )w and z0 , z2 on the other. Then the bundle L˜ has a natural trivialization over the set {w ∈ S 2 : w  = z0 , z1 , z2 }. It is not hard to check that this trivialization extends over the points z0 , z2 , but that one negative twist is introduced when z1 is added. The argument is very similar to the proof of Lemma 3.9 below and is left to the reader. (ii) It follows from the general theory that LY → Y is smooth over the smooth points of Y . Moreover, at σ01 = [S 2 , h], the automorphism γ : S 2 → S 2 such that h ◦ γ = h, γ (zi ) = zi acts on Tz1 S 2 by the map v → −v. (To see this, note that we can identify S 2 with C ∪ {∞} in such a way that z0 = p0 = 0, z1 = p1 = ∞. Then h(z) = z2 and γ (z) = −z.) Hence the local structure of L at σ01 is given by

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

159

quotienting the trivial bundle D 4 × C by the map (x, y) × v → (−x, −y) × −v. This is precisely the structure of the quotient of L˜ by τ˜ at the singular point. Moreover, we can identify S(LY ) with S 5 /τ globally since LY → Y pulls back to L˜ → Y˜ under the map Y˜ → Y . The quotient S 5 /τ is smooth except possibly at the fixed points (x, y, 0) of τ . Since S(LY ) is smooth at these points, S(LY ) is smooth everywhere. It may be identified with S 5 by the map    S5 5 2 2 2 S(LY ) ≡ −→ S : (x, y, z) −→ x 1 + |z| , y 1 + |z| , z . τ The last statement may be proved by noting that the formula   (x, y, z) −→ x 2 : y 2 : z : xy ∈ CP 3 defines a diffeomorphism from the orbit space of the given circle action to CP 2/τ = Y . 3.2.4. Attaching the strata. The next step is to understand how the two strata ᐂ1,J and ᐂ2,J fit together. The two zero sections ᐆ1,J and ᐆ2,J intersect at the point σ∗ (see Figure 1). Recall that the domain ) of σ∗ has four components with )0 , )2 , )3 , all meeting )1 and a marked point z0 ∈ )3 . The map h0 : )0 → J parametrizes J as a section, h1 takes )1 onto the point x1 = F0 ∩J , and h2 , h3 have image F0 with h3 (z0 ) = x0 . The stratum of ᐆ1,J containing σ∗ consists just of this one point. Hence the local coordinates of σ∗ in ᐆ1,J are given by two gluing parameters (a0 , a1 ). If we write zij for the point )i ∩ )j , these are (a0 , a1 ),

where a0 ∈ Tz12 )1 ⊗ Tz12 )2 , a1 ∈ Tz13 )1 ⊗ Tz13 )3 .

Similarly, the local coordinates for a (deleted) neighborhood of σ∗ in ᐂ1,J are (b, a0 , a1 ), where (a0 , a1 ) are as before, and b ∈ Tz01 )0 ⊗ Tz01 )1 is a gluing parameter at the point z01 , where the component )0 mapping to J is attached. On the other hand, the natural coordinates for a neighborhood of σ∗ in ᐂ2,J are triples (w, b, a) where b is a gluing parameter at the point z03 where the component )0 that maps to J is attached to the fixed fiber )3 , w is the point where the moving fiber )2 (the one not containing z0 ) is attached to )0 , and a is a gluing parameter at w. Lemma 3.9. The attaching map α at σ∗ has the form (b, a0 , a1 ) → (wb , ba0 , ba1 ), where b  = 0 and |b| is small. Here the map b → wb identifies a small neighborhood of 0 in Tz01 ) with a neighborhood of x1 in J in the obvious way. Proof. The attaching of ᐆ1,J to ᐆ2,J comes from gluing at the point z01 via the parameter b. Thus we are gluing the “ghost component” )1 to the component )0

160

DUSA MCDUFF

that maps to J in the space of stable A-curves that are holomorphic for a fixed J . (It is only when one glues at a0 or a1 that one changes the homology class of the curve J and, hence, has to change J .) In particular, we can forget the components )2 , )3 of the domain ) of σ∗ , retaining only the points z12 , z13 on )1 where they are attached. Therefore, we can consider the domain of the attaching map α to be the 2-dimensional space    )0 ∪z01 )1 , z12 , z13 , h ; b : b ∈ C = Tz01 )0 ⊗ Tz01 )1 , and its range to be the space of all elements [)0 , q0 , w, h , J ] where )0 = S 2 and w moves in a small disc about x1 . Here the map h : )0 → J is fixed and parametrizes J as a section. We can encode this by picking two points q1 , q2 in )0 that are different from q0 = πJ (x0 ) and then considering h to be the map that takes these two marked points to two other fixed points on J . Thus the attaching map α is equivalent to the following map α  that attaches different strata in the moduli space ᏹ0,4 (S 2 ) of four marked points on S 2 : 
  

 α  : )0 ∪ )1 , q1 , q2 , z12 , z13 ; b : b ∈ C −→ )0 , q1 , q2 , z12 , z13 ∈ ᏹ0,4 S 2 . Here, as before, each )i is a copy of S 2 . On the left, q1 , q2 are two marked points on )0 and z12 , z13 are two marked points on )1 . On the right, we should consider the three points q1 , q2 , z13 to be fixed, while z12 = w moves, since this corresponds to our previous trivialization of the neighborhood of σ∗ in ᐂ2,J . Thus α  may be considered as a map taking b to wb = z12 ∈ J . It remains to check that as b moves once (positively) around 0, wb moves once positively around z13 . This follows by examining the identification of the glued domain
 
)b = )0 − D(z01 ) ∪glb )1 − D(z01 ) with )0 = S 2 . Observe that the two points q1 , q2 in )0 −D(z01 ) and the single point z13 in )1 −D(z01 ) must be taken to the corresponding three fixed points on S 2 = )1 . Hence the identification on )0 −D(z01 ) is fixed, while that on )1 −D(z01 ) can rotate about z13 as b moves. Hence when b moves round a complete circle, so does wb . It remains to check the direction of the rotation. Now, as we saw in Proposition 2.7, as b moves once around this circle positively as seen from z01 , the point pb on ∂D(z01 ) ⊂ )1 that is matched with a fixed point p on )0 − D(z01 ) moves once positively around ∂D(z01 ). In order to line up pb with p, )1 must be rotated in the opposite direction, that is, positively as seen from the fixed point z13 (see Figure 2). Hence wb rotates positively around z13 . To complete the proof of the lemma, we must understand how the gluing parameters a0 , a1 fit into this picture. Since nothing is happening in the vertical (i.e., fiberwise) direction, we may consider the ai to be elements of the following tangent spaces: a0 ∈ Tz12 )1 ,

a1 ∈ Tz13 )1 .

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

z13

z12 = wb )1

Pb

P

q1

161

q2

)0 Figure 2

As b rotates positively, the image of a0 in the glued curve rotates once positively in the tangent space of z12 , and a1 ∈ Twb )1 also rotates once with respect to the standard trivialization of the tangent spaces Twb )1 ⊂ T ()1 )|D(z12 ) , hence the result. Proof of Proposition 3.4. We have identified the orbibundle ᐂ1,J → ᐆ1,J with L → Y and the bundle ᐂ2,J → ᐆ2,J with ᏻ(−2) ⊕ C → S 2 . The previous lemma shows that these are attached by first twisting ᐂ1,J to ᏻ(−3)⊕ ᏻ(−1) and then plumbing it to L. Hence
 ᏸᐆ = S ᏻ(−3) ⊕ ᏻ(−1) ≡ ! S(LY ) as claimed. The identification of the latter space with S(ᏻ(−1) ⊕ C) follows from Proposition 3.3. 3.3. The projection ᐂJ → ᏶. In order to complete the calculation of the link ᏸ2,0 of ᏶2 in ᏶, it remains to understand the projection ᐂJ → ᏶. This is 1-to-1 except over the points of ᏶1 . In ᐂ2,J , it is clearly the points with zero gluing parameter at the moving fiber that get collapsed. Thus the subbundle R− of the circle bundle S(LP ) → ᏼ(ᏻ(−2)⊕C) that lies over the (rigid) section S− = ᏼ({0}⊕C) must be collapsed to a single circle. The subbundle R+ lying over the other section S+ = ᏼ(ᏻ(−2) ⊕ {0}) maps to a family of distinct elements in ᏶1 . The story on ᐂ1,J is, of course, more complicated. Here the points that concern us are the maps in ᏿2 where the branch points coincide. Thus, if we identify ᏹ0,0 (S 2 , 2) with CP 2 as in Lemma 3.7, these are the points of the quadric Q = {(u + v + w)2 = 4uv}. Note that the attaching point σ∗ ∈ ᏹ0,2 (S 2 , p0 , p1 , 2) sits over 
[1 : 0 : −1] = 1 ∩ Q ∈ CP 2 = ᏹ0,0 S 2 , 2 . The lift of Q to ᏹ0,2 (S 2 , 2) has two components Q± , given by the intersection Y ∩H± where H± is the hyperplane 2t = ±(u+v +w). Since we can assign these at will, we say that Q− corresponds to elements with the two marked points z0 , z1 on the same component of )w = )0 ∪w=w )1 and that Q+ corresponds to elements with z0 , z1 on different components. Then, when one glues at z1 , the resulting A-curve is the union of an (A − F )-curve with an F curve. It is not hard to check that the points

162

DUSA MCDUFF

on Q− give rise to a (A − F )-curve through x0 , which is independent of w, while those on Q+ give rise to a varying (A − F )-curve that meets the Jw -holomorphic fiber through x0 at the point w. Note that the intersection Q+ ∩ Q− consists of two points, p∗ = [1 : 0 : −1 : 0] (corresponding to σ∗ ) and q∗ = [0 : 1 : −1 : 0]. Moreover, in the coordinates (a0 , a1 ) of a neighborhood of σ∗ used in Lemma 3.9, 

 (a0 , a1 ) : a0 = 0 ⊂ Q− ,

  (a0 , a1 ) : a1 = 0 ⊂ Q+ .

This confirms that when ᐂ2,J ⊗ ᏻ(−1) = ᏻ(−3) ⊕ ᏻ(−1) is plumbed to ᐂ1,J , Q− is plumbed to the subbundle {0} ⊕ ᏻ(−1) corresponding to R− , and Q+ is plumbed to the subbundle ᏻ(−3) ⊕ {0} corresponding to R+ . Let S(Q± ) → Q± denote the restriction of S(LY ) → Y to Q± . Then the plumbing ᏻ(−3) ⊕ ᏻ(−1) ≡ ! S(LY ) contains the plumbings R− ≡ ! S(Q− ) and R+ ≡ ! S(Q+ ). ! S(Q− ) = S(C) = S 2 ×S 1 , and R+ ≡ ! S(Q+ ) = S(ᏻ(−2)). Lemma 3.10. (i) R− ≡ (ii) The subsets R− ≡ ! S(Q− ) and R+ ≡ ! S(Q+ ) of ᏻ(−3)⊕ ᏻ(−1) ≡ ! S(LY ) intersect in a circle. Proof. Since Q− and Q+ do not meet the singular point of Y , both bundles S(Q± ) → Q± have Euler number −1. Hence

 R− ≡ ! S(Q− ) = S ᏻ(−1) ≡ ! S ᏻ(−1) = S(C) = S 2 × S 1 , and




 R+ ≡ ! S(Q+ ) = S ᏻ(−3) ≡ ! S ᏻ(−1) = S ᏻ(−2) .

This proves (i). To prove (ii), note that the inverse image (in S(Q± )) of the intersection point p∗ = [1 : 0 : −1] of Q− with Q+ disappears under the plumbing. But the other one remains. Proof of Theorem 1.5. It follows from part (i) of the preceding lemma that it is possible to collapse the subset R− ≡ ! S(Q− ) of ᏸᐆ to a single circle. Moreover, it is not hard to see that under the identification of ᏸᐆ = S(ᏻ(−3)⊕ ᏻ(−1)) ≡ ! S(LY ) with S(ᏻ(−1) ⊕ C), this collapsing corresponds to collapsing the circle bundle over the exceptional divisor. Since the intersection of R− ≡ ! S(Q− ) with R+ ≡ ! S(Q+ ) is a single circle, this collapsing does not affect R+ ≡ ! S(Q+ ). Note that R+ ≡ ! S(Q+ ) is the 2 ! S(Q+ ) = inverse image of some 2-dimensional submanifold of CP . Because R+ ≡ S(ᏻ(−2)), this submanifold must be a quadric. 4. Analytic arguments. In §4.1 we prove the (easy) Lemma 2.2. §4.2 contains a detailed analysis of gluing. The exposition here is fairly self-contained, though some results are quoted from [MS] and [FO]. 4.1. Regularity in dimension 4. The theory of J -holomorphic spheres in dimension 4 is much simplified by the fact that any injective J -holomorphic map h : S 2 → X

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

163

that represents a class A with A · A ≥ −1 is regular; that is, the linearized delbar operator  


2 ∗ Dh : W 1,p h∗ (T X) −→ Lp J0,1 J S ⊗ h (T X) is surjective. This remains true even if h is a multiple covering. (For a proof, see Hofer, Lizan, and Sikorav [HLS]. The notation is explained in the proof of Lemma 2.2 below.) Therefore, regularity is automatic: one does not have to perturb the equation in order to achieve it. The analogous statement when A · A < −1 is that Coker Dh always has rank equal to 2 + 2A · A. As is shown below, this almost immediately implies that the ᏶k are submanifolds of ᏶. Proof of Lemma 2.2. We begin by proving that ᏶k is a Fréchet manifold. This is obvious when k = 0, since ᏶0 is an open subset of ᏶. For k > 0, let Ꮿk denote the space of all symplectically embedded spheres in the class A−kF , and let Ꮿk (᏶) be the bundle over Ꮿk whose fiber at C is the space of all smooth almost complex structures on C that are compatible with ω|C . Then Ꮿk (᏶) fibers over Ꮿk , and it is easy to check that both spaces are Fréchet manifolds. (Note that Ꮿk is an open submanifold in the space of all embedded spheres in the class A − kF . Because these spheres are not parametrized, the tangent space to Ꮿk at C is the space of all sections of the normal bundle to C.) Further, ᏶k fibers over Ꮿk (᏶) with fiber at (C, J |C ) are equal to all ω-compatible almost complex structures that restrict to J on T C. This proves the claim. To see that πk is bijective when k > 0, note that each J ∈ ᏶k admits a holomorphic curve in class A − kF by definition and that this curve is unique by positivity of intersections. A similar argument works when k = 0 since the curves in ᏹ(A, ᏶) are constrained to go through x0 . Hence ᏹk inherits a Fréchet manifold structure from ᏶k . To show that ᏶k is a submanifold of ᏶ when k > 0, we must use the theory of J holomorphic curves, as explained in Chapter 3 of [MS], for example. Let ᏹks , ᏶sk , ᏶s denote the similar spaces in the C s -category for some large s. These are all Banach manifolds. It is easy to check that the tangent space TJ ᏶s is the space End(T X, ω, J ) of all C s -sections Y of the endomorphism bundle of T X such that J Y + Y J = 0,

ω(Y x, y) = ω(x, Yy).

These conditions imply that ω(Y x, x) = ω(Y x, J x) = 0 for all x. It follows easily that Y is determined by its value on a single nonzero vector x that it has to take to the ω-orthogonal complement to the J -complex line through x. Observe further that there is an exponential map exp : TJ ᏶s −→ ᏶s that preserves smoothness and is a local diffeomorphism near the zero section. Next, note that the tangent space T[h,J ] ᏹks is the quotient of the space of all pairs (ξ, Y ) such that 1 Dh(ξ ) + Y ◦ dh ◦ j = 0 2

(∗)

164

DUSA MCDUFF

by the 6-dimensional tangent space to the reparametrization group PSL(2, C). Here j is the standard almost complex structure on S 2 , and Dh is the linearization of the delbar operator that maps the Sobolev space of W 1,p -smooth sections of h∗ (T X) to anti-J -holomorphic 1-forms, namely,  


2 ∗ Dh : W 1,p S 2 , h∗ (T X) −→ Lp J0,1 (1) J S , h (T X) , where the norms are defined using the standard metric on S 2 and a metric on T X. In [HLS], Hofer, Lizan, and Sikorav show how to interpret elements of Ker Dh and of Ker Dh∗ (where Dh∗ is the formal adjoint) as J -holomorphic curves in their own right. Using the fact that the domain is a sphere and that X has dimension 4, they then use positivity of intersections to show that Ker Dh is trivial when k > 0, that is, it consists only of vectors that generate the action of PSL(2, C). Hence Ker Dh∗ is a bundle over ᏹks ∼ = ᏶sk of rank 4k − 2 = −indexDh, and it is not hard to see that it is isomorphic to the normal bundle of ᏶sk in ᏶s . In other words, TJ ᏶s = TJ ᏶sk ⊕ Ker Dh∗ . To see this, observe that the map 1 ι : Y −→ Y ◦ dh ◦ j 2

(2)

2 ∗ maps TJ ᏶s onto the space of C s -sections of J0,1 J (S , h (T X)), and that the kernel of this projection consists of elements Y that vanish on the tangent bundle to the image of h and so lie in TJ ᏶sk whenever [h, J ] ∈ ᏹks . It follows from equation (1) that the image of TJ ᏶sk under this projection is precisely equal to the image of Dh, and so its complement is isomorphic to Ker Dh∗ . (For more details on all this, see the appendix to [A].) It now remains to show that ᏶k is a submanifold of ᏶ whose normal bundle has fibers Ker Dh∗ . This means in particular that the codimension of ᏶k is − ind Dh = 4k − 2. We therefore have to check that each point in ᏶k has a neighborhood U in ᏶ that is diffeomorphic to the product (U ∩ ᏶k )×R4k−2 . It is here that we use the exponential map exp. Clearly, one can use exp to define such local charts for ᏶sk in ᏶s . The point here is that the derivative of the putative chart is the identity along (U ∩ ᏶sk )×{0}, and so by the implicit function theorem for Banach manifolds, it is a diffeomorphism on a neighborhood. Then, because Ker Dh∗ consists of C ∞ sections when J is C ∞ , and because exp respects smoothness, this local diffeomorphism takes (U ∩ ᏶k ) × R4k−2 onto a neighborhood of J in ᏶.

4.2. Gluing. The next task is to complete the proof of Propositions 2.7 and 2.11. The standard gluing methods are local and work in the neighborhood of one stable map, and so our main problem is to globalize the construction. The first step in doing this is to show that one can still glue even when the elements of the obstruction

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

165

bundle are nonzero at the gluing point. We use the gluing method of McDuff and Salamon [MS] and Fukaya and Ono [FO]. Much of the needed analysis appears in [MS], but the conceptual framework of that work has to be enlarged to include the idea of stable maps as in [HS]. No doubt the other gluing methods can be adapted to give the same results. Our aim is to construct a gluing map Ᏻ : ᏺᐂ (ᐆ) −→ ᏹ(A − kF, ᏶),

where ᐆ = ᏹ(A − kF, ᏶m ) is the space of stable maps in class A − kF with one component in class A − mF , and ᏺᐂ (ᐆ) is a neighborhood of ᐆ in the space ᐂ of gluing parameters. Once and for all, choose a (4m − 2)-dimensional subbundle K of T ᏶|᏶m that is transverse to ᏶m . As explained in §3.1, the exponential map exp maps a neighborhood of the zero section in K diffeomorphically onto a neighborhood of ᏶m in ᏶. For each J ∈ ᏶m , let ᏷J ⊂ ᏶ be the slice through J (i.e., the image under exp of a small neighborhood ᏺJ (K) of 0 in the fiber of K at J ). We prove the following sharper version of Proposition 2.11. Proposition 4.1. Fix J ∈ ᏶m and let ᏺᐂ (ᐆJ ) be the fiber of the map ᏺᐂ (ᐆ) → ᏶m at J . Then, if the neighborhood ᏺᐂ (ᐆ) is sufficiently small, there is a homeomorphism









ᏳJ : ᏺᐂ ᐆJ −→ ᏹ A − kF, ᏷J



onto a neighborhood of ᏹ(A − kF, J ) in ᏹ(A − kF, ᏷J ). Moreover, the union of all the sets Im ᏳJ , J ∈ ᏶m , is a neighborhood of ᏹ(A − kF, ᏶m ) in ᏹ(A − kF, ᏶). Let πJ : ᏺᐂ (ᐆJ ) → ᐆJ denote the projection. We first construct the map ᏳJ in the fiber at one point σ = [)σ , hσ , J ] of ᐆJ , and then show how to fit these maps together to get a global map over ᏺᐂ (ᐆJ ) with the stated properties. For the next paragraphs (until §4.2.4), we fix a particular representative hσ : )σ → X of σ , and we define ˜ as a map into the space of parametrized stable maps. In order to understand a full Ᏻ neighborhood of σ , we have to glue not only at points where the branches meet the stem )0 , but also at points internal to the branches. Therefore, for the moment, we forget the stem-branch structure of our stable maps and consider the general problem of gluing at the points zi ∈ )i0 ∩ )i1 with parameter  a = ⊕ i ai ∈ Tzi )i0 ⊗ Tzi )i1 . i

4.2.1. Construction of the pregluing ha . In Proposition 2.11 we showed how to construct the glued domain )a . Since this construction depends on a choice of metric on ), we must assume that the domain ) of each stable map is equipped with a Kähler metric that is flat near all double points and is invariant under the action of

166

DUSA MCDUFF

the isotropy group 5σ . Fukaya and Ono point out in [FO, §9] that it is possible to choose such a metric continuously over the whole moduli space: one just has to start at the strata containing elements σ with the largest number of components, extend the choice of metric near these strata by using the gluing construction (which is invariant by 5σ ), and then continue inductively, strata by strata. In what follows, we assume this has been done. We also suppose that the cutoff functions χr used to define )a have been chosen once and for all. The approximately holomorphic map ha : )a → X is defined √ from hσ by using cutoff functions. As before, we write ri , or simply r, instead of |ai |. Hence, if R is as in [FO] or [MS], r = 1/R. We choose a small δ > 0 once and for all so that r/δ is still small.3 Set xi = hσ (zi ). Then for α = 0, 1, define  α 2r ha (z) = hσ (z) for z ∈ )iα − Dzi , δ r  = xi for z ∈ Dzαi − Dzαi (r), δ and interpolate on the annulus Dzαi (2r/δ) − Dzαi (r/δ) in )iα by setting   δ|z| ha (z) = expxi ρ ξiα (z) , r

where ρ is a smooth cutoff function that equals 1 on the interval [2, ∞) and 0 on [0, 1], and the vectors ξiα (z) ∈ Txi X exponentiate to give hσ (z) on )iα : 
 2r hσ (z) = expxi ξiα (z) for z ∈ Dzαi . δ The whole expression is defined provided that 2r/δ is small enough for the exponential maps to be injective. Later it will be useful to consider the corresponding map hσ,r with domain ). This map equals ha on ) − ∪i,α Dzαi (ri ) and is set equal to xi on each disc Dzαi (ri ). Note that hσ,r : ) → X converges in the W 1,p -norm to hσ as r → 0. 4.2.2. Construction of the gluing Ᏻ˜ (hσ , a). Let 

ᏺ0 (Wa ) = ᏺ0 W 1,p )a , h∗a (T X) be a small neighborhood of 0 in W 1,p ()a , h∗a (T X)). Note that if )a has several components )a,j , the elements σ of Wa can be considered as collections ξj of sections in W 1,p ()a,j , (ha,j )∗ (T X)) that agree pairwise at the points zi . (This makes sense since the ξj are continuous.) Further, we may identify ᏺ0 (Wa ) via the exponential logic is that one chooses δ > 0 small enough for certain inequalities to hold and then chooses r ≤ r(δ). See Lemma 4.7. 3 The

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

167

map with a neighborhood of ha in the space of W 1,p -maps )a → X. We write ha,ξ for the map )a → X given by
 ha,ξ (z) = expha (z) ξ(z) , z ∈ )a . Recall that ᏺJ (K) is a neighborhood of 0 in the fiber of K. Given Y ∈ ᏺJ (K), we write JY for the almost complex structure exp(Y ) in the slice ᏷J . Now consider the locally trivial bundle Ᏹ = Ᏹa → ᏺ0 (Wa ) × ᏺJ (K) whose fiber at (ξ, Y ) is
 Ᏹ(ξ,Y ) = Lp J0,1 ()a ) ⊗JY h∗a,ξ (T X) . We wish to convert the pregluing ha to a map that is JY -holomorphic for some Y by using the implicit function theorem for the section Ᏺa of Ᏹa defined by Ᏺa (ξ, Y ) = ∂ JY (ha,ξ ).

Note that Ᏺa (ξ, Y ) = 0 exactly when the map ha,ξ is JY -holomorphic. The linearization ᏸ(Ᏺa ) of Ᏺa at (0, 0) equals  

ᏸ(Ᏺa ) = D(ha ) ⊕ ιa : W 1,p h∗a (T X) ⊕ K −→ Lp J0,1 ()a ) ⊗J h∗a (T X) , where ιa is defined by ιa (Y ) = (1/2)Y ◦ dha ◦ j as in equation (2) in §4.1. Lemma 4.2. Suppose that there is a continuous family of right inverses Qa to

ᏸ(Ᏺa ) that are uniformly bounded for |a| ≤ r0 . Then there is r1 > 0 such that for all

a satisfying |a| ≤ r1 , there is a unique element (ξa , Ya ) ∈ Im Qa such that Ᏺa (ξa , Ya ) = 0.

Moreover, (ξa , Ya ) depends continuously on the initial data. Proof. This follows from the implicit function theorem as stated in [MS, §3.3.4]. It also uses [MS, Lemma A.4.3]. See also [FO, §11]. We construct the required family Qa in §4.2.3. The above lemma allows us to define the gluing map. Definition 4.3. We set Ᏻ˜ (hσ , a) = ()a , ha,ξa , JYa ) where (ξa , Ya ) is the unique element in Lemma 4.2. Further, Ᏻ(hσ , a) = [)a , ha,ξa , JYa ]. The next proposition states the main local properties of the gluing map Ᏻ. Proposition 4.4. Each σ ∈ ᐆJ has a neighborhood ᏺᐂ (σ ) in ᐂJ such that the map ᏺᐂ (σ ) −→ ᏹ(A − kF, ᏷J ) : (σ  , a  ) −→ Ᏻ(hσ  , a  ) takes ᏺᐂ (σ ) bijectively onto an open subset in ᏹ(A − kF, ᏷J ). Moreover, this map depends continuously on J ∈ ᏶m .

168

DUSA MCDUFF

Proof. This is a restatement of [FO, Theorem 12.9]. Note that the stable map

Ᏻ(hσ  , a  ) depends on the choice of representative ()  , hσ  ) of the equivalence class

σ  = [)  , hσ  ]. However, it is always possible to choose a smooth family of such representatives in a small enough neighborhood of σ in ᐆJ . (This point is discussed further in §4.2.4.) Moreover, if σ is an orbifold point (i.e., if 5σ is nontrivial), then hσ is 5σ -invariant and we can define Ᏻ˜ so that it is equivariant with respect to the natural action of 5σ on the space of gluing parameters a and its action on a neighborhood of σ in the space of parametrized maps. The composite Ᏻ of Ᏻ˜ with the forgetful map is therefore 5σ -invariant. (Cf. the discussion before Lemma 4.9.) This shows that Ᏻ is well defined. We prove that it is a local homeomorphism as in [FO, §§13, 14], and we say no more about this except to observe that our adding of K to the domain of Dhσ is equivalent to their replacement of the range of Dhσ by the quotient Lp /ιa (K). 4.2.3. Construction of the right inverses Qa . This is done essentially as in [MS, A.4] and [FO, §12]. However, there are one or two extra points to take care of, firstly because the stem of the map hσ is not regular, so that the restriction of Dhσ to )0 is not surjective, and secondly because the elements of the normal bundle K → ᏶m do not necessarily vanish near the points xi in X where gluing takes place. For simplicity, let us first consider the case when ) has just two components )0 , )1 intersecting at the point w, and suppose that hσ maps )0 onto the (A−kJ )-curve J and )1 onto a fiber. (For the general case, see Remark 4.8.) Then the linearization of ∂ J at hσ has the form  

∗ Dhσ : W 1,p ), h∗σ (T X) −→ Lp J0,1 J ()) ⊗ hσ (T X) . Here the domain consists of pairs (ξ0 , ξ1 ) where ξj is a W 1,p -smooth section of the bundle h∗σj (T X) → )j subject to the condition ξ0 (w) = ξ1 (w), and the range consists of pairs of Lp -smooth (0, 1)-forms over )j with values in h∗σj (T X) and with no condition at w. For short, we denote this map by Dhσ : Wσ → Lσ0 ⊕ Lσ1 . Recall from the discussion before Proposition 4.1 that we chose K so that Dhσ0 ⊕ ι0 : Wσ0 ⊕ K −→ Lσ0 is surjective and ι0 : K → Lσ0 is injective. (All maps ι are defined as in equation (2): it should be clear from the context what the subscripts mean.) Lemma 4.5. There are constants c, r0 > 0 so that the following conditions hold for all r < r0 :

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

169

(i) ιa is injective for all |a| ≤ r; (ii) the projection pr K : Lσ0 → K that has kernel Im Dσ0 and satisfies pr K ◦ι0 = idK has norm less than or equal to c; (iii) for all Y ∈ K and j = 0, 1,      1/p  1/p   1 ιj (Y )p ιj (Y )p ≤ , j 12c )j Dw (r) j

where Dw (r) is the disc in )j on which gluing takes place and integration is with respect to the area form defined by the chosen Kähler metric on )σ . Proof. There is c so that (ii) holds because Im Dσ0 is closed and Im ι0 is finitedimensional. Then there is r0 = r0 (c) satisfying (i) and (iii) since the elements of K are C ∞ -smooth (as are the elements of ᏶). Lemma 4.6. The operator Dhσ ⊕ (ι0 , ι1 ) : Wσ ⊕ K −→ Lσ0 ⊕ Lσ1 , is surjective and has kernel ker Dhσ . Proof. We know from the proof of Lemma 2.2 that  

Dhσ0 ⊕ ι0 : W 1,p )0 , h∗σ0 (T X) ⊕ K −→ Lp J0,1 ()0 ) ⊗J h∗σ0 (T X) = Lσ0 is surjective. Similarly, Dhσ1 is surjective. Therefore, to prove surjectivity, we just need to check that the compatibility condition ξ0 (w) = ξ1 (w) for the elements of Wσ causes no problem. However, the pullback bundle h∗σ1 T X splits naturally into the sum of a line bundle with Chern class 2d (where d ≥ 0 is the multiplicity of hσ,1 ) and a trivial line bundle, the pullback of the normal bundle to the fiber Im hσ1 . Hence there is an element ξ1 of ker Dhσ1 with any given value ξ1 (w) at w. The result follows. Note that an appropriate version of this argument applies for all σ , not just those with two components, since there is just one condition to satisfy at each double point z of ), and the maps ker Dhσj → C2 : ξ → ξ(z) are surjective for j > 0. The second statement holds because ι0 is injective. Note that the right inverse Qσ to Dhσ ⊕(ι0 , ι1 ) is completely determined by choosing a complement to the finite-dimensional subspace ker Dhσ in Wσ . Consider the composite pr σ0 : Lσ0 ⊕ Lσ1 −→ Lσ0 −→K, where the second projection is as in Lemma 4.5(ii). The fiber (pr σ0 )−1 (Y ) at Y has the form (Im Dhσ0 + ι0 (Y )) ⊕ Lσ1 , and we write
 QYσ : Im Dhσ0 ⊕ Lσ1 + ι0 (Y ), ι1 (Y ) −→ Wσ for the restriction of Qσ to this fiber.

170

DUSA MCDUFF

We now use the method of [MS, A.4] to construct an approximate right inverse Qa,app to ᏸ(Ᏺa ) = Dha ⊕ ιa : Wa ⊕ K → La , where


Wa = W 1,p h∗a (T X) ,


La = Lp J0,1 ()a ) ⊗J h∗a (T X) .

It is convenient to use the approximations hσ,r : )σ → X to hσ that are defined at the end of §4.2.1 where r 2 = |a|. We write hσj ,r for the restriction of hσ,r to the component )j . Since hσ,r converges W 1,p to hσ as r → 0, Dhσ,r has a uniformly bounded inverse Qσ,r : Lσ0 ,r ⊕ Lσ1 ,r −→ Wσ,r ⊕ K. (In the notation of [MS], Qσ,r = QuR ,vR .) As above, there is a projection pr σ0 ,r : Lσ0 ⊕ Lσ1 → K, and we write QYσ,r for the restriction of Qσ,r to the fiber over Y . As a guide to defining Qa,app , consider the diagram of spaces Wσ,r ⊕ K o  Wa ⊕ K o

Lσ0 ,r ⊕ Lσ1 ,r O Qa,app

La ,

where the maps are given by (ξ0 , ξ1 , Y ) o  (ξ, Y ) ∈ Wa ⊕ K o

Qσ,r Qa,app

(η0 ,O η1 ) η ∈ La .

We define the horizontal arrow Qa,app by following the other three arrows. Here ηα , α (r) extended by 0 as in [MS]. Note for α = 0, 1, is the restriction of η to )σ,α − Dw p that the ηα are in L even though they are not continuous. Next decompose
 η0 = η0 + ισ0 ,r (Y ) ∈ Im Dσ0 ,r + ισ0 ,r (K) = Lσ0 ,r , η1 = η1 + ισ1 ,r (Y ) ∈ Lσ1 ,r . Then (ξ0 , ξ1 ) = QYσ,r (η0 , η1 ). Note that ξ0 (w) = ξ1 (w) = v. We then define the α (r/δ) for α = 0, 1 and then extending section ξ by putting it equal to ξα on )α − Dw it over the neck using cutoff functions so that it equals ξ0 + ξ1 − v on the circle 0 (r) = ∂D 1 (r) ⊂ ) . In the formula below, we think of the gluing map : of ∂Dw a a w Proposition 2.11 as inducing identifications   0 r 0 1 1 − Dw (r) −→ Dw (r) − Dw (rδ), :a : A 0 = D w δ r  0 0 1 1 :a : A 1 = D w − Dw (r) − Dw (rδ) −→ Dw (r). δ

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

171

Then ξ is given by

    zδ

 ξ0 (z) + β ξ1 :a (z) − v if z ∈ A0 , r ξ(z) =     ξ
: (z) + 1 − β z
ξ (z) − v  if z ∈ A , 1 a 0 1 r

where β : C → [0, 1] is a cutoff function that equals 1 if |z| ≤ δ and equals 0 for |z| ≥ 1. The next lemma is the analog of [MS, Lemma A.4.2]. It shows that Qa,app is the approximate inverse that we are seeking. The norms used are the usual Lp -norms with respect to the chosen metric on )σ and the glued metrics on )a . Note that we suppose that the metrics on )a agree√with the standard model χr (|x|)|dx|2 on the annuli Diα (r/δ) − Diα (rδ) (where r = |a|) so that :a is an isometry. Lemma 4.7. For all sufficiently small δ, there is r(δ) > 0 and a cutoff function β such that for all η ∈ La , |a| ≤ r(δ)2 , we have 
   Dha ⊕ ιa Qa,app η − η ≤ 1 !η!. 2 Proof. It follows from the definitions that 
Dha ⊕ ιa Qa,app η = η j

on each set )j −Dw (r/δ). (Observe that hσj ,a = ha on this domain so that ισj ,a = ιa here.) Therefore, we have only to consider what happens on the subannuli   
0 0 r 0 0 A0 = D w − Dw (r), :a (A1 ) = :a Dw (r) − Dw (rδ) δ of )a . In this region the maps hσj ,a , as well as the glued map ha , are constant so that the maps ισj ,a , ιa are constant. Further, the linearizations Dhσj ,a and Dha are all equal and on functions coincide with the usual ∂-operator. We consider what happens in :a (A1 ), leaving the similar case of A0 to the reader. It is not hard to check that for z ∈ A1 , 
Dha ξ0 (z) = η0 (z) = −ιa (Y ), 
Dha ξ1 (:a z) = η1 (:a z). Let us write βr for the function βr (z) = β(z/r). Then if r 2 = |a| and (ξ, Y ) = Qa,app η, we have for z ∈ A1 , 
Dha ξ + ιa Y − η (:a z)
 = η1 (:a z) + (1 − βr ) − ιa Y − Dha (v) (z) − ∂(βr ) ⊗ (ξ0 − v)(z) + (ιa Y − η)(:a z)
 = (βr − 1) ιa Y + Dha (v) (z) − ∂(βr ) ⊗ (ξ0 − v)(z).

172

DUSA MCDUFF

Therefore, taking the Lp -norm, 
   Dha ξ + ιa Y − η ◦ :a  p L ,A1         ≤ ιa (Y ) Lp ,A + Dha (v)Lp ,A + ∂(βr ) ⊗ (ξ0 − v)Lp ,A . 1

1

1

If r is sufficiently small, by Lemma 4.5 we can suppose that !ιa (Y )!Lp ,A1 ≤ !η!/12. Moreover, because v is a constant section, Dha acts on v just by its zeroth order part, and so there are constants c1 , c2 such that   Dha (v) p ≤ c1 !v!(area A1 )1/p ≤ c2 !v!r 2/p . L ,A1

Furthermore, by [MS, Lemma A.1.2], given any ε > 0, we can choose δε > 0 and β so that   ∂(βr ) ⊗ (ξ0 − v) p ≤ ε!ξ0 − v! 1,p , W L ,A 1

for all δ ≤ δε . Hence



 !η! !Dha ξ + ιa Y − η!Lp ,A1 ≤ c2 r 2/p + ε !v! + !ξ0 − v!W 1,p + 12
2/p 
   !η! ≤ c3 r + ε  η0 , η1 Lp + 12  
2/p !η! ≤ c4 r + ε (η0 , η1 )Lp + 12   1
2/p = c4 r + ε + !η!, 12

where the second inequality holds because of the uniform estimate for the right inverse QYσ,r , and the third inequality holds because the projection of Lσ0 ,r ⊕ Lσ1 ,r onto the subspace ImDhσ,r ⊕Lσ1 ,r is continuous. Then if we choose δε so small that c4 ε < 1/12 and r # δε so small that c4 r < 1/12, we find  
  Dha ξ + ιa Y − η ◦ :a  p ≤ 1 !η!. L ,A1 4 Repeating this for A0 gives the desired result. Finally, we define the right inverse Qa by setting

 −1 Qa = Qa,app Dha ⊕ ιa Qa,app . It follows easily from the fact that the inverses Qσ,r are uniformly bounded for 0 < r ≤ r0 that the Qa are too. It remains to remark that the above construction can be carried out in such as way as to be 5σ -equivariant. The only choice left unspecified above is that of the right inverse Qσ,r . This in turn is determined by the choice of a subspace Rσ,r of 
W 1,p ), h∗σ,r (T X) complementary to the kernel of Dhσ,r . But since 5σ is finite, we can arrange that Rσ,r is 5σ -equivariant. For example, since Dhσ,r is a finite-dimensional space consisting of

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

173

C ∞ sections, we can take Rσ,r to be the L2 -orthogonal complement of Dhσ,r defined with respect to a 5σ -invariant norm on h∗σ,r (T X).4 Note that because hσ,r = hσ,r ◦ γ for γ ∈ 5σ , we can obtain a 5σ -invariant norm on h∗σ,r (T X) by integrating the pullback by hσ,r of any norm on the tangent bundle T X with respect to a 5σ -invariant area form on the domain )σ . We can achieve this uniformly over ᐆJ by choosing a suitable metric on each domain )σ as described at the beginning of §4.2.1. Remark 4.8. If one is gluing two branch components )j , j = 0, 1 of ), then both linearizations Dhj are surjective, and one can construct the inverse Qa to have image in Wa , thus forgetting about the summand K. The general gluing argument combines both of these cases. If one is gluing at N different points, then one needs to choose r so small that one has an inequality of the form   Dha ξ + ιa Y − η

Lp ,A

≤

1 !η! 4N

on each of the 2N-annuli A. Note that the number of components of ) is bounded above by some number that depends on m (where J ∈ ᏶m ). Hence there is always r0 > 0 such that gluing at σ is possible for all r < r0 , provided that one is looking at a family of parametrized stable maps σ = (), hσ , J ) that is compact for each J and where J ∈ ᏶m is bounded in C ∞ -norm. This completes the proof of Lemma 4.2 and hence of Proposition 4.4. 4.2.4. Aut K ())-equivariance of Ᏻ˜ . Note that there is an action of S 1 on the pair (hσ , a) that rotates one of the components (say )1 ) of ) = )0 ∪ )1 , fixing the intersection point w = )0 ∩ )1 . We claim that by choosing an invariant metric on ), we can make the whole construction invariant with respect to the action of this compact group, that is, so that as unparametrized stable maps,    
)a , Ᏻ˜ (hσ , a) = )θ·a , Ᏻ˜ hσ · θ −1 , θ · a . To see this, note that there is an isometry ψ from the glued domain )a to )θ·a such that ha = hθθ·a ◦ ψ, where hθb denotes the pregluing of hθσ = hσ ·θ −1 with parameter b. There is a similar formula for the maps hσ,r . It is not hard to check that the rest of the construction can be made compatible with this S 1 action. It is important here to use the Fukaya-Ono choice of Rσ,r , as described above, instead of cutting down the domain of Dh fixing the images of certain points as in [LiT], [LiuT1], and [Sieb]. More generally, consider a parametrization σ˜ = (), h) and an arbitrary element 4 As

pointed out in [FO], the map ξ −→ ξ −

$ξ, ej %ej ,

ξ ∈ Wσ,a

j

is well defined whenever e1 , . . . , ep is a finite set of C ∞ -smooth sections.

174

DUSA MCDUFF

σ ∈ ᐆJ . Recall that a component of ) is said to be unstable if it contains less than three special points, that is, points where two components of ) meet. Each unstable branch component has at least one special point where it attaches to the rest of ), and so the identity component of its automorphism group has the homotopy type of a circle. Therefore, if there are k such unstable components, the torus group T k is a subgroup of Aut()). It is not hard to see that if the automorphism group 5σ˜ of σ˜ is nonzero, we can choose the action of T k to be 5σ˜ -equivariant so that the groups fit together to form the compact group Aut K ()) of Definition 2.9. Note further that if ()  , h ) is obtained from σ˜ = (), h) by gluing, then Aut K ()  ) can be considered as a subgroup of Aut K ()). To see this, suppose for example that )  is obtained by gluing )i to )ji with parameter a, and that both these components have at most one other special point. Then we can choose metrics on )i ∪ )ji that are invariant under an S 1 action in each component and so that the glued metric on )a is invariant under the action of an S 1 in AutK ()  ). Note that the diagonal subgroup S 1 × S 1 of AutK ()) acts trivially on the gluing parameters at the double point )i ∩ )ji since it rotates in opposite directions in the two tangent spaces. It is now easy to check that if we write θˆ for the image of θ ∈ S 1 in the diagonal subgroup of S 1 × S 1 , then 


ˆ a) . )a , h ◦ θ = )a , Ᏻ˜ (h, a) ◦ θ = )a , Ᏻ˜ (h ◦ θ, Observe also that if b is a gluing parameter at the intersection of )i with some other component )k of σ˜ , then it can also be considered as a gluing parameter for σ˜  . Moreover, under this correspondence, θˆ · b corresponds to θ · b. These arguments prove the following result. Lemma 4.9. Let σ˜ = (), h) and suppose that the metric on ) is Aut K ())invariant. Then the following statements hold. (i) The composite Ᏻ of Ᏻ˜ with the forgetful map into the space of unparametrized stable maps is Aut K ())-invariant. (ii) Divide the set P of double points of ) into two sets Pb , Ps and correspondingly write the gluing parameter a as ab + as . Suppose that ()  , h ) = ()ab , Ᏻ˜ (h, ab )) and consider as as a gluing parameter at σ˜  . Then we can choose metrics and choose the groups Aut K ()), Aut K ()  ) so that there is an inclusion
 Aut K )  −→ AutK ()) : θ −→ θˆ such that



  )ab , h ◦ θ −1 ; θ · as = )ab , Ᏻ˜ h ◦ θˆ −1 , ab ; θˆ · as .




Further, this can be done continuously as ab (and hence h ) varies, and smoothly if )ab varies in a fixed stratum. Remark 4.10. In [LiuT2, §5], Liu and Tian also develop a version of gluing that is invariant with respect to a partially defined torus action.

ALMOST COMPLEX STRUCTURES ON S 2 × S 2

175

4.2.5. Globalization. The preceding paragraphs construct the gluing map Ᏻ˜ (hσ , a) over a neighborhood ᏺ(σ ) of one point σ ∈ ᐆJ . We now show how to define a gluing map ᏳJ : ᏺᐂ (ᐆJ ) → ᏹ(A − kF, ᏷J ) on a whole neighborhood ᏺᐂ (ᐆJ ) of ᐆJ in the space of gluing parameters ᐂJ . The only difficulty in doing this lies in choosing a suitable parametrized representative s(σ ) = (), hσ ) of the equivalence class σ = [), h] as σ varies over ᐆJ . In other words, in order to define Ᏻ˜ (hσ , a), we need to choose a parametrization hσ : ) → X of the stable map σ , and now we have to choose this consistently as σ varies. We now show that although we may not be able to make a single-valued choice s(σ ) = hσ continuously over ᐆJ , we can find a section that at each point is well defined modulo the action of a suitable subgroup of Aut K ()). More precisely, we claim the following. Lemma 4.11. We may choose a continuous family of metrics gσ on )σ for σ ∈ ᐆJ and a family of parametrizations s(σ ) for each σ ∈ ᐆJ such that (i) s(σ ) consists of a Gσ -orbit of maps hσ : )σ → X, and gσ is Gσ -invariant, where Gσ ⊂ Aut K ()); (ii) the assignment σ → s(σ ) is continuous in the sense that near each σ , there is a (single-valued) continuous map σ → hσ ∈ s(σ ) whose restriction to each stratum is smooth. Moreover, gσ varies smoothly on each stratum. Proof. The strata in ᐆJ can be partially ordered with ᏿ ≤ ᏿ if there is a gluing that takes an element in the stratum ᏿ to one in ᏿ , that is, if the stratum ᏿ is contained in the closure of ᏿ . If ᏿ is maximal under this ordering and σ ∈ ᏿, then each branch component in ) is mapped to a fiber by a map of degree less than or equal to 1. It is easy to check that in this case there is a unique identification of the domain )σ with a union of spheres such that the map hσ is a section on the stem and, on each branch component, is either constant or the identity map (cf. Example 2.10). We assume this and then extend the choice of parametrization to a neighborhood of each of these maximal strata by gluing. We now start extending our choice s(σ ) = {hσ } of parametrization to the whole of ᐆJ by downwards induction over the partially ordered strata. Clearly we can always choose a parametrization modulo the action of AutK ()). In order for the image of the fiber πJ−1 (σ ) = {(σ, a) : |a| < ε} under the gluing map to be independent of this choice, we need the metric gσ on )σ to be AutK ()σ )-invariant. This choice of metric can be assumed to be smooth as σ varies in a stratum. However, it cannot always be chosen continuously as σ goes from one stratum to another. For example, if σ has one component )i with three special points at 0, 1, ∞ and that is glued to some component )ji at 1 with gluing parameter a, then the resulting component )a is unstable if )ji has no other special points. But for small |a|, the metric on )a is determined by the metrics on ) = )i ∪ )ji by the gluing construction and cannot be chosen to be S 1 -invariant. On the other hand, if both )i and )ji have at most one other special point, then the glued metric on )a is S 1 -invariant provided that the original metrics on )i , )ji are also S 1 -invariant.

176

DUSA MCDUFF

The above remarks show that suitable gσ᏿ , s(σ )᏿ and G᏿ σ can be defined over each stratum ᏿ and, in particular, over maximal strata. If gσ , s(σ ) and Gσ are already suitably defined over some union Y of strata, then the above remarks about gluing gl show that they can be extended to a neighborhood ᐁ(Y ) of Y . Let us write gσ , s(σ )gl gl and Gσ for the objects obtained by gluing when σ ∈ ᐁ(Y ). It is not hard to see that gl we can suppose that Gσ ⊂ G᏿ σ . Then if β : ᐁ(Y ) ∪ ᏿ → [0, 1] is a smooth cutoff function that equals 0 near Y and 1 near the boundary of ᐁ(Y ), set
 gσ = 1 − β(σ ) gσgl + β(σ )gσ᏿ , ᏿

s(σ ) = s(σ ) ,

if β(σ ) = 1,

= s(σ )gl , ᏿

Gσ = Gσ , = Ggl σ,

σ ∈ ᏿,

otherwise,

if β(σ ) = 1, otherwise.

It is easy to check that the required conditions are satisfied. Proof of Proposition 4.1. By Lemmas 4.9 and 4.11, there is a well-defined continuous gluing map
 ᏳJ : ᏺᐂ (ᐆJ ) −→ ᏹ A − kF, ᏷J that restricts on ᐆJ to the inclusion. Therefore, because ᐆJ is compact, the injectivity of ᏳJ on a small neighborhood ᏺᐂ (ᐆJ ) follows from the local injectivity statement in Proposition 4.4. Similarly, the local surjectivity of Proposition 4.4 implies that the image of ᏳJ is open in ᏹ(A − kF, ᏷J ). Note that all the restrictions made on the size of ᏺᐂ (ᐆJ ) vary smoothly with J (and involve no more than the C 2 norm of J ). Hence ∪J Im ᏳJ is an open subset of ᏹ(A − kF, ᏶). References [A] [AM] [BL] [FO] [G] [HLS] [HS] [K]

M. Abreu, Topology of symplectomorphism groups of S 2 × S 2 , Invent. Math. 131 (1998), 1–23. M. Abreu and D. McDuff, Topology of symplectomorphism groups of rational ruled surfaces, in preparation. J. Bryan and C. Leung, The enumerative geometry of K3 surfaces and modular forms, preprint. K. Fukaya and K. Ono, Arnold conjecture and Gromov-Witten invariant, Topology 38 (1999), 933–1048. L. Godinho, Circle actions on symplectic manifolds, Ph.D. thesis, State Univ. of New York at Stony Brook, 1999. H. Hofer, V. Lizan, and J.-C. Sikorav, On genericity for holomorphic curves in fourdimensional almost-complex manifolds, J. Geom. Anal. 7 (1997), 149–159. H. Hofer and D. Salamon, Gromov compactness and stable maps, preprint, 1997. P. Kronheimer, Some nontrivial families of symplectic structures, preprint, 1998.

ALMOST COMPLEX STRUCTURES ON S 2 × S 2 [LM]

[LL1] [LL2] [LiT]

[LiuT1] [LiuT2] [Lo] [MP] [MS] [R] [S] [YZ]

177

F. Lalonde and D. McDuff, “J -curves and the classification of rational and ruled symplectic 4-manifolds” in Contact and Symplectic Geometry (Cambridge, 1994), Publ. Newton Inst. 8, Cambridge Univ. Press, Cambridge, 1996, 3–42. T. J. Li and A. Liu, Symplectic 4-manifolds with torsion canonical classes, in preparation. , Family Seiberg-Witten invariants and wall crossing formulas, preprint. Jun Li and Gang Tian, “Virtual moduli cycles and Gromov-Witten invariants of general symplectic manifolds” in Topics in Symplectic 4-Manifolds (Irvine, Calif., 1996), First Int. Press Lect. Ser., Internat. Press, Cambridge, Mass., 1998, 47–83. Gang Liu and Gang Tian, Floer homology and Arnold conjecture, J. Differential Geom. 49 (1998), 1–74. , Weinstein conjecture and GW invariants, preprint, 1997. W. Lorek, Generalized Cauchy-Riemann operators in symplectic geometry, Ph.D. thesis, State Univ. of New York at Stony Brook, 1996. D. McDuff and L. Polterovich, Symplectic packings and algebraic geometry, Invent. Math. 115 (1994), 405–434. D. McDuff and D. A. Salamon, J -Holomorphic Curves and Quantum Cohomology, Univ. Lecture Ser. 6, Amer. Math. Soc. Providence, 1994. Yongbin Ruan, Virtual neighborhoods and pseudo-holomorphic curves, preprint, http://xxx.lanl.gov/abs/alg-geom/9611021. B. Siebert, Gromov-Witten invariants for general symplectic manifolds, preprint, 1996. S.-T. Yau and E. Zaslow, BPS states, string duality, and nodal curves on K3, Nuclear Phys. B 471 (1996), 503–512.

Department of Mathematics, State University of New York at Stony Brook, Stony Brook, New York 11794-3651, USA; [email protected]

Vol. 101, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

TANGENT VECTORS TO HECKE CURVES ON THE MODULI SPACE OF RANK 2 BUNDLES OVER AN ALGEBRAIC CURVE JUN-MUK HWANG

The moduli space of semistable bundles of a fixed determinant over an algebraic curve has been studied by many authors from various points of view. Especially wellunderstood is the case of rank 2 bundles. In this case, quite a detailed study of the geometry of the moduli space has been done in [Be], [BV], and [DR]. This moduli space is a Fano variety of Picard number 1. In a series of joint works with N. Mok, we have studied the geometry of Fano manifolds of Picard number 1 by investigating the projective geometry of the variety of tangent directions of the minimal rational curves (see [HM3] for a survey). Our aim here is to apply this study to the moduli space Mi of rank 2 bundles of a fixed determinant of degree i = 0, 1 over an algebraic curve of genus g ≥ 2. In this case, the minimal rational curves in the sense of [HM3] turn out to be “Hecke curves,” originally introduced by Narasimhan and Ramanan [NR1], [NR2]. Using the results in [NR2, Section 5], we study the variety of tangent vectors to Hecke curves through a fixed point on Mi . As a consequence of this study and our previous work [H], we get the following result, which seems to have been speculated by experts in this field. Theorem 1. Let M1 be the moduli space of stable bundles of rank 2 with a fixed determinant of odd degree over an algebraic curve of genus g ≥ 2. Then the tangent bundle of M1 is stable. Our next result is on the deformation rigidity of generically finite morphisms over Mi . A result of this type is expected for many Fano manifolds of Picard number 1, and the result here can be regarded as an example (see [HM1] for a general discussion). To streamline the presentation, we assume g ≥ 4 for M0 and g ≥ 5 for M1 in the next theorem. It may be possible to extend our result to some cases of lower genus, but one would need a different idea to cover all the cases, especially the case of g = 2. Theorem 2. Assume g ≥ 4 for i = 0 and g ≥ 5 for i = 1. Let Y be any projective manifold of dimension 3g − 3, and let f : Y → Mi be a surjective holomorphic map. Received 20 April 1998. Revision received 2 March 1999. 1991 Mathematics Subject Classification. Primary 14H60; Secondary 14J45. Author supported by Seoul National University Research Fund and by grant number 98-0701-01-5-L from the Korea Science and Engineering Foundation. 179

180

JUN-MUK HWANG

Then f is locally rigid in the sense that any deformation {ft : Y → Mi , t ∈ C, |t| < , f0 = f } satisfies ft = f for all t. Theorem 2 follows from a detailed study of the variety of tangent vectors to Hecke curves using the deep results of [Be] and [BV]. Theorem 2 can be regarded as a generalization of Narasimhan and Ramanan’s result that Mi does not have any nonzero vector field (see [NR1]). Indeed our proof is well in the spirit of [NR2, Remark 5.17(iii)] concerning the original proof of results in [NR1]. This paper consists of three sections. The first section is a collection of relevant results of Narasimhan and Ramanan from [NR2]. The proofs of Theorems 1 and 2 are given in the second and third sections, respectively. 1. Hecke curves. Throughout this paper, X is a nonsingular complex algebraic curve of genus g ≥ 2. Let Mi be the moduli space of semistable bundles of rank 2 over X with a fixed determinant of degree i, i = 0, 1. They have dimension 3g −3. Let Mis ⊂ Mi be the locus of stable bundles. Then M1s = M1 , and M0s is the smooth part of M0 . We use the notation W ∈ Mi to denote the point corresponding to a semistable bundle W on X. It is well known that Pic(Mi ) ∼ = Z. We use the additive notation for the group multiplication of Pic(Mi ). When ᏸi is the ample generator of Pic(Mi ), −KM0 = 4ᏸ0 and −KM1 = 2ᏸ1 (see [Be], [R]). Hecke curves on Mi were introduced by Narasimhan and Ramanan in [NR1], [NR2]. We recall some basic properties of Hecke curves. Almost all the results in this section can be found in [NR2, Section 5]. Given two nonnegative integers k and l, a vector bundle W of rank 2 and degree i is (k, l)-stable if, for every line subbundle L of W , we have deg(L) + k < (1/2)(i + k − l). The (0, 0)-stability is equivalent to the usual stability. A (k, l)-stable bundle is (k, l −1)-stable for l > 0. The dual bundle of a (k, l)-stable bundle is (l, k)-stable. We need the next two propositions, proved in [NR2], to explain the definition of Hecke curves. Proposition 1 [NR2, 5.4]. A generic W ∈ M1 is (1, 1)-stable if g ≥ 3. A generic W ∈ M0 is (1, 1)-stable if g ≥ 4. A generic W ∈ Mi is (1, 2)-stable if g ≥ 5. Proposition 2 [NR2, 5.5]. Given an exact sequence 0 → V → W → ᏻx → 0, where V , W are vector bundles of rank 2 and ᏻx is the skyscraper sheaf at a point x ∈ X, if W is (k, l)-stable, then V is (k, l − 1)-stable. From now on, we assume that g ≥ 3 for i = 1 and g ≥ 4 for i = 0 so that a generic point of Mi is (1, 1)-stable. For a rank 2 bundle W , its projectivization PW is canonically isomorphic to the projectivization of its dual PW ∗ . Given a point η ∈ PW , we use the same letter η to denote the 1-dimensional spaces in W and W ∗ corresponding to η. Let W ∈ Mi be a (1, 1)-stable bundle. Let π : PW → X be the natural projection. Given a point η ∈ PW with x = π(η) ∈ X, the canonical projection Wx → Wx /η

HECKE CURVES

181

defines a new rank 2 bundle W η by 0 −→ W η −→ W −→ ᏻx ⊗ (Wx /η) −→ 0. Let V be W η∗ , the dual of W η . Then det(V ) = − det(W ) + Lx , where Lx is the line bundle on X corresponding to the divisor x. For each ν ∈ PVx , we have 0 −→ V ν −→ V −→ ᏻx ⊗ (Vx /ν) −→ 0. V ν∗ has the same determinant as W , and V ν is stable from Proposition 2. Thus the family {V ν∗ , ν ∈ PVx } defines a rational curve on Mis , called the Hecke curve associated to η ∈ PW . By a Hecke curve, we mean a rational curve on Mi which is the Hecke curve associated to some η ∈ PW for some (1, 1)-stable bundle W . η At the point x, the map Wx → Wx has 1-dimensional kernel. Let η ∈ PVx be its annihilator. By writing down transition matrices explicitly, one can see easily that the  dual of V η is isomorphic to W (see [NR1, 4.11] for a proof). So the Hecke curve associated to η ∈ PW is a rational curve through W ∈ Mi . The next four propositions were proved in [NR2]. Proposition 3 [NR2, 5.16]. A Hecke curve has degree 4 with respect to −KMi . Proposition 4 [NR2, 5.9]. A Hecke curve is a smooth rational curve on Mis . Proposition 5 [NR2, 5.15]. The normal bundle of a Hecke curve in Mis is generated by global sections. Proposition 6 [NR2, 5.13]. For any (1, 1)-stable W ∈ Mi , the Hecke curves associated to two distinct η1 , η2 ∈ PW are distinct rational curves on Mi . For the next proposition, we need some definitions. The family of bundles {V ν , ν ∈ PVx } whose duals define the Hecke curve associated to η ∈ PW can be viewed  as a deformation of the bundle W ∗ = V η . Let ζ : Tη (PVx ) = Hom(η , Vx /η ) → η

η

Hom(Vx , Vx /η ) be the map induced by β : Vx → η in the defining exact sequence
  β 0 −→ V η −→ V −→ ᏻx ⊗ Vx /η −→ 0. η



Let δ : Hom(Vx , Vx /η ) → H 1 (X, End(V η )) be the connecting homomorphism for the long exact sequence coming from 

  0 −→ End(V η ) −→ Hom(V η , V ) −→ Hom V η , ᏻx ⊗ Vx /η −→ 0. Proposition 7 [NR2, 5.10]. The Kodaira-Spencer map Tη (PVx ) → H 1 (X,   ad(V η )) associated to the deformation {V ν , ν ∈ PVx } of V η is equal to δ ◦ ζ up to sign. Since the next two propositions are not given explicitly in [NR2], we provide proofs.

182

JUN-MUK HWANG

Proposition 8. Let W ∈ Mi be a generic point. Then Hecke curves through W have minimal degree among rational curves through W . Proof. For M0 , this is immediate from −KM0 = 4ᏸ0 and Proposition 3. For M1 , we have to show that there exists no rational curve of degree 1 with respect to ᏸ1 through W . By Kodaira’s stability (see [K]), if a rational curve of degree 1 exists at a generic point of M1 for some X, such a curve exists at a generic point of M1 for any X of the same genus. In particular, such a curve exists when X is hyperelliptic. From [DR, Theorem 1], M1 for a hyperelliptic curve is the set of (g −2)-dimensional linear subspaces in the intersection of two quadrics in P2g+1 determined by the hyperelliptic curve. A line on M1 gives a (g − 1)-dimensional linear subspace in the intersection of the two quadrics. If lines exist through generic points of M1 , we have at least a (3g − 3) − (g − 1) = (2g − 2)-dimensional family of (g − 1)-dimensional linear subspaces in the intersection of the two quadrics. But [DR, Theorem 2] says that the set of (g − 1)-dimensional linear subspaces of the intersection of the two quadrics is equivalent to the Jacobian of X, a contradiction to our assumption g ≥ 3. Proposition 9. Let ᏷ be an irreducible component of the Hilbert scheme of Mi containing a Hecke curve. Then ᏷ is smooth at the point corresponding to the Hecke curve, and all Hecke curves belong to ᏷. Let W ∈ Mi be (1, 1)-stable, and let ᏷W be the subscheme of ᏷ consisting of curves passing through W . Then ᏷W is smooth and consists of Hecke curves through W . In particular, ᏷W is naturally biholomorphic to PW . Proof. From [NR2, 5.15], the set of all Hecke curves form an irreducible family. Thus all Hecke curves belong to ᏷. Let C be a Hecke curve through W , and let N be its normal bundle in Mi . From Proposition 5, H 1 (C, N) = 0 and H 1 (C, N(−1)) = 0, which is equivalent to the smoothness of ᏷ and ᏷W at the point C. From Proposition 3, dim(᏷) = h0 (C, N ) = 3g − 2 and dim(᏷W ) = h0 (C, N(−1)) = 2. Thus Hecke curves are dense in ᏷, and ᏷W consists of Hecke curves through W . From Proposition 6, the classifying map PW → ᏷W is biholomorphic. 2. Proof of Theorem 1. In this section, we prove Theorem 1. For g = 2, the stability of the tangent bundle of a Fano 3-fold of Picard number 1 is well known. So we assume g ≥ 3. The proof uses a result of [H], which we briefly recall here. Let Z be any n-dimensional Fano manifold of Picard number 1. Fix an irreducible component ᏷ of the Hilbert scheme of rational curves on Z so that members of ᏷ cover Z and a generic member has minimal degree among rational curves through generic points of Z. For a torsion-free sheaf Ᏺ of rank r > 0 on Z, choose a ᏷-curve C disjoint from the singular loci of Ᏺ and define the slope of Ᏺ to be µ(Ᏺ) := (1/r) det(Ᏺ) · C. A vector bundle E of rank k on Z is stable if, for every subsheaf Ᏺ of rank r with 0 < r < k, the inequality µ(Ᏺ) < µ(E) holds. For a generic point z ∈ Z, let ᏷z be the subscheme of ᏷ corresponding to curves through z and let ᏷zo ⊂ ᏷z be the subscheme corresponding to curves smooth at z.

HECKE CURVES

183

For each member of ᏷zo , we associate its tangent vector at z to define a rational map *z : ᏷z → PTz (Z), which is called the tangent map. Let Ꮿz ⊂ PTz (Z) be the strict image of *z . Let p be the dimension of ᏷z . The following is Proposition 4 of [H]. Proposition 10. In the above situation, if the tangent bundle of Z is not stable, then there exists a subsheaf Ᏺ of rank r in T (Z) such that, for a generic point z ∈ Z and a generic point α of any component of Ꮿz , the intersection of the projective tangent space to Ꮿz at α and PᏲz in PTz (Z) is nonempty and has dimension larger than or equal to (r/n)(p + 2) − 1. In fact, if we choose Ᏺ as the subsheaf of maximal slope, it is not difficult to translate µ(Ᏺ) ≥ µ(T (Z)) into the above form, by examining the splitting type of Ᏺ on a generic ᏷-curve. See [H] for details. We apply the above result to the case Z = M1 , where p = 2 and n = 3g − 3. We choose ᏷ as the component of the Hilbert scheme containing Hecke curves given in Proposition 9. Then z is a general (1, 1)-stable W ∈ M1 . Since all Hecke o = ᏷ ∼ PW . So the tangent map * : curves are smooth (see Proposition 4), ᏷W W = W PW → PTW (Mi ) is a morphism, associating its tangent vector at W to a Hecke curve through W . To use Proposition 10 here, we need an explicit description of *W . For later use, we include the case of M0 in the next proposition. Although M0 is not smooth, we can define the tangent morphism *W : PW → PTW (Mi ) for any (1, 1)-stable W ∈ M0 , just as in the case of M1 . Proposition 11. Assume that g ≥ 3 for i = 1 and g ≥ 4 for i = 0. For a generic point W ∈ Mi , the tangent morphism *W is given by the complete linear system |2π ∗ KX − KPW | and is finite over its image. Proof. We can describe the map φ defined by the complete linear system |2π ∗ KX− KPW | as follows. Note that 2π ∗ KX − KPW = π ∗ KX + .π , where .π is the relative tangent sheaf. From π∗ .π = ad(W ) and R j π∗ .π = 0, j > 0 (see [NR1, 2.3]), H 0 (PW, 2π ∗ KX − KPW ) = H 0 (X, KX ⊗ ad(W )). Given a point η ∈ PWx , we have the corresponding η ∈ PWx∗ , which we denote by η⊥ in this proof to avoid confusion. Consider η⊥ ⊗ η ∈ P ad(Wx ) ⊂ PWx∗ ⊗ Wx . A representative of η⊥ ⊗ η in ad(Wx ) defines a linear functional H 0 (X, KX ⊗ ad(W )) → KX, x by taking the trace of endomorphisms of Wx , which gives the element φ(η) ∈ PH 0 (X, KX ⊗ ad(W ))∗ . On the other hand, from the definition of the Hecke curve associated to η ∈ PWx , *W (η) ∈ PTW (Mi ) = PH 1 (X, ad(W )) is represented by the Kodaira-Spencer class  of the family {V ν∗ , ν ∈ PVx } at W = V η ∗ , where η ∈ PVx is the annihilator of the η kernel of the map Wx → Wx . Since we are interested in the projectivization, we may  consider the Kodaira-Spencer class of the family {V ν , ν ∈ PVx } at V η , or the image of the map δ ◦ ζ in Proposition 7. We can find a cocycle representing this class as follows. Choose a coordinate covering {U, U1 , . . . , UN } of X so that all vector bundles are

184

JUN-MUK HWANG 

trivial on U, Uj and x ∈ U , x  ∈ Uj . On Uj , let us identify W ∗ = V η and V once   and for all by the map β : V η → V in the exact sequence defining V η . Choose a  coordinate z on U centered at x and choose frames {e1 , e2 } of V η and {f1 , f2 } of η V over U so that e1, x ∈ η⊥ ⊂ Wx∗ = Vx , f2,x ∈ η , β(e1 ) = zf1 , and β(e2 ) = f2 . η Given a nonzero vector v ∈ Hom(η , Vx /η ), ζ (v) = v ◦ β ∈ Hom(Vx , Vx /η ) sends e1, x to zero and e2, x to a nonzero element, namely, an element of Vx /η represented by a constant multiple of f1, x . After multiplying v by a suitable nonzero constant,  ˜ 1 ) = 0, v(e ˜ 2 ) = f1 . ζ (v) can be extended to v˜ ∈ H 0 (U, Hom(V η , V )) satisfying v(e   0 η η Then δ(ζ (v)) is defined by the cocycle vˆj ∈ H (U ∩ Uj , Hom(V , V )) obtained  by composing v˜ with the inverse of the isomorphism β|U ∩Uj : V η |U ∩Uj → V |U ∩Uj . ∗ ∗ Then vˆj (e1 ) = 0, vˆj (e2 ) = (1/z)e1 . Thus when {e1 , e2 } is the dual frame, *W (η) is represented by the cocycle {(1/z)e2∗ ⊗ e1 on U ∩ Uj }. ∗ ∈ η. It follows that the cocycle From the choice of e1 and e2 , e1, x ∈ η⊥ and e2, x ∗ ⊥ 0 {(1/z)e2 ⊗e1 } corresponds to η ⊗η ∈ PH (X, KX ⊗ad(W ))∗ via the residue pairing giving the Serre duality H 1 (X, ad(W )) = H 0 (X, KX ⊗ ad(W ))∗ . Thus *W (η) = φ(η). It remains to show that *W is finite over its image. In the notation of [Ha, V.2], the numerical class of 2π ∗ KX −KPW is 2C0 +(2g −2+e)f with e < 0 for a stable bundle W (see [Ha, Remark 2.16.1, page 379]). Thus it is ample by [Ha, Proposition 2.21, page 382], and *W is finite over its image. We are ready to prove Theorem 1. From Proposition 10, it suffices to have the following proposition. Proposition 12. Let ᏯW ⊂ P3g−4 = PTW (M1 ) be the image of PW under the finite morphism *W defined by 2π ∗ KX − KPW . Given any linear subspace PF ⊂ PTW (M1 ) of dimension r − 1, its intersection with the projective tangent space at a generic point of ᏯW is either empty or has dimension smaller than (4r/(3g −3))−1. Proof. Suppose that the intersection is nonempty and of dimension greater than or equal to (4r/(3g − 3)) − 1. Since the surface ᏯW is nondegenerate in PTW (M1 ), the intersection can have dimension 0 or 1. If the intersection has dimension 1, then the projection from PF sends the tangent space at a generic point of ᏯW to zero. Thus the projection sends ᏯW to a point. This implies that ᏯW is contained in some linear subspace Pr containing PF , a contradiction to the nondegeneracy of ᏯW . It follows that the intersection has dimension 0 and r ≤ (3/4)(g − 1). Moreover, the projection from PF projects ᏯW to a curve l in P3g−4−r . Suppose the *W -image of a generic fiber of π : PW → X is dominant over l. Since the image of this fiber under *W is of degree less than or equal to 2, l must be contained in a plane. This implies that ᏯW is contained in some Pr+2 containing PF , a contradiction to the nondegeneracy of ᏯW again. Thus the projection to P3g−4−r contracts generic fibers of π to a point. It follows that the *W -image of a generic

HECKE CURVES

185

fiber of π is contained in some linear subspace Pr containing PF as a hyperplane, and it intersects PF . Let 5 ⊂ |2π ∗ KX − KPW | be the subsystem of dimension 3g − 4 − r defining the projection of PW to P3g−4−r from PF . Let D ⊂ PW be the fixed component of 5. D corresponds to the intersection of ᏯW with PF . Hence generic fibers of π : PW → X intersect D twice, counting multiplicity. Using the notation of [Ha, V.2], the numerical class of D is of the form 2C0 +df for some integer d. From [Ha, V.2], the numerical class of 2π ∗ KX − KPW is 2C0 + (2g − 2 + e)f . Thus the moving part of the system 5 is just the pullback of a linear system on X of degree 2g − 2 + e − d. By Nagata, 0 < C02 = −e ≤ g (see [Na]). Since C0 is ample (see [Ha, V.2, Proposition 2.21]), 0 < D · C0 and −2e + d > 0. So 5 is the pullback of a linear system of degree less than or equal to 3g − 3. By the Riemann-Roch theorem and Clifford’s theorem (see [Ha, page 343]), dim(5) ≤ max((3/2)(g − 1), 2g − 3) = 2g − 3. Combined with dim(5) = 3g − 4 − r, we get g ≤ r + 1, a contradiction to r ≤ (3/4)(g − 1). 3. Proof of Theorem 2. For the proof of Theorem 2, we need some refinements of Proposition 11. These follow from the results of [Be] and [BV] rather easily. Proposition 13. Assume g ≥ 4. Then *W is an embedding for any (1, 1)-stable W ∈ M0 , except possibly when X is hyperelliptic and W is a fixed point of the involution of M0 induced by the hyperelliptic involution of X. Proof. We use the following result from [Be] and [BV]. Theorem (Beauville, Brivio-Verra). The complete linear system associated to ᏸ0 is base point free, defining a morphism ψ : M0 → PH 0 (M0 , ᏸ0 )∗ . If X is not hyperelliptic, ψ is an embedding on M0s . If X is hyperelliptic, ψ is equivalent to the quotient by the involution of M0 induced by the hyperelliptic involution of X. Since Hecke curves on M0 have degree 1 with respect to ᏸ0 , the tangent morphism *W at a point W ∈ M0 (where ψ is immersive), is just associating to lines through ψ(W ), their tangent vectors. Thus *W is an embedding of PW for any (1, 1)-stable W that is not a fixed point of the hyperelliptic involution. Proposition 14. Assume g ≥ 5. Then *W is birational for any generic W ∈ M1 . Proof. For η ∈ PW , PW η is related to PW by the elementary transformation as follows. The blow-up of PW at η is naturally biholomorphic to the blow-up of PW η at η . The exceptional divisor over η corresponds to the strict transform of the fiber π −1 (π(η )), and the exceptional divisor over η corresponds to the strict transform of the fiber π −1 (π(η)). Namely, the projectivized tangent space at η corresponds to the fiber π −1 (π(η )). Under this correspondence, a section of 2π ∗ KX − KPW η on PW η vanishing at η can be lifted to the blow-up of PW η at η and then pushed to a section of 2π ∗ KX − KPW on PW vanishing at η. For a generic (1, 2)-stable W ∈ M1 and a generic point η ∈ PW , W η ∈ M0 is

186

JUN-MUK HWANG

(1, 1)-stable and is not a fixed point of the hyperelliptic involution. Thus 2π ∗ KX − KPW η is very ample from Proposition 13. For any ζ ∈ PW with π(η)  = π(ζ ), we can find a section of 2π ∗ KX − KPW η on PW η vanishing at η and nonvanishing at ζ . Thus η and ζ can be separated by a section of 2π ∗ KX − KPW . If η  = ζ ∈ PW with π(η) = π(ζ ), they correspond to two different tangent vectors at η in PW η and can be separated by a section of 2π ∗ KX − KPW . This shows that *W is birational. To prove Theorem 2, we combine Propositions 13 and 14 with the result in [HM2, Section 1]. To explain the latter result, we need some definitions. Let g : S → Z be a regular map between two quasiprojective complex algebraic varieties. We can stratify S and Z into finitely many nonsingular quasiprojective subvarieties. On the other hand, given g : S → Z with both S and Z smooth, we can stratify S into finitely many quasiprojective subvarieties on each of which g has constant rank. Applying these two stratifications repeatedly, we can stratify S naturally into finitely many irreducible quasiprojective nonsingular subvarieties S = S1 ∪· · ·∪Sk , such that for each i, the image g(Si ) is nonsingular and the holomorphic map g|Si : Si → g(Si ) is of constant rank. It is called the g-stratification of S. Let Y be a projective manifold, and let y ∈ Y be a point. We consider the subscheme of the Hilbert scheme of curves on Y passing through y, parametrizing irreducible and reduced curves smooth at y with a fixed geometric genus, and we denote by ᏿y the underlying quasiprojective variety of an irreducible component of that subscheme. For each member l of ᏿y , let *y (l) ∈ PTy (Y ) be the tangent to the curve l at y. This defines the tangent map *y : ᏿y → PTy (Y ). Let {Si } be the *y -stratification of ᏿y . A subvariety of PTy (Y ) is called a variety of distinguished tangents in PTy (Y ), if it is the closure of the image *y (Si ) for some choice of ᏿y and Si . Note that there exist only countably many subvarieties in PTy (Y ) which can serve as varieties of distinguished tangents, because the Hilbert scheme has only countably many components. Let Z be a Fano variety and let ᏷ be as in the beginning of Section 2, namely, it is an irreducible component of the Hilbert scheme of curves so that a generic member of ᏷ is a rational curve of minimal degree through a generic point of Z. When Z has singularity, we assume that, for a generic point z ∈ Z, all members of ᏷z lie on the smooth part of Z. For a generic point z ∈ Z, let Ꮿz ⊂ PTz (Z) be the strict image of the tangent map *z : ᏷z → PTz (Z). The main result of [HM2, Section 1] is the following. The statement is slightly different from [HM2], but the proof works verbatim. Proposition 15 [HM2, Proposition 3]. Let f : Y → Z be a generically finite morphism from a projective manifold Y onto a Fano variety Z of Picard number 1 which has a family of rational curves ᏷ with the above mentioned properties. Let z ∈ Z and y ∈ f −1 (z) so that df : Ty (Y ) → Tz (Z) is an isomorphism. If we choose z and y generically, then each irreducible component of df −1 (Ꮿz ) ⊂ PTy (Y ) is a variety of distinguished tangents. Now we are ready to prove Theorem 2. We apply Proposition 15 to Z = Mi . Let ᏷

HECKE CURVES

187

be the Hilbert component containing Hecke curves. It satisfies the above requirement. Let ᏯW ⊂ PTW (Mi ) be the image of *W for (1,1)-stable W ∈ Mi . For a generic point y ∈ Y , let Wt = ft (y). By Proposition 15, dft,−1 y (ᏯWt ) is a family of varieties of distinguished tangents, and it must be a fixed subvariety of PTy (Y ) from the countability of varieties of distinguished tangents. It follows that ᏯWt is biholomorphic to ᏯW0 for all t. But from Proposition 13 for i = 0, g ≥ 4, and Proposition 14 for i = 1, g ≥ 5, this implies that PWt ∼ = PW0 . It follows that Wt = W0 and ft = f0 . Acknowledgement. We wish to thank Professor Ngaiming Mok for encouragement, Dr. Julee Kim for a special help, and the referee for many helpful suggestions. References [Be] [BV] [DR] [Ha] [H] [HM1] [HM2] [HM3] [K] [Na] [NR1] [NR2] [R]

A. Beauville, Fibrés de rang 2 sur une courbe, fibré déterminant et fonctions thêta, Bull. Soc. Math. France 116 (1988), 431–448. S. Brivio and A. Verra, The theta divisor of SU C (2, 2d)s is very ample if C is not hyperelliptic, Duke. Math. J. 82 (1996), 503–552. U. V. Desale and S. Ramanan, Classification of vector bundles of rank 2 on hyperelliptic curves, Invent. Math. 38 (1976/77), 161–185. R. Hartshorne, Algebraic Geometry, Grad. Texts in Math. 52, Springer-Verlag, New York, 1977. J.-M. Hwang, Stability of tangent bundles of low-dimensional Fano manifolds with Picard number 1, Math. Ann. 312 (1998), 599–606. J.-M. Hwang and N. Mok, Cartan-Fubini type extension of holomorphic maps for Fano manifolds with Picard number 1, preprint, 1998. , Holomorphic maps from rational homogeneous spaces of Picard number 1 onto projective manifolds, Invent. Math. 136 (1999), 209–231. , Varieties of minimal rational tangents on uniruled projective manifolds, to appear in the Proceedings of MSRI Special Year in Several Complex Variables. K. Kodaira, On stability of compact submanifolds of complex manifolds, Amer. J. Math. 85 (1963), 79–94. M. Nagata, On self-intersection number of a section on a ruled surface, Nagoya Math. J. 37 (1970), 191–196. M. S. Narasimhan and S. Ramanan, Deformations of the moduli space of vector bundles over an algebraic curve, Ann. of Math. (2) 101 (1975), 391–417. , “Geometry of Hecke cycles, I” in C. P. Ramanujam—A Tribute, Tata Inst. Fund. Res. Studies in Math. 8, Springer-Verlag, Berlin, 1978, 291–345. S. Ramanan, The moduli spaces of vector bundles over an algebraic curve, Math. Ann. 200 (1973), 69–84.

Department of Mathematics, Seoul National University, Seoul, 151-742, Korea; [email protected]

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

ON THE MORGAN-SHALEN COMPACTIFICATION OF THE SL(2, C) CHARACTER VARIETIES OF SURFACE GROUPS G. DASKALOPOULOS, S. DOSTOGLOU, and R. WENTWORTH 1. Introduction. Let 6 be a closed, compact, oriented surface of genus g ≥ 2 and fundamental group 0. Let X(0) denote the SL(2, C) character variety of 0, and D(0) ⊂ X(0) the closed subset consisting of conjugacy classes of discrete, faithful representations. Then X(0) is an affine algebraic variety admitting a compactification X(0) (due to Morgan and Shalen [MS1]), whose boundary points ∂ X(0) = X(0) \ X(0) correspond to elements of PL(0), the space of projective classes of length functions on 0 with the weak topology. Choose a metric σ on 6, and let MHiggs (σ ) denote the moduli space of semistable rank-2 Higgs pairs on 6 (σ ) with trivial determinant, as constructed by Hitchin [H]. Then MHiggs (σ ) is an algebraic variety, depending on the complex structure defined by σ (cf. [Si]). By the theorem of Donaldson [D], MHiggs (σ ) is homeomorphic to X(0), though not complex-analytically so. Let us denote this map h : X(0) → MHiggs (we henceforth assume the choice of base point σ ). We define a compactification of MHiggs as follows: Let QD (more precisely, QD(σ )) denote the finite-dimensional complex vector space of holomorphic quadratic differentials on 6. Then there is a surjective, holomorphic map MHiggs → QD taking the Higgs field 8 to ϕ = det 8. We compose this with the map ϕ −→ where kϕk =

R

6 |ϕ|,

4ϕ , 1 + 4kϕk

and obtain

f : MHiggs −→ BQD = {ϕ ∈ QD : kϕk < 1} . det Let SQD = {ϕ ∈ QD : kϕk = 1} be the space of normalized holomorphic quadratic differentials. We then define MHiggs = MHiggs ∪ SQD with the topology given via the Received 6 August 1998. 1991 Mathematics Subject Classification. Primary 58E20; Secondary 20E08, 30F30, 32G13. Daskalopoulos’s work partially supported by National Science Foundation grant number DMS9803606. Dostoglou’s work partially supported by the Research Board of the University of Missouri and the Arts and Science Travel Fund of the University of Missouri, Columbia. Wentworth’s work partially supported by National Science Foundation grant number DMS-9971860 and a Sloan Fellowship. 189

190

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

f The aim of this paper is to compare the two compactifications X(0) and map det. MHiggs . The points of PL(0) may be regarded as arising from the translation lengths of minimal, nontrivial 0 actions on R-trees. Modulo isometries and scalings, this correspondence is one-to-one, at least in the nonabelian case (cf. [CM] and our Section 2). The boundary ∂ D(0) consists of small actions, that is, those for which the arc-stabilizer subgroups are all cyclic. With our choice of conformal structure σ , we can define a continuous, surjective map (1.1)

H : PL(0) −→ SQD .

When the length function [`] is realized by the translation length function of a tree dual to the lift of a normalized holomorphic quadratic differential ϕ, then H ([`]) = ϕ; the full map is a continuous extension of this (see Theorem 3.9) with the fibers of H corresponding more generally to foldings of dual trees. Let PMF(0) denote the space of projective classes of measured foliations on 6, modulo isotopy and Whitehead equivalence (cf. [FLP, exposé 5]). By the theorem ∼ of Hubbard-Masur [HM] we also have a homeomorphism HM : PMF(0) −→ SQD. It is not clear how to lift H to factor through PMF(0) in a manner independent of σ . However, it follows essentially by Skora’s theorem [Sk] that if H is restricted to PSL(0), the small actions, then it factors through HM by a homeomorphism ∼ PSL(0) −→ PMF(0). With this understood, we define a (set-theoretic) map (1.2)

h¯ : X(0) −→ MHiggs

by extending the map h to H on the boundary. We prove the following. Main Theorem. The map h¯ is continuous and surjective. Restricted to the compactification of the discrete, faithful representations D(0), it is a homeomorphism onto its image. Note that the second statement follows from the first, since ∂ D(0) consists of small actions, and therefore the restricted map is injective by the above-mentioned theorem of Skora. The full map is not bijective: For example, quadratic differentials that are squares of holomorphic 1-forms are images of the length functions of their dual trees, but they also appear as images of the limits of abelian representations (see Section 3). It would be interesting to determine the fibers of h¯ in general; this question will be taken up elsewhere. We also remark that the SL(2, R) version of the above theorem leads to a harmonic-maps description of the Thurston compactification of Teichmüller space and was first proved by Wolf [W1]. Generalizing this result to SL(2, C) is one of the motivations for this paper. This paper is organized as follows: In Section 2 we review the Morgan-Shalen compactification, the definition of the Higgs moduli space, and the notion of a harmonic map to an R-tree. In Section 3, we define the boundary map H . The key point

THE MORGAN-SHALEN COMPACTIFICATION

191

is that the nonuniqueness in the correspondence between abelian length functions and R-trees alluded to above nevertheless leads, via harmonic maps, to a well-defined geometric object on 6, in this case, a quadratic differential. The most important result here is Theorem 3.7. Along the way, we give a criterion, Theorem 3.3, for uniqueness of harmonic maps to trees, using the arguments in [W3]. The main theorem is then proven in Section 4 as a consequence of our previous work [DDW]. In the last section, a somewhat more concrete analysis of the behavior of high energy harmonic maps is outlined, illustrating previous ideas. 2. Definitions. Let 0 be a hyperbolic surface group as in the introduction. We denote by R(0) the set of representations of 0 into SL(2, C), and by X(0) the set of characters of representations. Recall that a representation ρ : 0 → SL(2, C) defines a character χρ : 0 → C by χρ (g) = Tr ρ(g). Two representations ρ and ρ 0 are equivalent if χρ = χρ 0 . It is easily seen (cf. [CS]) that equivalent irreducible representations are conjugate. If ρ is a reducible representation, then we can write   a(g) λρ (g) ρ(g) = 0 λρ (g)−1 for a representation λρ : 0 → C∗ . The character χρ determines λρ up to the inversion coming from the action of the Weyl group and is, in turn, completely determined by it. It is shown in [CS] that the set of characters X(0) has the structure of an affine algebraic variety. In [MS1], a (nonalgebraic) compactification X(0) of X(0) is defined as follows: Let C be the set of conjugacy classes of 0, and let P(C) = P(RC ) be the (real) projective space of nonzero, positive functions on C. Define the map ϑ : X(0) → P(C) by
ª  ¡ ϑ(ρ) = log |χρ (γ )| + 2 : γ ∈ C and let X(0)+ denote the 1-point compactification of X(0) with the inclusion map ı : X(0) → X(0)+ . Finally, X(0) is defined to be the closure of the embedded image of X(0) in X(0)+ × P(C) by the map ı × ϑ. It is proved in [MS1] that X(0) is compact and that the boundary points consist of projective length functions on 0 (see the definition ª below). Note that in its definition, ϑ(ρ) could be replaced by the  function `ρ (γ ) γ ∈C , where `ρ denotes the translation length for the action of ρ(γ ) on H3 :

 ª `ρ (γ ) = inf distH3 (x, ρ(γ )x) : x ∈ H3

(see [Cp]). Recall that an R-tree is a metric space (T , dT ) such that any two points x, y ∈ T are connected by a segment [x, y] (that is, a rectifiable arc isometric to a compact (possibly degenerate) interval in R whose length realizes dT (x, y)) and that [x, y] is the unique embedded path from x to y. We say that x ∈ T is an edge point (resp., vertex) if T \ {x} has two (resp., more than two) components. A 0-tree is an R-tree

192

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

with an action of 0 by isometries, and it is called minimal if there is no proper 0invariant subtree. We say that 0 fixes an end of T (or more simply, that T has a fixed end) if there is a ray R ⊂ T such that for every γ ∈ 0, γ (R) ∩ R is a subray. When the action is understood, we often refer to “trees” instead of “0-trees.” Given an R-tree (T , dT ), the associated length function `T : 0 → R+ is defined by `T (γ ) = inf x∈T dT (x, γ x). If `T 6≡ 0, which is equivalent to 0 having no fixed point in T (cf. [MS1, Prop. II.2.15]), then the class of `T in P(C) is called a projective length function. We denote by PL(0) the set of all projective length functions on 0-trees. A length function is called abelian if it is given by |µ(γ )| for some homomorphism µ : 0 → R. We use the following result. Theorem 2.1 [CM, Cor. 2.3 and Thm. 3.7]. Let T be a minimal 0-tree with nontrivial length function `T . Then `T is nonabelian if and only if 0 acts without fixed ends. Moreover, if T 0 is any other minimal 0-tree with the same nonabelian length function, then there is a unique equivariant isometry T ' T 0 . It is a fact that abelian length functions, in general, no longer determine a unique minimal 0-tree up to isometry (e.g., see [CM, Example 3.9]), and this presents one of the main difficulties dealt with in this paper. We now give a quick review of the theory of Higgs bundles on Riemann surfaces and their relationship to representation varieties. Let 6, 0 be as in the introduction. A Higgs pair is a pair (A, 8), where A is an SU(2) connection on a rank-2 smooth vector bundle E over 6; and 8 ∈ 1,0 (6, End0 (E)), where End0 (E) denotes the bundle of traceless endomorphisms of E. The Hitchin equations are (2.1)

FA +[8, 8∗ ] = 0, 00 8 = 0. DA

The group G of (real) gauge transformations acts on the space of Higgs pairs and preserves the set of solutions to (2.1). We denote by MHiggs the set of gauge equivalence classes of these solutions. Then MHiggs is a complex analytic variety of dimension 6g − 6 (the holomorphic structure depending upon the choice σ on 6), which admits a holomorphic map (cf. [H]) (2.2)

det : MHiggs −→ QD = H 0 (6, K6⊗2 ) : (A, 8) 7→ det 8 = − Tr 82 .

By associating to [(A, 8)] ∈ MHiggs the character of the flat SL(2, C) connection A+8+8∗ , one obtains a homeomorphism h : MHiggs → X(0) (cf. [D], [C]). Implicit in the definition of h is a 0-equivariant harmonic map u from the universal cover H2 of 6 to H3 . It is easily verified that the Hopf differential of u, Hopf(u) = ϕ˜ = huz , uz idz2 , descends to a holomorphic quadratic differential ϕ on 6 equal to det 8 (up to a universal nonzero constant). Having introduced harmonic maps, we now give an alternative way to view the

THE MORGAN-SHALEN COMPACTIFICATION

193

Morgan-Shalen compactification. First, it follows by an easy application of the Bochner-Weitzenböck formula that a sequence of representations ρi diverges to the boundary only if the energies E(uρi ) of the associated equivariant harmonic maps uρi are unbounded. Furthermore, given such a sequence, it is shown in [DDW] that if the ρi converge to a boundary point in the sense of Morgan-Shalen, then the harmonic maps uρi converge (perhaps after passing to a subsequence) in the sense of Korevaar-Schoen to a 0-equivariant harmonic map u : H2 → (T , dT ), where (T , dT ) is a minimal 0-tree having the same projective length function as the Morgan-Shalen limit of the ρi . As pointed out before, the tree is not necessarily uniquely defined, and even in the case where the tree is unique, uniqueness of the harmonic map is problematic. Recall that a harmonic map to a tree means, by definition, an energy minimizer for the energy functional defined in [KS1]. Given such a map, its Hopf differential ϕ˜ can be defined almost everywhere, and by [S1, Lemma 1.1], which can be adapted to the singular case, one can show that the harmonicity of u implies that ϕ˜ is a holomorphic quadratic differential. The equivariance of u implies that ϕ˜ is the lift of a differential on 6. Note also that if u : H2 → T is harmonic, then Hopf(u) ≡ 0 if and only if u is constant. In the equivariant case, this in turn is equivalent to `T ≡ 0 (cf. [DDW]). For the rest of the paper, we tacitly assume `T 6≡ 0. A particular example is the following: Consider a nonzero holomorphic quadratic differential ϕ, and denote by ϕ˜ its lift to H2 . Locally away from the zeros, ϕ˜ may be written as dz2 with respect to a local conformal coordinate z = ξ + iη. The lines ξ = const (the vertical leaf space) and transverse measure |dξ | give the structure of a metric space Tϕ˜ , which is independent of the choice of coordinate z and naturally extends past the zeros. According to [MS2] (and using the correspondence between measured foliations and geodesic laminations), Tϕ˜ is an R-tree with an action of 0, and the projection π : H2 → Tϕ˜ is a 0-equivariant continuous map. We note two ˜ important facts: (1) The vertices of Tϕ˜ are precisely the image by π of the zeros of ϕ. (2) Since the action of 0 on Tϕ˜ is small, Tϕ˜ has no fixed ends (cf. [MO]). ˜ Proposition 2.2. The map π : H2 → Tϕ˜ is harmonic with Hopf differential ϕ. Proof. Since Tϕ˜ has no fixed ends, the existence of a harmonic map follows from [KS2, Cor. 2.3.2]. The fact that π is itself an energy minimizer seems to be well known. See, for example, [W2] and the introduction to [GS]: Although the definition of harmonic map in [W2] is a priori different from the notion of an energy minimizer, a proof follows easily. Indeed, for fixed ϕ 6= 0 and positive real numbers ti → ∞, we can find a sequence of hyperbolic metrics σi on 6 such that the unique harmonic maps 6(σ ) → 6(σi ) homotopic to the identity have Hopf differentials ti ϕ (cf. [W1] and [Wan]). Uniformizing the σi , we obtain a sequence ρi of discrete faithful SL(2, R) ⊂ SL(2, C) representations and ρi -equivariant harmonic maps ui : ˜ Let di denote the pullback distance functions H2 → H2 with Hopf differentials ti ϕ. on H2 by the ui , and let d∞ denote the pseudometric obtained by pulling back the

194

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

metric on Tϕ˜ by the projection π . Extend all of these to pseudometrics, also denoted di and d∞ , on the space H2∞ constructed in [KS2]. Then the natural projection H2 → H2∞ /d∞ ' Tϕ˜ coincides with the map π. On the other hand, by [W2, Section 4.2], di → d∞ pointwise, locally uniformly. Therefore, by [KS2, Thm. 3.9], π is an energy minimizer. Next, we consider 0-trees that are not necessarily of the form Tϕ˜ . We need the following. Definition 2.3. A morphism of R-trees is a map f : T → T 0 such that every nondegenerate segment [x, y] has a nondegenerate subsegment [x, w] such that f restricted to [x, w] is an isometry onto its image. The morphism f is said to fold at a point x ∈ T if there are nondegenerate segments [x, y1 ] and [x, y2 ] with [x, y1 ] ∩ [x, y2 ] = {x} such that f maps each segment [x, yi ] isometrically onto a common segment in T 0 . It is a fact that a morphism f : T → T 0 is an isometric embedding unless it folds at some point (cf. [MO, Lemma I.1.1]). We also note that, in general, foldings T → T 0 may take vertices to edge points. Conversely, vertices in T 0 need not lie in the image of the vertex set of T . Proposition 2.4 (cf. [FW]). Let T be an R-tree with 0 action, and let u : H2 → T be an equivariant harmonic map with Hopf differential ϕ. ˜ Then u factors as u = 2 f ◦π , where π : H → Tϕ˜ is as in Proposition 2.2 and f : Tϕ˜ → T is an equivariant morphism. Proof. Consider f = u ◦ π −1 : Tϕ˜ → T . We first show that f is well defined: Indeed, assume z1 , z2 ∈ π −1 (w). Then z1 and z2 may be connected by a vertical leaf e of the foliation of ϕ. ˜ Now, by the argument in [W3, p. 117], u must collapse e to a point, and so u(z1 ) = u(z2 ). In order to show that f is a morphism, consider a ˜ Moreover, segment [x, z] ∈ Tϕ˜ . We may lift x to a point x˜ away from the zeros of ϕ. we may choose a small horizontal arc e˜ from x˜ to some y˜ projecting to [x, y] ⊂ [x, z], still bounded away from the zeros. The analysis in [W3] again shows that this must map by u isometrically onto a segment in T . Remark. It is easily shown (cf. [DDW]) that images of equivariant harmonic maps to trees are always minimal subtrees; hence, throughout this paper we assume, without loss of generality, that our trees are minimal. Thus, for example, the factorization f : Tϕ˜ → T above either folds at some point or is an equivariant isometry. 3. The map H . The Hopf differential for a harmonic map to a given tree is uniquely determined, as shown by the following statement. Proposition 3.1. Let T be a minimal R-tree with a nontrivial 0 action. If u, v are equivariant harmonic maps H2 → T , then Hopf(u) = Hopf(v).

THE MORGAN-SHALEN COMPACTIFICATION

195

Proof. This is proven in [KS1], where in fact the full pullback “metric tensor” is considered. In our situation, the result can also be seen as a direct consequence of the leaf structure of the Hopf differential. First, by [KS1, p. 633], the function z 7 → dT2 (u(z), v(z)) is subharmonic; hence, by the equivariance it must be equal to a constant c. We assume c 6= 0, since otherwise there is nothing to prove. Set ˜ and let 1 be a small ϕ˜ = Hopf(u), ψ˜ = Hopf(v). Suppose that p ∈ H2 is a zero of ϕ, ˜ except perhaps neighborhood of p containing no other zeros of ϕ˜ and no zeros of ψ, p itself. Then by Proposition 2.4 it follows that u is constant and equal to u(p) on every arc e ⊂ 1 of the vertical foliation of ϕ˜ with endpoint p. On the other hand, v(e) is a connected set satisfying dT (u(p), v(z)) = c for all z ∈ e. Since spheres are discrete in trees, v is constant and equal to v(p) on e as well. Referring again to ˜ In this Proposition 2.4, this implies that e must be contained in a vertical leaf of ψ. way, one sees that the zeros of ϕ˜ and ψ˜ coincide with multiplicity in H2 . Thus, the same is true for ϕ and ψ on 6. Since the quadratic differentials are both normalized, they must be equal. We also need the following restriction on the kinds of foldings that arise from harmonic maps. Lemma 3.2. Let Tϕ˜ → T arise from a harmonic map as in Proposition 2.4. Then folding occurs only at vertices, that is, the images of zeros of ϕ. ˜ At the zeros of ϕ, ˜ adjacent edges may not be folded. In particular, folding cannot occur at simple zeros. Proof. The argument is similar to that in [W2, p. 587]. Suppose p ∈ H2 is a zero at which a folding occurs, and choose a neighborhood 1 of p contained in a fundamental domain and containing no other zeros. We can find distinct segments e, e0 of the horizontal foliation of ϕ˜ with a common endpoint p that map to segments of Tϕ˜ . We may further assume that the folding Tϕ˜ → T carries each of e and e0 isometrically onto a segment e¯ of T . Suppose that e and e0 are adjacent. Then there is a small disk 10 ⊂ H2 that, under the projection π : H2 → Tϕ˜ , maps to π(e) ∪ π(e0 ) and whose center maps to π(p) (see Figure 1). Then the harmonic map u : H2 → T maps 10 onto the segment e¯ with the center mapping to an endpoint. Let q denote the other endpoint of e. ¯ The function z 7→ (dT (u(z), q))2 is subharmonic on 10 with an interior maximum. It therefore must be constant, which contradicts ϕ 6≡ 0. For the last statement, recall that the horizontal foliation is trivalent at a simple zero, so that any two edges are adjacent. Though the following is not important in this paper, we find it interesting that a uniqueness result for equivariant harmonic maps to trees follows from these considerations, in certain cases. Theorem 3.3. Let u : H2 → T be an equivariant harmonic map with ϕ˜ = Hopf(u). Suppose there is some vertex x of Tϕ˜ such that the map f : Tϕ˜ → T from Proposition 2.4 does not fold at x. Then u is the unique equivariant harmonic map to T .

196

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

e p e0

= vertical foliation = horizontal foliation Figure 1

Proof. Let p be a zero of ϕ˜ projecting via π to x, and let v be another equivariant harmonic map to T . Choose a neighborhood 1 of p as in the proof of Proposition 3.1, and again suppose that the constant c = dT (u(z), v(z)) 6= 0. Recall that x is a vertex of Tϕ˜ . By the assumption of no folding at x, there must be a segment e of the vertical foliation of ϕ˜ in 1, with one endpoint being p, having the following property: For any z 6= p in e there is a neighborhood 10 ⊂ 1 of z such that u(10 ) ∩ [u(p), v(p)] = {u(p)}. By Proposition 3.1 and Lemma 3.2, we see that for such 10 , v(10 ) 6⊂ [u(p), v(p)]. Thus, there is a q ∈ 1 such that u(q) 6∈ [u(p), v(p)] and v(q) 6∈ [u(p), v(p)]. But then dT (u(q), v(q)) > dT (u(p), v(p)) = c, a contradiction. Corollary 3.4. Let ϕ 6 ≡ 0 be a holomorphic quadratic differential on 6. Then the map π : H2 → Tϕ˜ in Proposition 2.2 is the unique equivariant harmonic map to Tϕ˜ . If u : H2 → T is an equivariant harmonic map and Hopf(u) has a zero of odd order, then u is unique. Proof. The first statement is clear from Theorem 3.3. For the second statement, notice that if p is a zero of odd order, we can still find a neighborhood 10 as in the proof of Theorem 3.3. Proposition 3.1 allows us to associate a unique ϕ ∈ SQD to any nonabelian length function. Proposition 3.5. Let [`] ∈ PL(0) be nonabelian. Then there is a unique choice ϕ ∈ SQD with the following property: If T is any minimal R-tree with length function ` in the class [`], and u : H2 → T is a 0-equivariant harmonic map, then Hopf(u) = ϕ. Proof. Let ` ∈ [`]. By Theorem 2.1, there is a unique minimal tree T , up to isometry, with length function ` and no fixed ends. By Proposition 3.1, any two harmonic maps u, v : H2 → T have the same normalized Hopf differential. Furthermore, if T 0 is isometric to T and u0 is a harmonic map to T 0 , then, composing with the

THE MORGAN-SHALEN COMPACTIFICATION

197

isometry, we see that u0 has the same Hopf differential as any harmonic map to T . If the length function ` is scaled, then the normalized Hopf differential remains invariant. Finally, since T has no fixed ends, it follows from [KS2, Cor. 2.3.2] that there exists an equivariant harmonic map u : H2 → T ; so we set ϕ = Hopf(u). We now turn our attention to the abelian length functions. These no longer determine a unique R-tree in general; nevertheless, we see that there is still a uniquely defined quadratic differential associated to them. Proposition 3.6. Let ` be an abelian length function, and let 0 act on R with translation length function equal to `. Then there is an equivariant harmonic function ˜ 2 , where u : H2 → R, unique up to translations of R, with Hopf differential ϕ˜ = (ω) 2 ω˜ is the lift to H of an abelian differential ω on 6. Moreover, ` is determined by the periods of Re(ω). Proof. The uniqueness statement is clear. By harmonic theory, there is a unique holomorphic one-form ω on 6 such that the real parts of its periods correspond to the homomorphism µ : π1 (6) −→ H1 (6, Z) −→ R. harmonic function is Choosing any base point ∗ of H2 , the desired equivariant Rz ˜ The Hopf differential is the real part of the holomorphic function f (z) = ∗ ω. (f 0 (z))2 = (ω) ˜ 2. It is generally true that harmonic maps to trees with abelian length functions have Hopf differentials with even-order vanishing and that the length functions are recovered from the periods of the associated abelian differential, as the next result demonstrates. Theorem 3.7. Let u : H2 → T be an equivariant harmonic map to a minimal R-tree with nontrivial abelian length function `. Then Hopf(u) = (ω) ˜ 2 , where ω˜ is 2 the lift to H of an abelian differential ω on 6. Moreover, ` is determined by the periods of Re(ω). Proof. We first prove that the Hopf differential ϕ˜ = Hopf(u) must be a square. It suffices to prove that the zeros of ϕ˜ are all of even order. Let p be such a zero, and choose a neighborhood 1 of p as above. Since T has an abelian length function, the action of 0 must fix an end E of T . Then, applying the construction of Section 5 of [DDW], we find a continuous family of equivariant harmonic maps uε obtained by “pushing” the image of u a distance ε in the direction of the fixed end. On the other hand, if ϕ˜ had a zero of odd order, this would violate Corollary 3.4. We may therefore express ϕ˜ = (ω) ˜ 2 for some abelian differential ω˜ on H2 . A priori, we can only conclude that ω˜ descends to an abelian differential ωˆ on an b of 6 determined by an index-2 subgroup b unramified double cover 6 0 ⊂ 0. Let L be a complete noncritical leaf of the horizontal foliation of ϕ. ˜ Choose a point x0 ∈ L and let x¯0 = u(x0 ). We assume that we have chosen x0 so that x¯0 is an edge point.

198

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

R

E

e

R0 q

e0

Figure 2

Then there is a unique ray R¯ with end point x¯0 leading out to the fixed end E. Let R denote the half-leaf of L starting at x0 and such that a small neighborhood of x0 in R ¯ maps isometrically onto a small subsegment of R. ¯ For suppose to the contrary that We claim that R itself maps isometrically onto R. there is a point y ∈ R such that the portion [x0 , y] of R from x to y maps isometrically ¯ but that this is not true for any y 0 ∈ R \ [x0 , y]. Clearly, the onto a subsegment of R, image of y by u must be a vertex of T . Recall the factorization f : Tϕ˜ → T from Proposition 2.4. Since f is a surjective morphism of trees, the vertices of T are either ˜ or they are images by f of vertices of Tϕ˜ and, hence, images by u of zeros of ϕ, vertices created by a folding of f . Thus, there are two cases to consider: (1) There is a point q such that y and q lie on the same vertical leaf and q is a zero of ϕ. ˜ 0 Moreover, there is a critical horizontal leaf R with one end point equal to q, a small subsegment of which maps isometrically onto a subsegment of R¯ with end point q¯ = u(q) (see Figure 2). (2) There is a point q such that y and q lie on the same vertical leaf, q is connected by a horizontal leaf to a zero p of ϕ, ˜ and the map f folds at π(p), identifying the segment [p, q] with a portion [p, q 0 ] of another horizontal leaf R 0 . Moreover, [p, q 0 ] maps isometrically onto a subsegment of the unique ray from p¯ = u(p) to the end E (see Figure 3). Consider case (1): As indicated in Figure 2, we can find a small neighborhood 1 of y and portions of horizontal leaves e and e0 meeting at q that map isometrically onto segments of T intersecting the image R¯ 0 = u(R 0 ) only in q. ¯ Now, as above, by pushing the image of u in the direction of E and possibly choosing 1 smaller, we can find a harmonic map uε that maps 1 onto a segment with end point q¯ and maps y to the opposite end point—a contradiction. The argument for case (2) is similar: We may find a disk 1 centered at y that maps to the union of segments [p, ¯ q] ¯ and [¯r , q], ¯ with y being mapped to q. ¯ Then, pushing the map in the direction of E as above again leads to a contradiction (see Figure 3). Next, we claim that for any g ∈ b 0 , `(g) is given by the period of Re(ω) ˆ around a curve representing the class [g]. First, by definition of a fixed end, the intersection ¯ contains a subray of R, ¯ and for all x¯ in this subray, `(g) = dT (x, ¯ g(x)) ¯ (cf. R¯ ∩ g(R) ¯ ⊂ R. ¯ Choose a lift of x¯ to [CM, Thm. 2.2]). For simplicity then, we assume g(R)

199

THE MORGAN-SHALEN COMPACTIFICATION

E

E R

r

q0 p

q

R0 u

--->

p

q = q0 r

Figure 3

¯ Suppose g(x) is connected by a (possibly empty) x ∈ R. Then u(g(x)) = g(x) ¯ ∈ R. 0 vertical leaf to a point x on R. Then the curve γ˜ consisting of the portion [x, x 0 ] of R from x to x 0 followed by the vertical leaf to g(x) projects to a curve γ on 6 ¯ `(g) is the length of representing g. Moreover, since R maps isometrically onto R, 0 by ϕ. ˜ Since R contains no [x, x ] with respect to the transverse measure determined R zeros of ϕ, ˜ the latter is simply the absolute value of [x,x 0 ] Re(ω). ˜ Futhermore, since the vertical direction lies in the kernel of Re(ω), ˆ we also have Z Z ˜ = Re(ω) ˆ `(g) = Re(ω) γ˜

γ

as desired. Now consider the possibility that g(x) ∈ g(R) is not connected to R by a vertical ¯ it follows from Proposition 2.4 and the fact that R maps onto R¯ leaf. Since g(x) ¯ ∈ R, that there is an intervening folding of a subray of g(R) onto R. Let y ∈ R project to the vertex in Tϕ˜ at which this occurs. The simplest case is where y is connected by a vertical leaf to a point w ∈ g(R), and the folding identifies the subray of R starting at y isometrically with the subray of g(R) starting at w. The same analysis as above then produces the closed curve γ . A more complicated situation arises when there are intervening vertices (see Figure 4(a)): For example, there may be zeros p, q of ϕ, ˜ a point w0 ∈ g(R), and segments 0 00 e, e , and e of the vertical, horizontal, and vertical foliations, respectively, with endpoints {y, p}, {p, q}, and {q, w 0 }, respectively. Moreover, the map u folds e0 onto a subsegment f of R with endpoints y and y 0 , and then it identifies the subray of R starting at y 0 isometrically with the subray of g(R) starting at w0 . In this way, we see that a subsegment f 0 of g(R) with endpoints w0 and w gets identified with f and e0 ; in particular, the transverse measures of these three segments are all equal. (Strictly speaking, y 0 need not lie on R as we choose it, but this does not affect the argument.) Now consider the prongs at the zero p, for example. These project to distinct segments in Tϕ˜ , which are then either projected to segments in T intersecting R¯ only in y; ¯ or alternatively there may be a folding identifying them with subsegments of

200

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

g(x)

E y

e

p

q

y0 R

w

e0

f

+

0 or − p

f0 w0

e00

− g(R)

x

+

(a)

+

0 or −

(b) Figure 4

¯ Let us label the prongs with a + sign if there is a folding onto a subsegment of R. [y, ¯ E), with a − sign if there is a folding onto a subsegment of [x, ¯ y], ¯ and with a 0 if no folding occurs or if the edge is folded along some other segment (see Figure 4(b)). Since p is connected by the vertical leaf e to R, we label the adjacent horizontal segments with + and − accordingly. Working our way around p in the clockwise direction, and repeatedly using the “pushing” argument from Section 5 of [DDW], we find that every second prong must be labeled + while the intervening prongs may get either − or 0 (recall Lemma 3.2). Therefore, there must be an odd number of prongs between e0 and the one adjacent to e, which is identified in the leaf space with a portion of f . A similar argument applies to q, e00 , and f 0 . Let γ˜ 0 be the path from y 0 to w obtained by following f, e, e0 , e00 , and then f 0 . Because of the to the folding of the prongs at p and q, one may easily R odd sign ˜ is the just the transverse measure of the segment f . Indeed, verify that γ˜ 0 Re(ω) suppose ϕ˜ has a zero of order 2n at some point p, and choose a local conformal coordinate z such that ϕ(z) ˜ = z2n dz2 . Then the foliation is determined by the leaves of ξ = zn+1 /n + 1. If ζ is a primitive 2n + 2 root of unity, then z 7→ ζ k z takes one radial prong to another, with k − 1pprongs in between (in the counterclockwise direction). The outward integrals of Re ϕ˜ along these prongs to a fixed radius differ by (−1)k . Our analysis implies that k −1 is odd, so k is even, and we have the correct 0 cancellation. If we R extend γ˜ along the horizontal leaves R and g(R) to a path γ˜ from ˜ = dT (x, ¯ g(x)) ¯ as required. In general, there are additional x to g(x), then γ˜ Re(ω) intervening zeros, and the procedure above applies to each of these with no further complication. Thus, ` restricted to b 0 is given by the periods of Re(ω). ˆ Since the real parts of the periods of an abelian differential determine the differential uniquely, ωˆ must agree b of the form in Proposition 3.6; in particular, it descends to 6. with the pullback to 6 This completes the proof of Theorem 3.7. We immediately have the following.

THE MORGAN-SHALEN COMPACTIFICATION

201

Corollary 3.8. Fix an abelian length function `. Then for any tree T with length function ` and any equivariant harmonic map v : H2 → T , we have Hopf(v) = Hopf(u) where u is the equivariant harmonic function from Proposition 3.6 corresponding to `. We are now prepared to define the map (1.1). Take a representative ` of [`] ∈

PL(0). There are two cases: If ` is nonabelian, use Proposition 3.5 to define

H ([`]) = ϕ. If ` is abelian, use Proposition 3.6. The main result of this section is the following. Theorem 3.9. The map H : PL(0) → SQD defined above is continuous.

Proof. Suppose [`i ] → [`], and assume, to the contrary, that there is a subsequence, which we take to be the sequence itself, such that H ([`i ]) → ϕ 6= H ([`]). Choose representatives `i → `. If there is a subsequence {i 0 } consisting entirely of abelian length functions, then ` itself must be abelian, and from the construction of Proposition 3.6, H (`i 0 ) → H (`), a contradiction. Thus, we may assume all the `i ’s are nonabelian. There exist R-trees Ti , unique up to isometry, and equivariant harmonic maps ui : H2 → Ti . We claim that the ui have uniform modulus of continuity (cf. [KS2, Prop. 3.7]). Indeed, by [GS, Thm. 2.4], it suffices to show that E(ui ) is uniformly bounded. If E(ui ) → ∞, then the same argument as in [DDW, proof of Thm. 3.1] would give a contradiction. It follows by [KS2, Prop. 3.7] that there is a subsequence {i 0 } (which we assume is the sequence itself) such that ui converges in the pullback sense to an equivariant harmonic map u : H2 → T , where T is a minimal R-tree with length function equal to `. In addition, by [KS2, Theorem 3.9], Hopf(ui ) → Hopf(u). If ` is nonabelian, we have a contradiction by Proposition 3.1; if ` is abelian, we have a contradiction by Corollary 3.8. 4. Proof of the main theorem. We show how the results of the previous section, combined with those in [KS2] and [DDW], give a proof of the main theorem. We first reduce the proof of the continuity of h¯ to the following. Claim. If [ρi ] ∈ X(0) is a sequence of representations converging to [`] ∈ PL(0), then h([ρi ]) → H ([`]). Suppose the claim holds and h¯ is not continuous. Then we may find a sequence ¯ i ) → y 6= h(x). ¯ If x ∈ PL(0) so that xi ∈ PL(0) ∪ X(0) such that xi → x but h(x ¯h(x) = H (x), the claim rules out the possibility that there is a subsequence of {xi } in X(0). In this case then, there must be a subsequence in PL(0). But this contradicts the continuity of H , by Theorem 3.9. Thus, x must be in X(0). But then we may assume that {xi } ⊂ X(0), so that h¯ = h on {xi }. The continuity of the homeomorphism h : X(0) → MHiggs then provides the contradiction. It remains to prove the claim. Again suppose to the contrary that [ρi ] → [`] but h([ρi ]) → ϕ 6 = H ([`]) for ϕ ∈ SQD. First, suppose that there is a subsequence [ρi 0 ] with reducible representative representations ρi 0 : 0 → SL(2, C). Up to conjugation,

202

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

which amounts to changing the choice of representative, we may assume each ρi 0 fixes a given vector 0 6 = v ∈ C2 , and that the action on the 1-dimensional line spanned by v is determined by a character χi 0 : 0 → C∗ . The associated translation length functions `i 0 are therefore all abelian, and so [`] must be abelian. We may assume there is a representative ` such that `i 0 → `. By Proposition 3.6 there are harmonic functions u, ui 0 : H2 −→ R ' C∗ /U (1) ,→ H3 , equivariant for the induced action of 0 on C∗ by χ and χi 0 , respectively. These converge (after rescaling) to a harmonic function u : H → R, equivariant with respect to an action on R with translation length function `. Since the length functions converge, it follows from the construction in Proposition 3.6 that Hopf(ui 0 ) → Hopf(u), and so by the definition of H , h([ρi 0 ]) → H ([`]), a contradiction. Second, suppose that there is a subsequence [ρi 0 ] of irreducibles. Then by the main result of [DDW] we can find a further subsequence (which we take to be the sequence itself) of ρi 0 -equivariant harmonic maps ui 0 : H2 → H3 converging in the sense of Korevaar-Schoen to a harmonic map u : H2 → T , where T is a minimal R-tree with an action of 0 by isometries and length function ` in the class [`]. As above, Hopf(ui 0 ) → Hopf(u), so by the definition of H , h([ρi 0 ]) → H ([`]), a contradiction. Since we have accounted for both possible cases, this proves the claim. 5. Convergence of length functions. In this final section we briefly sketch an alternative argument for the convergence to the boundary in the main theorem, based on a direct analysis of length functions, more in the spirit of [W1]. The generalization of estimates for equivariant harmonic maps with target H2 to maps with target H3 has largely been carried out by Minsky [M]. We discuss this point of view, however, since it reveals how and why the folding of the dual tree Tϕ˜ occurs. The first step is to analyze the behavior of the induced metric for a harmonic map u : H2 → H3 of high energy (at the points where u is an immersion). As usual we denote by ϕ˜ the Hopf differential for the map u. Because of equivariance, ϕ˜ is the lift of a holomorphic quadratic differential ϕ on 6. Recall the norm kϕk from the introduction, and let Z(ϕ) ⊂ 6 denote the zero set of ϕ. We also set µ to be the 2 . Beltrami differential associated to the pullback metric u∗ dsH 3 Lemma 5.1. Fix δ, T > 0. Then there are constants B, α > 0 such that for all u, µ, and ϕ as above, kϕk ≥ T , and all p ∈ 6 satisfying dist σ (p, Z(ϕ)) ≥ δ, we have   1 (p) < Be−αkϕk . log |µ| Proof. This result is proven in [M, Lemma 3.4]. One needs only a statement concerning the uniformity of the constants appearing there. However, by using the compactness of SQD, one easily shows the following: For δ > 0 there is a constant c(δ) > 0 such that, for all ϕ ∈ SQD and all p ∈ 6 such that dist σ (p, Z(ϕ)) ≥ δ,

THE MORGAN-SHALEN COMPACTIFICATION

203

the disk U of radius c(δ) ˜ (with respect to the singular flat metric |ϕ|) around p is embedded in 6 and contains no zeros of ϕ. Then the result cited above applies. This estimate is all that is needed to prove convergence in the case where there cannot be a folding of the dual tree Tϕ˜ such that the composition of projection to Tϕ˜ with the folding is harmonic. From Lemma 3.2, this is guaranteed, for example, if ϕ has only simple zeros. For simplicity, in this section we assume all representations are irreducible. Theorem 5.2. Given an unbounded sequence ρj of representations with MorganShalen limit [`], let uj : H2 → H3 be the associated ρj -equivariant harmonic maps. Suppose that for ϕ˜j = Hopf(uj ) we have ϕj /kϕj k → ϕ ∈ SQD, where ϕ has only simple zeros. Then [`] = [`T ], where T = Tϕ˜ . Proof. We prove the convergence of length functions in two steps. First, we compare the length of closed curves γ in the free isotopy class [γ ] with respect to the induced metric from uj to the length with respect to the transverse measure. Second, we compare the length of the image by uj of a lift γ˜ to H2 of γ to the translation length in H3 of the conjugacy class that [γ ] represents. The basic idea is that the image of γ˜ very nearly approximates a segment of the hyperbolic axis for ρj ([γ ]). For ϕ and [γ ] as above, let `ϕ ([γ ]) denote the infimum over all representatives γ of [γ ] of the length of γ with respect to the vertical measured foliation defined by ϕ. If u : H2 → H3 is a differentiable equivariant map, we define `u ([γ ]) as follows: For each representative γ of [γ ], where [γ ] corresponds to the conjugacy class of g ∈ 0, lift γ to a curve γ˜ at a point x ∈ H2 , terminating at gx. We then take the infimum over all such γ˜ of the length of u(γ˜ ). This is `u ([γ ]), and by the equivariance of u it is independent of the choice of x. Finally, recall that the translation length `ρ ([γ ]) for a representation ρ : 0 → SL(2, C) is defined in Section 2. Given ε > 0, let QDε ⊂ QD \{0} denote the subset consisting of holomorphic quadratic differentials ϕ having only simple zeros, and such that the zeros are pairwise at least a σ -distance ε apart. Notice that for t 6= 0, t QDε = QDε . The next result is a consequence of Lemma 5.1. Proposition 5.3. For all classes [γ ] and differentials ϕ ∈ QDε , there exist constants k and η depending on kϕk, [γ ], and ε, so that k `ϕ ([γ ]) + η ≥ `u ([γ ]) ≥ `ϕ ([γ ]) where k → 1 and ηkϕk−1/2 → 0 as kϕk → ∞ in QDε . Sketch of proof. We first need to choose an appropriate representative for the class of [γ ]. Such a choice was explained in [W1]. Namely, for δ > 0 and a given ϕ, we can find a representative γ consisting of alternating vertical and horizontal segments and having the transverse measure of the class [γ ]. Moreover, because the zeros of ϕ are

204

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

simple, for sufficiently small δ we can also guarantee that γ avoid a δ neighborhood of the zeros. Now the proof follows as in [W1, Lemma 4.6]. Note that along a harmonic maps ray (that is, a sequence ui such that Hopf(ui ) is of the form ti ϕ for a fixed ϕ and an increasing unbounded sequence ti ), we no longer necessarily have monotonicity of the norm of the Beltrami differentials |µ(ti )|. The argument for the estimate still applies, however, since the representatives γ are uniformly supported away from the zeros. There, we apply the estimate Lemma 5.1. The details are omitted. Next, we compare `u with the translation length in H3 . Proposition 5.4. Let ρ : 0 → SL(2, C) and u : H2 → H3 be the ρ-equivariant harmonic map with ϕ˜ = Hopf(u). Suppose ϕ ∈ QDε . For all classes [γ ] there exist constants m and ζ depending on kϕk, [γ ], and ε, so that m `ρ ([γ ]) + ζ ≥ `u ([γ ]) ≥ `ρ ([γ ]), where m → 1 and ζ kϕk−1/2 → 0 as kϕk → ∞ in QDε . Combining Propositions 5.3 and 5.4 proves Theorem 5.2. Sketch of proof of Proposition 5.4. One observes that away from the zeros, the images of the horizontal leaves of the foliation of ϕ˜ closely approximate (long) geodesics in H3 , while by Lemma 5.1 the images of vertical leaves collapse. More precisely, the following is proven in [M, Thm. 3.5]. Lemma 5.5. Fix δ > 0, a representation ρ : 0 → SL(2, C), and let u : H2 → be the ρ-equivariant harmonic map with Hopf differential ϕ. ˜ Let β˜ be a seg˜ ment of the horizontal foliation of ϕ˜ from x to y and suppose that, for all p˜ ∈ β, distσ (p, Z(ϕ)) ≥ δ. Then there is an ε, exponentially decaying in kϕk, such that the following hold: ˜ is uniformly within ε of the geodesic in H3 from u(x) to u(y). (1) u(β) ˜ is within ε of dist H3 (u(x), u(y)). (2) The length of u(β)

H3

The following is a key result. Lemma 5.6. Given g ∈ SL(2, C), let `(g) denote the translation length for the action of g on H3 . Suppose that s ⊂ H3 is a curve that is g invariant and satisfies the following property: For any two points x, y ∈ s, the segment of s from x to y is uniformly within a distance 1 of the geodesic in H3 joining x and y. Then there is a universal constant C such that ¡
inf distH3 x, g(x) ≤ `(g) + C.

x∈s

Proof. The intuition is clear; such an s must be an “approximate axis” for g. The proof proceeds as follows: Choose x ∈ s, and let c denote the geodesic in H3 from x to

THE MORGAN-SHALEN COMPACTIFICATION

205

g(x). By [Cp, Lemma 2.4] there exists a universal constant D and a subgeodesic c˜ of c with the property that |length(c) ˜ − `(g)| ≤ D. Let a and b be the endpoints of c˜ closest to x and g(x), respectively. By the construction of c˜ in the reference cited above, it follows that dist H3 (b, g(a)) ≤ D; hence, distH3 (b, g(b)) ≤ `(g) + 2D. Now by the assumption on c, there is a point y ∈ s close to b, so that dist H3 (y, g(y)) ≤ `(g) + C, where C = 2(D + 1). Proceeding with the proof of Proposition 5.4, choose the representative γ as discussed in Proposition 5.3. We may then lift to γ˜ ⊂ H2 so that γ˜ is invariant under the action of g. Now γ˜ is written as a union of horizontal and vertical segments of the foliation of ϕ. ˜ Let s = u(γ˜ ).Then Lemmas 5.1 and 5.5 imply that s satisfies the hypothesis of Lemma 5.6. Moreover, using Lemma 5.5 again, along with some elementary hyperbolic geometry, one can show that inf x∈s distH3 (x, g(x)) is approximated by the length of a segment of u(γ˜ ) from a point u(x) to u(gx). We leave the precise estimates to the reader. From Lemma 3.2, we see that foldings can only arise when the Hopf differentials converge in SQD to differentials with multiplicity at the zeros. From the point of view taken here, this corresponds to the fact that the representatives for closed curves γ chosen above may be forced to run into zeros of the Hopf differential where the estimate Lemma 5.1 fails. These may cause nontrivial angles to form in the image u(γ˜ ) which, in the limit, may fold the dual tree. Consider again the situation along a harmonic maps ray with differential ϕ. Given [γ ] corresponding to the conjugacy class of an element g ∈ 0, representatives γ still may be chosen as in the proof of Proposition 5.3 so that the horizontal segments remain bounded away from the zeros. However, it may happen that a vertical segment passes through a zero of order two or greater. For simplicity, assume this happens once. Divide γ into curves γ1 , γ2 , and γv , where γv is the offending vertical segment, and lift to segments γ˜1 , γ˜2 , and γ˜v in H2 . Note that one end point of each of the γ˜i ’s corresponds to either end point of γ˜v , and the other end points of the γ˜i ’s are related by g. By the Lipschitz estimate for harmonic maps to nonpositively curved spaces, we have a bound on the distance in H3 between the end points of u(γ˜v ) in terms of the length of γv and the energy E(u)1/2 (cf. [S2]). Thus, the rescaled length is small; in fact, since the length of γv is arbitrary, the distance converges to zero. On the other hand, the previous argument applies to the segments u(γ˜1 ) and u(γ˜2 ), which are connected by u(γ˜v ). Adding the geodesic in H3 joining the other end points of u(γ˜1 ) and u(γ˜2 ) forms an approximate geodesic quadrilateral, which, in the rescaled limit, converges either to an edge | (no folding) or a possibly degenerate tripod a (folding). In both cases, there is an edge that, by the same argument as in the proof of Proposition 5.4, approximates the axis of ρj (g) for large j . At the same time, the rescaled length of this segment is approximated by the translation length of the element g acting on a folding of Tϕ˜ at the zero. An interesting question is whether this approach may be used to determine precisely

206

DASKALOPOULOS, DOSTOGLOU, AND WENTWORTH

the fibers of the map h¯ in the main theorem. While the essential ideas are outlined here, a complete description is not yet available. We will return to this issue in a future work. References [Cp] [C] [CM] [CS] [DDW] [D] [FW] [FLP] [GS]

[H] [HM] [KS1] [KS2] [M] [MO] [MS1] [MS2] [S1]

[S2]

[Si] [Sk] [Wan] [W1]

D. Cooper, Degenerations of representations into SL(2, C), preprint, 1998. K. Corlette, Flat G-bundles with canonical metrics, J. Differential Geom. 28 (1988), 361–382. M. Culler and J. Morgan, Group actions on R-trees, Proc. London Math. Soc. (3) 55 (1987), 571–604. M. Culler and P. Shalen, Varieties of group representations and splittings of 3-manifolds, Ann. of Math. (2) 117 (1983), 109–146. G. Daskalopoulos, S. Dostoglou, and R. Wentworth, Character varieties and harmonic maps to R-trees, Math. Res. Lett. 5 (1998), 523–534. S. K. Donaldson, Twisted harmonic maps and the self-duality equations, Proc. London Math. Soc. (3) 55 (1987), 127–131. B. Farb and M. Wolf, Harmonic splittings of surfaces, preprint, 1998. A. Fathi, F. Laudenbach, and V. Poénaru, eds., Travaux de Thurston sur les surfaces: Séminaire Orsay, Astérisque 66–67, Soc. Math. France, Paris, 1979. M. Gromov and R. Schoen, Harmonic maps into singular spaces and p-adic superrigidity for lattices in groups of rank one, Inst. Hautes Études Sci. Publ. Math. 76 (1992), 165–246. N. J. Hitchin, The self-duality equations on a Riemann surface, Proc. London Math. Soc. (3) 55 (1987), 59–126. J. Hubbard and H. Masur, Quadratic differentials and foliations, Acta Math. 142 (1979), 221–274. N. Korevaar and R. Schoen, Sobolev spaces and harmonic maps for metric space targets, Comm. Anal. Geom. 1 (1993), 561–659. , Global existence theorems for harmonic maps to non-locally compact spaces, Comm. Anal. Geom. 5 (1997), 333–387. Y. Minsky, Harmonic maps into hyperbolic 3-manifolds, Trans. Amer. Math. Soc. 332 (1992), 607–632. J. Morgan and J.-P. Otal, Relative growth rates of closed geodesics on a surface under varying hyperbolic structures, Comment. Math. Helv. 68 (1993), 171–208. J. Morgan and P. Shalen, Valuations, trees, and degenerations of hyperbolic structures, I, Ann. of Math. (2) 120 (1984), 401–476. , Free actions of surface groups on R-trees, Topology 30 (1991), 143–154. R. Schoen, “Analytic aspects of the harmonic map problem” in Seminar on Nonlinear Partial Differential Equations (Berkeley, Calif., 1983), ed. S. S. Chern, Math. Sci. Res. Inst. Publ. 2, Springer-Verlag, New York, 1984, 321–358. , “The role of harmonic mappings in rigidity and deformation problems” in Complex Geometry (Osaka, 1990), Lecture Notes in Pure and Appl. Math. 143, Dekker, New York, 1993, 179–200. C. T. Simpson, Moduli of representations of the fundamental group of a smooth projective variety, II, Inst. Hautes Études Sci. Publ. Math. 80 (1994), 5–79. R. K. Skora, Splittings of surfaces, J. Amer. Math. Soc. 9 (1996), 605–616. T. Wan, Constant mean curvature surface, harmonic maps, and universal Teichmüller space, J. Differential Geom. 35 (1992), 643–657. M. Wolf, The Teichmüller theory of harmonic maps, J. Differential Geom. 29 (1989), 449–479.

THE MORGAN-SHALEN COMPACTIFICATION [W2] [W3]

207

, Harmonic maps from surfaces to R-trees, Math. Z. 218 (1995), 577–593. , On realizing measured foliations via quadratic differentials of harmonic maps to R-trees, J. Anal. Math. 68 (1996), 107–120.

Daskalopoulos: Department of Mathematics, Brown University, Providence, Rhode Island 02912, USA; [email protected] Dostoglou: Mathematical Sciences Building, University of Missouri, Columbia, Missouri 65211, USA; [email protected] Wentworth: Department of Mathematics, University of California, Irvine, California 92697, USA; [email protected]

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

S-ARITHMETICITY OF DISCRETE SUBGROUPS CONTAINING LATTICES IN HOROSPHERICAL SUBGROUPS HEE OH 0. Introduction. Let Qp be the field of p-adic numbers, and let Q∞ = R. Let Gp be a connected semisimple Qp -algebraic group. The unipotent radical of a proper parabolic Qp -subgroup of Gp is called a horospherical subgroup. Two horospherical subgroups are called opposite if they are the unipotent radicals of two opposite parabolic subgroups. In [5] and [6], we studied discrete subgroups generated by lattices in two opposite horospherical subgroups in a simple real algebraic group with real rank at least 2. This work was inspired by the following conjecture posed by G. Margulis. Conjecture 0.1. Let G be a connected semisimple R-algebraic group such that R-rank (G) ≥ 2, and let U1 , U2 be a pair of opposite horospherical R-subgroups of G. For each i = 1, 2, let Fi be a lattice in Ui (R) such that H ∩ Fi is finite for any proper normal R-subgroup H of G. If the subgroup generated by F1 and F2 is discrete, then it is an arithmetic lattice in G(R). We settled the conjecture in many cases, including the case when G is an absolutely simple real split group with G(R) not locally isomorphic to SL3 (R) (see [5]). In this paper, we study a problem analogous to the conjecture in a product of real and p-adic algebraic groups. The following is a special case of the main theorem, Theorem 4.3. Theorem 0.2. Let S be a finite set of valuations of Q including the archimedean valuation ∞. For each p ∈ S, let Gp be a connected semisimple algebraic Qp group without any Qp -anisotropic factors, Q and let U1p , U2p be Qa pair of opposite horospherical subgroups of Gp . Set G = p∈S Gp (Qp ), U1 = p∈S U1p (Qp ), and Q U2 = p∈S U2p (Qp ). Assume that G∞ is absolutely simple R-split with rank at least 2 and that if G∞ (R) is locally isomorphic to SL3 (R), then U1∞ is not the unipotent radical of a Borel subgroup of G∞ . Let F1 and F2 be lattices in U1 and U2 , respectively. If the subgroup generated by F1 and F2 is discrete, then it is a nonuniform S-arithmetic lattice in G. If p is a nonarchimedean valuation of Q, then no horospherical subgroup of Gp (Qp ) admits a lattice. Moreover, there is no infinite unipotent discrete subgroup in a p-adic Lie group. Therefore it is necessary to assume in Theorem 0.2 that S contains the archimedean valuation ∞. Received 12 January 1999. 1991 Mathematics Subject Classification. Primary 22E40; Secondary 22E46, 22E50. Author’s work partially supported by National Science Foundation grant number DMS-9801136. 209

210

HEE OH

In the group G = SLm (R) × SLn (Qp ), one can ask if it is possible to generate a discrete subgroup by taking lattices from two opposite horospherical subgroups of G. One interesting aspect of Theorem 0.2 says that, in general, the answer is no. Q Corollary 0.3. Keeping the same notation as in Theorem 0.2, set G = p∈S Gp . Suppose that there exist lattices F1 and F2 in U1 and U2 , respectively, that generate a discrete subgroup of G. Then (1) G is typewise homogeneous; that is, for each p ∈ S, there is an isogeny fp : Gp → G∞ ; in particular, Gp is absolutely almost simple; (2) for each p ∈ S, U1p is isomorphic to U1∞ . In particular, we have the following corollary. Corollary 0.4. With the same notation as in Corollary 0.3, suppose that G is not typewise homogeneous. Then any subgroup generated by lattices in a pair of opposite horospherical subgroups of G is not discrete. Corollary 0.3 follows from Theorem 0.2 simply by the definition of an S-arithmetic subgroup of G (see Section 1.6). Examples. In the following groups there are no discrete subgroups containing lattices in opposite horospherical subgroups: (1) G = SLm (R) × SLn (Qp ) for any m 6= n such that m ≥ 4 and n ≥ 2; (2) G = SLm (R) × SLn1 (Qp ) × SLn2 (Qp ) for any n1 , n2 ≥ 2 and m ≥ 4; (3) G = SO(m, m)R × SLn (Qp ) for any n ≥ 2 and m ≥ 2. As a corollary of Theorem 0.2, we obtain that as long as a discrete subgroup of G intersects a pair of opposite horospherical subgroups as lattices, then it is a lattice in the ambient group G as well. This is not always true in rank-1 simple groups, for instance, in SL2 (R) (see the remark after Theorem 0.2 in [5]). Corollary 0.5. Let S, Gp , p ∈ S, and G be as in Theorem 0.2. Let 0 be a discrete subgroup of G. Then 0 is a nonuniform S-arithmetic lattice in G if and only if for each p ∈ S there exists a pair U1p , U2p of opposite Q horospherical subgroups of Gp such that 0 ∩ Ui is a lattice in Ui , where Ui = p∈S Uip (Qp ) for each i = 1, 2. In that case, G is typewise homogeneous. For the proof of Theorem 0.2, denote by 0F1 ,F2 the subgroup generated by F1 and F2 , and denote by 0F∞1 ,F2 the image of the subgroup 0F1 ,F2 ∩ G∞ (R) × Q p∈S, p6 =∞ Gp (Zp ) under the natural projection G → G∞ (R). Using the results from [5], we first obtain a Q-form on G∞ with respect to which the subgroup 0F∞1 ,F2 is an arithmetic lattice in G∞ (R). Then applying a special case of Margulis’s superrigidity (see Theorem 4.2), we show that this Q-form of G∞ endows a Q-form on each Gp , p ∈ S, so that 0F1 ,F2 becomes an S-arithmetic subgroup in G. We also need some results on the classification of lattices in the product of real and p-adic nilpotent Lie groups (see Corollary 2.7). In fact our method shows that in Theorem 0.2 we can

S-ARITHMETICITY OF SOME DISCRETE SUBGROUPS

211

remove the assumption that G∞ is R-split as long as G∞ is absolutely simple with real rank at least 2 and Conjecture 0.1 holds for G∞ . In [5], we proved directly that any discrete subgroup of G∞ (R) containing lattices in opposite horospherical subgroups is an arithmetic subgroup, rather than using Margulis’s arithmeticity theorem or superrigidity theorem. Therefore, the methods used provide an alternative proof of the arithmeticity theorem in the case of nonuniform lattices for the groups considered in [5]. In the present paper, however, we use a special case of Margulis’s superrigidity theorem in order to extend the arithmetic structure of 0F∞1 ,F2 , obtained in [5], to an S-arithmetic structure of 0F1 ,F2 . Before we close the introduction, we describe the following open case of Margulis’s conjecture, which is believed to be a challenging case. Open problem. Consider the following two subgroups of PSL2 (R) × PSL2 (R):     1 0 1 R2 , U2 = . U1 = R2 1 0 1 For i = 1, 2, choose two linearly independent vectors ui and vi in R2 such that {nui + mvi | n, m ∈ Z} does not contain any element of the form (x, 0) or (0, x) for any x 6 = 0. By the natural isomorphism of Ui with R2 , we consider ui and vi as elements of Ui . Then the question dealt with by Conjecture 0.1 can be regarded as the following discreteness criterion problem: Which four elements u1 , u2 , v1 , and v2 generate a discrete subgroup? From the classification of the Q-forms of PSL2 (R) × PSL2 (R) (cf. [11]), it is not hard to see that Conjecture 0.1 implies that u1 , v1 , u2 , and v2 can generate a discrete subgroup only in the case when the elements u1 , v1 , u2 , and v2 are from some Hilbert modular group of PSL2 (R) × PSL2 (R). It then follows from the results of [12] that the discrete subgroup generated by those four elements is in fact a Hilbert modular group. Here we say that 0 is a Hilbert modular group of PSL2 (R)×PSL2 (R) if there is a real quadratic extension field k of Q such that 0 is conjugate to a subgroup of finite index in ª ¡ σ
g, g | g ∈ PSL2 (J ) , where J is the ring of integers of k and σ : k → k is the nontrivial Galois automorphism of k. It seems plausible that an analogous conjecture in the setting P of Theorem 0.2 holds under the assumption that the S-rank of G, that is, the p∈S Qp -rank of G, is at least 2 (without any assumption on G∞ ). The first question in this regard would be to ask whether the conjecture is true for G = PSL2 (R) × PSL2 (Qp ). Acknowledgments. Thanks are due to Professor G. Margulis who drew my attention to Theorem 4.2. This work was done during my visits to the University of Bielefeld and the University of Chicago. I would like to thank the members of the mathematics departments at both universities for their hospitality as well as their invitations.

212

HEE OH

1. Notation 1.1. For a set S of valuations of Q, denote by Sf the subset of S consisting of nonarchimedean valuations (i.e., Sf = S − {∞}), where ∞ denotes the archimedean valuation of Q. Denote by Qp the field of p-adic numbers with the normalized absolute value | |p and set Q∞ = R. Denote by Zp the ring of p-adic integers, that is, Zp = {x ∈ Qp | |x|p ≤ 1}. For a valuation p on Q and a connected Qp -algebraic group Gp , we denote by Gp the Qp -rational points of Gp and by Gp+ the subgroup of Gp generated by all of its unipotent 1-parameter subgroups. 1.2. By Lie Gp we denote the Lie algebra of the group Gp considered as a Lie group over Qp , which is naturally identified with the Qp -points of the Lie algebra Lie Gp . Q 1.3. For each p ∈ S, we denote by prp the natural projection map from p∈S Gp Q to Gp . The notation pr ∞ denotes the projection of G∞ × p∈Sf G(Zp ) to G∞ . For Q a subgroup H of p∈S Gp , the notation H ∞ denotes the image of H ∩ (G∞ × Q ∞ p∈Sf Gp (Zp )) under the projection pr . 1.4. The notation ZS denotes the subring of Q generated by Z and {(1/p) | p ∈ Sf }. Q 1.5. For G = p∈S Gp , we say that G has a Q-form if there exist a connected algebraic Q-group H and a Qp -isomorphism φp : H → Gp for each p ∈ S. A subgroup M ofQG is said to be defined over Q if there is a Q-subgroup M0 of H such that M = p∈S φp (M0 ). For a subring J of Q and a subgroup M of G defined Q over Q, the notation M(J ) denotes the set { p∈S φp (x) ∈ G | x ∈ M0 (J )}, where Q M = p∈S φp (M0 ). 1.6. Q Let Gp be a connected algebraic Qp -group for each p ∈ S. A subgroup 0 of G = p∈S Gp (Qp ) is called an S-arithmetic (or simply arithmetic if S = {∞}) -isogeny fp : subgroup of G if there exist a connected algebraic Q Qp -group G0p , a QpQ 0 0 0 Gp → Gp for each p ∈ S, and a Q-form on G = p∈S Gp such that ( p∈S fp )(0) is commensurable to G0 (ZS ). An epimorphism with finite kernel is called an isogeny. 1.7. A discrete subgroup 0 of a locally compact group G is called a lattice if G/ 0 has a finite G-invariant Borel measure. A lattice 0 in G is called uniform if G/ 0 is compact, and it is nonuniform otherwise. 2. Discrete unipotent subgroups 2.1. Let S be a finite set of valuations of Q Qincluding ∞. For each p ∈ S, let Gp be a connected algebraic Qp -group. Let G = p∈S Gp . Proposition 2.1. If 0 is a discrete subgroup (resp., lattice) in G, then 0 ∞ is a discrete subgroup (resp., lattice) in G∞ .

S-ARITHMETICITY OF SOME DISCRETE SUBGROUPS

213

compact, the subgroup 0 ∞ is discrete if 0 is. Proof. Since the kernel of pr ∞ isQ Let 0 be a lattice in G. Since G∞ × p∈Sf Gp (Zp ) is an open subgroup of G, the Q Q intersection 0 ∩ (G∞ × p∈Sf Gp (Zp )) is also a lattice in G∞ × p∈Sf Gp (Zp ). Hence, 0 ∞ is a lattice in G∞ by the compactness of the kernel of pr ∞ . It also follows from the above proof that if 0 is a uniform lattice in G, then 0 ∞ is a uniform lattice in G∞ as well. Lemma 2.2. For p ∈ Sf , let Gp be unipotent. If L is a closed subgroup of Gp such that Gp /L carries a finite Gp -invariant Borel measure, then L = Gp . Proof. Suppose this is not so. The general case is easily reduced to the case when Gp is abelian. Then there is a 1-parameter unipotent subgroup U = Qp x of Gp that is not contained in L. If {p−n x | n > n0 } ∩ L = ∅ for some positive integer n0 , then it contradicts the assumption that Gp /L carries a finite Gp -invariant Borel measure (since Qp /p−n0 Zp is an infinite countable set). Hence there exists a sequence xi ∈ L ∩ U for all i ≥ 1 such that xi → ∞ as i → ∞. Since L is closed and Z(xi ) ⊂ L, we have Zp (xi ) ⊂ L for all i ≥ 1. Note that U = ∪i≥1 Zp (xi ) since xi → ∞ as i → ∞. Therefore U ⊂ L, contradicting the assumption. Lemma 2.3. For each p ∈ S, let Gp be unipotent. If 0 is a lattice in G, then (1) prp (0) is dense in Gp for each p ∈ Sf ; (2) pr ∞ (0) is Zariski-dense in G∞ . Proof. Let p ∈ Sf . Denote by L the closure of prp (0) in Gp . Since G/ 0 has a finite G-invariant measure, then so does Gp /L (see, e.g., [4, Chap. II, Lemma 6.1]). Therefore by Lemma 2.2, L = Gp . This implies (1). For (2), the subgroup pr ∞ (0) is a lattice in G∞ by Proposition 2.1. Note that G∞ is a connected and simply connected nilpotent Lie group as is any real unipotent algebraic group. It is well known that any lattice in a connected and simply connected nilpotent Lie group is Zariski-dense (cf. [8]). This completes the proof. Let V be a connected unipotent algebraic Q-group, Vp = V(Qp ) and V = V . For a subring J of Q, we identify V (J ) with its image under the diagonal p p∈S embedding of V(Q) into V . It is well known that V (ZS ) is a uniform lattice in V (cf. [7, Thm. 5.7]).

Q 2.2.

Lemma 2.4. Let F be a discrete subgroup of V . Then the restriction pr∞ |F is injective. Proof. Without loss of generality, we may assume that V ⊂ GLN . Suppose that there is a nontrivial element x ∈ F such that pr∞ (x) = e. Since x is unipotent, we have that for each (x − e)n = 0 for some n ∈ N. Then, by the binomial formula, Q m m p ∈ Sf , prp (x s ) tends to e as m → ∞, where s = p∈Sf p. Hence, x s → e as m → ∞. This contradicts the assumption that F is discrete and thus proves our claim.

214

HEE OH

Lemma 2.5. For any nonzero integers m and d, there exists a nonzero integer k such that V (kZS ) ⊂ mV (dZS ), where mV (ZS ) = {x m | x ∈ V (ZS )}. Proof. Since V is unipotent, there exists an integer n such that (x − e)n = e for any x ∈ V . Without loss of generality, we may assume that V ⊂ GLN and  ª V (dZS ) = x ∈ V | (x − e) is a matrix whose entries are in dZS . P j +1 /j )uj ). Therefore For any x = e + u ∈ V , we have log x 1/m = (1/m)( n−1 j =1 ((−1) we can find k such that if x ∈ V (kZS ), then log x 1/m ∈ log V (dZS ). Hence V (kZS ) ⊂ mV (dZS ). 2.3. It is well known that any lattice in a real algebraic unipotent group is an arithmetic subgroup (cf. [3]). Analogously, we now prove that any lattice in V = Q p∈S Vp is an S-arithmetic subgroup. Q We denote by V the product p∈S Lie Vp . There exists an integer b such that for any subgroup U in V , bhlog U i ⊂ log U , where hlog U i denotes the subring of V generated by log U (cf. [3, Lemma 5.2]). For a discrete subgroup F in V , we set 1F = bhlog F i. It is then clear that 1F is a discrete subgroup in V such that b log F ⊂ 1F ⊂ log F . Proposition 2.6. Let F be a discrete subgroup of V such that pr ∞ (F ) is Zariskidense in V∞ and prp (F ) is dense in Vp for each p ∈ Sf . Then F is an S-arithmetic subgroup of V . Proof. Since log : Vp → Vp is both a rational map and a homeomorphism for each p ∈ S, we have that pr ∞ (1F ) is Zariski-dense in V∞ and prp (1F ) is dense in Vp for each p ∈ Sf . We first show that there exists a Q-form on V such that 1F ⊂ V(Q). Since 1F ∞ is a Zariski-dense discrete subgroup in V∞ , which is a connected and simply connected nilpotent Lie group, then 1F ∞ is a lattice in V∞ (see, e.g., [8]). Therefore there exists a Q-form on V∞ such that 1F ∞ = V∞ (Z). (Note that this Q-form on V∞ does not necessarily coincide with the Q-form on V∞ given by the original Q-form on V with which we started.) We denote by V∞ (Qp ) the completion of V(Q) with respect to the p-adic norm. Note that 1F ∞ is a basis of the vector space V∞ (Qp ) over Qp . Therefore, in order to define a Qp -linear map φp : V∞ (Qp ) → Vp , it is enough to define it on 1F ∞ . For each x ∈ 1F ∞ , there exists an element yx ∈ 1F such that pr∞ (yx ) = x. By Lemma 2.4, such an element yx is unique. We set φp (x) = prp (yx ). We show that the map φp is a Qp -isomorphism. Since dim V∞ (Qp ) = dim Vp , it suffices to show that φp is onto. To show this, it is again enough to show that prp (1F ) ⊂ Im φp , since prp (1F ) is dense in Vp by assumption. For x = log y ∈ 1F , there is an Q n n ∈ N such that pr∞ (y s ) ∈ F ∞ where s = p∈Sf p. Then pr∞ (s n x) ∈ 1F ∞ , and hence prp (x) = φp (s −n pr∞ (s n x)). Therefore prp (1F ) ⊂ Im φp , proving that φp is

S-ARITHMETICITY OF SOME DISCRETE SUBGROUPS

215

an isomorphism over Qp for each p ∈ Sf . Set φ∞ to be the identity map of V∞ . Hence, (V∞ , (φp , p ∈ S)) provides a Q-form on V such that 1F ⊂ V(Q). Using the exponential map, we obtain a Q-form on V such that F ⊂ V (Q). We now show that k1V (ZS ) ⊂ 1F for some nonzero integer k. It is easy to see that 1V (mZ) ⊂ 1V (ZS ) ∩ 1F for some nonzero integer m. Now let B be a basis of 1V (mZ) over Z. To show that 1F contains the ZS -module generated by B, it is enough to show that for any x ∈ B, we have p −n x ∈ 1F for all n ≥ 1 and for all p ∈ Sf , since the Z-span of {p−n | p ∈ Sf , n ≥ 1} is equal to ZS . For p ∈ Sf , since prp (1F ) is dense in Vp , there exists ni ∈ N, going to infinity as i → ∞, such that p −ni x ∈ 1F . Let n ≥ 1, and take any integer i such that ni ≥ n. Since pn = p ni −n p −ni and p−ni x ∈ 1F , we have p−n x ∈ 1F . Therefore 1F contains the ZS -module generated by 1V (mZ) as well as by B. Since V (mZ) has finite index in V (Z), we can find a nonzero integer k such that n k1VQ (Z) ⊂ 1V (mZ) . For any x ∈ 1V (ZS ) , there exists n such that s x ∈ 1V (Z) for n s = p∈Sf p, and hence ks x ∈ 1V (mZ) . Therefore kx ∈ 1F since 1F contains the ZS -module generated by 1V (mZ) . This proves that k1V (ZS ) ⊂ 1F . Since kb log V (ZS ) ⊂ k1V (ZS ) , 1F ⊂ log F , we have kbV (ZS ) ⊂ F . By Lemma 2.5, there exists a nonzero integer j such that V (j ZS ) ⊂ kbV (ZS ). Therefore V (j ZS ) ⊂ F and F is commensurable with V (ZS ). This shows that F is an S-arithmetic subgroup of V . 2.4. By Lemma 2.3 and the remark in Section 2.2 that any S-arithmetic subgroup of V is a uniform lattice in V , we obtain the following two corollaries of Proposition 2.6. Corollary 2.7. Any lattice in V is an S-arithmetic subgroup of V . Corollary 2.8. Let F be a discrete subgroup of V . Then the following are equivalent: (1) F is a lattice in V ; (2) pr ∞ (F ) is Zariski-dense in V∞ and prp (F ) is dense in Vp for each p ∈ Sf ; (3) V /F is compact. Proposition 2.9. Let F be a lattice in V . If F ⊂ V (Q), then F is commensurable with V (ZS ). Proof. By Proposition 2.6, there is a Q-form on V with respect to which F is an S-arithmetic subgroup. Since F ⊂ V (Q), this Q-form must coincide with the original Q-form of V. Therefore F is commensurable with V (ZS ). 3. Discrete subgroups in semisimple groups 3.1. Throughout this section, let S be a finite set of valuations of Q including ∞. For each p ∈ S, let Gp be a connected adjoint semisimple Qp -algebraic group without

216

HEE OH

any Qp -anisotropic factors,Q and let U1p , U2pQbe a pair of opposite horospherical Q subgroups of Gp . Set U1 = p∈S U1p , U2 = p∈S U2p , U1 = p∈S U1p (Qp ), and Q U2 = p∈S U2p (Qp ). For lattices F1 and F2 in U1 and U2 , respectively, we denote by 0F1 ,F2 the subgroup generated by F1 and F2 . Lemma 3.1. (1) The subgroups U1p (Qp ) and U2p (Qp ) generate the subgroup Gp+ (see [2]). (2) Any subgroup of Gp normalized by Gp+ is either trivial or contains Hp+ for some nontrivial normal simple Qp -subgroup Hp of Gp (see [10]). If Gp is Qp -simple, it is well known [10] that any subgroup of Gp normalized by Gp+ is either central (and hence trivial in our case since Gp is adjoint) or contains Gp+ . It is not difficult to see that this implies (2) of the above lemma, since a connected adjoint semisimple Qp -algebraic group is a direct product of adjoint Qp -simple groups. Lemma 3.2. Let F1 and F2 be lattices in U1 and U2 , respectively. Then for each p ∈ Sf , prp (0F1 ,F2 ) is dense in Gp+ . Proof. By Lemma 2.3, the closure of prp (Fi ) contains Uip (Qp ). Therefore the closure of prp (0F1 ,F2 ) contains the subgroup generated by U1p (Qp ) and U2p (Qp ), which is Gp+ by Lemma 3.1. Proposition 3.3. If 0 is a discrete subgroup of G containing F1 and F2 , then the restriction pr∞ |0 of pr∞ is injective. Proof. We show that the subgroup 00 = {γ ∈ 0 | pr∞Q(γ ) = e} is trivial. Without loss of generality, we may assume that 00 ⊂ GSf = p∈Sf Gp . Note that 00 is normalized by prSf (0) as well as by 0. We claim that 00 is normalized by G+ Sf = Q + . For each g ∈ G+ , there is a sequence {g | i = 1, 2, . . . } in pr (0) G i Sf p∈Sf p Sf converging to g as i → ∞, since prSf (0) is dense in G+ Sf by Lemma 3.2. Note that gi xgi−1 ∈ 00 for any x ∈ 00 and any i ≥ 1. But 00 is discrete, and in particular, it is closed. Therefore gxg −1 ∈ 00 , proving that 00 is normalized by G+ Sf . Let p ∈ + Sf . Since prp (00 ) is normalized by Gp and prp (00 ) is countable, it follows from Lemma 3.1 that prp (00 ) is trivial. Therefore 00 is trivial, yielding that pr∞ |0 is injective. Theorem 3.4 (See [1] and [4, Chap. I, Thm. Q 3.2.4]). Let G be a connected semisimple Q-algebraic group, and let G = p∈S G(Qp ). Then the S-arithmetic subgroup G(ZS ) is a lattice in G. 3.2. Let G be a connected P Q-simple algebraic group with Q-rank at least 1, S-rank at least 2 (S-rank of G = p∈S Qp -rank of G), and U1 , U2 a pair of opposite horospherical Q-subgroups of G. It was proved by Raghunathan [9] for Q-rank at least 2 and by Venkataramana [12] for Q-rank 1 that for any ideal A of ZS , the

S-ARITHMETICITY OF SOME DISCRETE SUBGROUPS

217

subgroup generated by U1 (A) and U2 (A) is of finite index in G(A). It is not hard to see that the following theorem is a consequence of the above result. Theorem 3.5. Let F1 and F2 be lattices in U1 and U2 commensurable to U1 (ZS ) and U2 (ZS ), respectively. If the subgroup 0F1 ,F2 is discrete, then it is commensurable with the S-arithmetic subgroup G(ZS ). 4. Main theorem 4.1. As before, let S be a finite set of valuations of Q including ∞, and for each p ∈ S, let Gp be a connected semisimple Qp -algebraic group without any Qp subgroups anisotropic factors and Q let U1p , U2p be Q a pair of opposite Q horospherical Q of Gp . We set G = p∈S Gp , G = p∈S Gp , U1 = p∈S U1p , U2 = p∈S U2p , Q Q U1 = p∈S U1p (Qp ), and U2 = p∈S U2p (Qp ). Theorem 4.1 (See [5] and [6]). Let S = {∞} and let G be an absolutely simple real algebraic group with R-rank at least 2. Denote by Z(Ui ) the center of Ui for each i = 1, 2. Let the pair (G, U1 ) be as follows: (1) for commutative U1 , assume that G 6= E62 ; (2) for Heisenberg U1 , assume that G 6= A22 , Bn2 , Dn2 ; (3) for U1 such that Z(U1 ) is not the root group of a highest real root, assume that G0 6 = E62 , where G0 is the algebraic subgroup generated by Z(U1 ) and Z(U2 ); (4) for U1 such that Z(U1 ) is the root group of a highest real root, assume that [U1 , U1 ] 6 = Z(U1 ) and G00 6= E62 , where G00 is the algebraic subgroup generated by Z(U10 ) and Z(U20 ) and where Ui0 is the centralizer of the subgroup {g ∈ Ui | gug −1 u−1 ∈ Z(Ui ) for all u ∈ Ui } in Ui . For any lattices F1 and F2 in U1 and U2 , respectively, the subgroup 0F1 ,F2 is discrete if and only if there exists a Q-form on G such that 0F1 ,F2 is a subgroup of finite index in G(Z) and hence a nonuniform arithmetic lattice in G = G(R). Remark. As for the hypothesis on the pair (G, U1 ), if G is split over R and G is not locally isomorphic to SL3 (R), then U1 can be any horospherical subgroup. If G is locally isomorphic to SL3 (R) (i.e., is of type A22 ), then the above hypothesis excludes only the case when U1 is Heisenberg. If R-rank (G) ≥ 3, then U1 can be any commutative or Heisenberg horospherical subgroup. 4.2. The following is a special case of Margulis’s superrigidity theorem (see [4, Chap. VIII, Thm. 3.6]). Theorem 4.2. Let G be a connected almost Q-simple algebraic group without any R-anisotropic factors. Assume that R-rank G ≥ 2 and that 0 ⊂ G(Q) is an arithmetic subgroup of G. Let l be any field of char 0, H a connected adjoint semisimple l-group, and j : 0 → H(l) a homomorphism with the image being Zariski-dense in H.

218

HEE OH

Then there exists a rational l-epimorphism φ : G → H such that φ(x) = j (x) for all x ∈ 0. 4.3. We now prove the main theorem of this paper. The notation continues from Section 4.1. Theorem 4.3. Let Gp be a connected semisimple adjoint Qp -algebraic group without any Qp -anisotropic factors for each p ∈ S. Let F1 and F2 be lattices in U1 and U2 , respectively, such that 0F1 ,F2 is discrete. Assume that (G∞ , U1∞ ) satisfies the conditions in Theorem 4.1. Then there exists a Q-form on G (in the sense of Section 1.5) such that 0F1 ,F2 is a subgroup of finite index in the S-arithmetic subgroup G(ZS ). Hence 0F1 ,F2 is a nonuniform S-arithmetic lattice in G. Proof. Since 0F∞1 ,F2 is a discrete subgroup of G∞ (by Proposition 2.1) containing the lattices F1∞ and F2∞ in U1∞ (R) and U2∞ (R), respectively, Theorem 4.1 implies that there exists a Q-form on G∞ such that 0F∞1 ,F2 is a subgroup of finite index in G∞ (Z). By Proposition 3.3, the map pr∞ |0F1 ,F2 is injective. Therefore we can define a map jp : 0F∞1 ,F2 → Gp as follows: For x ∈ 0F∞1 ,F2 , set jp (x) = prp ◦ (pr ∞ )−1 (x). It is clear from the definition of 0F∞1 ,F2 that jp (0F∞1 ,F2 ) ⊂ Gp (Zp ). We claim that jp (0F∞1 ,F2 ) is Zariski-dense in Gp . Since the subgroup generated by U1p and U2p is Zariski-dense in Gp , it suffices to show that the subgroup jp (Fi∞ ) is Zariski-dense in Uip for each i = 1, 2. It is clear for p = ∞ since j∞ (Fi∞ ) = Fi∞ is a lattice in Ui ∞ (R). For p ∈ Sf , note that jp (Fi∞ ) = prp (Fi ) ∩ Uip (Zp ). Since prp (Fi ) is dense in Uip by Lemma 2.3 and since Uip (Zp ) is open in Uip , jp (Fi∞ ) is dense in Uip (Zp ). Therefore the Zariski closure of jp (Fi∞ ) contains Uip (Zp ) and hence Uip , since it is well known that Uip (Zp ) is Zariski-dense in Uip . By Theorem 4.2, for each p ∈ S, there exists a Qp -epimorphism φp : G∞ → Gp such that φp (x) = jp (x) for all x ∈ 0F∞1 ,F2 . Since G∞ is absolutely simple in our case and hence has no nontrivial normal subgroup, φp is in fact an isomorphism. Therefore (G∞ , (φp , p ∈ S)) endows a Q-form on G with respect to which U1 and U2 are defined over Q. Since Fi ⊂ Ui (Q), Fi is commensurable with Ui (ZS ) by Proposition 2.9. Since 0F1 ,F2 is discrete, it follows from Theorem 3.5 that the subgroup 0F1 ,F2 is commensurable with the S-arithmetic subgroup G(ZS ). Since each Gp is adjoint, we the adjoint representation of Gp . Morecan assume that Gp ⊂ SLN by considering Q over we may assume G(Q) ⊂ { p∈S g | g ∈ SLN (Q)} = SLN (Q) by considering the isomorphisms φp . Since 0F1 ,F2 is an S-arithmetic subgroup contained in G(Q), there exists a ZS -module L in QN of rank N that is invariant by 0F1 ,F2 (cf. [7, Prop. 4.2]); hence 0F1 ,F2 ⊂ GL = {g ∈ G(Q) | g(L) ⊂ L}. Now, by applying the automorphism of SLN (C) that changes the standard basis to a basis of L, we may assume G(ZS ) = GL so that 0F1 ,F2 ⊂ G(ZS ). By Theorem 3.4, 0F1 ,F2 is a lattice in G. Since the lattice 0F∞1 ,F2 in G∞ contains a nontrivial unipotent element, 0F∞1 ,F2 is a nonuniform lattice by Godement’s criterion

S-ARITHMETICITY OF SOME DISCRETE SUBGROUPS

219

(cf. [8]). Therefore, by the remark following Proposition 2.1, the lattice 0F1 ,F2 is nonuniform. Proof of Theorem 0.2. The hypothesis on (G∞ , U1∞ ) in Theorem 4.2 is satisfied for the groups considered in Theorem 0.2 by the remark following Theorem 4.1. To go from an adjoint group to its finite covers, we now give a standard argument. For each p ∈ S, there exists a connected semisimple adjoint Qp -group Gp0 and a Qp Q isogeny fp : Gp → Gp0 (cf. [4, Chap. I, Prop. 1.4.11]). Set f = p∈S fp , the direct product of the fp ’s. Set Fi0 = f (Fi ) for each i = 1, 2, and let 0F0 1 ,F2 be the subgroup generated by F10 and F20 . Since the kernel of f is finite, it follows that Fi0 is a lattice in f (Ui ) and 0F0 1 ,F2 is discrete since 0F0 1 ,F2 ⊂ f (0F1 ,F2 ). Hence by Theorem 4.3, there Q exists a Q-form on G0 = p∈S Gp0 such that 0F0 1 ,F2 is a subgroup of finite index in G0 (ZS ). Since f (0F1 ,F2 ) is a discrete subgroup containing the S-arithmetic subgroup 0F0 1 ,F2 , the subgroup f (0F1 ,F2 ) is commensurable with G0 (ZS ). Hence 0F1 ,F2 is an S-arithmetic subgroup of G by the definition in Section 1.6. Hence Theorem 0.2 is proved. References [1] [2] [3]

[4] [5] [6] [7] [8] [9] [10] [11]

[12]

A. Borel, Some finiteness properties of adèle groups over number fields, Inst. Hautes Études Sci. Publ. Math. 16 (1963), 5–30. A. Borel and J. Tits, Homomorphismes “abstraits” de groupes algébriques simples, Ann. of Math. (2) 97 (1973), 499–571. G. A. Margulis, “Non-uniform lattices in semisimple algebraic groups” in Lie Groups and Their Representations (Budapest, 1971), ed. I. M. Gelfand, Wiley, New York, 1975, 371–553. , Discrete Subgroups of Semisimple Lie Groups, Ergeb. Math. Grenzgeb. (3) 17, Springer-Verlag, Berlin, 1991. H. Oh, Discrete subgroups generated by lattices in opposite horospherical subgroups, J. Algebra 203 (1998), 621–676. , On discrete subgroups containing a lattice in a horospherical subgroup, Israel J. Math. 110 (1999), 333–340. V. Platonov and A. Rapinchuk, Algebraic Groups and Number Theory, Pure Appl. Math. 139, Academic Press, Boston, 1994. M. S. Raghunathan, Discrete Subgroups of Lie Groups, Ergeb. Math. Grenzgeb. (3) 68, Springer-Verlag, New York, 1972. , A note on generators for arithmetic subgroups of algebraic groups, Pacific J. Math. 152 (1992), 365–373. J. Tits, Algebraic and abstract simple groups, Ann. of Math. (2) 80 (1964), 313–329. , “Classification of algebraic semisimple groups” in Algebraic Groups and Discontinuous Subgroups (Boulder, Colo., 1965), Proc. Sympos. Pure Math. 9, Amer. Math. Soc., Providence, 1966, 33–62. T. N. Venkataramana, On systems of generators of arithmetic subgroups of higher rank groups, Pacific J. Math. 166 (1994), 193–212.

Department of Mathematics, Princeton University, Princeton, New Jersey 08544, USA; [email protected]

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

A MODULAR INVARIANCE ON THE THETA FUNCTIONS DEFINED ON VERTEX OPERATOR ALGEBRAS MASAHIKO MIYAMOTO

To Professor Toshiro Tsuzuku on his seventieth birthday 1. Introduction. Throughout this paper, V denotes a vertex algebra, or Poperator −n−1 denotes V , Y, 1, ω) with central charge c and Y (v, z) = v(n)z VOA, (⊕∞ n=0 n a vertex operator of v. (Abusing the notation, we also use it for vertex operators of v for V -modules.) o(v) denotes the grade-keeping operator of v, which is given by v(m − 1) for v ∈ Vm and defined by extending it for all elements of V linearly. In particular, o(ω) equals L(0) = ω(1) for the Virasoro element ω of V and o(v) = v(0) for v ∈ V1 . In order to simplify the situation, we assume that dim V0 = 1 so that there is a constant hv, ui ∈ C such that v1 u = −hv, ui1 for v, u ∈ V1 . We call V a rational vertex operator algebra in the case when each V -module is a direct sum of simple modules. Define C2 (V ) to be the subspace of V spanned by elements u(−2)v for u, v ∈ V . We say that V satisfies condition C2 if C2 (V ) has finite codimension in V . For a V -module M with grading M = ⊕Mm , we define the formal character as X (1) dim Mm q m = tr M q −c/24+L(0) . chq M = q −c/24 In this paper, we consider these functions less formally by taking q to be the usual local parameter q = qτ = e2π ιτ at infinity in the upper half-plane  ª H = τ ∈ C | Iτ > 0 . Although it is often said that a VOA is a conformal field theory with mathematically rigorous axioms, the axioms of VOA do not assume the modular invariance. However, Zhu [Z] showed the modular (SL2 (Z)) invariance of the space E D |a | q1 1 · · · qn|an | tr W Y (a1 , q1 ) · · · Y (an , qn )q L(0)−c/24 : W irreducible V -modules (2) for a rational VOA V with central charge c and ai ∈ V|ai | under condition C2 , which are satisfied by many known examples, where qj = qzj = e2πιzj and |ai | denotes the Received 13 October 1998. 1991 Mathematics Subject Classification. Primary 17B69; Secondary 11F11. Author’s work supported by the Grants-in-Aid for Scientific Research, number 09440004 and number 10974001, the Ministry of Education, Science and Culture, Japan. 221

222

MASAHIKO MIYAMOTO

weight of aj . For example, the space  ­ chq W : W irreducible V -modules is SL2 (Z)-invariant. Recently, Dong, Li, and Mason [DLiM] extended Zhu’s idea and proved a modular invariance of the space  ­ (3) tr U φ i q L(0)−c/24 : i ∈ Z, U φ-twisted modules by introducing the concept of φ-twisted modules for a finite automorphism φ. An easy example of automorphism of a VOA is given by a vector v ∈ V1 as φ = e2πιv(0) . Especially if the eigenvalues of o(v) (= v(0)) on modules are in (1/n)Z, then the order of e2π ιo(v) is finite. So for a V -module W and u, v ∈ V1 , we define ZW (v; u; τ ) = tr W e2πι(o(v)−(hv,ui/2)) q L(0)+o(u)−(c+12hu,ui)/24

(4)

and we call Z )c a theta function of W , where u1 u = −hu, ui1 and QW∞(v; 0; τ )η(τ 1/24 n η(τ ) = q n=1 (1 − q ) is the Dedekind eta function. It is worth noting that c + 12hu, ui is the central charge of conformal element ω − L(−1)u and o(ω − L(−1)u) is equal to L(0) + o(u); see [DLnM]. For example, let V be a lattice VOA V2Zx constructed from a 1-dimensional lattice L = 2Zx with hx, xi = 1. It has exactly four irreducible modules (see [D]): W0 = V2Zx ,

W1 = V(2Z+(1/2))x , W2 = V(2Z+1)x , W3 = V(2Z−(1/2))x . P Let θh,k (z, τ ) (:= n∈Z exp(π ι(n + h)2 τ + 2π(n + h)(z + k))) be theta functions for h, k = 0, 1/2. By the construction of a lattice VOA (see [FLMe]), it is easy to check ¡ ¡
¡

θh,k (z, τ ) = η(τ ) (ι)4hk ZW2h zx(−1)1; 0; τ + (−1)k (ι)4hk ZW2+2h zx(−1)1; 0; τ (5) for h, k = 0, 1/2; their modular transformations     πιz2 z −1 , = (i)4hk (−ιτ )1/2 exp θk,h (z, τ ) θh,k τ τ τ are well known (see [Mu]). In particular, there are constants Ahk ∈ C such that  X    −1 1 2 h = Ak ZWk 0; zx(−1) + z , τ . ZWh zx(−1)1; 0; τ 2

(6)

(7)

Namely, the modular transformations of ZWh (zx(−1)1; 0 : τ ) are expressed by linear combinations of ZWk (u; v; τ ) of (ordinary) modules Wk , but not twisted modules. By this result, for an automorphism φ = ev(0) , we can expect to obtain a modular transformation by using only the ordinary modules, which offers some information about twisted modules. This is the motivation of this paper, and we actually show that the above result is generally true, that is, we prove the following modular transformation by using Zhu’s result (2).

223

THETA FUNCTIONS OF VERTEX OPERATOR ALGEBRAS

Main Theorem. Let V be a rational vertex operator algebra with the irreducible modules {Wi : i = 1, . . . , m} and u, v ∈ V1 . Assume v(0)v = v(0)u = u(0)v = u(0)u = 0 and v(1)v, v(1)u, u(1)u ∈ C1. If V satisfies Zhu’s finite condition C2 , then ª  (8) ZWh (v; u; τ ) : h = 1, . . . , m ¡
satisfies a modular invariance. That is, for α = fa db ∈ SL2 (Z), there are constants Ahα,k (see Theorem 4.1) such that  ZWh

aτ + b v; u; f τ +d

 =

m X k=1

Ahα,k ZWk (av + bu; f v + du; τ ).

(9)

2. Vertex operator algebras Definition. A vertex operator algebra is a Z-graded vector space V = ⊕n∈Z Vn

(10)

satisfying dim Vn < ∞ for all n and Vn = 0 for n  0, equipped with a linear map   V −→ (End V ) z, z−1 , X v(n)z−n−1 v −→ Y (v, z) = n∈Z

and with two distinguished vectors, vacuum element 1 ∈ V0 and conformal vector ω ∈ V2 , satisfying the following conditions for u, v ∈ V : u(n)v = 0

for n sufficiently large; Y (1, z) = 1;

Y (v, z)1 ∈ V [[z]]

and

lim Y (v, z)1 = v;

z→0

(z − x)N Y (v, z)Y (u, x) = (z − x)N Y (u, x)Y (v, z)

for N sufficiently large,

where (z1 −z2 )n (n ∈ Z) are to be expanded in nonnegative integral powers of second variable z2 ; 

 1 L(m), L(n) = (m − n)L(m + n) + (m3 − m)δm+n,0 c 12

for m, n ∈ Z, where L(m) = ω(m + 1) and c is called central charge; L(0)v = nv for v ∈ Vn ; ¡
d Y (v, z) = Y L(−1)v, z . dz

(11)

224

MASAHIKO MIYAMOTO

We also have the notion of modules: Let (V , Y, 1, ω) be a VOA. A weak module W of (V , Y, 1, ω) is a C-graded vector space W = ⊕n∈C Wn

(12)

equipped with a linear map  ¡
 V −→ End(W ) z, z−1 , X v W (n)z−n−1 v −→ Y W (v, z) =

¡




vn ∈ End(W )

n∈Z

satisfying the following conditions: For u, v ∈ V , w ∈ M, v W (m)w = 0 for m  0; Y W (1, z) = 1; LW (0)w = nw for w ∈ Wn , LW (0) = ωW (1); ¡
d W Y (v, z) = Y L(−1)v, z ; dz and the following Jacobi identity holds:     z1 − z2 z2 − z1 Y W (u, z1 )Y W (v, z2 ) − z0−1 δ Y W (v, z2 )Y W (u, z1 ) z0−1 δ z0 −z0   ¡
z1 − z0 −1 = z2 δ Y W Y (u, z0 )v, z2 . z2 A weak module W is called a module if every finitely generated weak submodule M = ⊕r∈C Mr of W satisfies (1) dim Mr < ∞, (2) Mr+n = 0 for n ∈ Z sufficiently large, for any r ∈ C. 3. Formal power series. We use the notation q and qz to denote e2πιτ and e2πιz , respectively. In this paper, the formal power series  ∞  X nqz−n q n nqzn 2 + P2 (qz , q) = (2πι) 1 − qn 1 − qn n=1

P ni plays an essential role, where 1/(1 − q n ) is understood as ∞ i=0 q . The limit of P2 (qz , q) (which we still denote as P2 (qz , q)) relates to p(z, τ ) by P2 (qz , q) = p(z, τ ) + G2 (τ ), where G2 (τ ) =

X π2 + 3

X

m∈Z−{0} n∈Z

1 (mτ + n)2

THETA FUNCTIONS OF VERTEX OPERATOR ALGEBRAS

225

is the Eisenstein series and p(z, τ ) is the Weierstrass p-function  X  1 1 1 p(z, τ ) = 2 + − . z (z − mτ − n)2 (mτ + n)2 (m,n)6=(0,0)

It is known that  G2 and

aτ + b f τ +d

 = (f τ + d)2 G2 (τ ) − 2π ιf (f τ + d)



aτ + b z , p f τ +d f τ +d

 = (f τ + d)2 p(z, τ ).

In particular,  P2

aτ + b z , f τ +d f τ +d

 = (f τ + d)2 P2 (z, τ ) − 2π ιf (f τ + d).

(13)

In this paper, we use variables {z1 , . . . , zn } and calculate the products of formal power series P2 (qzi −zj , τ ). In order to simplify notation, we use a transposition (i, j ) of symmetric groups 6n on  = {1, . . . , n}. For {(i11 , i12 ), . . . , (it1 , it2 )} with is1 < is2 and iab 6 = icd for (a, b) 6= (c, d), we view σQ= (i11 , i12 ) · · · (it1 , it2 ) as an involution (element of order 2) of 6n and denote tj =1 P2 (qzj 2 −zj 1 , τ ) by Q 2 i<σ (i) P2 (qzσ (i) −zi , τ ). Let I (n) denote the set of all elements g in 6n with g = 1. For σ ∈ 6n , set  ª m(σ ) = i ∈  | σ (i) 6= i ,  ª f (σ ) = i ∈  | σ (i) = i .

(14) (15)

For σ1 , σ2 ∈ 6n , σ1 , σ2 are called disjoint if m(σ1 ) ∩ m(σ2 ) = ∅. The notation σ1 + · · · + σn = σ expresses that {σ1 , . . . , σn } are mutually disjoint and the product σ1 · · · σn is equal to σ . For v ∈ V1 and u ∈ V , v(0) acts on the finite-dimensional homogeneous subspaces Vm and satisfies [v(0), u(m)] = (v(0)u)(m) and ∞ X ¡
  (−1)i ω(i)v (−1 − ι)1 v(0)ω = v(0)ω(−1)1 = − ω(−1), v(0) 1 = −

¡




¡




i=0

¡
= − ω(0)v (−1)1 + ω(1)v (−2)1 = −v(−2)1 + ω(1)v (−2)1 = 0. P n v(0) ω = ω and Therefore, ev(0) = ∞ n=0 (v(0) /n!) is well defined and satisfies that e v(0) v(0) v(0) v(0) is an automorphism of V . e (um w) = (e u)m (e w). In particular, e

226

MASAHIKO MIYAMOTO

Definition. For a V -module W and u, v ∈ V1 , define ZW (v; u; τ ) = tr W e2πι(v(0)−hv,ui/2) q (u(0)−hu,ui/2)+L(0)−c/24 .

(16)

τ ) = ZW (v; 0; τ )η(τ )c and call it a theta function of W , where We set θW (v,Q ∞ 1/24 n η(τ ) = q n=1 (1 − q ) is the Dedekind eta function. For example, let V be a lattice VOA V2Zx associated with a 1-dimensional lattice L = 2Zx with hx, xi = 1. Then W0 = V2Zx ,

W1 = V(2Z+(1/2))x ,

W2 = V(2Z+1)x ,

W3 = V(Z−(1/2))x

are the irreducible V2Zx -modules by [D] and hx(−1)1, x(−1)1i = −1. We then have X ¡
exp πιn2 τ + 2π ιnz θ0,0 (z, τ ) = n∈Z

¡
¡
= θW0 zx(−1)1, τ + θW2 zx(−1)1, τ ,    X 1 exp πιn2 τ + 2πιn z + θ0,1/2 (z, τ ) = 2 ¡
¡
= θW0 zx(−1)1, τ − θW2 zx(−1)1, τ ,       X 1 2 1 z exp πι n + τ + 2πι n + θ1/2,0 (z, τ ) = 2 2 ¡
¡
= θW1 zx(−1)1, τ + θW3 zx(−1)1, τ ,       X 1 1 2 1 z+ exp πι n + τ + 2πι n + θ1/2,1/2 (z, τ ) = 2 2 2 ¡
¡
= ιθW1 zx(−1)1, τ − ιθW3 zx(−1)1, τ ; and their modular transformations     z −1 πιz2 , = (−ιτ )1/2 exp θ0,0 (z, τ ), θ0,0 τ τ τ     πιz2 z −1 1/2 , = (−ιτ ) exp θ1,0 (z, τ ), θ0,1/2 τ τ τ     πιz2 z −1 , = (−ιτ )1/2 exp θ0,1 (z, τ ), θ1/2,0 τ τ τ     πιz2 z −1 1/2 , = −(−ιτ ) exp θ1,1 (z, τ ) θ1/2,1/2 τ τ τ are well known (see [Mu]). Therefore, the modular transformations ZWh (zx(−1)1; 0 : −1/τ ) are expressed by linear combinations of ZWk (u; v; τ ) of (ordinary) modules Wk but not twisted modules.

THETA FUNCTIONS OF VERTEX OPERATOR ALGEBRAS

227

4. Modular invariance. In this section, we prove a modular invariance. Throughout this section, we assume (A1) V = ⊕∞ n=0 Vn is a rational VOA; (A2) {W 1 , . . . , W m } is the set of all irreducible V -modules; (A3) fixed v1 , . . . , vn ∈ V1 satisfies vr (0)vj = 0 and vr (1)vj ∈ C1 for any r, j . By (A3), we have 



X m¡


vr (k)vj (m + n − ι) k ¡
= m vr (1)vj (m + n − 1) = δm,−n mh−vr , vj i,

vr (m), vj (n) =

where vr (1)vj = h−vr , vi i1. For a grade-keeping endomorphism formal power series ψ ∈ End(W )((qz1 , . . . , qzn , qτ )) of W , set
¡
¡ SW (ψ; z1 , . . . , zn , τ ) = tr W ψY v1 , qz1 · · · Y vn , qzn qz1 · · · qzn q L(0)−(c/24) . By the same argument as in the proof of Proposition 4.3.2 of [Z], we have the following. Proposition 4.1. Assume [ψ, vr (n)] = 0 for r = 1, . . . , n. Then we have SW (ψ; z1 , z2 , . . . , zn , τ )  Ã !  ¡ −1
Y X Y ­  P2 zσ (j ) zj , τ o(vr ); τ , (17) − vj , vσ (j ) SW ψ = (2πι)2 σ ∈I (n) j <σ (j )

r∈f (σ )

where I (n) is the set of all elements σ in the symmetric group 6n on n point set  = {1, 2, . . . , n} with σ 2 = 1, and f (σ ) denotes the set of fixed points of σ . Proof. For k ∈ Z, we have
¡ ; z2 , . . . , zn , τ SW ψv1 (k)qz−k 1
¡
¡ · · · Y v qz1 · · · qzn q L(0)−c/24 Y v , q , q = tr W ψv1 (k)qz−k 2 z n z n 2 1  ¡
¡
 = tr W ψ v1 (k), Y v2 , qz2 · · · Y vn , qzn qz−k qz2 · · · qzn q L(0)−c/24 1 ¡
¡
+ tr W ψY v2 , qz2 · · · Y vn , qzn v(k)qz−k qz2 · · · qzn q L(0)−c/24 1 n X  X ¡
¡
k k−ι = qzj tr W ψY v2 , qz2 · · · Y v1 (i)vj , qzj i j =2 i∈N

¡ ¡ qz2 · · · qzn q L(0)−c/24 + tr W ψY v2 , qz2 · · · Y vn , qzn qz−k 1
¡ · · · Y vn , qzn qz−k qz2 · · · qzn q L(0)−c/24 v(k)q k 1

228

MASAHIKO MIYAMOTO

=

n X j =2

¡
¡
kqzk−1 qz−k tr W ψY v2 , qz2 · · · Y h−vj , v1 i1, qzj j 1



¡ ¡ Y v2 , qz2 · · · Y vn , qzn qz2 · · · qzn q L(0)−c/24 + tr W ψv1 (k)qz−k 1
¡ · · · Y vn , qzn qz2 · · · qzn q L(0)−c/24 q k =

n X j =2

¡ ¡

¡
tr W h−vj , v1 ikqzkj −z1 ψY v2 , qz2 · · · Y \ vj , qzj · · · Y vn , qzn qz2

¡
L(0)−c/24 · · · qc + tr W ψv1 (k)qz−k Y v2 , qz2 zj · · · qzn q 1
¡ · · · Y vn , qzn qz2 · · · qzn q L(0)−c/24 q k =

n X j =2

¡ ¡

¡
tr W h−vj , v1 ikqzkj −z1 ψY v2 , qz2 · · · Y \ vj , qzj · · · Y vn , qzn qz2

¡
L(0)−c/24 · · · qc + tr W ψv1 (k)qz−k Y v, qz2 zj · · · qzn q 1
¡ · · · Y v, qzn qz2 · · · qzn q L(0)−c/24 q k =

n X ¡
h−v1 , vj ikqzkj −z1 S ψ; z2 , . . . , zbj , . . . , zn , τ j =2


¡ ; z2 , . . . , zn , τ q k , + S ψv1 (k)qz−k 1 where qc zj means that we take off the term qzj . Hence, for k 6 = 0, we have
¡ ; z2 , . . . , zn , τ SW ψv1 (k)qz−k 1 =

n X j =2

h−vj , v1 i

kqzkj −z1 1 − qk


¡ SW ψ; z2 , . . . , zj −1 , zj +1 , . . . , zn , τ . (18)

Therefore, we have SW (ψ; z1 ,z2 , . . . , zn , τ )
X ¡
¡ SW ψv(k)qz−k ; z2 , . . . , zn , τ = SW ψv1 (0); z2 , . . . , zn , τ + 1 k6=0

¡




= SW ψv1 (0); z2 , . . . , zn , τ +

n XX

h−vj , v1 i

k6=0 j =2

kqzkj −z1 1 − qk

× SW (ψ; z2 , . . . , zj −1 , zj +1 , . . . , zn , τ ) n P2 (qzj −z1 , τ )
X ¡ hvu , v1 i = SW ψv1 (0); z2 , . . . , zn , τ − (2πι)2 j =2

× SW (ψ; z2 , . . . , zj −1 , zj +1 , . . . , zn , τ ).

229

THETA FUNCTIONS OF VERTEX OPERATOR ALGEBRAS

By substituting ψv1 (0) into φ and repeating these steps, we have SW (ψ; z1 , z2 , . . . , zn , τ ) =

X

Y

à ­

− vi , vσ (i)

¡
!  P2 qzσ (i) −zi , τ (2πι)2

σ ∈I (n) i<σ (i)

à SW ψ

Y

! vi (0); τ .

(19)

i∈f (σ )

The main result we quote from [Z] is the following. Theorem 4.1 (Zhu). ¡
Let (V , Y, 1, ω) be a rational VOA satisfying (A1) and (A2). Then for any α = fa db ∈ SL2 (Z), we have  SWh

zn aτ + b z1 ,..., , 1; f τ +d f τ +d f τ +d



n

= (f τ + d)

m X j =1

Ahα,k SWk (1; z1 , . . . , zn , τ ), (20)

where the Ahα,k are constants depending only on α, h, k. To simplify notation, Sk (· · · ), α(v), and d(α) denote SWk (· · · ), (f τ + d)o(v), and f τ + d, respectively. Set  

2 ¡ 1 2¡ d(α) P2 qzr −zj , τ , D(r, j ) = h−vr , vj i 2πι     ¡
¡
2 1 f d(α) , E(r, j ) = h−vr , vj i d(α) P2 (qzr −zj , τ ) − 2πι Y Y ¡ ¡

D σ (j ), j , and Eσ = E σ (j ), j . Dσ = j <σ (j )

j <σ (j )

Lemma 4.1. If |m(σ )| = 2p, then X (−1)t Eσt · · · Eσ2 Eσ1 = (−1)p Eσ .

(21)

σ1 +···+σt =σ

Proof. We first note that Eσt · · · Eσ2 Eσ1 = Eσ . Therefore, we have to count the number of Eσ in the left side. We prove it by induction on p. If p = 1, then it is trivial. For σ = ¡(r1
, r2 ) · · · (r2p−1 , r2p ), the number of σ1 with σ1 + · · · + σt = σ and |m(σ1 )| = 2r is pr . Therefore, by induction, we have X (−1)t Eσt · · · Eσ2 Eσ1 σ1 +···+σt =σ

=− For α =

¡a b
f d

p   X p j =1

j


¡ (−1)p−j Eσ = − − (−1)p Eσ = (−1)p Eσ .

∈ SL2 (Z), we have the following.

(22)

230

MASAHIKO MIYAMOTO

Lemma 4.2. We have ! Ã n Y aτ + b o(vr ); Sh f τ +d r=1 =

m X k=1

Ahα,k

X

Y

σ ∈I (n) j <σ (j )

! Ã ¡ Ã
! Y f d(α) α(vr ); τ . (23) h−vj , vσ (j ) iSk 2πι r∈f (σ )

Proof. By Theorem 4.1 and Proposition 4.1, we have à   Y  X 1 2 h−vσ (r) , vr i 2πι σ ∈I (n) r<σ (r) !   à Y  ¡
aτ + b aτ + b Sh × P2 q(zσ (r) −zr )/(f τ +d) , o vj ; f τ +d f τ +d j ∈f (σ )   zn aτ + b z1 ,..., , = Sh 1; f τ +d f τ +d f τ +d m ¡
n X h Aα,k Sk (1; z1 , . . . , zn , τ ) = d(α) k=1

m X
n X Ahα,k = d(α)

¡

k=1

σ ∈I (n)

Ã

Y ­

− vr , vσ (r)

r<σ (r)

¡

×P2 qzσ (r) −zr , τ

 1 2 2πι ! Ã








Sk

Y

! ¡
o vj ; τ .

j ∈f (σ )

Hence, we have ! Ã n Y aτ + b o(vr ); Sh f τ +d r=1 Ã m Y  X X ¡
2 h Aα,k = h−vr , vσ (r) i d(α) k=1

σ ∈I (n)

r<σ (r)

!  ! Ã Y
¡
1 2 ¡ × P2 qzσ (r) −zr , τ Sk α vj ; τ 2π ι j ∈f (σ ) Ã    Y  X ¡
2 1 2 h−vr , vσ (r) i d(α) − 2πι 16 =σ ∈I (n) r<σ (r) !   ! Ã Y
¡
aτ + b ¡ ¡
1 ×P2 qzσ (r) −zr , τ − o vj ; f d(α) Sh 2πι f τ +d j ∈f (σ ) 

231

THETA FUNCTIONS OF VERTEX OPERATOR ALGEBRAS

=

m X k=1

Ahα,k

X Ã

16 =σ ∈I (n)

=

m X k=1

−

Ahα,k

¡
D σ (r), r Sk !

¡

E σ (r), r

r<σ (r)

Ã

Ã

Eσ Sh

16 =σ ∈I (n)




Y

Dσ Sk

σ ∈I (n)

X

Y

Y

X

!

r<σ (r)

σ ∈I (n)

X

−

Ã

à Sh

Ã

Y

¡
α vj ; τ

!

j ∈f (σ )

Y

¡
aτ + b o vj ; f τ +d j ∈f (σ ) !

!

¡
α vj ; τ

j ∈f (σ )

! aτ + b o(vr ); . f τ +d r∈f (σ ) Y


¡Q Substituting the above equation into Sh r∈f (σ ) o(vr ) in the last term by replacing {1, . . . , n} into f (σ ), we have ! Ã n Y aτ + b o(vr ); Sh f τ +d r=1 Ã ! m Y ¡
X X h Aα,k Dσ Sk α vj ; τ = k=1

−

X

Eσ

X

m X k=1

16 =σ ∈I (n)

+

j ∈f (σ )

σ ∈I (n)

Ahα,k

Dσ 0 Sk

σ 0 ∈I (n),m(σ 0 )∩m(σ )=∅

X

Eσ

16 =σ ∈I (n)

Ã

X

Ã

Eσ 0 Sh

16 =σ 0 ∈I (n),m(σ 0 )∩m(σ )=∅

Y r∈f (σ 0 +σ )

! α(vr ); τ

! aτ + b o(vj ); . f τ +d j ∈f (σ )∩f (σ 0 ) Y

Repeating the above steps and by the equations  X  ¡
aτ + b = Ahα,k Sk o(v); τ , Sh o(v); f τ +d  X  aτ + b = Ahα,k Sk (1; τ ) Sh 1; f τ +d in [Z], we have ! Ã n Y aτ + b o(vr ); Sh f τ +d r=1   m X Y X = Ahα,k Sk  Dσ α(vj ); τ  − k=1

σ ∈I (n)

j ∈f (σ )

X 16=σ ∈I (n)

à Eσ Sh

Y

aτ + b o(vr ); f τ +d r∈f (σ )

!

232 =

MASAHIKO MIYAMOTO

m X k=1

( Ahα,k

X

(−1)t−1

σ1 ,...,σt ∈I (n):|f (σ2 +···+σt )|≤n−2

Ã

× Eσt · · · Eσ2 Dσ1 Sk (if n is odd) −

Y r∈∩tj =1 f (σj )

n X

!

α(vr ); τ  ¡ ¡

(−1)t Eσt · · · Eσ2 Eσ1 Sk α vj ; τ

X

j =1 σ1 ,...,σt ∈I (n):f (σ1 +···+σt )={j }

X

(if n is even) −

)

(−1)t Eσt · · · Eσ2 Eσ1 Sk (1; τ ) .

σ1 ,...,σt ∈I (n):f (σ1 +···+σt )=∅

By Lemma 4.1 this is equal to m X Ahα,k

(

k=1

X

Ã

X

(−1)|σ1 |/2 Eσ1 Dσ2 Sk

σ ∈I (n) σ1 ,σ2 ∈I (n):σ1 +σ2 =σ, |f (σ1 )|≥2 n X X

(if n is odd) −

X

=

m X k=1

α(vr ); τ

(n−1)/2

(−1)



¡ ¡

Eσ Sk α vj ; τ )

n/2

(−1)

Eσ Sk (1; τ )

σ ∈I (n):f (σ )=∅

!) Ã Y  1 ¡ ¡ Y 
p
­ α(vr ); τ . f d(α) − vj , vσ (j ) Sk 2πι σ ∈I (n) j <σ (j ) r∈f (σ )

( Ahα,k

!

r∈f (σ )

j =1 σ ∈I (n):f (σ )={j }

(if n is even) −

Y

X

Theorem A. For α =

¡a b
f d

∈ SL2 (Z), we have

 X  m
¡ aτ + b Ahα,k Sk e(h−v,vi/2)(1/2πι)f (d(α))+(d(α))o(v) , τ . = Sh eo(v) ; f τ +d

(24)

k=1

In particular,  X  m aτ + b = Ahα,k ZWk (dv; f v; τ ). ZWh v; 0; f τ +d

(25)

k=1

Proof. Let us calculate the coefficient of (h−u, vi(1/2π ι)f (d(α)))p (α(v))r in (23) by setting¡ n = 2p
+ r and v1 = · ·p· = vn = v. The number of involutions σ with × ((2p + r)!/p!2 ). Therefore, we have |f (σ )| = r is 2p+r r

THETA FUNCTIONS OF VERTEX OPERATOR ALGEBRAS



Sh eo(v) ;

233



aτ + b f τ +d Ã∞ n X 1 Y

! ! Ã n ∞ X Y aτ + b aτ + b 1 Sh = o(v); o(v); = Sh n! i=1 f τ +d n! f τ +d n=0 i=1 n=0 ! Ã ¡
|m(σ )|/2 m ¡ X X 1 X
|f (σ )|  f d(α) Ahα,k Sk α(v) ;τ h−v, vi = n! 2πι =

r,p∈N m X k=1

k=1

Ahα,k

X r,p∈N

σ ∈I (n)

  r + 2p (2p)! 1 (r + 2p)! r p!2p

¡
!|m(σ )|/2  ¡
|f (σ )|  f d(α) × h−v, vi Sk α(v) ;τ 2π ι ¡
!p  Ã m X X ¡¡
r
f d(α) r + p h−v, vi 1 Ahα,k Sk α(v) ; τ = (r + p)! p 2 2πι k=1 r,p∈N ¡
!n−r  Ã n m ∞ X X X ¡¡
r
h−v, vi f d(α) 1 n h Aα,k Sk α(v) ; τ = n! p 2 2πι k=1 n=0 r=0 Ã ¡
  n ! m ∞ X X h−v, vi f d(α) 1 h Aα,k Sk + α(v) ; τ = n! 2 2πι Ã

r=1

=

m X r=1

since

n=0


Ahα,k Sk e(h−v,vi/2)(f (d(α))/2πι)+α(v) ; τ , ¡

  r + 2p (2p)! (r + 2p)! (2p)! 1 1 = (r + 2p)! r p!2p (r + 2p)! k!(2p)! p!2p   r +p 1 1 = . (r + p)! r 2p

Let us show the final version of our result. ¡
Main Theorem. For α = fa db ∈ SL2 (Z), we have  X  m aτ + b = ZWh v; u; Ahα,k ZWk (dv + bu; f v + au; τ ), f τ +d k=1

where Ahα,k are the coefficients in the equations given by Zhu [Z]. Proof. Fix s, t ∈ N so that n = s + t. Assume that v1 = v2 = · · · = vs (say, = v),

us+1 = · · · = vs+t (say, = u).

(26)

(27)

234

MASAHIKO MIYAMOTO

For p, q, r ∈ Z, set k = s − 2p − r ≥ 0, h = t − 2q − r ≥ 0. In order to simplify notation, for u, v ∈ V1 and σ ∈ I (n), we use the following notation: ¡
f d(α) h−v, ui, α(−v, u) = ¡ 2πι
α(v) = d(α) o(v), m11σ = |{r ∈  | r < σ (r) ≤ s}|, m12σ = |{r ∈  | r ≤ s < σ (r)}|, m22σ = |{r ∈  | s < r < σ (i)}|, f 1σ = |{r ∈  | r = σ (r) ≤ s}|, f 2σ = |{r ∈  | s < i = σ (r)}|. By Lemma 4.2, we have  X  m s t aτ + b Ahα,g = Sh o(v) o(u) ; f τ +d g=1

X

α(−v, v)m11σ α(−v, u)m12σ

σ ∈I (s+t)

  × α(−u, u)m22σ Sg α(v)f 1σ α(u)f 2σ ; τ .

Hence,  X  m X (2p + r + k)!(2q + r + h)! s t aτ + b = Ahα,g Sh o(v) o(u) ; f τ +d k!h!r!p!q!2p+q r,p,q g=1
¡ × α(−v, v)p α(−v, u)r α(−u, u)q Sg α(v)k α(u)h ; τ . Expanding the exponential, we have
¡ Sh e2π ι(o(v)+h−v,ui/2)+2π ιτ (o(u)+h−u,ui/2) ; τ   ∞ X h−v, ui h−u, ui n L(0)−c/24 (2π ι)n q = tr Wh o(v) + τ o(u) + +τ n! 2 2 n=0     X (2π ι)α+β+γ +δ ¡
β h−v, ui γ h−u, ui δ L(0)−c/24 o(v)α τ o(u) τ q = tr Wh . α!β!γ !δ! 2 2 α,β,γ ,δ

Hence, for α = −1/τ , we have   2π i(o(v)+h−v,ui/2+τ (o(u)+h−u,ui/2)) −1 ; Sh e τ β     X (2π ι)α+β+γ +δ −1 h−v, ui γ Sh o(v)α o(u) = α!β!γ !δ! τ 2 α,β,γ ,δ      h−u, ui δ −1 δ −1 × ; 2 τ τ

THETA FUNCTIONS OF VERTEX OPERATOR ALGEBRAS

=

=

X (2π ι)α+β+γ +δ  h−u, ui δ  −1 δ α!β!γ !δ! 2 τ γ ,δ,α,β       h−v, ui γ −1 β α β −1 × Sh o(v) o(u) ; 2 τ τ         δ δ X (2π ι)α+β+γ +δ h−u, ui −1 h−v, ui γ −1 β γ ,δ,α,β m X

α!β!γ !δ!

τ

2

2

τ

X

α!β! h!k!r!p!q!2p+q g=1 p,q,r,h,k:2p+r+k=α, 2q+r+h=β       
k ¡
h h−v, vi p h−v, ui r h−u, ui q ¡ τ τ × Sg τ τ o(v) τ o(u) ; τ 4π ι 4πι 4πι      (p+k+r+q+h+γ +δ) X −1 h−u, ui δ h−v, ui γ (2π ι) ×

=

235

Ahα,g

γ !δ!h!k!r!p!q!

τ

2

2

γ ,δ,p,q,r,h,k      m X
r 1 h−u, ui q ¡
k ¡
h h−v, vi p ¡ × Ahα,g Sg τ − h−v, ui τ o(v) − o(u) ; τ 2 τ 2 g=1 m X ¡
Ahα,g Sg e2π ι((−1/τ )h−u,ui/2+h−v,ui/2+τ h−v,vi/2−h−v,ui+(1/τ )h−u,ui/2+τ o(v)−o(u)) ; τ = g=1 m X
¡ Ahα,g Sg e2π ι(−h−v,ui/2+τ h−v,vi/2+τ o(v)−o(u)) ; τ = g=1 m X ¡
Ahα,g Sg e2π ι(o(−u)+hu,vi/2)+2πιτ (o(v)+h−v,vi/2) ; τ . = g=1

Namely,  ZWh

−1 v; u; τ

 =

X

Ahα,k ZWk (−u; v; τ ).

On the other hand, since Wh is irreducible, ZWh (v; u; τ + 1) = tr W e2π i(o(v)−hu,vi/2+L(0)+o(u)−c/24−hu,ui/2 q L(0)+o(u)−(c+12hu,ui)/24 = tr W e2π i(o(v+u)−hu,v+ui/2+L(0)−c/24 q L(0)+o(u)−(c+12hu,ui)/24 = λh ZWh (v + u; u; τ ) for some λh ∈ C. Therefore, for α =

¡a b
f d

, we have

(28)

236

MASAHIKO MIYAMOTO

 ZWh

aτ + b v; u; f τ +d



m X

=

k=1

Ahα,k ZWk (dv + bu; f v + au; τ ).

(29)

Example. Let V be a lattice VOA V2Zx with hx, xi = 1 and W a module VZx . For z ∈ C, set zx = zx(−1)1 ∈ (V2Zx )1 . Then it is easy to check by the definition of lattice VOAs that tro(1)e2π izx(0) q L(0)−1/24 |M = For α =

¡ 0 −1
1 0

1 θ(τ, z). η(τ )

,

  X −1 1 2 θ ,z = λM tro(1)q L(0)+zx(0)−(1/24)+(z /2) |M , η(−1/τ ) τ M

where M runs over the following modules: {VL , Vx+L , V(1/2)x+L , V−(1/2)x+L }. Taking τ = i and several z, we have the known formula:   −1 1 1 2 θ ,z = θ(τ, τ z)eπiz τ . η(−1/τ ) τ η(τ ) References [B] [CN] [D] [DLiM] [DLnM]

[FHL] [FLMe] [Mu] [Z]

R. E. Borcherds, Vertex algebras, Kac-Moody algebras, and the Monster, Proc. Nat. Acad. Sci. U.S.A. 83 (1986), 3068–3071. J. H. Conway and S. P. Norton, Monstrous moonshine, Bull. London Math. Soc. 11 (1979), 308–339. C. Dong, Vertex algebras associated with even lattices, J. Algebra 161 (1993), 245–265. C. Dong, H. Li, and G. Mason, Modular-invariance of trace functions in orbifold theory, preprint. C. Dong, Z. Lin, and G. Mason, “On vertex operator algebras as sl2 -modules” in Groups, Difference Sets, and the Monster (Columbus, Ohio, 1993), Ohio State Univ. Math. Res. Inst. Publ. 4, de Gruyter, Berlin, 1996, 349–362. I. B. Frenkel, Y.-Z. Huang, and J. Lepowsky, On axiomatic approaches to vertex operator algebras and modules, Mem. Amer. Math. Soc. 104 (1993), no. 494. I. B. Frenkel, J. Lepowsky, and A. Meurman, Vertex Operator Algebras and the Monster, Pure Appl. Math. 134, Academic Press, Boston, 1988. D. Mumford, Tata Lectures on Theta, I, Progr. Math. 28, Birkhäuser, Boston, 1983. Y. Zhu, Modular invariance of characters of vertex operator algebras, J. Amer. Math. Soc. 9 (1996), 237–302.

Institute of Mathematics, University of Tsukuba, Tsukuba 305, Japan; miyamoto@math. tsukuba.ac.jp

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES IN TORIC VARIETIES ILIA ZHARKOV

1. Introduction. Strominger, Yau, and Zaslow [SYZ] conjectured that any CalabiYau manifold X having a mirror partner X∨ should admit a special Lagrangian fibration π : X → B. (A mathematical account of their construction can be found in [M].) If so, the mirror manifold X ∨ is obtained by finding some suitable compactification of the moduli space of flat U (1)-bundles along the nonsingular fibers, which restricts the fibers to be tori. More precisely, if B0 ⊆ B is the largest set such that π0 = π |π −1 (B0 ) is smooth, then X∨ should be a compactification of the dual fibration R 1 π0∗ (R/Z) → B0 . The conjecture is trivial in the elliptic curve case. On a K3-surface, the hyperKähler structure translates the theory of special Lagrangian T 2 -fibrations to the standard theory of elliptic fibrations in another complex structure. However, little progress has been made thus far in higher dimensions. Gross and Wilson [GrW] have worked out some aspects of the conjecture for the Voisin-Borcea 3-folds of the form (K3×T 2 )/Z2 . Also Bryant [Br] found special Lagrangian tori inside some hypersurfaces in CPn for n = 2, 3, and 4 as components of the set of real points, though it is unclear if there is a fibration. But the general question of finding special Lagrangian fibrations on Calabi-Yau manifolds still remains open. We examine the case of regular anticanonical hypersurfaces in smooth toric varieties. The main result of this paper is that such a hypersurface in a neighborhood of the large complex structure admits a torus fibration over a sphere. Unfortunately, we were unable to control the fibers to be special Lagrangians. However we argue that on some open patches the fibers do possess some calibration property. The application of the moment map for constructing torus fibrations was observed by Gross and Wilson [GrW] and also by Morrison and collaborators using Batyrev’s construction (see [B]) for mirror symmetry. Batyrev showed that toric varieties X1 with ample anticanonical bundles are given by reflexive polyhedra. Such a polyhedron 1 contains a unique integral interior point P {0}. A Calabi-Yau hypersurface Y ⊂ X1 is defined by an equation in the form ω∈1(Z) aω x ω = 0, where ω runs over the integral points in 1. The image of Y under the moment map µ : X1 → 1 has the shape of an amoeba (cf. [GKZ, Chap. 6]), a blob with holes around some lattice points ω in 1(Z). The sizes of the holes are determined by the corresponding coefficients aω . If Y is near the large complex structure, then a{0} is large and µ(Y ) has exactly Received 25 June 1998. 1991 Mathematics Subject Classification. Primary 14J32, 14M25; Secondary 83E30. 237

238

ILIA ZHARKOV

one interior hole corresponding to {0}; that is, µ(Y ) is homeomorphic to S N−1 × I . The idea is to choose the right trivialization of this product, so that for general s ∈ S N −1 ' ∂1 the preimage of the interval µ−1 ({s} × I ) is an (N − 1)-dimensional torus T N −1 . Singular fibers come from the intervals Is := {s}×I , which intersect the (N − 2)-dimensional skeleton of 1. For example, consider Y , a K3-surface given by a quartic in CP 3 = X1 , where 1 is the integral 3-simplex in Z3 with the vertices (−1, −1, −1), (−1, −1, 3), (−1, 3, −1), (3, −1, −1). There are exactly four points in µ(Y ) on each of the six one-dimensional edges of 1, corresponding to four points of the intersection of Y with the projective line determined by this edge. Altogether they give twenty-four singular fibers. Instead of pursuing this idea, we modify the moment map and deform the original hypersurface. An explicit parameterization of the fibers allows us to analyze the action of the monodromy for one-parameter families of hypersurfaces and to construct a dual fibration. We speculate that this dual fibration represents the mirror Calabi-Yau. Let us demonstrate most of our essential ideas with a simple example. Consider a family of elliptic curves Et in CP2 given by the equations txyz + x 3 + y 3 + z3 = 0, where t plays the role of a parameter. As t → ∞, the curve Et degenerates to three lines with normal crossings. The main idea is roughly to consider the asymptotic behavior of Et up to the next order to keep the curve smooth. There are six regions in Et according to its image under the moment map (see Figure 1). In each of them, there y3 Uy

Uyz

z3

Uz

Uxy xyz Uzx

Ux x3

Figure 1. The image of the elliptic curve txyz + x 3 + y 3 + z3 = 0 (shaded area) in CP2 under the moment map

239

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

are different terms in addition to txyz that are dominant. For instance, in Uz the elliptic curve Et for large t is approximated by txyz + z3 = 0, in Uzx by txyz + x 3 + z3 = 0, and so on. It is easy to introduce a coordinate on a curve (which is still smooth in the corresponding region) defined by the abbreviated equation. In Uz either x/z or y/z is a coordinate, in Uzx we can use x/z or z/x, and so on in the other regions. The circle fibration is provided by fixing an absolute value of the coordinate. The set of all possible absolute values in all six regions clearly forms a circle, which is the base of the fibration. The partition of unity technique of gluing these six pieces into one curve, which approximates the original elliptic curve, constitutes Section 3. To compute the monodromy as arg(t) 7→ arg(t)+2πi, we need to understand how the identification of the fibers changes in the overlaps. Nothing happens in Uzx , Uxy , or Uyz , as the parameterization of fibers changes by reversing an orientation of the circles, which does not depend on t at all. Monodromy is nontrivial only in Uz , Uy , or Ux . For example, consider Uz , where we can use two coordinates y/z or x/z, which are related by the equation t (x/z)(y/z) = −1. As arg(t) 7→ arg(t) + 2πi, the circles parameterized by arg(x/z) and by arg(y/z) are twisted by 2πi with respect to each other. Combining all together we get a triple Dehn twist. The monodromy calculations are performed more carefully in Section 4. Uy Vzx Uyz

Vz

Vx

Uxy

Vy Uz

Uzx

Ux

Vyz

Vxy

Figure 2. The mirror pair of elliptic curves

Section 5 is devoted to dual fibrations and mirror symmetry. Leung and Vafa [LV] describe the idea of the mirror construction as follows. T -duality interchanges small circle fibers in the corner regions of one family with large circle fibers in the facet regions of the mirror family. More precisely, we consider the polytope 1∨ dual to the above family of elliptic curves. In this case, it is just the polar polytope 1D , and hence again it is reflexive. The associated toric variety X1D is singular, but the elliptic curve in X1D is smooth because it misses all singular points (the vertices of the triangle). 1D is combinatorially dual to 1, and the dual elliptic curve also breaks into six regions according to its image under the moment map (see Figure 2). We speculate

240

ILIA ZHARKOV

that this dual curve also admits similar fibration by circles and that the fibers in the corresponding regions Uα and Vα are naturally dual circles. This example captures all the features except for the fact that in higher dimensions there are always singular fibers. Unfortunately, in our construction the discriminant locus has codimension 1 in ∂1 for N ≥ 4. However, it retracts to an (N −3)-dimensional simplicial complex inside the (N − 2)-skeleton of ∂1, which is of codimension 2. Acknowledgments. I am grateful to Ron Donagi, my thesis advisor, for suggesting this problem to me and constantly supervising my work on it, and to Tony Pantev for illuminating discussions. 2. Hypersurfaces in toric varieties. In this section we review some basic constructions in the theory of toric varieties, and we set up the notation. For more details, see [C] and references therein. Let M ' RN be an N -dimensional real affine space and ZN an integral lattice in M. We choose an integral point {0} in ZN to be the origin. This endows M with a vector space structure, and ZN becomes a free abelian group. An N -dimensional convex integral polyhedron 1 in M is called reflexive if it contains the origin {0} as an interior point and if its polar polyhedron  ª 1D = u ∈ M ∗ : hu, m − {0}i ≥ −1 for all m ∈ 1 ⊂ M ∗ is also integral. We denote by 1(Z) the lattice points in the polyhedron 1 and by ∂1 its boundary. It follows that each (N −1)-dimensional face 6 (which we call a facet) of 1 is defined by an equation hu6 , mi = −1 for some u6 ∈ M ∗ . This easily implies that {0} is the unique integer interior point. For an arbitrary face 2 ⊆ ∂1, we denote by 2Z the affine sublattice of ZN generated by the integer points of 2. Also we assume that 1 is nonsingular. This means that every vertex of 1 is Nvalent; that is, exactly N edges e1 , . . . , eN emanate from it, and the integer points of these edges (ei )Z , i = 1, . . . , N , generate the lattice ZN . Because 1 is reflexive (i.e., the integral distance from the origin {0} to any facet 6 ⊂ 1 is 1), the lattice Zn is generated by 6Z together with {0}. Let us denote by X1 the projective toric variety corresponding to 1. The normal projective embedding is given, for example, by the closure of the image of the map (C∗ )N ,→ P|1(Z)|−1 , x 7 → {x ω1 : x ω2 : · · · : x ω|1(Z)| }. Because 1 is nonsingular, the toric variety X1 is smooth. We use {x ω } as projective coordinates on X1 . One may want to restrict the set of monomials {x ω } to a subsystem of the anticanonical linear system. In this case we allow ω to vary among A ⊂ 1(Z), which is a subset of integer points in 1, such that {0} ∈ A and 1 is the convex hull of A. The moment map X1 → 1 is given by P ω ω∈A |x | · ω . µ(x) = P ω ω∈A |x | It is a well-defined function on X1 , because both the top and bottom are polynomials

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

241

of the same homogeneous degree. A triangulation of a convex polyhedron is a decomposition of it into a finite number of simplices such that the intersection of any two of these simplices is a common face of both of them (which may be empty). By a triangulation T of (1, A), we simply mean a triangulation of 1 with vertices in A. Note that we do not require every element of A to appear as a vertex of a simplex. A continuous function ψ : 1 → R is called T – piecewise linear if it is affine-linear on every simplex of T . Such a function ψ is convex if for any x, y ∈ 1, we have ψ(tx + (1 − t)y) ≥ tψ(x) + (1 − t)ψ(y), 0 ≤ t ≤ 1. We call it strictly convex if the (maximal-dimensional) simplices of T are the maximal domains of the linearity of ψ. A triangulation T of (1, A) is called coherent (some mathematicians call it regular or projective) if there exists a strictly convex T –piecewise linear function. We call a coherent triangulation T central if {0} is a vertex in every (maximal-dimensional) simplex of T . Let ∂T denote the collection of simplices of T lying in ∂1. We use σ to denote maximal-dimensional simplices in ∂T and τ for arbitrary simplices of ∂T . Denote by Cτ ⊂ 1 the corresponding simplex in T of one dimension higher with base τ and vertex {0}. We denote by τZ the sublattice of ZN generated by the integer points in τ . 3τ denotes the sublattice in τZ of index (dim τ )!·vol(τ ) generated by the vertices of τ . Given a triangulation T of (1, A), every function λR : A → R defines a characteristic function ψλR : 1 → R, a T –piecewise linear function, by its values on the vertices of T . Denote by C(T ) ⊂ R|A| a subset of such functions λR , whose corresponding characteristic functions ψλR are convex and ψλR (ω) ≥ λR (ω) for any ω ∈ A. Anyone familiar with toric varieties immediately recognizes a secondary cone, which is the normal cone to the secondary polyhedron at the vertex corresponding to the triangulation T . In particular C(T ) has a nonempty interior if T is a coherent triangulation. The last piece of data we need is an integral vector λ in the interior of CT ⊂ R|A| . This means that the characteristic function ψλ : 1 → Z is strictly convex with respect to the triangulation T and ψλ (ω) ≥ λ(ω), with equality holding exactly for the vertices of T . From now on, we fix a nonsingular reflexive integral polyhedron 1, a subset of its integral points A, a central coherent triangulation T , and an integral vector λ ∈ C(T ). Given such data,we can define the one-parameter family of Calabi-Yau hypersurfaces Ft by the equations X t λ(ω) x ω = 0. t λ(0) x {0} − ω∈A∩∂1

P To any hypersurface given by an equation in the form ω∈A aω x ω = 0, we can associate the vector λa := {log |aω1 |, . . . , log |aω|A| |} ∈ R|A| . If this vector λa is sufficiently far from the walls of the secondary fan, then the corresponding hypersurface in X1 is nonsingular. Note that for |t|  0, log |t λ(ω) | = λ · log |t| lies deeply inside CT , so that the hypersurface Ft is smooth (cf. [GKZ, Chap. 10]). The main object

242

ILIA ZHARKOV

of this study is the behavior of this family as |t| → ∞. In the limit we approach the large complex structure limit point. It may be convenient to have local coordinates on affine subsets of X1 . We define τ -associated coordinates for any k-dimensional simplex τ ∈ ∂T in the triangulation. Choosing coordinates on the open orbit (C∗ )N ⊂ X1 is equivalent to choosing an affine basis {, ω1 , . . . , ωN } for the lattice ZN together with a marked reference point . Consider the minimal face 2τ ⊂ 1 containing τ . Let l be its dimension, k ≤ l ≤ N − 1. Using the fact that 1 is reflexive and nonsingular, we choose {, ω1 , . . . , ωN } such that the reference point  is a vertex of τ , the first k + 1 vertices {, ω1 , . . . , ωk } generate the lattice τZ , and the first l +1 vertices {, ω1 , . . . , ωl } generate the lattice (2τ )Z . Moreover, because of the unit integral distance from the facets of 1 intersecting at 2τ to the origin {0}, we can have the rest of the basis satisfy {0} −  = (ωl+1 − ) + · · · + (ωN − ). Notice that the last N − l points are uniquely determined modulo (2τ )Z . Then {y1 , . . . , yN } := {x ω1 − , . . . , x ωN − } give local coordinates on (C∗ )N . In fact, they extend to coordinates on the affine subset of X1 obtained by removing the divisors that correspond to the facets of 1 not containing . It may also be useful to identify 1 with the closure of the positive real part (X1 )≥0 of X1 . The homeomorphism (X1 )≥0 ' 1 is provided by the restriction of the moment map. In particular, {|y1 |, . . . , |yN |} provides coordinates in 1 with facets not containing  as excluded above. Inspired by the proof of Viro’s theorem (see [GKZ, Chap. 11]), we use the weighted moment map µt : X1 → 1 defined as λ(ω) ω P · |x | · ω ω∈A t . µt (x) = P λ(ω) · |x ω | ω∈A t The right way to think about this weighted moment map is the following. Add one extra dimension to M ∼ = RN and extend the lattice to ZN+1 ⊂ RN+1 . Let P be the convex hull of {(ω, λ(ω))}ω∈A in RN+1 (see Figure 3). Then we can think of the whole family {Ft } as a hypersurface in XP (t is considered a coordinate). The vertical projection p : P → 1 splits the boundary of P into two pieces, ∂P = ∂+ P ∪ ∂− P . In fact ∂+ P is exactly the graph of the characteristic function ψλ , and the projection p : ∂+ P → 1 identifies the faces of ∂+ P with the simplices in the triangulation T . (P ) The weighted moment map µt0 is just the composition of the restriction µt0 := (P ) (P ) to the hypersurface µ |t=t0 : XP |t=t0 → P of the ordinary moment map µ Ht0 := {t = t0 } in XP with the vertical projection p : P → 1. The image of µ(P ) (Ht0 ) ⊂ P is the graph of a function 9t0 : 1 → R with the following crucial property (see [GKZ, Chap. 11]). Proposition 2.1. As |t0 | → ∞ the function 9t0 : 1 → R is continuous and smooth outside ∂1, converging uniformly to the characteristic function ψλ , whose graph is ∂+ P .

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

243

t 2x3

txyz x3

y3

z3

Figure 3. The extended polyhedron P for the family txyz − t 2 x 3 − y 3 − z3 = 0 in CP2 and the image of the weighted moment map as t → ∞

The image of the hypersurface {Ft } ⊂ XP under the moment map µ(P ) misses all the vertices of P (cf. [GKZ, Chap. 6]), and, in particular, (because ψλ is convex) it misses some neighborhood of ({0}, λ(0)) ∈ P . Applying the above proposition, we see that for sufficiently large |t| the image of the weighted moment map µt (Ft ) ⊂ 1 misses some ball B around {0} ∈ 1. Denote by 10 := 1\B our polytope with that ball removed. Clearly 10 is homeomorphic to S N−1 × I , and the next step is to find a good trivialization of this product. We conclude this section with a list of notation used throughout the rest of the paper. M = ZN ⊗ R ' RN the real affine space ZN an N-dimensional lattice in M 1 ⊂ M a convex nonsingular integral reflexive polyhedron, ∂1 its boundary 1(Z) the set of integral points in 1 {0} ∈ 1(Z) the unique integral interior point A a subset of 1(Z) containing {0} and all vertices of 1 2 a face of 1 6 a maximal-dimensional face of 1 T a central coherent triangulation of (1, A) λ ∈ C(T ) an integral vector in the interior of the secondary cone at T ∂T the induced triangulation of ∂1 τ a k-dimensional simplex in ∂T σ a maximal-dimensional simplex in ∂T 2τ the minimal face of 1 containing τ

244

ILIA ZHARKOV

Cτ the (k + 1)-dimensional simplex in T over τ with the vertex {0} O(τ ) the center of a simplex τ 2Z or τZ the affine sublattices of ZN generated by the integral points in 2 or τ 3τ the sublattice of τZ generated by the vertices of τ X1 the toric variety associated to 1 Ft the family of the Calabi-Yau hypersurfaces in X1 µt the weighted moment map 0 := µ−1 (10 ) 10 the polyhedron 1 with a small ball around {0} removed, X1 t 3. Torus fibrations. We must now leave the realm of beautiful algebraic geometry and employ some analysis techniques such as partitions of unity and transversality theory. Let Bar(∂T ) be the first barycentric subdivision of the triangulation ∂T . The vertices in this subdivision are the centers O(τ ) of the simplices τ in ∂T . Consider the subdivision {Vτ0 }τ ⊂∂T dual to Bar(∂T ). Namely, take the second barycentric subdivision Bar (2) (∂T ) of ∂T and define Vτ0 to be the union of all simplices in Bar (2) (∂T ) having O(τ ) as a vertex. Every Vτ0 contains the point O(τ ) for a unique τ and is labeled correspondingly (see Figure 4). For each Vτ0 we take its small open neighborhood Vτ to get an open cover of ∂1. By construction, Vτ and Vτ 0 intersect if and only if either τ ⊂ τ 0 or τ 0 ⊂ τ . So every point in ∂1 lies in at most N different Vτi ’s for {τi } forming a nested sequence.

Figure 4. Vτ0 subdivision of ∂1

Define the subsets of ∂1 [ V τ0, Uτ := Vτ − τ 0 6 =τ

Wτ := V τ −

[

Vτ 0

(set-theoretic difference),

τ 0 ⊃τ

where V τ ⊂ ∂1 denotes the closure of Vτ . The collection of cells {Wτ }τ ∈∂T provides a CW-decomposition of ∂1 homeomorphic to that given by {Vτ0 } (see Figure 5). Uτ are the “pure” open subsets in Wτ ⊆ V τ .

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

245

Figure 5. Wτ -cell decomposition of ∂1 0 := µ−1 (10 ) Now we construct a trivialization of 10 ' ∂1 × I . This provides X1 t with the structure of a fibration via the composition of the maps µt

pr 1

0 −−→ 10 ' ∂1 × I −−→ ∂1. X1

Inside the small central ball B 0 , choose a concentric minicopy 10 of 1 with the induced triangulation ∂T 0 of ∂10 . To construct a trivialization, we need to connect the two boundaries ∂1 and ∂10 by nonintersecting intervals. To do this, we need a rule describing how to choose which pairs of points s ∈ ∂1 and s 0 ∈ ∂10 should be connected and how to choose the interval connecting them. Then we must make sure that these intervals do provide a trivialization of 10 . We denote by 20 ⊂ ∂10 and τ 0 ∈ ∂T 0 the minicopies of 2 and τ , correspondingly. Let s 0 be an interior point of an l-dimensional face 20 ⊂ ∂10 . We choose τ associated coordinates {y1 , . . . , yN } for some τ , such that 2τ is the minimal face containing τ . Then {|y1 |, . . . , |yN |} provides coordinates in the open subset of 1 corresponding to τ (see the previous section) by means of the weighted moment map. Let mi = |yi |(m) be the coordinates of a point m ∈ 1. Define the curved normal cone to 20 at the point s 0 ∈ 20 by  ª n(s 0 ) = m ∈ 1 : mi = si0 , i = 1, . . . , l, and mi ≤ si0 , i = l + 1, . . . , N . Note that the definition does not depend on the choice of τ -coordinates. Combining the curved normal cones forms a fat curved normal fan to the polyhedron 10 . Namely, to every k-dimensional face 20 ⊂ ∂10 we can associate an (N − l) × l-dimensional fat curved normal cone N (20 ) ' n(20 ) × 20 , where n(20 ) is the curved normal cone to 20 at any point s 0 ∈ 20 . This fat fan provides a fat cone decomposition of 1 with 10 deleted. The trivialization of 10 ' ∂1×I is achieved by connecting any point s 0 ∈ ∂10 with all the points s ∈ ∂1 lying in the curved normal cone n(s 0 ). The connecting intervals are given by the unique line (given by affine linear equations in the remaining (N −l)

246

ILIA ZHARKOV

Figure 6. The fat cone decomposition and Is -fibration of 10 for X1 = CP1 × CP1

coordinates) in n(s 0 ) passing through both points s and s 0 (see Figure 6). Because every point s ∈ ∂1 belongs to a unique interval, we denote this interval by Is . Notice that the interval Is does not depend on the choice of τ -coordinates, for a change of coordinates does not affect the affineness of the defining equations in the last (N − l) coordinates. But Is does depend on the value of t through the dependence of the weighted moment map on t, and this t-dependence is crucial in the next proposition. For any k-dimensional simplex τ ∈ ∂T , we refer to s ∈ ∂1 as a τ -point if in any τ -associated coordinate system the first k coordinates {|y1 |, . . . , |yk |} are constant along the interval Is . In other words, Is lies in a fat cone N(20 ) for some 20 ⊃ 20τ . Again, this does not depend on the choice of τ -associated coordinates. We say that a collection {Is }s∈∂1 is W -supported if for any τ ∈ ∂T all the points in the cell Wτ are τ -points. Proposition 3.1. Given a cell decomposition {Wτ }, there exists a positive real number R, such that for |t| ≥ R the collection {Is }s∈∂1 providing a trivialization of 10 ' S N −1 × I is W -supported. Proof. We claim that in the limit t → ∞ the intervals Is provide a bijective correspondence between points in τ and points in τ 0 for all τ ∈ ∂T . In particular, this means that all interior points in any simplex τ eventually become τ -points. Assuming this claim, the proposition follows immediately from the next observation. Wτ is compact and contained in the interior of the union of the facets of the polytope 1 containing the face 2τ , and every point in this interior becomes a τ -point for large |t|. To show the claim, we consider the extended polytope P . Keeping the first k τ -associated coordinates fixed is equivalent to fixing (k + 1) projective coordinates {|t λ(ω) x ω |}ω∈τ . The restriction of these equations to p−1 (Cτ ) ⊂ ∂+ P , where p : ∂+ P → 1 is the vertical projection, defines a ray Rs from the origin ({0}, λ(0)) to

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

247

a point in p −1 (τ ). Thus we see that every point in the interior of p−1 (τ 0 ) connects to a unique point in the interior of p −1 (τ ). But because the function 9t0 : 1 → R converges uniformly to the characteristic function ψλ as |t| → ∞, Is is given in the limit by the projection of Rs to 1. 0 over a smooth sphere (as opposed to just Remark. If one wants a fibration of X1 topological), then some care should be taken to smooth out Is near the boundaries of the fat cone decomposition of 10 . From now on, we assume that ∂1 ' S N−1 is endowed with a smooth structure compatible with the fibration.

For any subset S ⊂ ∂1, we denote by IS ' S × I the union of Is for s ∈ S, and we 0 f f let e S := µ−1 t (IS ) ⊂ X1 . In particular, we use Uτ and Wτ . 0 Next we choose a partition of unity {ρτ } subordinate to the cover Vτ , and we define the cutoff P functions ρω : ∂1 → [0, 1] for ω ∈ ∂T , a vertex of the triangulation, by ρω := τ 3ω ρτ0 . In particular, it is clear that ( 0, unless s ∈ Vτ for some τ containing ω, ρω (s) = 1, if s ∈ Vτ only for those τ that contain ω. To make the notation uniform, let ρ{0} ≡ 1 and ρω ≡ 0 for the ω ∈ A that are not vertices in the triangulation. Extend ρ’s to the entire 10 by setting ρω (Is ) := ρω (s) for the entire interval Is . By abuse of notation, we continue to write ρω for the pullback 0 := µ−1 (10 ) via the moment map. of the cutoff functions to X1 t Now we are in a position to define the auxiliary object Ht , a real (nonanalytic) codimension-2 submanifold in X1 , which we still call a hypersurface. It is defined by the equation X ρω t λ(ω) x ω = 0. t λ(0) x {0} − ω∈A∩∂1

fτ defines an open set (in complex topology) of an algebraic The restriction of Ht to U subvariety in X1 given by the equation X t λ(ω) x ω = 0. t λ(0) x {0} − ω∈τ

The cutoff functions ρ are designed to connect these pieces. We have shown that µt (Ft ) misses a ball around {0} ∈ 1. A similar argument applies to show that µt (Ht ) also lies in 10 . Namely, every point of Ht can be thought of as a point in an algebraic hypersurface in X1 defined by the same equation as Ht but with constant ρ’s. Every such algebraic hypersurface clearly misses a ball around {0} ∈ 1. Using the compactness of Ht , we get the claim. 0 → ∂1, The structure of the rest of this section is as follows. The fibration X1 when restricted to the auxiliary hypersurface, induces a torus fibration ht : Ht → ∂1. Then we show that the auxiliary hypersurface is, in fact, diffeomorphic to the original

248

ILIA ZHARKOV

one. Moreover, this diffeomorphism extends to a diffeomorphism between the entire families {Ft } and {Ht }. This provides a torus fibration of the original Calabi-Yau hypersurface. The first step is to show that the collection {Is }s∈∂1 defines a torus fibration ht : Ht → ∂1. Of course, some of the fibers are degenerate, and we try to describe them as explicitly as possible. Let τ be a k-dimensional simplex in ∂T , let 2τ ⊂ 1 be the minimal face containing τ , and let l := dim 2τ . Let {y1 , . . . , yl } be the first l of τ -associated coordinates. A τ -point s defines an l-dimensional torus T l ⊂ (C∗ )l by setting |yi | = si . Let P (y1 , . . . , yk ) = 0 be the equation X ρω t λ(ω) x ω = 0, ω∈τ

written in the local coordinates. Denote by Dτ,s ⊂ T l the zero locus of the polynomial P (y1 , . . . , yk ). To get an idea of what Dτ,s looks like, we consider a k-dimensional 0 torus T k ⊂ (C∗ )k = Spec[z1±1 , . . . , zk±1 ], defined by fixing |z|. Denote by Dτ,|z| the P k 0 k intersection of T with a plane ρ0 + i=1 ρi zi = 0. For generic |z| and ρ, Dτ,|z| is 0 can be a single (real) point. either empty or a (k −1)-torus. In exceptional cases, Dτ,|z| 0 can be described as follows. The substitution The relation between Dτ,s and Dτ,|z| zi = t λ(ωi )−λ() x ωi − , where {, ω1 , . . . , ωk } are the vertices of the simplex τ , defines a covering map πτ : T k → T k . The degree of πτ is equal to the index of the sublattice 3τ inside the lattice τZ , which is given by k! · vol(τ ). Then Dτ,s ⊂ T l is a pullback 0 ⊂ T k under the composition of maps of Dτ,|z| pr

πτ

T l −−→ T k −−→ T k , where pr : {y1 , . . . , yl } → {y1 , . . . , yk } is a projection onto the first k coordinates. Proposition 3.2. Let s ∈ Wτ as above be a point in the interior of an L-dimensional face of 1, L ≥ l. Then the fiber Ts := µ−1 t (Is ) itself has a structure of a fibration ps : Ts → T l with a generic fiber T N−1−l and fibers T L−l over the discriminant locus Dτ,s . Thus, Ts is homeomorphic to

¡¡ T L−l × T l × T N −1−L / ∼ , where (d, t1 ) ∼ (d, t2 ), if d ∈ Dτ,s . In particular, if the point s is in the interior of an (N − 1)-dimensional face, then Ts is a smooth (N − 1)-torus. Proof. We choose τ -associated coordinates {y1 , . . . , yN }. In particular, they profτ . The equation of the auxiliary hypersurface restricted to Ts vide coordinates in W becomes yl+1 yl+2 · · · yN = Pt (y1 , . . . , yk ), where Pt is a polynomial (all ρ are constants on Ts ). Because s is a τ -point, |yi | are fixed and nonzero for i = 1, . . . , l. This gives a projection pr s : Ts → T l . A fiber of

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

249

this projection is determined by fixing a point Y := {yi }, i = 1, . . . , l, on the base. After that, we are left with the equation yl+1 · · · yN = Pt (Y ) = const. At this point we must remember that Is is given by a line in chosen coordinates, and hence it intersects the hyperbola |yl+1 | · · · |yN | = |Pt (Y )| in exactly one point in 1. Thus it determines the remaining |yi | uniquely. We see that a generic fiber of the projection prs : Ts → T l is an (N − l − 1)-torus. The dimension drops to (L − l), if Pt (Y ) = 0, that is, exactly if the point Y is in the discriminant locus Dτ,s . We want to say some words about the discriminant locus D(Ht ) of the fibration ht : Ht → ∂1. The above proposition says that D(Ht ) consists of all points in ∂∂1, the (N − 2)-dimensional skeleton of ∂1, which are in the image of the moment map µt . Thus D(Ht ) is homeomorphic to ∂∂1, with some neighborhoods of its (N−3) , which is the (N − 3)vertices removed, which has a homotopy type of Sk T skeleton of the subdivision dual to the triangulation of ∂∂1 ⊂ ∂1. Moreover, with an appropriate choice of the V -subdivision (Wτ should have small volume for all τ with 1 ≤ dim τ ≤ N − 2), the discriminant locus D(Ht ) lies in an arbitrarily small (N−3) . We refer to this limit as the right W -decomposition limit. neighborhood of Sk T To deform the auxiliary hypersurface Ht and to use transversality theory, we must make sure that it is smooth. Lemma 3.3. For a generic choice of {ρ}, the hypersurface Ht is smooth. fτ := µ−1 Proof. It is enough to show that Ht is smooth in W t (IWτ ) for every fτ . Let Gt be the defining τ ⊂ ∂T . We use τ -associated coordinates {y1 , . . . , yN } in W fτ (see the previous proposition): equation of the auxiliary hypersurface in W Gt := yl+1 yl+2 · · · yN − b ρ (|y|) −

k X

bi ρi (|y|)y αi = 0,

i=1

where t-dependence is encoded into bi ’s. Denote by R the family of {ρω } constructed from the family of partitions of unity ρ f ρ subordinate to {Vτ }. Consider the map Gt : W τ → C. The statement that Gt = 0 is ρ smooth in this language translates as Gt is transversally regular to 0 ∈ C. We want to ρ show that there are enough functions in R that a generic Gt is transversally regular to 0 ∈ C. According to restricted transversality theory, it is enough to show that the fτ × R → C is transversally regular to 0 ∈ C (cf., e.g., ft : W map of the entire family G ft −1 (0), we need to show that the tangent space [DNF]). For any point x˜ = (x, ρ) ∈ G ft to the slice in a small neighborhood at x˜ maps onto C. Consider the restriction of G ft |ρ=const becomes algebraic, and it is a of x, ˜ given by ρ = const. The function G straightforward calculation to show that the tangent space to that slice is transversal to 0 ∈ C. Just notice that all yi , i = 1, . . . , k, are nonzero, and at least one of the ρi is nonzero, too. Hence a generic choice of ρ provides a smooth preimage of 0 ∈ C. This completes the proof.

250

ILIA ZHARKOV

0 := The next step is to find a small deformation of Ht and a diffeomorphism of X1 −1 0 µt (1 ) inside X1 , which transforms the deformed equation for Ht into the equation of a genuine hypersurface F(0·t) for some real number 0. For this we need a technical

lemma. Lemma 3.4. There exists a function χ (s, γ , ω) : ∂1×R≥0 ×A → R, smooth with respect to (s, γ ) and affine linear with respect to ω, which satisfies • χ(s, 0, ω) ≡ 0 and χ (s, γ , {0}) ≡ γ · λ(0), • as γ → ∞, the function eγ λ(ω)−χ (s,γ ,ω) converges (uniformly) to ρω (s) for every ω ∈ A. Proof. First, for every τ ⊂ ∂T in the triangulation, we choose an affine linear function χτ : A → R with the following property: χτ (ω) ≥ λ(ω) with equality holding exactly for ω ∈ τ ∪ {0}. Note that for σ , an (N − 1)-dimensional simplex, χσ is uniquely determined by ψλ |Cσ , and the inequality condition is satisfied because ψλ is a strictly convex function. By the same reasoning, we can satisfy the inequality for the simplices of smaller dimension. For every (N − 1)-dimensional simplex σ , we define the function à ! X 0 −γ ·χτ ρτ e χσ (s, γ , ω) := − log τ ∈∂T

first for ω ∈ σ ∪ {0} and then extend it by linearity to all ω ∈ A. The function χ (s, γ , ω) is constructed by gluing the functions χσ (s, γ , ω) together in the following way. The collection of maximal-dimensional simplices σ ∈ ∂T provides a triangulation of ∂1. We take small open neighborhoods of each σ to get an open cover {e σ } of ∂1, and we choose a partition of unity {ασ } subordinateSto it. We require the open enlargements of the σ ’s to be small enough, so that e σ ⊂ τ ⊂σ Wτ . ≡ 0 unless τ ⊂ σ . Now we can define the desired function In particular, ρτ0 |e σ X χ (s, γ , ω) := ασ (s)χσ (s, γ , ω), σ ∈∂T

which is smooth with respect to (s, γ ) and affine-linear with respect to ω by construction. It is also clear that à ! X ρτ0 = 0 χσ (s, 0, ω) = − log τ ∈∂T

and

¡




χσ s, γ , {0} = − log

Ã

X

τ ∈∂T

! ρτ0 e−γ ·λ(0) = γ · λ(0)

for all σ . Hence χ(s, 0, ω) ≡ 0 and χ (s, γ , {0}) ≡ γ · λ(0).

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

251

The last thing to check is the behavior of χ (s, γ , ω) as γ → ∞. Fix a point s ∈ Wτ0 , partition function ασ (s) vanishes unless τ0 ⊂ σ ; and first consider vertices ω ∈ τ0 . TheP hence ω ∈ σ and χ (s, γ , ω) = − log( τ ∈∂T ρτ0 e−γ ·χτ (ω) ). Thus, lim e−χ (s,γ ,ω) eγ ·λ(ω) = lim

γ →∞

X

γ →∞

τ ∈∂T

ρτ0 eγ ·(λ(ω)−χτ (ω)) =

X τ ∈ω

ρτ0 = ρω ,

because λ(ω) − χτ (ω) ≤ 0 with equality holding exactly for ω ∈ τ ∪ {0}. If ω ∈ / τ0 , then to show that limγ →∞ e−χ (s,γ ,ω) eγ ·λ(ω) = ρω (s) = 0 we look at the asymptotics of the affine functions χσ (s, γ , ω) as γ → ∞. Let Q be the collection of simplices τ with the minimal value of χτ (ω) among those with nonzero ρτ0 (s). Note that if ρτ0 (s) 6 = 0, then τ ⊆ τ0 , and hence χτ (ω) > λ(ω) as ω ∈ / τ . So we see that   X ρτ0  + γ · χτ (ω). χσ (s, γ , ω) ∼ − log  τ ∈Q

Combining these together for all σ ’s, we get   X ρτ0  + γ · χτ (ω), χ (s, γ , ω) ∼ − log  τ ∈Q

and hence e−χ (s,γ ,ω) eγ ·λ(ω) ∼

X τ ∈Q

ρτ0 eγ ·(λ(ω)−χτ (ω)) −→ 0

as γ → ∞. This completes the proof. Remark. In the above proof, we used the crucial fact that the characteristic function ψλ is strictly convex. 0 via As for ρω , we use the same notation for both χ (γ , ω) and its pullback to X1 the moment map. Now we have all the tools to prove the main theorem. Let R be the positive real number as in Proposition 3.1. Denote by HR the one- (real) parameter family of the auxiliary hypersurfaces {Ht , |t| = R}. Let F00 R be the one- (complex) parameter family of the original Calabi-Yau hypersurfaces {Ft , |t| ≥ 00 R}.

Theorem 3.5. There is a positive real number 00 , such that there exists a diffeomorphism between the families {HR } × (00 , +∞) and F00 R that specializes to a diffeomorphism between the hypersurfaces (Ht , 0) and F0t . 0 by the equation Proof. First we define a hypersurface Htε ⊂ X1

t λ(0) x {0} −

X ¡


ρω + εω t λ(ω) x ω = 0,

ω∈A∩∂1

where εω := eγ λ(ω)−χ (γ ,ω) − ρω .

252

ILIA ZHARKOV

According to Lemma 3.4, all εω uniformly vanish as γ → ∞; that is, for γ ≥ γ0 Htε is indeed a small deformation of Ht and, hence, diffeomorphic to it. Using the substitution x 0 ω := x ω e−χ (γ ,ω) , we get eγ λ(0) t λ(0) x 0

{0}

X

−

ω

eγ λ(ω) t λ(ω) x 0 = 0,

ω∈A∩∂1

which is exactly the equation of the hypersurface F(0·t) for 0 = eγ . A priori this 0 → P|1(Z)|−1 . But because χ(ω) is an affine funcsubstitution defines a map X1 tion, the image of this map, in fact, lies in X1 . Hence the above equation defines a hypersurface in X1 . Notice that this construction works for the entire families, because the deformation diffeomorphisms clearly form a trivial system and the substitutions depend only on the absolute values of the parameters of the families. According to [GKZ, Chap. 10], all hypersurfaces that lie inside a translated cone C(T ) + b, where b is some vector in the interior of C(T ), are smooth and hence diffeomorphic to each other. Combining the above theorem with Proposition 3.2, we get the main result of the paper. Corollary 3.6. A Calabi-Yau hypersurface in X1 , which is sufficiently far away from the walls of the secondary fan to 1 and sufficiently close to the large complex structure, admits a fibration over a sphere S N−1 with generic fibers (N − 1)dimensional tori. Remark. It should be possible to remove the smoothness requirement. In this case one has to be more careful with deforming the equation of a nonsmooth hypersurface. For a 1-regular hypersurface in the translated cone C(T ) + b, all singularities come from the singularities of X1 . There is a natural stratification of X1 by µ−1 (2), as 2 runs over open parts of the faces of 1, which induces a stratification of Ft . All diffeomorphisms should then be understood in this stratified sense (see, e.g., [GoM]). The ultimate goal would be, of course, to construct a special Lagrangian fibration. Our construction, unfortunately, leaves this problem open. But there are some features that may be worth mentioning. For instance, our fibration is quite special because it tends to concentrate the singularities of the fibers into a smaller number of fibers with worse degenerations. As an example, let us consider the family of quartic K3-surfaces in CP3 , given by the equations t · x {0} +

X

¡
x  + O t −1 = 0.

 vertex of 1

A generic fibration is expected to have twenty-four degenerate fibers, and each one of them is homeomorphic to the standard I1 -degenerate elliptic curve. In our fibration the terms O(t −1 ) do not matter, and we get just six singular fibers of type I4 .

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

253

There is a local special Lagrangian structure on the algebraic pieces of the auxiliary hypersurface. However, for this we should have defined the cutoff functions ρω slightly differently. (We did not do so in the first placeP because it would have spoiled the uniformity of the definition.) Namely, let ρω := τ ρτ0 , where τ runs over the simplices in ∂T containing ω but of dimension at most (N − 2). This reduces the support of ρω to a neighborhood of the (N −2)-skeleton of ∂1. Then for a maximalfτ would be just x {0} = 0, which dimensional simplex σ ⊂ ∂T , the equation of Ht in U defines an open subset of the toric divisor corresponding to σ . Notice that µt (Ts ) is just one point in 1 (which in this case lies on the boundary ∂1). So, with respect to the standard symplectic form on the toric variety X1 , Ts is clearly Lagrangian. fσ according to the equation Moreover, if we define the top holomorphic form in U (every hypersurface is locally Calabi-Yau), then it restricts to a volume form on each fω for any vertex in the triangulafiber. It is easy to check that the same is true in U tion, where the local equation is y1 · · · yN = const. The fibration is given by fixing |yi |, 1 ≤ i ≤ N , which is clearly Lagrangian. A top holomorphic form can be written as (dy1 /y1 ) ∧ · · · ∧ (dyN −1 /yN−1 ), which restricts to a volume form on Ts . Thus, in the right W -decomposition limit, every Ts becomes Lagrangian with respect to the deformed symplectic structure except for s in the singular locus. Unfortunately we cannot say the same thing about the special Lagrangian property. Although the local holomorphic forms on the auxiliary hypersurface do give the volume forms when restricted to the fibers, it is not at all clear what are their pullbacks to the original hypersurface and how to patch them together in the transition regions. 4. Monodromy. In this section we want to show an application of the constructed fibration to the monodromy calculations. Gross [Gr] has made a conjecture about the monodromy transformation in a family of Calabi-Yau manifolds X → S. Let f

X = Xt −→ B, t ∈ S, be a torus fibration with a section δ0 . Then X] , the complement of the critical locus of f , has a structure of a fiber space of abelian groups with the zero section δ0 . Given another section δ, one obtains a diffeomorphism Tδ : X ] → X] given by x 7 → x + δ(f (x)), which extends to a diffeomorphism of the entire X. Given a degeneration divisor in S passing through the large complex structure point and a loop around this divisor, we can consider the monodromy transformation on cohomology. The conjecture says that this monodromy is induced by Tδ for some section δ. We are going to construct such a section for our family of hypersurfaces Ft . Because Theorem 3.5 establishes the diffeomorphism between the entire families, the monodromy question is identical for the auxiliary family HR . Without loss of generality, we may assume that the base point in the family is given by a hypersurface Ht0 with t0 = R, a real positive number. The monodromy loop is parameterized by t = t0 e2π iγ , 0 ≤ γ ≤ 1. But first we need a zero section. Lemma 4.1. The fibration ht0 : Ht0 → ∂1 has a section δ0 , which misses all singular points of the fibers.

254

ILIA ZHARKOV

Proof. The section is given by the set of all real positive points of Ht0 . Let s ∈ Wτ 0 be a τ -point. We just have to show that the interval Is of positive real points in X1 has a unique solution to the equation X t λ(0) x {0} = ρω t λ(ω) x ω . ω∈τ 0 with 10 , the real However, with the identification of the real positive points of X1 positive points satisfying this equation form a hypersurface separating {0} from those ω for which ρω 6 = 0 (cf. [GKZ, Chap. 11]). In particular, it has a unique point of intersection with the line Is , which, moreover, lies in the interior of 10 . But all the singular points of the fiber Ts are mapped to the boundary of 1. Thus we get the desired section δ0 : ∂1 → Ht0 .

To construct the other section δ, we consider a Delzant-type polytope 1∨γ ∈ M ∗ defined by the inequalities ­  ¡
m, ω − {0} ≥ γ · λ(ω) − λ(0) , where ω runs over the vertices in the triangulation T . This is a convex polytope with a nonempty interior for γ > 0 because of strict convexity of the characteristic function ψλ . By the same reasoning, it is combinatorially dual to (1, T ); namely, to each k-dimensional simplex τ in ∂1, there corresponds an (N − 1 − k)-dimensional face τ ∨ in ∂1∨ λ with the reverse incidence relation. The bijective correspondence between the centers of the dual pairs, τ and τ ∨ , gives rise to a simplicial map νγ : Bar(∂T ) → Bar(∂1∨ γ ) between the first barycentric subdivisions. Considering the W -decompositions, which are dual to the barycentric ones, we get a homeomorphism νγ : ∂1 → ∂1∨ γ satisfying νγ (Wτ ) = Wτ ∨ , where ∨ {Wτ ∨ } provide a CW-decomposition of ∂1γ . Because each Wτ ∨ contains the center of the simplex τ ∨ , there is a map νγ0 : ∂1 → ∂1∨ γ , homotopic to νγ , with the property that νγ0 (Wτ ) ⊂ τ ∨ . 0 . Let us define a We use the same notation for both νγ0 and its pullback to X1 0 → X 0 by diffeomorphism Dγ : X1 1 0

x ω 7−→ x ω e2πihνγ (x), ω−{0}i . This is well defined, as νγ0 (x) is a linear functional with respect to ω. In fact, this diffeomorphism is equivariant with respect to the toric action, that is, µt ◦ Dγ = µt . For convenience we drop the index γ in all notation whenever γ = 1. The desired section δ : ∂1 → Ht0 is given by applying the diffeomorphism D to the zero section. Thus we define δ := D ◦ δ0 . Theorem 4.2. The section δ : ∂1 → Ht0 is well defined and induces the monodromy transformation on Ht0 .

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

255

Proof. The correctness of the definition follows easily from the following observation. For s ∈ Wτ notice that hνγ0 (s), ω − {0}i = γ (λ(ω) − λ(0)) for all ω ∈ τ . This means that for γ = 1, the diffeomorphism D has no effect on any monomial x ω for ω ∈ τ , as it gets multiplied by the factor of e2πi(λ(ω)−λ(0)) . Thus, the equation of Ht0 fτ , in W X λ(0) λ(ω) ρω t0 x ω = 0, t0 x {0} − ω∈τ

is still satisfied for δ = D(δ0 ). Notice that the diffeomorphism D respects the fibration ht0 : Ht0 → ∂1, and hence the action in a fiber Ts is just the translation by δ(s)−δ0 (s). This action is also well defined on singular fibers. Indeed, a singular fiber is itself a fibration ps : Ts → T dim τ , according to Proposition 3.2. The action on Ts translates points along the fibers of ps , which generically are (N −1−dim τ )-dimensional tori. To see that the diffeomorphism D induces the monodromy transformation, we fτ the notice that Dγ provides a diffeomorphism of Ht0 with Ht0 e2π iγ . Indeed, in W defining equation of Ht0 translates into λ(0) {0}

t0

x

−

X ω∈τ

λ(ω) ω 2πiγ (λ(ω)−λ(0))

ρω t0

x e

= 0,

which is exactly the defining equation for Ht0 e2π iγ . As γ runs from 0 to 1, the family of diffeomorphisms Dγ provides the monodromy along the loop t0 e2πiγ . This completes the proof. 5. Dual fibrations and mirror symmetry. This section is rather speculative in character, but it is impossible to overlook a connection of our construction with the mirror symmetry. A triple (1, T , λ) defines a family of complex structures on a Calabi-Yau hypersurface. For simplicity we assume that the subset A ⊂ ZN coincides with the set of vertices of the triangulation T . On the mirror side, we want to get a family of Kähler structures on some other Calabi-Yau hypersurfaces. This family is provided by the monomial-divisor map (see [AGM]). To construct this map, we consider the polytopes 1∨γ defined in the previous section. Let N (1∨γ ) be the normal fan to 1∨γ . N(1∨γ ) is a rational convex polyhedral fan, and the corresponding toric variety X1∨γ is a blow-up of the variety X1D for the polar polytope 1D . The one-dimensional cones in N(1∨γ ) are in one-to-one correspondence with the vertices of the triangulation ∂T . The exceptional divisors are labeled by those that are not the vertices of 1, and N(1∨γ ) is a simplicial cone subdivision of N(1D ) by means of the triangulation T . The vector λ lies in the interior of the Kähler cone of X1∨γ . It defines the Kähler class (in the orbifold sense) by the linear combination of the toric divisors [ω∨ ] corresponding to the facets ω∨ in X1∨γ : X ¡
   γ λ(ω) − λ(0) ω∨ . κγ := −



ω∈∂T

256

ILIA ZHARKOV

We consider the family {X1∨γ }, where γ runs over positive real numbers. The sym→ 1∨γ (cf. [Gu]). Now we choose plectic form κγ defines the moment map µ∨ γ : X1∨ γ a regular anticanonical hypersurface Z D in X1D with a large complex structure (e.g., with large central coefficient). Denote by Z ⊂ X1∨γ its proper transform induced by the blow-up X1∨γ → X1D . Z is a Calabi-Yau hypersurface (cf. [B]) endowed with a Kähler structure by restriction from X1∨γ in the orbifold sense. This is the mirror family. We let γ = 1 for future consideration as the behavior of the family changes by a simple rescaling for other γ . To study the geometry of Z, we again use the moment map µ∨ : X1∨ → 1∨ . At this point, we assume that Z possesses a torus fibration analogous to that of a smooth hypersurface. All singularities of Z are mapped by µ∨ to the (N −2)-skeleton of 1∨ . A generic fiber is thus still a smooth T N−1 . But degenerations in singular tori may give rise to singularities in the total space. First, let us introduce some notation. We use τZ to denote the subgroup of ZN modeled on the affine sublattice τZ . In other words, τR := τZ ⊗R is the k-dimensional vector subspace parallel to τ and passing through {0}. Denote by τZ∗ the quotient of (ZN )∗ dual to τZ ⊂ ZN , and let τR∗ := τZ∗ ⊗ R be the corresponding quotient of M ∗ . Now we consider an explicit parameterization of nonsingular fibers in the original family. Let s ∈ Wτ , for τ ⊂ ∂T , be a point in ∂1. According to Proposition 3.2, a fiber Ts is a fibration itself, ps : Ts → T k , and T k is naturally isomorphic to the torus τR∗ /τZ∗ . Choosing a point in T k determines the phases not only of x ω , ω ∈ τ , but also of x {0} . Considering the fact that τR∨ is defined by the equations hu, ω −{0}i = 0, ω ∈ τ , we conclude that an (N − k − 1)-dimensional fiber T N−k−1 can be identified with τR∨ /τZ∨ . Hence the fiber Ts is isomorphic to the torus τR∗ /τZ∗ ⊕ τR∨ /τZ∨ , though splitting into the direct sum is not natural. The dual fiber Ts ∨ , where s∨ = ν(s) is a point in Wτ ∨ ⊂ ∂1∨ , is isomorphic to τR /τZ ⊕ (τR∨ )∗ /(τZ∨ )∗ . To conclude, we need to take the singular tori into consideration. Remember that the discriminant locus D(Ht ) is homotopy-equivalent (and, in fact, can be made (N−3) arbitrarily close in the appropriate W -decomposition limit) to Sk T , the (N − 3)(N−3) is a simplicial skeleton of the subdivision dual to the triangulation of ∂∂1. Sk T complex consisting of the simplices (O(τi1 ), . . . , O(τik )), with vertices O(τij ), the centers of τij , and τi1 ⊂ · · · ⊂ τik running over all nested chains of simplices in ∂∂1 of positive dimension. On the mirror side, the discriminant locus is again homotopy-equivalent to a sim(N−3) ∨ ) with the vertices O(τ ∨ ) and the simplices labeled by plicial complex (Sk T ∨ the nested chains of τ ’s. However, the simplices τ ∨ ⊂ ∂∂1∨ that have appeared as a result of the blow-up X1∨ → X1D do not contain any points in the image of the moment map µ∨ (Z), and hence they should be excluded from the discriminant locus. They correspond exactly to the simplices τ ∈ ∂T for which the minimal face 2τ is a facet of 1, that is, to those not in ∂∂1. The simplicial map ν : ∂1 → ∂1∨ provides a one-to-one correspondence between

TORUS FIBRATIONS OF CALABI-YAU HYPERSURFACES

257

the points O(τ ) and O(τ ∨ ), and thus it establishes the simplicial isomorphism be(N−3) (N−3) ∨ and (Sk T ) . Thus in the right limit we get the identification tween Sk T between the two discriminant loci. This suggests that we should consider the corresponding singular fibers Ts and Tν(s) to be dual to each other. References [AGM] [B] [Br] [C]

[DNF] [GKZ] [GoM] [Gr] [GrW] [Gu] [LV] [M] [SYZ]

P. Aspinwall, B. Greene, and D. Morrison, The monomial-divisor mirror map, Internat. Math. Res. Notices 1993, 319–337. V. Batyrev, Dual polyhedra and mirror symmetry for Calabi-Yau hypersurfaces in toric varieties, J. Algebraic Geom. 3 (1994), 493–535. R. Bryant, Some examples of special Lagrangian tori, http://xxx.lanl.gov/abs/math. DG/9902076. D. Cox, “Recent developments in toric geometry” in Algebraic Geometry—Santa Cruz 1995, Proc. Sympos. Pure Math. 62, Part 2, Amer. Math. Soc., Providence, 1997, 389–436. B. Dubrovin, S. Novikov, and A. Fomenko, Modern Geometry: Methods and Applications (in Russian), 2d ed., Nauka, Moscow, 1986. I. Gelfand, M. Kapranov, and A. Zelevinsky, Discriminants, Resultants, and Multidimensional Determinants, Math. Theory Appl., Birkhäuser, Boston, 1994. M. Goresky and R. MacPherson, Stratified Morse Theory, Ergeb. Math. Grenzgeb. (3) 14, Springer-Verlag, Berlin, 1988. M. Gross, “Special Lagrangian fibrations, I: Topology” in Integrable Systems and Algebraic Geometry (Kobe/Kyoto, 1997), World Sci., River Edge, N.J., 1998, 156–193. M. Gross and P. M. H. Wilson, Mirror symmetry via 3-tori for a class of Calabi-Yau threefolds, Math. Ann. 309 (1997), 505–531. V. Guillemin, Kaehler structures on toric varieties, J. Differential Geom. 40 (1994), 285– 309. N. Leung and C. Vafa, Branes and toric geometry, Adv. Theor. Math. Phys. 2 (1998), 91–118. D. Morrison, The geometry underlying mirror symmetry, http://xxx.lanl.gov/abs/alggeom/9608006. A. Strominger, S.-T. Yau, and E. Zaslow, Mirror symmetry is T -duality, Nuclear Phys. B 479 (1996), 243–259.

Department of Mathematics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA; [email protected]

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

AN Lp –Lq ESTIMATE FOR RADON TRANSFORMS ASSOCIATED TO POLYNOMIALS JONG-GUK BAK Let S(x, y) be a polynomial of degree n ≥ 2 with real coefficients. Thus (1)

S(x, y) =

n X X

j k

aj k x y =

d=0 j +k=d

d n X X

ad−k, k x d−k y k ,

d=0 k=0

where x, y, and aj k are real numbers. We always assume that a1, n−1 6= 0 or an−1, 1 6= 0. The Radon transform of f associated to the polynomial S(x, y) is defined by Z ∞ ¡
(2) f t + S(x, y), y ψ(t, x, y) dy, Rf (t, x) = −∞

where ψ ∈ Cc∞ (R3 ) is a cutoff function. (For the background information on the well-developed theory of Radon transforms and related oscillatory integral operators, we refer the reader to the papers [P], [PS3], [S], [Se2], and the references contained there.) When S(x, y) is a homogeneous polynomial of degree n, that is, ad−k, k = 0 for d < n in (1), the operator R was studied by Phong and Stein [PS2] as a model for degenerate Radon transforms. They proved among other things that, if a1,n−1 6= 0 and an−1,1 6 = 0, then R is bounded from Lp (R2 ) to Lq (R2 ), when (1/p, 1/q) is in the set τ defined as follows. First let 1 be the closed convex hull (a trapezoid) of the points O = (0, 0), A = (2/(n + 1), 1/(n + 1)), A0 = (n/(n + 1), (n − 1)/(n + 1)), and O 0 = (1, 1) in the plane (see Figure 1). Then τ is defined to be 1 minus the half-open segments (O, A] and [A0 , O 0 ). Phong and Stein also proved that, for R to be bounded from Lp (R2 ) to Lq (R2 ), it is necessary that (1/p, 1/q) ∈ 1. When n = 2, 3, it is known that R is bounded precisely in 1 (see [PS1], [Se1], and [Se2]). When n ≥ 4, the endpoint (Lp , Lq ) estimates have been open except in the translation-invariant case S(x, y) = c(x −y)n , for which R is known to be bounded in all of 1 (see [PS2, p. 720] and [B]). (We want to point out here that the new method in [C2] may be used to prove a restricted weak type at the endpoints.) The purpose of this paper is to give a positive answer to these remaining endpoint questions. In fact, our results are somewhat more general. Since homogeneity plays Received 11 November 1998. 1991 Mathematics Subject Classification. Primary 42A85, 42B15. Author’s work partially supported by Korea Science and Engineering Foundation grant number 971-0102-009-2 and Korea Research Foundation (1998). 259

260

JONG-GUK BAK

1 q O0

1

A0

B

B0   2 1 D , 3 3

A O

1

1 p

Figure 1

no role in our arguments, these results actually cover arbitrary polynomials that satisfy the conditions a1,n−1 6 = 0 and an−1,1 6= 0. Let us now define a noncompact version of the operator R by Z ∞ ¡
(3) f t + S(x, y), y dy, Tf (t, x) = −∞

where f is, say, a continuous function with compact support. We first state our result for the point A = (2/(n + 1), 1/(n + 1)). Theorem 1. Suppose that n ≥ 2, and let T be defined by (3) with S(x, y) as in (1). If a1,n−1 6 = 0, then there exists a constant C(n) such that (4)

kTf kLn+1 (R2 ) ≤ C(n)|a1,n−1 |−1/(n+1) kf kL(n+1)/2 (R2 ) .

Here the constant C(n) depends only on the degree n. In particular, C(n) is independent of the function f and the coefficients aj k . The proof relies heavily on a method of Oberlin [O], which was used to give a new proof of the optimal convolution properties of the arclength measure on the unit circle. It is based on multilinear interpolation in contrast to the usual approach that uses Fourier transform and complex interpolation. By duality and interpolation, Theorem 1 immediately implies the following result. Theorem 2. Suppose that S(x, y) is a homogeneous polynomial of degree n ≥ 2. Let R and T be defined by (2) and (3), respectively. Suppose that a1,n−1 6= 0 and

RADON TRANSFORMS

261

an−1,1 6 = 0. Then R is bounded from Lp (R2 ) to Lq (R2 ), if and only if (1/p, 1/q) is in 1. Moreover, T is bounded from Lp (R2 ) to Lq (R2 ), if and only if (1/p, 1/q) is on the closed segment [A, A0 ]. Proof. Theorem 1 implies that T is bounded at A = (2/(n + 1), 1/(n + 1)). The adjoint of T is given by Z ∞ ¡
∗ g t − S(x, y), x dx. T (g)(t, y) = −∞

If an−1,1 6 = 0, then the proof of Theorem 1, with the roles of x and y reversed, shows that T ∗ is bounded at the point A, with the operator norm C(n)|an−1,1 |−1/(n+1) . Therefore, by duality, T is bounded at A0 = (n/(n + 1), (n − 1)/(n + 1)). Also, R is clearly bounded on the diagonal 0 ≤ 1/p = 1/q ≤ 1. Since R is dominated by T , the rest follows by interpolation and the known necessary conditions. (For a proof of the necessary condition for T , see the proof of Corollary 5.) In what follows, the letters C and c denote positive constants that may not be the same at each occurrence. We also use the notation g(t) ≈ h(t) to mean that the two functions are comparable; that is, there exist some positive constants C1 , C2 (uniform in t) such that C1 h(t) ≤ g(t) ≤ C2 h(t). The following lemma concerning polynomials plays a key role in our arguments. It is used to estimate the Jacobian of a certain transformation. Lemma 3. Let F (x) be a polynomial of degree k ≥ 1 with real coefficients. Suppose that the leading coefficient of F (x) is 1. Then there exist positive integers N and N (k), with 1 ≤ N ≤ N (k), and positive constants C1 (k), C2 (k), and c(k) such that, for each 1 ≤ j ≤ N, there exist an interval Ij , with ∪N j =1 Ij = R, and a polynomial wj which satisfy the inequalities C1 (k) wj (x) ≤ F (x) ≤ C2 (k) wj (x) and (5)

(k−1)/k dwj dx ≥ c(k) wj (x)

for x ∈ Ij . Moreover, the constants N(k), C1 (k), C2 (k), and c(k) depend only on the degree k. In particular, these constants are independent of the coefficients of F (x). Remark 4. Clearly, each function wj in Lemma 3 can have, at most, one root in the interval Ij . By dividing Ij into two subintervals if necessary, we may take wj to be nonnegative and strictly monotone on Ij . Proof of Lemma 3. Since F (x) has real coefficients, we can factorize and write it in the form Y ¡ Y
2  x − sj + δj , F (x) = (x − A` ) · `

j

262

JONG-GUK BAK

where the first product is over ν indices `, and the second product is over m indices j . We have k = ν + 2m. Here A` , sj are real numbers, and δj > 0. The proof is based on induction. We treat the case m = 0, ν ≥ 1 in Step 1 and the case m ≥ 1, ν ≥ 0 in Step 2. Step 1: The case m = 0, ν ≥ 1. We use induction on k = ν. The case k = 1 is trivial. Fix k ≥ 2. Assuming that the case k − 1 is true, we want to show that the case k holds. We may assume that A1 ≤ A2 ≤ · · · ≤ Ak = Aν . When A1 = · · · = Ak = A, we have F (x) = (x − A)k . Let w = (x − A)k on I = I1 = R. Then |dw/dx| = |k(x − A)k−1 | = k|w|(k−1)/k . Let us now assume A1 < Ak . By translation we may assume that A1 = −A and Ak = A for some A > 0. We may write F = Ak (t − r1 ) · · · (t − rk ), where t = x/A, rj = Aj /A. We have −1 = r1 ≤ r2 ≤ · · · ≤ rk = 1. We consider three cases. (i) The case 0 ≤ t ≤ 2. Here we have 1 ≤ t − r1 = t + 1 ≤ 3. Let v(t) = (t − r2 ) · · · (t − rk ). Since v has degree k − 1, it follows by the induction hypothesis that there exist intervals I1 , . . . , IN , N ≤ N(k − 1), whose union is R, and functions v1 , . . . , vN such that the inequalities v(t) ≈ vj (t) and (6)

dvj (k−2)/(k−1) dt ≥ c|vj |

hold on each Ij . Let Ij0 = Ij ∩ [0, 2]. On each nonempty Ij0 , define wj = Ak vj . Then |F (x)| ≈ |wj (x)| on Ij0 . Since dwj /dx = A−1 dwj /dt = Ak−1 dvj /dt and |vj | ≈ |v| ≤ 3k−1 on Ij0 ⊂ [0, 2], it follows from (6) that dwj k−1 (k−2)/(k−1) ≥ cAk−1 |vj |(k−1)/k = c|wj |(k−1)/k dx ≥ cA |vj | on each Ij0 . (ii) The case −2 ≤ t < 0. This case is similar to (i). (iii) The case |t| > 2. Since |rj | ≤ 1, we have |t − rj | ≈ |t|, and so |F (x)| ≈ |x|k . Let w1 , w2 = x k on I1 = (−∞, −2) and I2 = (2, ∞), respectively. Finally, rename the nonempty intervals Ij0 from (i) and (ii) as I3 , . . . , IN 0 , N 0 ≤ N (k), and renumber wj accordingly. Notice that we have N(k) ≤ 2N(k − 1) + 2. Thus the case k (and m = 0) of the lemma holds. This finishes the proof of the case m = 0. Step 2: The case m ≥ 1, ν ≥ 0. We use induction on m. The case m = 0 (and ν ≥ 1) has just been established in Step 1. Now fix m ≥ 1 and assume that the lemma is true in the case m − 1 (for ν ≥ 1 if m = 1, and for ν ≥ 0 if m ≥ 2). We want to show that the lemma is then true for m.

RADON TRANSFORMS

263

By writing t = x − s1 , rj = Aj − s1 , and uj = sj − s1 , we get u1 = 0. We may assume that 0 < δ1 ≤ δ2 ≤ · · · ≤ δm . Let us write δ = δ1 . Then ¡
    (7) F = (t − r1 ) · · · (t − rν ) t 2 + δ (t − u2 )2 + δ2 · · · (t − um )2 + δm . We may assume that |r1 | ≤ |r2 | ≤ · · · ≤ |rν | (if ν ≥ 1). Let us first show that it suffices to consider the interval I0 defined to be the intersection of the intervals given by the inequalities p (8) |t − uj | < δj for 1 ≤ j ≤ m. p To see this, suppose that |t − uj | ≥ δj for some j = 1, . . . , m. Then (t − uj )2 ≤ (t − uj )2 + δj ≤ 2(t − uj )2 , and so |F | ≈ |F˜ |, where Y  0 (t − ui )2 + δi F˜ = (t − r1 ) · · · (t − rν )(t − uj )2 Q and 0 means that the index j is omitted from the product. Since F˜ has m − 1 quadratic factors, we may apply the induction hypothesis. This takes care of the set R \ I0 . Thus we√may assume (8) in the rest of the proof. In particular, we have |t| = |t −u1 | < δ. To obtain estimates for I0 , we consider two cases. (If I0 is empty, there is of course nothing to prove.)√ (i) The case when |ri | > 2 δ for all i. Since |ri | > 2|t|, we have |t −ri | ≈ |ri | for each i. Also, (8) implies that (t − uj )2 + δj ≈ δj . Hence we have |F | ≈ Rεδ, where R = |r1 · · · rν | (we take R = 1 if ν = 0), and ε = δ2 · · · δm . We now define w to be the linear function √
√ ¡ w = Rε δ t + 2 δ √ on I0 . Then |w| ≈ Rεδ ≈ |F |, since |t| < δ. Finally, since Rε ≥ cδ ν/2 δ m−1 = cδ (k−2)/2 , it follows that √ (k−1)/k dw (k−1)/k ≈ w(t) on I0 . dt = Rε δ ≥ c|Rεδ| √ (ii) The case when |ri | ≤ 2 δ for exactly j (1 ≤ j ≤ ν) indices i. We have |F | ≈ Rε|v|, where v = (t −r1 ) · · · (t −rj ), R = |rj +1 · · · rν | (take R = 1 if j = ν), and ε = δ1 · · · δm . An application of the case m = 0 (Step 1) to v yields intervals I1 , . . . , IN , N ≤ N (j ), and functions v1 , . . . , vN that satisfy the conditions |v| ≈ |vi | and |dvi /dt| ≥ c|vi |(j −1)/j on Ii . Now define wi = Rεvi on Ii0 = Ii ∩ I0 , if this set is nonempty. Then, for 1 ≤ i ≤ N , dwi dvi = Rε ≥ cRε|vi |(j −1)/j on I 0 . i dt dt

264

JONG-GUK BAK

We see that the right-hand side is bounded below by c|Rεvi |(k−1)/k = c|wi |(k−1)/k since Rε ≥ cδ (ν−j )/2 δ m = cδ (k−j )/2 and |vi | ≈ |v| ≤ Cδ j/2 on Ii0 . This finishes the proof of the case m, and the proof of Lemma 3 is complete. Proof of Theorem 1. We may clearly assume that f is nonnegative. To prove (4), it suffices to prove the multilinear estimate Z

n+1 Y

R2 j =1

Tfj (t, x) dt dx ≤ C

n+1 Y

kfj k(n+1)/2

j =1

with C = C(n)n+1 |a1,n−1 |−1 . This follows from the estimate Z

n+1 Y

(9)

R2 j =1

Tfj (t, x) dt dx ≤ Ckf1 k1

n+1 Y

kfj kn,1

j =2

(and n similar estimates obtained by permuting the indices) by the so-called multilinear trick (see [C1] and [D]). Here k · kp,q denotes the Lorentz space norm on R2 . The left-hand side of (9) may be rewritten as Z

∞

Z

−∞ R2

Y ¡
n+1 f1 t + S(x, y − x), y − x Tfj (t, x) dt dx dy Z =

j =2

∞Z −∞ R2

f1 (s, z) Z

Z =

R2

f1 (s, z)

n+1 Y

¡
Tfj s − S(y − z, z), y − z ds dz dy

j =2 ∞ n+1 Y

−∞ j =2

¡
Tfj s − S(y, z), y dy ds dz.

So (9) is a consequence of Z (10)

∞ n+1 Y −∞ j =2

n+1 Y ¡
Tfj s − S(y, z), y dy ≤ C kfj kn,1 , j =2

where C is independent of (s, z) ∈ R2 . But (10) follows, by the multiple Hölder inequality, from Z

∞ −∞



¡
n Tf s − S(y, z), y dy

1/n ≤ Ckf kn,1 .

RADON TRANSFORMS

265

This is equivalent to the estimate Z ∞ ¡
(11) Tf s − S(y, z), y g(y) dy ≤ Ckf kn,1 kgkn/(n−1) , −∞

where g is a nonnegative function on R. The left-hand side of this inequality is equal to Z ¡
f s + S(y, x) − S(y, z), x g(y) dx dy. I= R2

Let us now make the change of variables ¡
(12) (ξ, η) = φ(y, x) = s + S(y, x) − S(y, z), x with s, z fixed. We claim that R2 minus the vertical line x = z may be decomposed into certain subsets U1 , . . . , UN , with 1 ≤ N ≤ n−1, such that φ is one-to-one on each Uj . To see this, assume that φ(y, x) = φ(y 0 , x 0 ). Then we have x = x 0 , and S(y, x)− S(y, z) = S(y 0 , x)−S(y 0 , z). Note that the partial derivative (∂/∂y)(S(y, x)−S(y, z)) is a polynomial in y of degree m, for some 0 ≤ m ≤ n − 2 (see (14)). If m > 0, then this partial derivative changes sign at most m times, as y varies over R. Therefore, there exist intervals I1 , . . . , IN , for some 1 ≤ N ≤ m + 1, such that S(y, x) − S(y, z) is a strictly monotone function of y on each Ij = Ij (x). Hence, if we assume further that y, y 0 ∈ Ij for some j , then it follows that y = y 0 , and so (x, y) = (x 0 , y 0 ). As x varies over R \ {z}, the interval Ij = Ij (x) sweeps over a set Uj . That is, we define Uj = {(x, y) : y ∈ Ij (x), x ∈ R, x 6= z}. Then φ must be one-to-one on Uj . On the P other hand, if m = 0, then (∂/∂y)(S(y, x) − S(y, z)) = nd=1 a1,d−1 (x d−1 − zd−1 ). Since a1,n−1 6 = 0, this is a polynomial in x of degree n−1. Let x = x1 , . . . , x` (1 ≤ ` ≤ n − 1) be the real roots of this polynomial. If x 6= x1 , . . . , x` , then (∂/∂y)(S(y, x) − S(y, z)) 6 = 0, and so S(y, x)−S(y, z) is a strictly monotone function of y. Therefore, in this case, φ is one-to-one on R2 minus the vertical lines x = x1 , . . . , x` . This proves the claim. Thus we have Z (13) f (ξ, η)G(ξ, η) dξ dη, I ≤C R2

where G(ξ, η) = g(y)/|J |, and J is the Jacobian of φ. A calculation shows that
∂ ¡ ∂(ξ, η) = S(y, x) − S(y, z) J (x, y, z) = ∂(y, x) ∂y (14) d−1 n X X ¡
(d − k)ad−k, k y d−k−1 x k − zk . = d=1 k=1

Fix y, z. Then J may be regarded as a polynomial of degree n−1 in x, with the leading coefficient a1, n−1 . Let us write a for a1, n−1 . By applying Lemma 3 to F (x) = J /a, and k = n − 1, we obtain intervals I1 , . . . , IN , with N ≤ N(n − 1), and functions w1 , . . . , wN such that

266

JONG-GUK BAK

J ≥ C wj (x) a

(15) and

∂wj ≥ c wj (x) (n−2)/(n−1) ∂x

(16)

on each Ij . To finish the proof of the inequality (11), we first estimate I by using (13) and the Hölder inequality for Lorentz spaces. We have I ≤ Ckf kn,1 kGkn/(n−1),∞ . It remains to show that kGkn/(n−1),∞ ≤ Ckgkn/(n−1) , which is equivalent to (17)

 ª (ξ, η) ∈ R2 : G(ξ, η) > λ ≤ C

Z

∞

−∞



g(y) λ

n/(n−1)

dy,

λ > 0.

The left-hand side of (17) is equal to the integral Z dξ dη. {(ξ,η)∈R2 : G(ξ,η)>λ}

Reversing the change of variables made above by (12) shows that this is bounded by Z ∞Z Z g(y) |J | dx dy ≤ C dx dy. C {(x,y)∈R2 : |J |
RADON TRANSFORMS

267

This implies (17), hence (11), with C = Cn |a|−1/n , where Cn is some constant depending only on n. (In the homogeneous case, we may first establish this estimate assuming |a| = 1, and then we can obtain the correct operator norm in (4) by a scaling argument.) The proof of (4) is complete. Finally, as an application of Theorem 1, in Corollary 5 we establish global convolution properties of the measure dµ = dy on the curve (P (y), y) in the plane, where P (y) is an arbitrary polynomial of degree n ≥ 2 with real coefficients. This corresponds to the translation-invariant case S(x, y) = P (y − x) in (3), and it may also be regarded as a generalization of the result for the case S(x, y) = c(x − y)n . According to this result, a necessary condition for the global convolution estimate from Lp to Lq is that 1 1 1 1 ≤ − ≤ , n+1 p q m+1 where m is the (finite) type of the curve. That is, the upper bound for the number 1/p −1/q, which may be called the degree of smoothing, is determined by the highest vanishing order of the curvature, while the lower bound is determined by the degree of the polynomial. Of course, we also have the usual necessary condition that (1/p, 1/q) is in the closed convex hull of the points O = (0, 0), O 0 = (1, 1), and D = (2/3, 1/3). The intersection of these two sets is the trapezoid 1nm . Namely, 1nm is the closed convex hull of the points A = (2/(n+1), 1/(n+1)), A0 = (n/(n+1), (n−1)/(n+1)), B = (2/(m+1), 1/(m+1)), and B 0 = (m/(m+1), (m−1)/(m+1)). (See Figure 1.) The condition (1/p, 1/q) ∈ 1nm is also sufficient. Corollary 5. Let P (y) be a polynomial of degree n ≥ 2 with real coefficients. Let ν ≥ 0 be the number of the distinct real roots Aj of P 00 (y). Let mj be the multiplicity of the root Aj . Set m = 2 if ν = 0, and set m = 2 + max{m1 , . . . , mν } if ν > 0. Define Z ∞ ¡
f t + P (y), x + y dy. Tf (t, x) = −∞

Then T is bounded from Lp (R2 ) to Lq (R2 ), if and only if (1/p, 1/q) is in 1nm . Proof. If ν = 0, we can show that T is bounded at D = (2/3, 1/3) by using the standard proof based on complex interpolation along with the fact that |P 00 (y)| is bounded away from zero. (In fact, this may by shown by considering the anaR1 R∞ lytic family Sz f (t, x) = (z + 1) −1 −∞ f (t + P (y) + s, x + y) dy|s|z ds. Since T is a constant multiple of S−1 , the required estimate follows from the estimates (i) kSz f k2 ≤ Cz kf k2 for Re z = −3/2 and (ii) kSz f k∞ ≤ Cz kf k1 for Re z = 0. The estimate (i) is a consequence of Plancherel’s theorem and van der Corput’s lemma, while (ii) follows from a change of variables.)

268

JONG-GUK BAK

Next assume that ν > 0. Let ε > 0 be a sufficiently small number. For each Aj , define Z ¡
f t + P (y), x + y dy. Tj f (t, x) = |y−Aj |<ε

We decompose T as the sum of operators T1 , . . . , Tν , and T0 = T − T1 − · · · − Tν . Fix j = 1, . . . , ν. By translation, we may assume Aj = 0. We can show that Tj is bounded at Bj = (2/(mj +3), 1/(mj +3)) by using a variant of the proof of Theorem 1. We give only a brief sketch. We may assume the domain of integration of Tj is 0 < y < ε. Since we are dealing with a translation-invariant operator here, it suffices to prove (10) with (s, z) = (0, 0). Then the counterpart of φ in (12) may be given the form φ(x, y) = (P (y) − P (x), y − x), and its Jacobian is given by Z y 0 0 P 00 (t) dt. J (x, y) = P (y) − P (x) = |P 00 (t)| ≈ |t|mj

x m |J | ≈ |x j +1 − y mj +1 |

for small t, we have for small x, y > 0, Since and so an application of Lemma 3 enables us to finish the proof. This shows that Tj is bounded at Bj and, hence, also in the closed convex hull of Bj , Bj0 , O = (0, 0), and O 0 = (1, 1). The proof that T0 is bounded at D = (2/3, 1/3) is the same as that for the case ν = 0, since |P 00 (y)| is bounded away from zero on the domain of integration of T0 . Now Theorem 1 implies that T is bounded at A; hence so is T0 . Thus, T0 is bounded in the closed convex hull of A, A0 , and D. Therefore, we may conclude that T is bounded in 1nm , since it is the intersection of the type sets of Tj for 0 ≤ j ≤ ν. To prove the necessary condition (1/p, 1/q) ∈ 1nm for the boundedness of T , we adapt the standard argument. Suppose that T is bounded from Lp (R2 ) to Lq (R2 ). Choose a point Aj such that m = mj + 2. We may assume Aj = 0. Taking f to be the characteristic function of the parallelogram {(x, y) : |y| ≤ δ, |x − P (0) − P 0 (0)y| ≤ δ m } gives Tf (t, x) ≥ cδ on the parallelogram {(t, x) : |x| ≤ δ/2, |t − P 0 (0)x| ≤ δ m /2}. So we obtain cδ 1+(m+1)/q ≤ kTf kq ≤ Ckf kp = Cδ (m+1)/p for small δ > 0. This implies that 1/p − 1/q ≤ 1/(m + 1), namely, (1/p, 1/q) is on or above the line BB 0 . Taking f to be the characteristic function of the rectangle [−δ n , δ n ] × [−δ, δ] for large δ > 0 shows that Tf (t, x) ≥ cδ on the rectangle [0, δ n /2] × [0, δ/2]. Hence, kTf kq ≥ cδ 1+(n+1)/q , while kf kp = Cδ (n+1)/p . So 1/p − 1/q ≥ 1/(n + 1); that is, (1/p, 1/q) is on or below the line AA0 . Taking f to be the characteristic function of the square [−δ, δ] × [−δ, δ] for small δ > 0 gives 1+1/q ≥ 2/p. That is, the point (1/p, 1/q) is on or above the line O 0 D, and duality implies that the point is on or above the line OD as well. Therefore, we may conclude that (1/p, 1/q) is in 1nm . Remark 6. Note that if P (y) has degree 3, then m = n = 3 in Corollary 5; hence it follows that the associated operator T is bounded precisely on the closed segment [A, A0 ], where A = (1/2, 1/4), A0 = (3/4, 1/2).

RADON TRANSFORMS

269

Acknowledgements. We would like to thank the referee for suggesting an apt title and for updating the references. References [B] [C1] [C2] [D]

[O] [P]

[PS1] [PS2] [PS3] [Se1] [Se2] [S]

J.-G. Bak, Multilinear proofs for convolution estimates for degenerate plane curves, to appear in Canad. Math. Bull. (1999). M. Christ, On the restriction of the Fourier transform to curves: Endpoint results and the degenerate case, Trans. Amer. Math. Soc. 287 (1985), 223–238. , Convolution, curvature, and combinatorics: A case study, Internat. Math. Res. Notices 1998, 1033–1048. S. Drury, “A survey of k-plane transform estimates” in Commutative Harmonic Analysis (Canton, N.Y., 1987), Contemp. Math. 91, Amer. Math. Soc., Providence, 1989, 43– 55. D. Oberlin, Multilinear proofs for two theorems on circular averages, Colloq. Math. 63 (1992), 187–190. D. H. Phong, “Singular integrals and Fourier integral operators” in Essays on Fourier Analysis in Honor of Elias M. Stein (Princeton, N.J., 1991), Princeton Math. Ser. 42, Princeton Univ. Press, Princeton, 1995, 286–320. D. H. Phong and E. M. Stein, Radon transforms and torsion, Internat. Math. Res. Notices 1991, 49–60. , Models of degenerate Fourier integral operators and Radon transforms, Ann. of Math. (2) 140 (1994), 703–722. , The Newton polyhedron and oscillatory integral operators, Acta Math. 179 (1997), 105–152. A. Seeger, Degenerate Fourier integral operators in the plane, Duke Math. J. 71 (1993), 685–745. , Radon transforms and finite type conditions, J. Amer. Math. Soc. 11 (1998), 869– 897. E. M. Stein, “Oscillatory integrals related to Radon-like transforms” in Proceedings of the Conference in Honor of Jean-Pierre Kahane (Orsay, 1993), J. Fourier Anal. Appl. 1995, special issue, CRC, Boca Raton, Fla., 535–551.

Department of Mathematics, Pohang University of Science and Technology, Pohang 790-784, Korea; [email protected]

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

EXTREMAL MANIFOLDS AND HAUSDORFF DIMENSION H. DICKINSON and M. M. DODSON

1. Introduction. The recent proof by D. Y. Kleinbock and G. A. Margulis [11] of Sprindžuk’s conjecture for smooth nondegenerate manifolds M means that the set Lv (M) of v-approximable points (this and other terminology is explained below) on M is of zero induced Lebesgue measure. This raises the question of its Hausdorff dimension. Bounds and indeed the exact dimension for manifolds satisfying a variety of arithmetic, geometric, and analytic conditions are known (see [2], [3], [5], [7]). In this paper ubiquity is used to obtain a lower bound for the Hausdorff dimension of a set more general than Lv (M) for any extremal C 1 manifold M. Hitherto volume estimates that depend on curvature conditions were used to overcome a “small denominators” problem. It turns out, however, that extremality, when combined with Fatou’s lemma, is all that is needed. We begin with some notation. Let |x| = max{|x1 |, . . . , |xn |} denote the supremum norm or height of the point x = (x1 , . . . , xn ) in n-dimensional Euclidean space Rn , and denote its Euclidean norm by |x|2 = (x12 +· · ·+xn2 )1/2 . Throughout, q = (q1 , . . . , qn ) is a vector in Zn , and q · x = q1 x1 + · · · + qn xn denotes the usual inner product. For positive numbers a, b, we use the Vinogradov notation a  b and b  a if a = O(b). If a  b  a, we write a  b. A point x ∈ Rn that satisfies (1)

kq · xk < |q|−v

for infinitely many q ∈ Zn is called v-approximable (kxk is the distance of the real number x from Z). Let M be an m-dimensional manifold in Rn . The set of v-approximable points in the manifold M is denoted by Lv (M). The manifold M is called extremal if for any v > n, Lv (M) has Lebesgue measure 0. Equivalently, by Khintchine’s transference principle, M is extremal if the set Sw (M) of points x ∈ M that are simultaneously w-approximable (i.e., for which kqxk < |q|−w for infinitely many q ∈ Z) is null (i.e., of measure zero) when w > 1/n. Khintchine’s theorem implies that the real line is extremal, and the terminology reflects the fact that Received 24 November 1998. 1991 Mathematics Subject Classification. Primary 11J83; Secondary 11J20. Dickinson supported by Engineering and Physical Sciences Research Council grant number GR/ K56407. 271

272

DICKINSON AND DODSON

the order of approximation given by Dirichlet’s theorem is unimprovable for almost all points on an extremal manifold (see [12]). Let U be an open set in Rm , where m 6 n. V. G. Sprindžuk conjectured that if the functions θj : U → R, j = 1, . . . , n are analytic and, together with 1, independent over R, then the manifold
ª ¡ θ1 (u), . . . , θn (u) : u ∈ U = θ(U ) ⊂ Rn is extremal (see Conjecture H1 in [19]). Manifolds satisfying a variety of additional or different analytic, geometric, and number-theoretic conditions have been shown to be extremal; references and further details can be found in [18], [19] (see also [4], [7], [9], [11], [20]). In the stronger Baker-Sprindžuk conjecture, the hypotheses on the manifold M are the same, but the approximation function |q|−v is replaced by a larger multiplicative anisotropic function. When v > n, if the set of points x ∈ M for which (2)

kq · xk <

n Y ¡


−v/n

|qj | + 1

j =1

for infinitely many q ∈ Zn is relatively null, then M is said to be strongly extremal (see Conjecture H2 in [19]). Points satisfying (2) for infinitely many q ∈ Zn are called multiplicatively v-approximable. Transference principles allow simultaneous and multiplicative approximation forms of these conjectures (see [11], [18]). The conjecture H2 was first proposed by A. Baker for the rational normal curve ¡
ª V = t, t 2 , . . . , t n : t ∈ R in [1] and proved for this case by V. I. Bernik [6]. J. Kubilius proved the parabola extremal in 1949 [13], and in 1964 W. M. Schmidt established the remarkable result that any C 3 planar curve with nonzero curvature almost everywhere is extremal [16]. About the same time, Sprindžuk proved Mahler’s conjecture, corresponding to the rational normal curve being extremal (see [17]). Recently, in [11], Kleinbock and Margulis have proved a result that implies not only Sprindžuk’s conjecture H1 , but also the Baker-Sprindžuk conjecture H2 . They used ideas from dynamical systems, namely, unipotent flows in homogeneous spaces of lattices and the correspondence between multiplicatively v-approximable points for v > n and unbounded orbits in the space of lattices. Although at the moment their techniques do not yield nontrivial upper bounds for the Hausdorff dimension, they do give a partial Khintchine-type result and might open the way to further progress. In [3], R. C. Baker refined Schmidt’s result [16] by showing that if the curvature of a C 3 planar curve vanishes only on a set with Hausdorff dimension 0, then for v > 2, dim Lv (M) =

3 . v +1

Using the idea of regular systems, A. Baker and Schmidt [2] showed that dim Lv (V) >

EXTREMAL MANIFOLDS AND HAUSDORFF DIMENSION

273

(n + 1)/(v + 1) for v > n. The complementary upper inequality was established by Bernik [5], giving n+1 dim Lv (V) = v +1 for v > n. For manifolds M with dimension m > 2 and satisfying a curvature condition that reduces to nonvanishing Gaussian curvature for surfaces in R3 , dim Lv (M) = m − 1 +

n+1 v +1

for v > n (see [7]). We use ubiquity (see [8]) to obtain the best possible lower bound for the Hausdorff dimension of the more general set  ª L(M; ψ) = x ∈ M : kq · xk < ψ(|q|) for infinitely many q ∈ Zn when M is a C 1 extremal manifold in Rn and the function ψ : N → R+ decreases. Note that when ψ(q) = q −v , we write Lv (M) for L(M; ψ). For more information about Hausdorff dimension, see [10], [14]. Acknowledgements. We are grateful to Dmitry Kleinbock and Gregor Margulis for helpful and stimulating discussions during the Dynamical Systems and Related Topics meeting held in 1997 at the Erwin Schrödinger Institute for Mathematical Physics in Vienna, to Chris Wood for his help with the differential geometry, and to Alan Baker and Bryan Rynne for their comments on earlier drafts. We are also grateful to the referee for suggesting ways of improving our presentation. We would like to thank the Engineering and Physical Sciences Research Council for enabling us to participate in the Vienna meeting. 2. Ubiquitous systems. Let U be a nonempty open subset of Rm . Let  ª R = Rj ⊂ U : j ∈ J be a family of sets indexed by J ; these sets are called resonant. Suppose further that each j ∈ J has a weight bj c > 0, and let ρ : N → R+ be a function converging to zero at infinity. Suppose that for each sufficiently large positive integer N, there exists a set A(N) ⊂ U for which (3) lim U \ A(N ) = 0. N→∞

Let (4)


ª
 ¡ ¡ B Rj ; δ = u ∈ U : dist u, Rj < δ ,

where dist(u, R) = inf{|u − r| : r ∈ R}. Let H /2 denote the hypercube H shrunk by 1/2 and with the same centre.

274

DICKINSON AND DODSON

Suppose that there exists a constant d ∈ [0, m] such that given any hypercube H ⊂ U with sidelength `(H ) = ρ(N) and H /2 meeting A(N ), there exists a j ∈ J with bj c 6 N such that for all δ ∈ (0, ρ(N )],
¡ H ∩ B Rj ; δ  δ m−d `(H )d . (5) Suppose further that given any other hypercube H 0 in U with `(H 0 ) 6 ρ(N), 0 ¡
H ∩ H ∩ B Rj ; δ  δ m−d `(H 0 )d . (6) Then the pair (R, b·c) is called a ubiquitous system with respect to ρ. In the one-dimensional case and when the resonant sets consist of points, ubiquitous and regular systems are virtually equivalent and essentially differ only in their formulation (see [15]). 3. Hausdorff dimension. The distribution of the resonant sets in ubiquitous systems allows the determination of a general lower bound for the lim-sup set
ª  ¡ 3(R; ψ) = u ∈ U : dist u, Rj < ψ(bj c) for infinitely many j ∈ J , where ψ : N → R+ is a decreasing function (see [8]). Theorem 1. Suppose (R, b·c) is ubiquitous with respect to ρ : N → R+ and that ψ : N → R+ is a decreasing function satisfying ψ(N ) 6 ρ(N) for N sufficiently large. Then dim 3(R ; ψ) > d + γ (m − d) where γ = lim supN →∞ (log ρ(N ))/(log ψ(N )) 6 1. The hypothesis that ψ(N ) 6 ρ(N) for N sufficiently large implies that γ 6 1. We now apply Theorem 1 to Diophantine approximation on a manifold. The lower order λ(f ) of the function f : N → R+ is defined by λ(f ) = lim inf N→∞

log f (N ) . log N

Theorem 2. Let M be an m-dimensional C 1 extremal manifold embedded in Rn . Let ψ : N → R+ be decreasing with the lower order of 1/ψ denoted by λ. Then for λ > n, dim L(M; ψ) > m − 1 +

n+1 . λ+1

Since dim L(M ∩ V ; ψ) 6 dim L(M; ψ), it suffices to consider the open subset M ∩ V of M, where V is a suitable open set in Rn . We assume without loss of

EXTREMAL MANIFOLDS AND HAUSDORFF DIMENSION

275

generality that M ∩V ⊂ [−1, 1]n and that V is sufficiently small. Let θ : U → M ∩V be the local parametrisation where the domain U is a sufficiently small hypercube in [−1, 1]m . We write MU = M ∩ V = θ(U ). Each point x ∈ MU can be written x = θ(u) for some u ∈ U . Since the manifold M is C 1 , by shrinking and closing U if necessary, we can assume that the geodesic distance between two points x, x 0 on MU is comparable with |x −x 0 | and that θ is bi-Lipschitz on U . Hence we can assume that the Hausdorff dimension of L(MU ; ψ) and that of ° ° ª  L(ψ) = u ∈ U : °q · θ (u)° < ψ(|q|) for infinitely many q ∈ Zn are the same (see [10]). We write Lv for L(ψ) when ψ(r) = r −v ; thus ° °  ª Lv = u ∈ U : °q · θ (u)° < |q|−v for infinitely many q ∈ Zn . By the inverse function theorem, we can also assume that MU is the graph of a C 1 (Monge) ordinate function ϕ : U → Rk , where k = n − m, so that  ª ¡
ª MU = θ(u) : u ∈ U = u, ϕ(u) : u ∈ U and θ = 1U × ϕ. The corresponding local chart h : MU → U is the restriction to MU of the projection Rm × Rk → Rm . Moreover, by shrinking and closing U again if necessary, we can assume |∂ϕj /∂ui | 6 Kij < ∞ for each u ∈ U , i = 1, . . . , m, j = 1, . . . , k. Indeed given δ > 0, we can choose U so that for any u ∈ U , ∂ϕj (u) 6 Kij . Kij − δ 6 ∂ui Thus we can assume that the change in the direction of a vector along any geodesic in MU is small. It follows that MU is not close to orthogonal to Rm × {0}, 0 = (0, . . . , 0) ∈ Rk , as indicated in Figure 1. More precisely, for each θ (u) in MU , the angle ϑ, say, between any vector in the tangent space Tθ(u) MU and Rm × {0}, satisfies cos ϑ > c for some constant c > 0 (i.e., in the Vinogradov notation, cos ϑ  1). Thus for any θ (u) in MU , the plane Rm × {0} is not close to being orthogonal to Tθ(u) MU . In other words, the normal space Tθ (u) MU⊥ is not close to being parallel to Rm × {0}. Since M is extremal, Lv (MU ) = θ(Lv ) is null for v > n in the induced measure on M and, since θ is bi-Lipschitz on U , the set Lv is null in Rm when v > n. To obtain a lower bound for the Hausdorff dimension of Lv (M) or equivalently for Lv , it suffices to find a sequence of suitable sets A(N ) ⊂ U that approximate U in measure and that satisfy the intersection conditions (5) and (6) above. Using the geometry of numbers, integer vectors q are chosen so that the hyperplanes  ª 5p,q = x ∈ Rn : q · x = p

276

DICKINSON AND DODSON

{0} × Rk

C 5p,q C C C C Cα C C C C

MU

U

Rm × {0}

:    q p p p p C    C C β  C   Figure 1. The manifold MU and a resonant set 5p,q

associated with the resonant sets Rp,q , defined below in (11), are not close to being parallel or tangential to MU (see Figure 1). This condition is stronger than 5p,q being transversal to MU . Let η > 0 be arbitrary and let N ∈ N be sufficiently large. By Minkowski’s linear forms theorem, given a point u ∈ U , there exist q = q(u) = (q1 , . . . , qn ) ∈ Zn satisfying 1 6 |q| 6 N , and p = p(u) ∈ Z such that  −n+kη (log N)k   q · θ(u) − p 6 N (7) |qi | 6 N, i = 1, . . . , m   |qm+j | 6 N 1−η (log N)−1 , j = 1, . . . , k. Hence for each N = 1, 2, . . . , the set U can be written (8)

U = A(N ) ∪ S(N) ∪ E(N ),

where E(N ) = {u ∈ U : dist(u, ∂U ) 6 1/N} (∂U is the boundary of U ), ª  S(N ) = u ∈ U : 1 6 |q| < N 1−η for some q satisfying (7) , and ¡
A(N ) = U \ E(N ) ∪ S(N) , so that A(N) consists of points u ∈ U \E(N ) for which there exist q ∈ Zn and p ∈ Z satisfying (7) and (9)

N 1−η 6 |q| 6 N.

EXTREMAL MANIFOLDS AND HAUSDORFF DIMENSION

277

Thus each u ∈ A(N) is at least 1/N from ∂U (in the supremum metric), and there exists a large vector q ∈ Zn and an integer p satisfying (7). The measure of E(N ) converges to 0 as N → ∞ since    m E(N ) = u ∈ U : dist(u, ∂U ) 6 1  `(U )m − `(U ) − 1  N −1 . N N The vector q = q(u) ∈ Zn can be written q = (q1 , . . . , qm , 0, . . . , 0) + (0, . . . , 0, qm+1 , . . . , qn ) = q0 + q00 , say, where q0 ∈ Rm ×{(0, . . . , 0)} and q00 ∈ {(0, . . . , 0)}×Rk . Since N is large enough, for each u ∈ A(N), the vector q is close to being parallel to q0 . Indeed the angle β that q makes with Rm × {0} satisfies cos β =

2  q 2 + · · · + qn2 q0 1 q · 0 > 1 − m+1 2 = 1−O |q|2 |q |2 log N |q|2

by (7) and (9). Hence the hyperplane 5p,q , which is normal to q, meets MU not close to tangentially. This implies that 5p,q ∩ MU is a connected (m − 1)-dimensional submanifold of MU . On replacing N by N 1/(1−η) in (7), it can be seen that the set S(N 1/(1−η) ) is contained in the set of points u ∈ U for which there exist p, q satisfying q · θ(u) − p < N −(n−kη)/(1−η) (log N)k (1 − η)−k with 1 6 |q| 6 N . Moreover, S(N 1/(1−η) ) is also a subset of  ª Tδ (N) = u ∈ U : q · θ(u) − p < N −n−δ for some q ∈ Zn , p ∈ Z, 1 6 |q| 6 N , where 0 < δ < η(n − k)/(1 − η). Lemma 1. For any δ > 0, lim sup Tδ (N ) = N →∞

∞ ∞ [ \

Tδ (N ) ⊆ Ln+δ .

k=1 N=k

∞ Proof. Let u ∈ ∩∞ k=1 ∪N =k Tδ (N ). Then u ∈ Tδ (Nj ) for an infinite subsequence Nj , j = 1, 2, . . . . Hence for each j there exist q(j ) ∈ Zn with 1 6 |q(j ) | 6 Nj and p (j ) ∈ Z such that

(j ) q · θ (u) − p(j ) < N −n−δ . j

278

DICKINSON AND DODSON

Suppose there are only finitely many different q(j ) for which the last displayed inequality holds and let ª  min q(j ) · θ (u) − p(j ) : j ∈ N = c, say. If c > 0, then choosing j so that Nj−n−δ < c gives a contradiction. If c = 0, then for each r ∈ N, r 6 |rq(j ) | 6 rNj and ¡ ¡
¡
−n−δ (j )
. · θ (u) − rp(j ) = 0 < rNj rq Thus there are infinitely many solutions, contradicting the supposition that there exist only a finite number of different q(j ) . But 1 6 |q(j ) | 6 Nj , whence (j ) q · θ (u) − p (j ) < q(j ) −n−δ holds for infinitely many j . Thus u ∈ Ln+δ . By Fatou’s lemma, for any δ > 0,

lim sup Tδ (N ) 6 lim sup Tδ (N ) 6 |Ln+δ | = 0 N →∞

N→∞

since M is extremal. Thus limN→∞ |Tδ (N )| = 0. But when 0 < δ < η(n−k)/(1−η), Tδ (N) ⊇ S(N 1/(1−η) ), and so ¡
lim S(N) = lim S N 1/(1−η) = 0. N →∞

N→∞

Applying this and the estimate for |E(N )| above to (8), it follows that U \ A(N ) 6 E(N ) + S(N) −→ 0 (10) as N → ∞ and A(N) satisfies (3). The resonant sets in U are now chosen to be  ª ¡
(11) Rp,q = u ∈ U : q · θ (u) = p = h 5p,q ∩ MU , where q and p are given by (7). Thus d, the dimension of Rp,q , is m − 1. For each u ∈ A(N), there exists a pair (p, q) satisfying (7) and N 1−η 6 |q| 6 N. For N sufficiently large, the hyperplane 5p,q is far from tangential to MU . Because of this and θ being bi-Lipschitz,
¡ ¡

q · θ (u) − p ¡ , dist u, Rp,q  dist θ (u), θ Rp,q  |q|2 | cos $ | where $ is the angle between the tangent plane Tθ(u) MU and q. Since 5p,q meets MU not close to tangentially, cos $  1, and so for any u ∈ U ,
q · θ (u) − p ¡ . dist u, Rp,q  |q|2

EXTREMAL MANIFOLDS AND HAUSDORFF DIMENSION

279

It follows from this and (7) that there are positive c∗ , c∗ such that ¡
(12) c∗ q · θ(u) − p |q|−1 6 dist u, Rp,q 6 c∗ N −n−1+η(k+1) (log N)k . Let (13)

ρ(N ) = 4c∗ N −n−1+(k+1)η (log N)k .

We now show that the other ubiquity properties (5) and (6) hold for the family R of resonant sets {Rp,q } where b(p, q)c = |q| and ρ : N → R+ is given by (13). Let H be a hypercube with `(H ) = ρ(N). The choice of q, which ensures that 5p,q meets MU not close to tangentially, together with the choice of ρ, implies that if in addition u ∈ H /4, then by (12), there exist p, q such that dist(u, Rp,q ) 6 `(H )/4. Hence the resonant set Rp,q meets the hypercube H substantially and
¡ H ∩ B Rp,q ; δ  `(H )m−1 δ, where B(Rp,q ; δ) is given by (4), as required for (5) to hold. It also follows that 5p,q meets MU in a connected (m − 1)-dimensional submanifold, so that any hypercube H 0 with `(H 0 ) 6 ρ(N) satisfies 0 ¡
 ª H ∩ H ∩ B Rp,q ; δ  `(H 0 )m−1 min δ, `(H 0 )  `(H 0 )m−1 δ, and (6) holds. Thus the family R = {Rp,q : q ∈ Zn \ {0}, p ∈ Z} is ubiquitous in U e : N → R+ , with respect to ρ. Hence by Theorem 1, for any decreasing function ψ
¡ e > m−1+γ, dim 3 R; ψ e) is the set of points u in U satisfying where 3(R; ψ
¡
¡ e b(p, q)c = ψ e(|q|) dist u, Rp,q < ψ e(N )). for infinitely many p, q and where γ = lim supN→∞ (log ρ(N ))/(log ψ −1 e(r) = c∗ r ψ(r). Then by (12), dist(u, Rp,q ) < ψ e(|q|) implies that Choose ψ e) implies that for infinitely many p, q, |q · θ(u) − p| < ψ(|q|). Therefore u ∈ 3(R; ψ q · θ (u) − p < ψ(|q|), e) ⊂ L(ψ). Thus and so 3(R; ψ ¡
e > m−1+γ, dim L(ψ) > dim 3 R; ψ where by (13) γ = lim sup N →∞

n + 1 − η(k + 1) log ρ(N)
= , −1 λ+1 log c∗ N ψ(N ) ¡

280

DICKINSON AND DODSON

where λ is the lower order of 1/ψ. Since η is an arbitrary positive number and U is a parametrisation domain, it follows that (14)

dim L(M ; ψ) > dim L(ψ) > m − 1 +

n+1 , λ+1

and Theorem 2 is proved. By [11], a C r m-dimensional manifold embedded in Rn and `-nondegenerate for some ` ≤ r almost everywhere is extremal (θ(U ) is `-nondegenerate if Rn is spanned by the partial derivatives of θ up to order `). Hence (14) holds for such manifolds with r ≥ 1 and so in particular for manifolds with at least one principal curvature nonzero almost everywhere. If M is not extremal, then dim Lw (M) = m for some w > n, and hence dim Lv (M) = m for v 6 w. Obtaining an upper bound for the Hausdorff dimension of L(M; ψ) involves estimating large contributions from near tangential resonant sets Rp,q and is much more difficult. The upper bound for L(M; ψ) has been shown to be m−1+(n+1)/(v +1) for v > n when M is C 3 , of dimension m > 2, and has at least two principal curvatures nonzero everywhere except on a set of Hausdorff dimension at most m − 1 (see [7]), so that the lower bound in Theorem 2 is best possible. It is likely that this is the Hausdorff dimension when at least one principal curvature is nonzero everywhere except on a set of Hausdorff dimension at most m − 1. Determining the Hausdorff dimension in the case of simultaneous Diophantine approximation seems harder and much less is known. References [1] [2] [3] [4] [5] [6] [7]

[8] [9] [10] [11]

A. Baker, Transcendental Number Theory, Cambridge Univ. Press, London, 1975. A. Baker and W. M. Schmidt, Diophantine approximation and Hausdorff dimension, Proc. London Math. Soc. (3) 21 (1970), 1–11. R. C. Baker, Dirichlet’s theorem on Diophantine approximation, Math. Proc. Cambridge Philos. Soc. 83 (1978), 37–59. V. Beresnevich and V. Bernik, On a metrical theorem of W. Schmidt, Acta Arith. 75 (1996), 219–233. V. I. Bernik, Application of the Hausdorff dimension in the theory of Diophantine approximations (in Russian), Acta Arith. 42 (1983), 219–253. , A proof of Baker’s conjecture in the metric theory of transcendental numbers (in Russian), Dokl. Akad. Nauk SSSR 277 (1984), 1036–1039. M. M. Dodson, B. P. Rynne, and J. A. G. Vickers, Metric Diophantine approximation and Hausdorff dimension on manifolds, Math. Proc. Cambridge Philos. Soc. 105 (1989), 547–558. , Diophantine approximation and a lower bound for Hausdorff dimension, Mathematika 37 (1990), 59–73. , Khintchine-type theorems on manifolds, Acta Arith. 57 (1991), 115–130. K. J. Falconer, The Geometry of Fractal Sets, Cambridge Tracts in Math. 85, Cambridge Univ. Press, Cambridge, 1986. D. Y. Kleinbock and G. A. Margulis, Flows on homogeneous spaces and Diophantine approximation on manifolds, Ann. of Math. (2) 148 (1998), 339–360.

EXTREMAL MANIFOLDS AND HAUSDORFF DIMENSION [12] [13] [14] [15] [16] [17] [18] [19]

[20]

281

J. F. Koksma, Diophantische Approximationen, Springer-Verlag, Berlin, 1974. J. Kubilius, On an application of I. M. Vinogradov’s method to the solution of a problem of the metrical theory of numbers (in Russian), Dokl. Akad. Nauk SSSR 67 (1949), 783–786. P. Mattila, Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability, Cambridge Stud. Adv. Math. 44, Cambridge Univ. Press, Cambridge, 1995. s -dense sequences, Mathematika 39 B. P. Rynne, Regular and ubiquitous systems, and M∞ (1992), 234–243. W. M. Schmidt, Metrische Sätze über simultane Approximation abhängiger Grössen, Monatsh. Math. 68 (1964), 154–166. V. G. Sprindžuk, Mahler’s Problem in Metric Number Theory, Transl. Math. Monogr. 25, Amer. Math. Soc., Providence, 1969. , Metric Theory of Diophantine Approximations, Scripta Ser. Math., Wiley, New York, 1979. , Achievements and problems of the theory of Diophantine approximations (in Russian), Uspekhi Mat. Nauk 35 (1980), 3–68, 248; English translation in Russian Math. Surveys 35 (1980), 1–80. A. I. Vinogradov and G. V. Chudnovsky, “The proof of extremality of certain manifolds” in Contributions to the Theory of Transcendental Numbers, Math. Surveys Monogr. 19, Amer. Math. Soc., Providence, 1984, 421–447.

Department of Mathematics, University of York, York YO1 5DD, England; hd3@ york.ac. uk; [email protected]

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

CONFORMAL GEOMETRY, CONTACT GEOMETRY, AND THE CALCULUS OF VARIATIONS JEFF A. VIACLOVSKY 1. Introduction. In the following we let (N, g0 ) denote a compact, connected smooth Riemannian manifold of dimension n ≥ 3. We denote the Ricci tensor and scalar curvature by Ric and R, respectively. In this paper we examine the nonlinear curvature equations   Rg · g = constant (1) σk Ricg − 2(n − 1) for metrics g in the conformal class of g0 , where we use the metric g to view the tensor as an endomorphism of the tangent bundle and where σk denotes the trace of the induced map on the kth exterior power; that is, σk is the kth elementary symmetric function of the eigenvalues. The case k = 1, R = constant is known as the Yamabe problem, and it has been studied in great depth (see [11] and [17]). We let M1 denote the set of unit volume metrics in the conformal class [g0 ]. We show that these equations have the following variational properties. Theorem 1. If k 6 = n/2 and (N, [g0 ]) is locally conformally flat, then a metric g ∈ M1 is a critical point of the functional   Z Rg Fk : g 7 → σk Ricg − · g dvolg 2(n − 1) N restricted to M1 if and only if

 σk Ricg −

 Rg · g = Ck 2(n − 1)

for some constant Ck . If N is not locally conformally flat, then the statement is true for k = 1 and k = 2. We compute the second variation of the above functionals and use this to examine the behavior of the functionals near a critical point. In particular, we show that they are elliptic when the eigenvalues are restricted to lie in a certain cone (see Section 6). Following [4], we call such a solution admissible. We prove Theorem 2 (k 6= n/2). Received 2 April 1999. 1991 Mathematics Subject Classification. Primary 53A30; Secondary 35J20, 35J60. Author’s work supported by a National Science Foundation Graduate Fellowship and a Sloan Dissertation Fellowship. 283

284

JEFF A. VIACLOVSKY

Theorem 2. If (N, [g0 ]) is locally conformally flat, then for odd (even) k, a negative k-admissible critical point of the functional Fk |M1 is a strict local minimum (maximum). For all k, a positive scalar curvature Einstein metric is a strict local minimum unless (N, [g]) is conformally equivalent to (S n , g0 ), in which case there is an (n + 1)-parameter family of local minima. If (N, [g0 ]) is not locally conformally flat, then the statement holds for k = 1 and k = 2. For k = 1, it is known that the local extrema in the above theorem are actually global extrema (see [17]). We conjecture that this is also true for k ≥ 2, provided we restrict the functionals to k-admissible unit volume metrics. We also do an ordinary differential equation (ODE) analysis for rotationally symmetric solutions on S n −{p1 , p2 }, where p1 and p2 are antipodal points. For k < n/2, we find periodic orbits, thus giving solutions which descend to S 1 ×S n−1 . For k = 1, these are known as Delaunay metrics (see [12] and [17]). We also prove the following uniqueness result. Theorem 3. Suppose (N, g0 ) is of unit volume and has constant sectional curvature K 6 = 0. Then for any k ∈ {1, . . . , n − 1}, g0 is the unique unit volume solution in its conformal class of (1) unless N is isometric to S n with the standard metric. In this case, we have an (n+1)-parameter family of solutions that are the images of the standard metric under the conformal diffeomorphisms of S n . For k = 1, the constant scalar curvature case, the theorem holds just assuming N is Einstein. This is a well-known theorem of Obata (see [14]). To prove this, we use the conformal frame bundle of Cartan to show that the manifold of 1-jets of sections of a density bundle is naturally isomorphic to a quotient of the total space of the principal bundle by the subgroup SO(n). The connection forms then descend to give nonlinear second-order equations on densities. These densities are just metrics in the conformal class, so we arrive at the above nonlinear curvature equations. Using the connection forms, we then derive an integral formula, which is the main part of the proof, and the theorem follows if k < n. The negative curvature case can be handled by a maximum principle argument. We will prove the following theorem. Theorem 4. Suppose (N, g0 ) is a compact Einstein manifold of unit volume with R < 0. Then for k ∈ {1, . . . , n}, g0 is the unique unit volume solution in its conformal class of (1). The only case left is k = n and K > 0. The techniques here do not work in this case, but for a partial differential equation (PDE) proof using the moving planes method (see [18]). The paper is organized as follows. We begin with a review of conformal geometry. In Section 3 we discuss the isomorphism between 1-jets and the quotient manifold. In Section 4 we introduce the curvature equations and discuss their variational

CONFORMAL GEOMETRY

285

properties. Section 5 is concerned with examples. We calculate the second variation in Section 6, examine the behavior of the functionals near a critical point, and show that the equations are elliptic at an admissible solution. Section 7 is devoted to proving the uniqueness theorems. Acknowledgements. This work is part of the author’s doctoral dissertation at Princeton University under the guidance of Phillip A. Griffiths, without whom none of this would have been possible. The author would also like to thank Robert Bryant, Jonathan Pakianathan, Dan Pollack, and Karen Uhlenbeck for their interest, time, and many helpful suggestions. 2. Review of conformal geometry. In this section we review the relevant notions from conformal geometry (see [5] and [10]). We use the Einstein summation convention. 2.1. Cartan’s principal conformal frame bundle. We begin with some definitions. Let Q denote the matrix 

0 0  0 In −1 0

 −1 0 . 0

We view SO(n + 1, 1) ⊂ GL(n + 2, R) as the subgroup of matrices A satisfying At QA = Q and det(A) = 1. We let SOo (n+1, 1) denote the subgroup of SO(n+1, 1) preserving time orientation, that is, A00 + An+1 > 0. 0 We let G denote the maximal parabolic subgroup of SOo (n + 1, 1), consisting of elements of the form 

r2 0 0

r 2 vt B B 0

 (r 2 /2)v t v , v r −2

where B is in SO(n), v ∈ Rn , and r 2 ∈ R+ . We denote by go the Lie algebra of SOo (n + 1, 1), consisting of matrices of the form 

2s x 0

yt z xt

 0 y , −2s

where s ∈ R, y and x are in Rn , and z is an n-by-n skew-symmetric matrix. Cartan [5] shows that given an oriented manifold N with dim(N ) ≥ 3 and an equivalence class [g] of conformal Riemannian metrics on N, there exists a right principal bundle P → N with structure group G. Moreover, there is a go -valued

286

JEFF A. VIACLOVSKY

1-form on P ,



2ρ ψ =  ωi 0

 0 βi  , −2ρ

βj αji ωj

with the following properties: (i) the forms ωi , αji (i > j ), βi , and ρ are linearly independent; (ii) the ωi are semibasic, that is, v y ωi = 0 for vertical vectors v; (iii) π ∗ [g] = [ω12 + · · · + ωn2 ]; (iv) Rg∗ (ψ) = g −1 ψg for all g ∈ G; (v) restricted to a fiber, ψ is the left-invariant Maurer-Cartan form of G. This bundle can be considered a solution to the equivalence problem for conformal structures in the sense that any conformal automorphism of N lifts to a unique automorphism of P , which preserves ψ, and vice versa. Furthermore, there exist real-valued functions Wjikl (the Weyl curvature) and Bij k (the Cotten tensor) on P with the following symmetries: j

Wjikl = −Wikl = −Wjilk , 0 = Wjiil , i 0 = Wjikl + Wklj + Wlji k ,

Bij k = −Bikj , 0 = Bij k + Bj ki + Bkij . We have the structure equations d(2ρ) = −βi ∧ ωi ,

(2)

i

(3)

dω = 2ρ ∧ ω

i

− αji ∧ ωj ,

1 j dβi = −2ρ ∧ βi + αi ∧ βj + Bij k ωj ∧ ωk , 2 1 i i k j dαj = −αk ∧ αj − βi ∧ ω + βj ∧ ωi + Wjikl ωk ∧ ωl . 2

(4) (5)

A necessary and sufficient condition for the conformal structure to be locally conformally flat is the vanishing of the Wjikl and the Bij k . For n > 3, if the Wjikl ≡ 0, then the Bij k are also identically zero. So, in this case the Wjikl are the only obstruction to local conformal flatness. To see this, we differentiate the structure equation (5), apply Cartan’s lemma, and find that i i + Wmkl αjm + Wjiml αkm + Wjikm αlm + Wjikl,m ωm dWjikl = −4Wjikl ρ − Wjmkl αm

for some functions Wjikl,m satisfying

CONFORMAL GEOMETRY

Wjikl,m + Wjilm,k + Wjimk,l = − Bj kl δim + Bikl δj m − Bj lm δik + Bilm δj k − Bj mk δil + Bimk δj l ,

287 (6)

where δij is the Kronecker delta symbol. Tracing (6) on i and m, we find that Wjikl,i = (3 − n)Bj kl − Biil δj k + Biik δj l .

(7)

Tracing (7) on j and k, we have 0 = (3 − n)Biil − nBiil + Biil = 2(2 − n)Biil . Therefore we have the conformal Bianchi identity Biik ≡ 0.

(8)

We see that if the Wjikl vanish and n > 3, then Bij k ≡ 0. For n = 3, the symmetry relations imply that Wjikl ≡ 0, so in this case the Bij k are the only obstruction to local conformal flatness. 2.2. Density bundles and metrics. There are some natural bundles that we can associate to the principal bundle π : P → N. Let h : G → R be the homomorphism sending g ∈ G to r 2 . We denote by D s/n the line bundle on N associated to P by the homomorphism hs ; that is, D s/n = P × R/ ∼, where ∼ is the equivalence relation (p, t) ∼ (Rg p, r −2s t). A section of the bundle D s/n is represented by a function u : P → R satisfying the condition ¡
−s u(p · g) = h(p) u(p). Differentiating this relation, using the structure equations, and using Cartan’s lemma, we see that there exist ui : P → R so that u satisfies the relation du = −2sρu + ui ωi .

(9)

Differentiating again, we see that there exist uij = uj i such that j

dui = −suβi − 2(s + 1)ui ρ + uj αi + uij ωj .

(10)

Differentiating one last time, we obtain ¡
¡
duij = −(s + 1) ui βj + uj βi + δij uk βk − 2(s + 2)uij ρ + uik αjk + ukj αik + uij k ωk , (11) where the uij k satisfy uij k = uj ik and uij k − uikj = −sBij k + Wijl k ul . One of the nicest features of the Cartan approach to conformal geometry is that the bundle P → N contains the Riemannian frame bundle of each of the metrics in

288

JEFF A. VIACLOVSKY

the conformal class [g], and positive sections of the density bundles are, in effect, metrics in the conformal class. We now restrict to the case s = −1. Given a density u ∈ 0(D −1/n ), relations (9) through (11) simplify to become du = 2ρu + ui ωi , j

dui = uβi + uj αi + uij ωj , ¡
duij = δij uk βk − 2uij ρ + uik αjk + ukj αik + uij k ωk .

(12) (13) (14)

If u > 0, we define the subset of P ,  ª Pu = p ∈ P , u(p) = 1, ui (p) = 0 . We let an overbar denote restriction to Pu ⊂ P . From (12), we see that ρ¯ = 0.

(15)

From the structure equation (3), we have that d ω¯ i = −α¯ji ∧ ω¯ j . This tells us that Pu → N is a principal SO(n) bundle. It is easy to verify that the symmetric form on P , ¡
¡
2  2 gu = u−2 ω1 + · · · + ωn , descends to a metric g¯ u ∈ [g], and it follows that Pu → N is the orthonormal frame bundle of the metric g¯ u , the ω¯ i are the canonical 1-forms, and the α¯ji are the LeviCivita connection forms. Equation (13) becomes β¯i = −u¯ ij ω¯ j .

(16)

Using the structure equations (5), we have 1 1 d α¯ji + α¯ ki ∧ α¯jk = −β¯i ∧ ω¯ j + β¯j ∧ ω¯ i + W¯ jikl ω¯ k ∧ ω¯ l = Rij kl ω¯ k ∧ ω¯ l , 2 2

(17)

where the Rij kl are the components of the Riemann curvature tensor of the metric g¯ u . Substituting (16) into this, we get
¡ Rij kl = − δil u¯j k − δj l u¯ ik − δik u¯j l + δj k u¯ il + W¯ jikl . Thus the Ricci curvature of g¯ u is Rij = Rlilj = (n − 2)u¯ ij + δij u¯ ll ,

(18)

289

CONFORMAL GEOMETRY

and the scalar curvature of g¯ u is R = Rll = 2(n − 1)u¯ ll .

(19)

Solving for u¯ ij , we get u¯ ij =

  1 R Rij − δij . n−2 2(n − 1)

(20)

Given another density v ∈ 0(D −1/n ), if we restrict the relations dv = 2ρv + vi ωi , j dvi = vβi + vj αi

+ vij ω

(21) j

(22)

to Pu , then we get d v¯ = v¯i ω¯ i , j

d v¯i = v¯ β¯i + v¯j α¯ i + v¯ij ω¯ j j

= v¯j α¯ i + (v¯ij − v¯ u¯ ij )ω¯ j . We see that v¯ descends to a function on N , and we have the formulas (∇ v) ¯ i = v¯i , ¡

∇ 2 v¯


ij

= v¯ij − v¯ u¯ ij = v¯ij −





v¯ R Rij − δij , n−2 2(n − 1)

(23) (24)

where the gradient and Hessian are taken with respect to the metric g¯ u . Since gv = v −2 ω, we have, still letting overbars denote restriction to Pu , g¯ v = v¯ −2 ω¯ = v¯ −2 g¯ u . We can think of v¯ −2 as the conformal factor taking us from the metric gu to the metric gv . 3. Contact geometry. There are two contact manifolds that arise naturally in this −1/n view of conformal geometry. One is J 1 (N, D+ ), the bundle of 1-jets of positive sections of the density bundle D −1/n , and the other is M = P / SO(n), the quotient of the total space of P by the subgroup SO(n) ⊂ G. In this section, we show that these two manifolds are isomorphic as R+ × Rn -bundles over N and are, in fact, isomorphic as contact manifolds. 3.1. P / SO(n). We look at the set of left cosets G/ SO(n). Given g ∈ G, the coset gSO(n) looks like

290 

r2 0 0

JEFF A. VIACLOVSKY

r 2 vt B B 0

¡






  2 r 2 /2 v t v 1 0 0 r     v 0 SO(n) 0 = 0 0 0 1 r −2 0

 ∗ ∗ B · SO(n) v  . 0 r −2

(25)

From property (v) of the connection, we see immediately that the fibers of P → P / SO(n) = M are given by {ρ, ωi , βi } = 0. From the structure equation (2), 2ρ is semibasic and 2dρ is semibasic; therefore 2ρ descends to M. Clearly 2ρ ∧(2dρ)n 6= 0, so we have that M is a contact manifold with 2ρ as a global contact form. From (25), it follows that G/ SO(n) is parameterized by (r 2 , v) ∈ R+ × Rn . We compute the left action of G on G/ SO(n), ¡
 2   2 2 t ∗ ∗ r s s w B s 2 /2 wt w   0 B · SO(n) v  0 C w −2 0 0 s 0 0 r −2   2 2 ∗ ∗ s r = 0 CB · SO(n) Cv + wr −2  . 0 0 s −2 r −2 Therefore the action of G on G/ SO(n) = R+ × Rn , which we denote by ρM : G → Aut(R+ × Rn ), is given by   ¡
(26) ρM (g) : r 2 , v −→ s 2 r 2 , Cv + r −2 w . Note that M, as the quotient space of P by the subgroup SO(n), is the R+ ×Rn -bundle over N associated to P via the representation ρM ; that is, M = P ×(R+ ×Rn )/ ∼M , where ∼M is the equivalence relation (p, (r 2 , v)) ∼M (Rg p, ρM (g −1 )(r 2 , v)). We write the equivalence class of (p, (r 2 , v)) as [p, (r 2 , v)]M . −1/n

−1/n

3.2. J 1 (N, D+ ). If we take u ∈ 0(D+ ui must satisfy are

), the differential relations that u and

du = 2ρu + ui ωi , j

dui = uβi + uj αi + uij ωj .

(27) (28)

We see that, taken together, (u, ui ) is a section of a bundle over N associated to P by a representation ρJ : G → Aut(R+ × Rn ). Since G is connected, we can read off the representation from (27) and (28); in fact, ¡
Rg∗ (u, ui ) = ρJ g −1 (u, ui ), where ¡
¡
ρJ (g) : r 2 , v −→ s −2 r 2 , Cv − r 2 w .

(29)

CONFORMAL GEOMETRY

291

To see this, given v ∈ g, the Lie algebra of G, we let g(t) be a curve in G, with ∗ (u, u ) = ρ (g(t)−1 )(u, u ) with respect g(0) = I and g(0) ˙ = v. Differentiating Rg(t) i J i to t, we get LV (u, ui ) = −ρJ ∗ (v)(u, ui ), where the left-hand side is the Lie derivative with respect to V , the fundamental vertical vector field on P corresponding to v. From this, (27) and (28) follow. −1/n We now define J 1 (N, D+ ) = P ×(R+ ×Rn )/ ∼J , where ∼J is the equivalence relation (p, (r 2 , v)) ∼J (Rg p, ρJ (g −1 )(r 2 , v)). We write the equivalence class of (p, (r 2 , v)) as [p, (r 2 , v)]J . We define the 1-jet of u, denoted by j 1 (u), as the section −1/n (u, ui ) of J 1 (N, D −1/n ). Note that given u ∈ 0(D+ ), λ a local section of P → U ⊂ N , and q ∈ U , the 1-jet of u is given locally as
  ¡ q −→ λ(q), u ◦ λ(q), ui ◦ λ(q) J . −1/n

Usually J 1 (N, D+ ) is defined as the union over all p ∈ N of equivalence classes −1/n of sections of D −1/n that agree to the first order at p. This way, J 1 (N, D+ ) embeds −1/n naturally into Gn (T D+ ), the Grassmanian of n-planes of the tangent bundle of −1/n (see [7]). This induces a natural contact structure such that graphs of 1-jets D+ −1/n of sections of D+ are Legendre submanifolds. We note that there is a natural isomorphism between these two definitions such that graphs of 1-jets map to graphs −1/n of 1-jets. Therefore, J 1 (N, D+ ) (using our original definition) has a natural contact structure. −1/n

3.3. J 1 (N, D+

) is isomorphic to M. Define F : R+ × Rn → R+ × Rn by

¡
¡¡ F r 2 , v = r −2 , −v .

We have ¡


¡
¡
¡
ρJ (g) ◦ F r 2 , v = ρJ (g) r −2 , −v = s −2 r −2 , −Cv − r −2 w ,
¡
¡
¡ F ◦ ρM (g) (r 2 , v) = F s 2 r 2 , Cv + r −2 w = s −2 r −2 , −Cv − r −2 w .

This shows that the representations ρM and ρJ are isomorphic; therefore F induces −1/n an isomorphism between the bundles J 1 (N, D+ ) and M. −1/n Given u ∈ 0(D+ ), we denote by Fu the map F (j 1 (u)) : N ,→ M. Let πM denote the projection P → M. Throughout the rest of the paper, we use the important fact that if λ is a local section of the bundle Pu = {p ∈ P , u(p) = 1, ui (p) = 0} → N, then πM ◦ λ = Fu . That is, λ is a local lifting of the map Fu . To see this, first note that πM sends p ∈ P to the equivalence class [p, (1, 0)]M . For q ∈ N , we have   πM ◦ λ(q) = λ(q), (1, 0) M ,

292

JEFF A. VIACLOVSKY

 ¡
  Fu (q) = F ◦ j 1 (u)(q) = F λ(q), u ◦ λ(q), ui ◦ λ(q) J    ¡
 = λ(q), F (1, 0) M = λ(q), (1, 0) M . For λ a section of Pu , we thus have from (15) that ¡
∗ j 1 (u)∗ F ∗ ρ = Fu∗ ρ = πM ◦ λ ρ = λ∗ ρ = 0. −1/n

Theorem 5. The map F : J 1 (N, D+ manifolds.

(30)

) → M is an isomorphism of contact

Proof. We already know that the map is a diffeomorphism. To show it is a contact map, we need to verify that F ∗ IM = IJ , where these are the respective contact line ⊥, subbundles of the cotangent bundles. This is equivalent to showing that F∗ (IJ⊥ ) = IM where the notation ⊥ means the rank-2n subbundle of the tangent bundle that is ⊥ for V an annihilated by the contact ideal. From (30), we know that F∗ (V ) ⊂ IM 1 n-plane tangent to graph of a 1-jet; that is, V is of the form j (u)∗ (Tp N). Since n-planes of this form are dense in IJ⊥ , the theorem follows. 4. Calculus of variations. In this section, we discuss how the above curvature equations arise, and we prove Theorem 1. 4.1. M. We first define the n-forms on P X Ek = (−1)k βI ∧ ω[I ] ,

(31)

|I |=k

where I = i1 < · · · < ik is an increasing multi-index of length k. The notation βI is short for βi1 ∧ · · · ∧ βik , and ω[I ] stands for ωI c (I c are the indices not appearing in I ), with sign determined by (no sum on ik ) ωik ∧ ω[i1 ···ik ] = ω[i1 ···ik−1 ] . We extend the definition to all strings of indices by skew-symmetry. For example, ωi ∧ ω[j ] = δji ω1 ∧ · · · ∧ ωn , ωl ∧ ω[ij ] = δjl ω[i] − δil ω[j ] , and so on. We thus have k ωi1 ∧ · · · ∧ ωik ∧ ω[j1 ···jk ] = δji11···i ···jk ,

the generalized Kronecker delta symbol. We have E0 = ω1 ∧ · · · ∧ ωn . We also refer to ω1 ∧ · · · ∧ ωn as ω.

CONFORMAL GEOMETRY

293

Proposition 6. If N is locally conformally flat, then we have the formulas for 0 ≤ k ≤ n, d Ek = (n − 2k)(2ρ) ∧ Ek .

(32)

If N is not locally conformally flat, the above equation still holds for 0 ≤ k ≤ 2. Proof. We first note that the following formulas hold for I = i1 < · · · < ik : dω[I ] = 2(n − k)ρ ∧ ω[I ] + αil1 ω[li2 ···ik ] + αil2 ω[i1 l···ik ] + · · · + αilk ω[i1 i2 ···l] ,

(33)

and in the conformally flat case, since Bij k = 0, d(βI ) = − 2kρ ∧ βI + αil1 βl ∧ βi2 ∧ · · · ∧ βik + αil2 βi1 ∧ βl ∧ · · · ∧ βik + · · · + αilk βi1 ∧ βi2 ∧ · · · ∧ βl .

(34)

Therefore we have for k = 0, . . . , n,   X βI ∧ ω[I ]  = dβI ∧ ω[I ] + (−1)k βI ∧ dω[I ] . (−1)k d Ek = d  |I |=k

Substituting (33) and (34) into this equation, we get   (−1)k d Ek = −2kρ ∧ βI + αil1 βl ∧ · · · ∧ βik + · · · + αilk βi1 ∧ · · · ∧ βl ∧ ω[I ]   + (−1)k βI ∧ 2(n − k)ρ ∧ ω[I ] + αil1 ω[li2 ···ik ] + · · · + αilk ω[i1 i2 ···l] . By reindexing, we see that the alpha terms cancel, and we are left with (32). Now if Bij k 6 = 0, then for k = 0 the statement is obvious since there are no βi terms. For k = 1, we have ¡ ¡

¡
−d E1 = d βi ∧ ω[i] = d βi ∧ ω[i] − βi ∧ dω[i]   1 j = − 2ρ ∧ βi + αi ∧ βj + Bij k ωj ∧ ωk ∧ ω[i] − βi ∧ dω[i] . 2 The Bij k terms vanish since ωj ∧ ωk ∧ ω[i] = 0, so the computation is the same as in the conformally flat case. For k = 2, we have (the 2 is in front because we sum on all i and j ) ¡
¡
2d E2 = d βi ∧ βj ∧ ω[ij ] ¡
= 2d βi ∧ βj ∧ ω[ij ] + βi ∧ βj ∧ dω[ij ]   1 j = 2 − 2ρ ∧ βi + αi ∧ βj + Bikl ωk ∧ ωl ∧ βj ∧ ω[ij ] + βi ∧ βj ∧ dω[ij ] . 2 We see that the only term different from the conformally flat computation is ¡
Bikl ∧ βj ∧ ωk ∧ ωl ∧ ω[ij ] = Biij − Bij i βj ∧ ω = 2Biij βj ∧ ω = 0 by the conformal Bianchi identity (8).

294

JEFF A. VIACLOVSKY

From the proposition, Ek and d Ek are both semibasic for P → M; therefore the

Ek descend to M.

Since M is a contact manifold, we can consider the calculus of variations on M. The general reference for this theory and the following computations is [3]. We consider the functional Z 1 Fk (N ) = Ek (n − 2k) N defined for N a Legendre submanifold of M. When n is even, we do not define Fk for k = n/2. Let H : (−, ) × N → M be a smooth variation of Legendre submanifolds. That is, for each t ∈ (−, ), Ht : N → M given by Ht = H|{t}×N is an immersed Legendre submanifold with image Nt . Then we have Z  Z d (n − 2k)F0 (N) = Ek = L∂/∂t H ∗ Ek dt Nt Nt   Z ∂ ∂ ∗ ∗ = y H d Ek + d y H Ek . ∂t Nt ∂t So by the proposition, we have in the locally conformally flat case (drop the H ∗ notation) Z Z ∂ F0 (N) = y (2ρ ∧ Ek ) = hEk , Nt ∂t Nt where h(t, p) is a smooth function on (−, )×N defined by H ∗ (2ρ) = h dt. We can think of h(0, p) : N → R as the “tangent vector” to the variation since it uniquely determines the normal vector field ∂/∂t. For t small, the function h can be chosen arbitrarily (see [3]), so we have the following theorem (k 6= n/2). Theorem 7. H0 : N → M is a critical point for the functional Fk if and only if H0∗ Ek = 0. This holds for all k in the locally conformally flat case and for 0 ≤ k ≤ 2 in the general case. Note that critical points of Fk restricted to the set of Legendre submanifolds such that n· F0 = 1 correspond to critical points of Fk −Ck F0 for some constant Ck , by the theory of Lagrange multipliers. These are then Legendre submanifolds H0 : N ,→ M such that H0∗ (Ek − Ck E0 ) = 0. −1/n

4.2. J 1 (N, D+ ). Using the isomorphism between the contact manifolds, we −1/n can transfer everything above to the jet space. If we take u ∈ 0(D+ ) and let λ be a local section of Pu on an open set U ∈ N , then we have πM ◦ λ = Fu . Therefore since Ek descends to M, it follows from (16) that ¡
¡
¡ ¡

(35) Fu∗ Ek = Fu∗ (−1)k βI ∧ ω[I ] = λ∗ (−1)k βI ∧ ω[I ] = λ∗ σk u¯ ij ω¯ . Note that Fu∗ E0 is the volume form of the metric gu on N. By Theorem 7, the remark above on Lagrange multipliers, and formula (20), we have proved Theorem 1.

CONFORMAL GEOMETRY

295

4.3. The case k = n/2. Notice that in the R conformally flat case, for k = n/2, Proposition 6 says that d Ek = 0. Therefore N Ek is a constant, independent of the choice of metric in the conformal class. This is no accident. In this case, it turns out that the integrand is a multiple of the Pfaffian of the curvature matrix, so we get the Euler characteristic by the generalized Chern-Gauss-Bonnet theorem. To see this, we argue as follows. Letting denote the Kulkarni-Nomizu product (see [2]), we can decompose the full curvature as   R 1 Ric − g g. Riem = Weyl + n−2 2(n − 1) We recall the definition of the Chern-Gauss-Bonnet integrand. We let {e1 , . . . , en } be a local orthonormal frame and {e1∗ , . . . , en∗ } be the dual frame. We view the curvature tensor as a skew-symmetric matrix of 2-forms ij = Rij kl ek∗ ∧ el∗ . Then we define ¡
Pfaffian(Riem) ≡ Pfaffian ij =

1 δ 1 2···n i i ∧ i3 i4 ∧ · · · ∧ in−1 in , n/2 2 · (n/2)! i1 i2 ···in 1 2

which is an n-form and is the Chern-Gauss-Bonnet integrand. Proposition 8. For a symmetric (0, 2)-tensor C, Pfaffian(C g) = 2n/2 (n/2)! · σn/2 (C) dvol . Proof. Let ij = (C g)ij kl ek∗ ∧ el∗ ¡
= Cik δj l − Cj k δil + Cj l δik − Cil δj k ek∗ ∧ el∗ = −2Cik ej∗ ∧ ek∗ + 2Cj k ei∗ ∧ ek∗ .

We may assume that C is diagonalized, Cij = λi δij , and we have ¡
ij = 2 λi + λj ei∗ ∧ ej∗ . We then have

¡
 ∧ i3 i4 ∧ · · · ∧ in−1 in 2n/2 · (n/2)! · Pfaffian ij = δi11 i2···n 2 ···in i1 i2 ¡
¡
= 2n/2 δi11 i2···n λi1 + λi2 (· · · ) λin−1 + λin ei∗1 ei∗2 · · · ei∗n 2 ···in X¡
¡
= 2n/2 λi1 + λi2 (· · · ) λin−1 + λin dvol Sn

=2

n

X Sn

λi1 λi2 · · · λin/2 dvol

¡
2 = 2n (n/2)! · σn/2 (C) dvol .

296

JEFF A. VIACLOVSKY

Therefore, in the locally conformally flat case, we have   R n/2 g dvol . Pfaffian(Riem) = 2 (n/2)! · σn/2 Ric − 2(n − 1) 4.4. Conformal factors. Fixing a background metric in the conformal class, we want to write out the above functionals and Euler-Lagrange equations with respect to a conformal factor. We let π denote the projection from P to N. Claim 9. The following formula holds on P : π

∗

Fu∗ Ek

P   1 (ul )2 δij ω. = n σk u · uij − u 2

(36)

Proof. In order to prove this, we first need a few lemmas. Lemma 10. On P , we have à  d σk

6(uj )2 δij u · uij − 2

!

≡ 0 mod ωi .

Proof. Let vij = u · uij − (6(uj )2 /2)δij . A computation shows that dvij = 0 + vik αjk + vkj αik + vij k ωk . Modulo ω terms, this is just the differential of conjugation in the fiber by the orthogonal part of a group element. This tells us that under the group action, vij transforms by conjugation of the orthogonal part of a group element, that is, Rg∗ vij = B −1 vij B. Taking σk of both sides, we see that σk (vij ) is a zero-density. Lemma 11. If u is an s/n-density and v is an r/n-density, then uv is an (r +s)/ndensity. Proof. We have d(uv) = v du + u dv ≡ v(−2suρ) + (−2rvρ) ≡ −2(r + s)uvρ mod ωi . Lemma 12. If f ∈ C ∞ (P , R), then f ω = π ∗ α for an n-form α on N if and only if f is a 1-density. Proof. The identity dα = 0 pulls up to (df + 2nfρ) ∧ ω = 0, so that df = −2nfρ + fi ωi . Conversely, given a function f satisfying this density equation, there is a unique n-form α on N such that π ∗ α = f ω.

CONFORMAL GEOMETRY

297

Applying the above lemmas, we see that the expression    6(uj )2 1 − δ σ u · u k ij ij un 2 is an (n + 0)/n = 1-density. Therefore    6(uj )2 1 − δ σ u · u ω = π ∗ (α) k ij ij un 2 with α an n-form on N . To prove the claim, we now need to verify that π ∗ Fu∗ Ek = π ∗ α. If we pull back both sides of this by the section λ, we get equation (35), so we are done. Previously, we required λ to be a local section of Pu ⊂ P . Using the claim, we can now write down the general formula for Fu∗ Ek , when λ is any local section. Pulling back (36) by λ, we have ∗ ∗

λ π

Fu∗ Ek

∗

= (π ◦ λ)

Fu∗ Ek

= Fu∗ Ek

=λ

∗



P    (ul )2 1 δij ω . σk u · uij − un 2

We now fix a u0 ∈ 0(D −1/n ) and denote by g0 the associated metric on N. We can now restate Theorem 1 as an equation on conformal factors. Given g ∈ M1 , we write g = v −2 g0 , where v ∈ C ∞ (N, R+ ). Using equations (23) and (24), we get the following theorem (k 6 = n/2). Theorem 13. If (N, [g0 ]) is locally conformally flat, a metric gw = w−2 g0 is a critical point of the functional Z Fk : v 7 →

N

     v R0 1 |∇v|2 2 σ v + − Ric g g v ∇ − k 0 0 0 dvol g0 vn n−2 2(n − 1) 2 (37)

restricted to M1 if and only if     w2 R0 |∇w|2 Ric0 − g0 − g0 = Ck σk w∇ 2 w + n−2 2(n − 1) 2

(38)

for some constant Ck . Here σk , ∇ 2 w, and ∇w are taken with respect to g0 . If (N, [g0 ]) is not locally conformally flat, the statement is true for k = 1 and k = 2. We summarize the results of this section in Table 1. 5. Examples. The scalar curvature case k = 1 is known as the Yamabe problem and has been completely solved. That is, for any compact Riemannian manifold

298

JEFF A. VIACLOVSKY

Table 1 ¡ −1/n
J 1 N, D+

M Fk (N)

=

(−1)k (n − 2k)

Fk (j 1 (u))

Z N

βI ∧ ω[I ]

=

1 (n − 2k)

N

  1 Ru Ricu − · gu volgu (n − 2) 2(n − 1)



Z σk

Critical point of Fk

Critical point of Fk

restricted to n · F0 = 1

restricted to M1

Critical point of Fk − Ck F0

Critical point of Fk − (Ck /n) Vol   R σk Ric − · g = (n − 2)k Ck 2(n − 1)

(−1)k βI ∧ ω[I ] = Ck ω

(N, g), there is a constant scalar curvature metric in the conformal class of g (see [11] and [17]). For an Einstein metric in [g] represented by a density u, we note that   R 1 R R δij − δij = δij . u¯ ij = n−2 n 2(n − 1) 2n(n − 1) So when viewed as a submanifold of M under the map Fu = F ◦ j 1 (u) : N ,→ M, a metric is Einstein if and only if βi = −(C1 /n)ωi , when restricted to Fu (N ) ⊂ M. Therefore an Einstein metric solves all of the equations Ek = Ck E0 , and then necessarily,  Ck = σk

R δij 2n(n − 1)



 =

R 2n(n − 1)

k   n . k

(39)

For k = 2, we do not require local conformal flatness; therefore, Einstein metrics are critical points as well as products of Einstein manifolds. A constant curvature manifold (space form) is exactly a locally conformally flat Einstein manifold. Therefore for k > 2, space forms are critical points. The product of space form with sectional curvature 1 and a space form with sectional curvature −1 is locally conformally flat and is a critical point. For any space form X, the product N = S 1 × X is locally conformally flat and is a critical point. An important example is (S n , [g0 ]), where g0 is the standard metric. We have the following theorem due to Obata (see [14]).

CONFORMAL GEOMETRY

299

Theorem 14. Any constant scalar curvature metric in the conformal class of g0 is necessarily Einstein and, moreover, is the image of the standard metric under a conformal diffeomorphism of S n . This theorem tells us that on S n we have an (n+1)-parameter family of unit volume Einstein metrics. Therefore we have an (n + 1)-parameter family of critical points of the functionals Fk |M1 (k 6 = n/2). In stereographic coordinates, fixing the flat metric, equation (38) becomes   ∂ 2u |∇u|2 δij = Ck . σk u · i j − ∂x ∂x 2 If we write g = u−2 gflat , then these critical metrics are given by u(x) = a|x|2 + bi x i + c for some constants a, bi , and c. The standard metric is represented by 1 + |x|2 . We now consider symmetric solutions on S 1 × S n−1 . To do this, we pass to the universal cover N = R ×S n−1 , the cylinder. We let gc denote the product metric. The Ricci tensor of N looks like   0 0 0 (n − 2)In−1 and R = (n − 1)(n − 2). Therefore,     R 0 0 (n − 2)/2 0 δij = − Ric − 0 (n − 2)In−1 0 −((n − 2)/2)In−1 2(n − 1)  −(n − 2)/2 0 . = 0 ((n − 2)/2)In−1 

If we assume that the conformal factor u is independent of S n−1 , then the equation reduces to an ODE, and u = u(t), where t is the coordinate on R. We have   00 u 0 2 ∇ (u) = 0 0 and |∇u|2 = (u0 )2 . Substituting this into equation (38), the PDE becomes the ODE  00 ¡ 2
¡ 0 2
 0 uu − u /2 − (u ) /2 ¡¡
¡

(40) = Ck . σk 0 u2 /2 − (u0 )2 /2 In−1 Note that u(t) = cosh(t) is a solution for all k. We briefly describe cylindrical coordinates on S n . Let p1 and p2 denote the north and south poles on S n , and let f : Rn → S n − {p2 } denote inverse stereographic projection. Cylindrical coordinates are then given by f

F : (t, v) −→ et v −→ S n − {p1 , p2 }.

300

JEFF A. VIACLOVSKY

It is easily verified that 4F ∗ g0 = (cosh(t))−2 gc , so cosh(t) represents the spherical metric. We now fix Ck corresponding to this solution. Since O(n−1) is compact, by the principle of symmetric criticality (see [15]), it follows that the ODE above is still variational for the functional restricted to symmetric functions. We observe that the Lagrangian does not depend explicitly on t; therefore, by Noether’s theorem, we have a conserved quantity associated to the time-translation symmetry. For any one-dimensional functional of the form Z


¡ L u, u0 , u00 dt,

then (see [8]) we have the first integral   d L − u0 Lu0 − Lu00 − u00 Lu00 = constant. dt With the Lagrange multiplier, our Lagrangian is (see Table 1)    00 ¡ 2
¡ 0 2
 1 Ck 1 uu − u /2 − (u ) /2 ¡¡ 0
¡

σk . − L= n 0 u2 /2 − (u0 )2 /2 In−1 u (n − 2k) n We then compute the conservation law, and since we are at a solution, we use (40) to solve for u00 and substitute this in to get a first-order Hamiltonian involving only u and u0 . After a tedious computation, we find that the conservation law takes the form
k ¡ 1 − u2 − (u0 )2 = Dk,n un , where Dk,n is a constant parameterizing the solutions. Instead of computing it this way, we can just show directly that this is indeed a conservation law by substitution. This verification may be found in [18]. If we look in the phase plane (u, u0 ), the spherical metric is represented by the hyperbola (cosh(t), sinh(t)). Note that there is a constant solution u0 , corresponding to the cylindrical metric. The orbits in the region to the right of the hyperbola stay inside of this region. In this region, for k < n/2 the un term dominates, and it is not difficult to see that all of the orbits are closed. These solutions, parameterized by Dk,n , orbit around the constant solution. Any one of these periodic solutions descends to give a solution on S 1 (T ) × S n−1 , where T is the radius of S 1 and 2πT is some multiple of the period of the solution. The analysis here is similar to the k = 1 case (Delaunay metrics), so we refer the reader to [17] for details. 6. The second variation. In this section we prove Theorem 2. We begin with some important definitions.

301

CONFORMAL GEOMETRY

Definition 1. Let (λ1 , . . . , λn ) ∈ Rn . We view the elementary symmetric functions as functions on Rn , X λi1 · · · λik , σk (λ1 , . . . , λn ) = i1 <···
and we let 0k+ equal the component of {σk > 0} containing the positive cone. For k even (odd), 0k− equals the component of {σk > (<) 0} containing the negative cone. For a symmetric linear transformation A : V → V , where V is an n-dimensional inner product space, the notation A ∈ 0k± means that the eigenvalues of A lie in the corresponding set. Definition 2. Let A : V → V be as above. For 0 ≤ q ≤ n, the qth Newton transformation associated with A is Tq (A) = σq (A) · I − σq−1 (A) · A + · · · + (−1)q Aq . It is proved in [16] that if Aij are the components of A with respect to some basis of V , then ij

Tq (A) =

1 i1 ···iq i δ Ai j · · · Aiq jq , q! j1 ···jq j 1 1

(41)

i ···i i

where δj11 ···jqq j is the generalized Kronecker delta symbol.

The following proposition describes some important properties of the sets 0k± . +(−)

is an open convex cone with vertex at the origin, Proposition 15. Each set 0k and we have the following sequences of inclusions: +(−)

+(−)

0n+(−) ⊂ 0n−1 ⊂ · · · ⊂ 01

. +(−)

+(−)

Furthermore, for symmetric linear transformations A ∈ 0k , B ∈ 0k , we have +(−) for t ∈ [0, 1]. If A ∈ 0k+ , then Tk−1 (A) is positive definite, and tA+(1−t)B ∈ 0k if k is odd (even) and A ∈ 0k− , then Tk−1 (A) is positive (negative) definite. The proof of this proposition is standard and may be found in [4], [6], and [19]. Definition 3. A metric g is positive k-admissible or negative k-admissible if Ricg −

Rg ·g 2(n − 1)

is everywhere in 0k+ or 0k− , respectively. Note that we are using the metric g to view this tensor as an endomorphism of the tangent space at any point, and we take the eigenvalues of this map. Since the Ricci tensor is symmetric, these eigenvalues are real.

302

JEFF A. VIACLOVSKY

When is a solution g of the equation  σk Ricg −

 Rg · g = Ck 2(n − 1)

admissible? Fixing a metric g0 in the conformal class and writing g = w−2 g0 , we have     Rg w2 R0 |∇w|2 w2 Ricg − g = w∇ 2 w + Ric0 − g0 − g0 . n−2 2(n − 1) n−2 2(n − 1) 2 +(−)

If the fixed metric g0 has Ric0 −(R0 /2(n − 1))g0 ∈ 0k , then looking at a minimum +(−) (maximum), we see, at the point, that Ricg −(Rg /2(n − 1))g ∈ 0k with respect to g, since g and g0 are conformal metrics. Therefore, since the cones are connected, +(−) everywhere. by continuity we have Ricg −(Rg /2(n − 1))g ∈ 0k Returning to the second variation, from the theory of Lagrange multipliers, at a critical point of Fk restricted to C = {N ⊂ M : n · F0 (N ) = 1}, we know the first variation is given by a Lagrange multiplier. Furthermore, we have at the critical point that  ª {F|C }00 = Fk00 − Ck F000 |C . That is, we can compute the second variation at a critical point using the Lagrange multiplier (see [1, page 125]). We let H : (−, ) × (−, ) × N → M be a two-parameter Legendre variation of the Legendre submanifold H(0,0) : N → M, satisfying the constraint n· F0 (H(s,t) ) = 1 ∗ (E − C E ) = 0. That is, we are at a critical point. Since H is and such that H(0,0) k k 0 a Legendre variation, we have H ∗ (2ρ) = h1 ds + h2 dt where h1 , h2 : (−, ) × (−, ) × N → R. We can think of the real-valued functions on N, h1 (0, 0, p) and h2 (0, 0, p) as “tangent vectors” to the two-parameter variation, since these uniquely determine the normal vector fields ∂/∂t and ∂/∂s. In this entire section, we make the following restriction to allowable variations. Since M is a bundle over N, we require that a variation move only in the fiber; that is, it is a variation of sections of the bundle. Note that for such variations, in the following computations all terms of the form ((∂/∂t)y ωi ) are zero, since the ωi are semibasic for P → N and therefore vanish on vertical vectors. Also note that in the following computations, the forms in the integrand are defined on P , but they descend to M. To be technically correct, we should be pulling back by a local section of P → M, but we take this as understood. At a critical point, we have the computation of the second variation (we omit the H ∗ notation and abbreviate N = N(0,0) )

303

CONFORMAL GEOMETRY


¡ (Fk |C )00 (N)(h1 , h2 ) = Fk00 − Ck F000 |C (N )(h1 , h2 ) Ã ÃZ !! Ck d 1 d Ek − E0 = dt ds n N(s,t) (n − 2k) =

d dt

N(0,t)

Z = Z =

N

N

(0,0)

!

ÃZ

h1 (Ek − Ck E0 )

t=0

h1 L∂/∂t (Ek − Ck E0 )  h1

  Z ∂ ∂ y d Ek + d y Ek − nCk h1 h2 ω. ∂t ∂t N

We let Ht = H(0,t) : (−, ) × N → M, and we have that Ht∗ βi = sij Ht∗ ωj + Pi dt, Ht∗ 2ρ

= h2 (0, t) dt

(42) (43)

for some functions Pi and sij . Note that for fixed t, since we are on Legendre submanifolds, d(2ρ) = −βi ∧ ωi restricts to zero. Therefore by Cartan’s lemma, βi = sij ωj with sij = sj i . 6.1. k = 1. In this case, H is a two-parameter variation of Legendre submanifolds ∗ (E − C E ) = 0, that satisfying the constraint n · F0 (H(s,t) ) = 1 and such that H(0,0) 1 1 0 is, βi ∧ ω[i] = −C1 ω. We then have (F1 |C )00 (N)(h1 , h2 )   Z  Z ∂ ∂ h1 − nC1 h1 h2 ω y d E1 + d y E1 = ∂t ∂t N N  Z   Z
∂ ∂ ¡ − nC1 h1 h2 ω. y 2(n − 2)ρ ∧ βi ∧ ω[i] + d y βi ∧ ω[i] = − h1 ∂t ∂t N N The first term in the integral is

¡
(n − 2)h1 h2 βi ∧ ω[i] − 2Pi ρ ∧ ω[i] .

The ρ term vanishes since we are on a Legendre submanifold. For the second term, we have  
¡
∂ ¡ y βi ∧ ω[i] = d Pi ω[i] d ∂t ¡
= dPi ω[i] + Pi 2(n − 1)ρ ∧ ω[i] + αil ω[l]
¡ = dPi − Pl αil ω[i] .

304

JEFF A. VIACLOVSKY

We define the functions Pij by dPi − Pl αil = Pij ωj . Therefore

Z

00

(F1 |C ) (N)(h1 , h2 ) = − =−

ZN N

h1 Pii ω + (n − 2)h1 h2 βi ∧ ω[i] + nC1 h1 h2 ω h1 Pii ω + 2C1 h1 h2 ω.

Now we pull back these computations to the jet space. If we take N to be of the form F (j 1 (u)), then we choose λ to be a section of Pu . Since this λ is a lifting of Fu , we just need to pull back the above computations under λ. We first identify the Pii . We have d(2ρ) = d(h2 dt) = d(h2 ) ∧ dt = −βi ∧ ωi ¡
= − Pi dt ∧ ωi + sij ωj ∧ ωi = Pi ωi ∧ dt, where the sij terms vanish from symmetry. Therefore from Cartan’s lemma we get that dh2 ≡ Pi ωi mod{dt}. So now when we pull back by the section λ, this tells us that Pi = (∇h2 )i ,

(44)

and by the definition of the Pij , we have ¡
Pij = ∇ 2 h2 ij .

(45)

Noting that C1 = (R/2(n − 1)), we then substitute this and (45) into the second variation to get Z
00 ¡ F1 |M1 (N)(h1 , h2 ) = − h1 Pii ω + 2C1 h1 h2 ω N   Z R h2 dvolgu . = − h1 1h2 + n−1 N Therefore the Jacobi operator of F1 |M1 is   R h . J1 (h) = − 1h + n−1

CONFORMAL GEOMETRY

305

Applying the Rayleigh-Ritz characterization of the first eigenvalue of the Laplacian, we have  Z  R F100 |M1 (N)(h, h) = − h 1h + h dvolgu n−1 N Z  R ≥ λ1 (−1) − h2 dvolgu . n−1 N We see immediately that if R ≤ 0, then a critical point is a strict local minimum for F1 |M1 . To analyze the case R > 0, we state the following theorem of Obata [13]. Theorem 16. Let N be a compact Einstein manifold with R > 0. Then λ1 (−1) ≥

R n−1

with equality if and only if N is isometric to S n with the standard metric. Using this theorem, we see that an Einstein metric is a strict local minimum for

F1 |M1 unless N is isometric to S n with the standard metric.

6.2. k = 2. We restrict H to be a two-parameter variation of Legendre submani∗ (E −C E ) = 0, folds satisfying the constraint n· F0 (H(s,t) ) = 1 and such that H(0,0) 2 2 0 that is, βi ∧ βj ∧ ω[ij ] = 2C2 ω. We compute    Z Z ∂ ∂ h1 y d E2 + d y E2 − nC2 h1 h2 ω (F2 |C )00 (N)(h1 , h2 ) = ∂t ∂t N N  Z ∂ 1 h1 y (n − 4)2ρ ∧ βi ∧ βj ∧ ω[ij ] = 2 N ∂t   Z
∂ ¡ +d y βi ∧ βj ∧ ω[ij ] − nC2 h1 h2 ω. ∂t N The 1/2 is in front because we are summing on all i, j . For the first term in the integral, we have ∂ y 2ρ ∧ βi ∧ βj ∧ ω[ij ] = h2 βi ∧ βj ∧ ω[ij ] − 2Pi (2ρ) ∧ βj ∧ ω[ij ] ∂t = h2 βi ∧ βj ∧ ω[ij ] since we are on a Legendre submanifold. For the second term we have  
¡
∂ ¡ y βi ∧ βj ∧ ω[ij ] = d 2Pi ∧ βj ∧ ω[ij ] d ∂t = 2(dPi )βj ∧ ω[ij ] + 2Pi dβj ∧ ω[ij ] − 2Pi ∧ βj ∧ dω[ij ] . (46)

306

JEFF A. VIACLOVSKY

From now on, since the ρ terms vanish, we leave them out. The middle term of this is   1 2Pi dβj ∧ ω[ij ] = 2Pi αjl ∧ βl + Bilm ωl ∧ ωm ∧ ω[ij ] 2 ¡
l = 2Pi αj ∧ βl ∧ ω[ij ] + Pi Bj ij − Bjj i ω (47) ¡
l = 2Pi αj ∧ βl ∧ ω[ij ] − 2Pi Bjj i ω = 2Pi αjl ∧ βl ∧ ω[ij ] by the conformal Bianchi identity (8). The last term in (46) is ¡
−2Pi βj ∧ dω[ij ] = −2Pi βj ∧ αil ∧ ω[lj ] + αjl ∧ ω[il] = −2Pl αil ∧ βj ∧ ω[ij ] − 2Pi αjl ∧ βl ∧ ω[ij ] .

(48)

The last term here cancels out with the term in (47), so we have Z Z
00 ¡ F2 |C (N)(h1 , h2 ) = −4 h1 h2 C2 ω + h1 Pil ωl ∧ βj ∧ ω[ij ] . N

N

Pulling back to the jet space, we have β¯j = −u¯j m ω¯ m , and using Definition 2, we get Z Z
00 ¡
¡ F2 |M1 (h1 , h2 ) = −4 h1 h2 C2 dvolgu − h1 ∇ 2 h2 il T1il (u¯ ∗∗ ) dvolgu . N

N

Therefore the Jacobi operator of F2 |M1 is ¡

ij




J2 (h) = −T1 (u¯ ∗∗ ) ∇ 2 h ij − 4C2 h.

Note that this is just the expression in a local orthonormal basis of a globally defined operator. Using Proposition 15, we see that if we are at a two-admissible critical metric, then the Jacobi operator is elliptic. The Jacobi operator is just the linearization of the Euler-Lagrange equations at a solution, so it follows that the Euler-Lagrange equations are elliptic at a two-admissible solution. It is easy to see from the derivation that the Jacobi operator must be of divergence ij form, that is, T1 (u¯ ∗∗ ),j = 0, so we see immediately from the second variation that a negative two-admissible solution is a strict local maximum for F2 |M1 . For the positive curvature case, assume we are at an Einstein metric. Then we have that C2 = (R 2 /8n(n − 1)), u¯ il = (R/2n(n − 1))δil , and R R δi l δ i1 i = δil . 2n(n − 1) 1 1 l1 l 2n

T1il (u¯ ∗∗ ) = δli11li u¯ i1 l1 = The second variation becomes ¡

F2 |M1


00

R (h1 , h2 ) = − 2n



Z N

h1

 R h2 dvolgu . 1h2 + (n − 1)

307

CONFORMAL GEOMETRY

Up to the factor (R/2n), we see that the linearization is exactly the same as that for the k = 1 case. Using Obata’s theorem, as in the section above, we conclude that a positive scalar curvature Einstein metric is a strict local minimum for F2 |M1 , unless N is isometric to S n with the standard metric. 6.3. k > 2. In this case, we require N to be locally conformally flat. A computation similar to that of the k = 2 case shows that at a critical point we have Z Z
00 ¡ (−1)k Fk |C (h1 , h2 ) = −2k h1 h2 Ck ω + h1 Pi1 l ωl βi2 · · · βik ω[i1 ···ik ] . (k − 1)! N N Pulling back under λ a section of Pu , we get Z Z ¡
ij = −2k h1 h2 Ck ω − h1 ∇ 2 h2 ij Tk−1 (u¯ ∗∗ ) dvolgu . N

N

Therefore the Jacobi operator of Fk |M1 is ¡

ij




Jk (h) = −Tk−1 (u¯ ∗∗ ) ∇ 2 h ij − 2kCk h.

We note that this is just the expression in a local orthonormal basis of a globally defined operator. From Proposition 15, we see that the equations are elliptic at a kadmissible solution. Again we see from the derivation that the Jacobi operator must be ij of divergence form, that is, Tk−1 (u¯ ∗∗ ),j = 0, so we see immediately from the second variation that for k odd, a negative k-admissible critical point is a strict local minimum for Fk |M1 . For k even, a negative k-admissible critical point is a strict local maximum, just as in the k = 1 and k = 2 cases. At a positive scalar curvature Einstein metric (which is a constant curvature metric since we are assuming N is locally conformally flat), we have βi = −(C1 /n)ωi . Therefore we have ¡

Fk |M1


00



C1 (h1 , h2 ) = − n

k−1 

Z

n−1 k −1

N

 h1

 R h2 dvolgu . 1h2 + n−1

Up to a positive factor, we see the linearization is exactly the same as for the case k = 1. Therefore we conclude that a positive constant sectional curvature metric is a strict local minimum for Fk |M1 , unless N is isometric to S n with the standard metric. 7. Uniqueness. In this section we prove Theorems 3 and 4. −1/n

7.1. Einstein metrics. Given a density v ∈ 0(D+

), we recall the formula (14):

dvij = (vl βl )δij − 2vij ρ + vil αjl + vlj αil + vij l ωl . ◦

◦

(49)

We let v ij denote the traceless vij , that is, v ij = vij − (vll /n)δij . We then have the following proposition.

308

JEFF A. VIACLOVSKY

Proposition 17. We have ¡◦
◦ ◦ ◦ d v ij = −2v ij ρ + v il αjl + v lj αil mod ωi .

(50)

Proof. Since the statement is modulo the ωi , we ignore these terms in the following computations. We trace formula (49) to get d(vii ) = n(vl βl ) − 2vii ρ + vil αil + vli αil = n(vl βl ) − 2vii ρ, since the vil are symmetric and the αil are skew. We thus have   1 ◦ dvll δij d v ij = dvij − n  
¡ 2 vll ρδij = vl βl δij − 2vij ρ + vil αjl + vlj αil − vl βl δij + n v ¡ ◦ kk j
αji + αi = −2v ij ρ + vil αjl + vlj αil − v ¡ n
v ¡
◦ kk kk l δil αjl + vlj αil − δj l αil = −2v ij ρ + vil αj − n n ◦

◦

◦

= −2v ij ρ + v il αjl + v lj αil . ◦

This proposition tells us that v ij scales and conjugates when we move in the fiber; that is, it is a section of a vector bundle. If we restrict to  ª Pv = p ∈ P , v(p) = 1, vi (p) = 0 , then we have ◦

 v¯ll δij n   R 1 R Rij − δij − δij = n−2 2(n − 1) 2n(n − 1)     1 R 1 ◦ = Rij . Rij − δij = n−2 n n−2 

v ij = v¯ij −

◦

So we have the fact that v ij restricts to be the traceless Ricci tensor of the metric gv . ◦ If gv should happen to be Einstein, then the traceless Ricci vanishes. Since v ij just scales and conjugates in the fiber of P → N , we have the following lemma. −1/n

Lemma 18. Given a density v ∈ 0(D+ on P .

◦

), if gv is an Einstein metric, then v ij ≡ 0

For the rest of this section we assume that (N, [g]) has an Einstein metric in the −1/n conformal class [g], and we now fix v ∈ 0(D+ ) to be an Einstein metric.

CONFORMAL GEOMETRY

309

7.2. k = 1. We state the following result of Obata (see [14]). Theorem 19. Let (N, g) be a compact Einstein manifold. Then any constant scalar curvature metric in the conformal class of g is also Einstein. Moreover, g is the unique constant scalar curvature metric in its conformal class (up to scaling by a constant) unless (N, g) is isometric to (S n , g0 ). If there are two distinct unit volume Einstein metrics in the conformal class, then Obata constructs an isometry to (S n , g0 ) to prove the second statement. We reprove the first statement to illustrate our methods. We assume N is oriented, noting that the nonorientable case follows by passing to the orientable double cover. We have the following proposition. Proposition 20. On a Legendre submanifold of M, we have
¡ n−1 vll βi ∧ ω[i] d vi βj ∧ ω[ij ] = vβi ∧ βj ∧ ω[ij ] + n n−1 vll E1 . = 2v E2 − n Proof. We omit ρ terms because we are restricting to a Legendre submanifold. We have
¡ (51) d vi βj ∧ ω[ij ] = dvi βj ∧ ω[ij ] + vi dβj ∧ ω[ij ] − vi βj ∧ dω[ij ] . Using Lemma 18, the first term on the right is ¡
dvi βj ∧ ω[ij ] = vβi + vl αil + vil ωl βj ∧ ω[ij ]   ¡
1 l vkk δil ωl βj ∧ ω[ij ] = vβi + vl αi ∧ βj ∧ ω[ij ] + n   ¡
1 vkk βj ωi ∧ ω[ij ] = vβi + vl αil ∧ βj ∧ ω[ij ] − n ¡
n−1 = vβi + vl αil ∧ βj ∧ ω[ij ] + vkk βj ∧ ω[j ] . n The middle term in (51) is   1 vi dβj ∧ ω[ij ] = vi αjl ∧ βl + Bj kl ωk ∧ ωl ∧ ω[ij ] 2 ¡
= vi αjl ∧ βl ∧ ω[ij ] − vi Bjj i ω ¡
= vi αjl ∧ βl ∧ ω[ij ] by the conformal Bianchi identity (8). The last term in (51) is ¡
−vi βj ∧ dω[ij ] = −vi βj ∧ αil ∧ ω[lj ] + αjl ∧ ω[il] .

310

JEFF A. VIACLOVSKY

Putting together all of the alpha terms, we get (omitting the wedges) vl αil βj ω[ij ] + vi αjl βl ω[ij ] − vi βj αil ω[lj ] − vi βj αjl ω[il] = vl αil βj ω[ij ] + vi αjl βl ω[ij ] − vl αil βj ω[ij ] − vi αjl βl ω[ij ] = 0. −1/n

Now we assume we have u ∈ 0(D+ ) of constant scalar curvature. Then βi ∧ ω[i] = −σ1 ω on F (j 1 (u)), where σ1 is a constant. We now let σk denote the function defined by Ek = σk ω when restricted to F (j 1 (u)), that is, σk = σk (u¯ ij ). We pull back the formula in the proposition to N and integrate to get Z  0=

N

2vσ ¯ 2−

 n−1 v¯ll σ1 dvolgu . n

We have a formula for vll restricted to the set Pu (see (24)), so we get Z 


n−1¡ 1v¯ + v¯ u¯ ii σ1 dvolgu n N   Z
n−1¡ = 1v¯ + vσ ¯ 1 σ1 dvolgu 2vσ ¯ 2− n N  Z  Z n−1 2 n−1 σ1 dvolgu − σ1 1v¯ dvolgu = 2 v¯ σ2 − 2n N N n  Z  n−1 2 σ dvolgu . = 2 v¯ σ2 − 2n 1 N

0=

2vσ ¯ 2−

(52)

Lemma 21. If A is a symmetric n × n matrix, then σ2 (A) ≤

n−1 σ1 (A)2 2n

with equality if and only if A = λI . Proof. If we let λi denote the eigenvalues of A, the lemma is a consequence of the factorization σ2 (A) −


2 n−1 1 X¡ λi − λj . σ1 (A)2 = − 2n 2n i<j

Using the lemma and the fact that v¯ is positive, we see that the integrand in (52) is nonpositive. Since the integral is equal to zero, we conclude that the integrand is identically zero, and therefore, by the lemma, the metric gu is Einstein.

CONFORMAL GEOMETRY

311

7.3. k > 1. In this case we assume that (N, [g]) is locally conformally flat. We prove Theorem 3 by showing that any solution is necessarily Einstein. The result then follows from Theorems 14 and 19. Omitting the wedge product notation we have the following. Proposition 22. On a Legendre submanifold of M, we have for k 6= n,
¡ n−k vll βi1 · · · βik ω[i1 ···ik ] d vi1 βi2 · · · βik+1 ω[i1 ···ik+1 ] = vβi1 · · · βik+1 ω[i1 ···ik+1 ] + n   n−k vll Ek . = (−1)k+1 (k + 1)! v Ek+1 − (k + 1)n Proof. We omit ρ terms because we are restricting to a Legendre submanifold. We have
¡ d vi1 βi2 · · · βik+1 ω[i1 ···ik+1 ] = dvi1 βi2 · · · βik+1 ω[i1 ···ik+1 ] + kvi1 dβi2 · · · βik+1 × ω[i1 ···ik+1 ] + (−1)k vi1 βi2 · · · βik+1 dω[i1 ···ik+1 ] .

(53)

Using Lemma (18), the first term on the right is dvi1 βi2 · · · βik+1 ω[i1 ···ik+1 ] ¡
= vβi1 + vl αil1 + vi1 l ωl βi2 · · · βik+1 ω[i1 ···ik+1 ]   ¡
1 l vjj δi1 l ωl βi2 · · · βik+1 ω[i1 ···ik+1 ] = vβi1 + vl αi1 βi2 · · · βik+1 ω[i1 ···ik+1 ] + n   ¡
1 = vβi1 +vl αil1 βi2 · · · βik+1 ω[i1 ···ik+1 ] + (−1)k vjj βi2 · · · βik+1 ωi1 ω[i1 ···ik+1 ] . n Next we note that ωi1 ω[i1 ···ik+1 ] = (−1)k (n − k)ω[i2 ···ik+1 ] , so we have dvi1 βi2 · · · βik+1 ω[i1 ···ik+1 ] = (vβi1 + vl αil1 )βi2 · · · βik+1 ω[i1 ···ik+1 ] n−k + vjj βi2 · · · βik+1 ω[i2 ···ik+1 ] . n The middle term in (53) is kvi1 dβi2 · · · βik+1 ω[i1 ···ik+1 ] = kvi1 αil2 βl βi3 · · · βik+1 ω[i1 ···ik+1 ] .

312

JEFF A. VIACLOVSKY

The last term in (53) is (−1)k vi1 βi2 · · · βik+1 dω[i1 ···ik+1 ]

¡
= (−1)k vi1 βi2 · · · βik+1 αil1 ω[l···ik+1 ] + · · · + αilk+1 ω[i1 ···l]

= (−1)k vl βi2 · · · βik+1 αli1 ω[i1 ···ik+1 ] + (−1)k vi1 βi2 ¡
· · · βik+1 αil2 ω[i1 l···ik+1 ] + · · · + αilk+1 ω[i1 ···l] = vl αli1 βi2 · · · βik+1 ω[i1 ···ik+1 ] + (−1)k vi1 βl · · · βik+1 αli2 ω[i1 i2 ···ik+1 ] i

+ · · · + (−1)k vi1 βi2 · · · βl αl k+1 ω[i1 i2 ···ik+1 ] = vl αli1 βi2 · · · βik+1 ω[i1 ···ik+1 ] + vi1 αli2 βl · · · βik+1 ω[i1 i2 ···ik+1 ] i

+ · · · + vi1 αl k+1 βi2 · · · βl ω[i1 i2 ···ik+1 ] = −vl αil1 βi2 · · · βik+1 ω[i1 ···ik+1 ] − kvi1 αil2 βl · · · βik+1 ω[i1 i2 ···ik+1 ] . We see that all of the alpha terms cancel, so we are left with
¡ n−k vll βi1 · · · βik ω[i1 ···ik ] d vi1 βi2 · · · βik+1 ω[i1 ···ik+1 ] = vβi1 · · · βik+1 ω[i1 ···ik+1 ] + n   n−k = (−1)k+1 (k + 1)! v Ek+1 − vll Ek . (k + 1)n −1/n

Now we assume we have u ∈ 0(D+ ) with σk = σk (u¯ ij ) constant. We pull back the formula in the proposition to N and integrate to get  Z  n−k k+1 v¯ll σk dvolgu . vσ ¯ k+1 − 0 = (−1) (k + 1)! (k + 1)n N We have a formula for vll restricted to the set Pu (see (24)), so we get  Z 
n−k ¡ 1v¯ + v¯ u¯ ii σk dvolgu vσ ¯ k+1 − 0= (k + 1)n N  Z 
n−k ¡ 1v¯ + vσ ¯ 1 σk dvolgu = vσ ¯ k+1 − (k + 1)n N  Z Z  n−k n−k σk σ1 dvolgu − = v¯ σk+1 − σk 1v¯ dvolgu (k + 1)n (k + 1)n N N  Z  n−k = v¯ σk+1 − σk σ1 dvolgu . (k + 1)n N Lemma 23. Let A be a symmetric n × n matrix. If A ∈ 0k+ , then σk+1 (A) ≤

n−k σk (A)σ1 (A) (k + 1)n

(54)

CONFORMAL GEOMETRY

313

with equality if and only if A = λI . If A ∈ 0k− , then for even (odd) k σk+1 (A) ≥ (≤)

n−k σk (A)σ1 (A) (k + 1)n

with equality if and only if A = λI . Proof. We let

 −1 n · σk (A). k The inequalities we want are then pk+1 ≤ pk p1 . For any n × n symmetric matrix, Newton’s inequalities (see [9]) state that pk =

pk−1 pk+1 ≤ pk2 . The proof proceeds by induction. The case k = 1 follows from Lemma 21. Assume the statement is true up to k −1. If A ∈ 0k+ , then from Proposition 15, we have pj > 0 for all j ≤ k. From Newton’s inequalities and since pk > 0, we use the inductive hypothesis to get pk−1 pk+1 ≤ pk pk ≤ pk p1 pk−1 . Since pk−1 > 0, we divide and arrive at the desired inequality. Assume pk+1 = p1 pk . Then from Newton’s inequalities we have pk−1 p1 pk = pk−1 pk+1 ≤ pk2 . Since pk > 0, we have pk−1 p1 ≤ pk . But by induction, pk ≤ p1 pk−1 ; therefore pk = p1 pk−1 . Again by induction we conclude that A = λI . The negative case follows easily. From the discussion after Definition 3, we have that gu is k-admissible, so we can apply Lemma 23. Since v¯ is positive, it follows that the integrand in (54) is always nonnegative or always nonpositive. Since the integral is equal to zero, we conclude that the integrand is identically zero, and therefore, by the lemma, the metric gu is Einstein. 7.4. Einstein manifolds with R < 0. We now prove Theorem 4. In the following we let Ck denote the constant corresponding to the solution g = w−2 g0 . Since g0 is an Einstein metric, writing out equation (1) with respect to a conformal factor, we get (see equation (38))     |∇w|2 R0 2 2 g0 − g0 . Ck = σk w∇ w + w 2n(n − 1) 2 Since N is compact, let p be a point where w is a maximum. We then have     R0 2 2 g0 = Ck . σk w(p)∇ w(p) + w(p) 2n(n − 1) To expand this, we need the following lemma.

(55)

314

JEFF A. VIACLOVSKY

Lemma 24. For any symmetric n × n matrix B and real number a, X ¡ ¡
¡

a k + a k−1 σ1 λj1 , . . . , λjk + · · · + σk λj1 , . . . , λjk , σk (aI + B) = j1 <···<jk

where I is the identity matrix and λi are the eigenvalues of B. Proof. We have ¡
σk (aI + B) = σk a + λ1 , . . . , a + λn = =

X ¡

k

a +a

k−1

¡

X ¡


¡
a + λj1 · · · a + λjk

j1 <···<jk


¡

σ1 λj1 , . . . , λjk + · · · + σk λj1 , . . . , λjk .

j1 <···<jk

This lemma implies that σk (aI + B) =

k X

cn,k,i σk−i (B)a i

i=0

for some positive combinatorial coefficients cn,k,i . We note that cn,k,k (the coefficient ¡n
k of a ) is equal to k . Equation (55) becomes     C1 (g0 ) g0 σk w(p)∇ 2 w(p) + w(p)2 n   k−1 X
C1 (g0 ) i ¡ w(p)k+i cn,k,i σk−i ∇ 2 w(p) + w(p)2k Ck (g0 ) = Ck . (56) = n i=0

7.4.1. k = 1. At the point p, since it is a maximum for w, we have that ∇ 2 w is negative semidefinite, so (56) becomes w(p)1w(p) + w(p)2 C1 (g0 ) = nonpositive term + w(p)2 C1 (g0 ) = C1 . Therefore w(p)2 C1 (g0 ) ≥ C1 .

(57)

Since C1 (g0 ) = (R0 /2(n − 1)) < 0, this implies that C1 < 0, and we have 1/2  C1 . w(p) ≤ C1 (g0 ) Since we are considering metrics of unit volume, we must have C1 ≤ C1 (g0 )

(58)

with equality if and only if w ≡ 1. The following proposition, which follows easily from Hölder’s inequality, implies that C1 = C1 (g0 ).

CONFORMAL GEOMETRY

315

Proposition 25. If (N, g0 ) is a compact Riemannian manifold with constant scalar curvature R0 ≤ 0, then R0 is a global minimum for the total scalar curvature functional restricted to M1 . 7.4.2. k = 2. In this case, at the maximum point, (56) becomes  
¡ 2 C1 (g0 ) 2 3 + w(p)4 C2 (g0 ) C2 = w(p) σ2 ∇ w(p) + (n − 1)w(p) 1w(p) n = nonnegative terms + w(p)4 C2 (g0 ). Therefore w(p)4 C2 (g0 ) ≤ C2 . Since C2 (g0 ) = (R02 /8n(n − 1)) > 0, this implies that C2 > 0, and we have  w(p) ≤

C2 C2 (g0 )

1/4 .

Again, since we are considering unit volume metrics, we must have C2 ≥ C2 (g0 ) with equality if and only if w ≡ 1. From Lemma 21, we have n−1 n−1 2 σ1 ≥ σ2 = C2 ≥ C2 (g0 ) = C1 (g0 )2 . 2n 2n Since g is negative two-admissible (Definition 3), we have that σ1 is negative by Proposition 15. Therefore σ1 ≤ C1 (g0 ). Proposition 25 then tells us that C2 = C2 (g0 ). 7.4.3. k > 2. These cases proceed exactly as in the k = 1 and k = 2 cases. Using the maximum principle, we can prove for odd k that Ck ≤ Ck (g0 ) and for even k that Ck ≥ Ck (g0 ). Lemma 23 implies the following. Lemma 26. Let A be a symmetric n×n matrix. If A ∈ 0k− , then for even k, pk ≤ p1k and, for odd k, pk ≥ p1k . Since any solution must necessarily be negative k-admissible, the equality of Ck and Ck (g0 ) follows from this lemma and Proposition 25. References [1]

Melvin S. Berger, Nonlinearity and Functional Analysis: Lectures on Nonlinear Problems in Mathematical Analysis, Pure Appl. Math. 74, Academic Press, New York, 1977.

316 [2] [3] [4]

[5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]

[18] [19]

JEFF A. VIACLOVSKY Arthur L. Besse, Einstein Manifolds, Ergeb. Math. Grenzgeb. (3) 10, Springer-Verlag, Berlin, 1987. Robert Bryant, Phillip A. Griffiths, and Lucas Hsu, The Poincaré-Cartan form, Institute for Advanced Study preprint, 1997. L. Caffarelli, L. Nirenberg, and J. Spruck, The Dirichlet problem for nonlinear secondorder elliptic equations, III: Functions of the eigenvalues of the Hessian, Acta Math. 155 (1985), 261–301. Élie Cartan, Les espaces à connexion conforme, Ann. Soc. Polon. Math. 2 (1923), 171–221. ˙ Lars Garding, An inequality for hyperbolic polynomials, J. Math. Mech. 8 (1959), 957–965. Robert B. Gardner, A differential geometric generalization of characteristics, Comm. Pure Appl. Math. 22 (1969), 597–626. I. M. Gelfand and S. V. Fomin, Calculus of Variations, rev. ed., Prentice-Hall, Englewood Cliffs, N.J., 1963. G. H. Hardy, J. E. Littlewood, and G. Pólya, Inequalities, Cambridge Math. Lib., Cambridge Univ. Press, Cambridge, 1988. Shoshichi Kobayashi, Transformation Groups in Differential Geometry, Classics Math., Springer-Verlag, Berlin, 1995. John M. Lee and Thomas H. Parker, The Yamabe problem, Bull. Amer. Math. Soc. (N.S.) 17 (1987), 37–91. Rafe Mazzeo, Daniel Pollack, and Karen Uhlenbeck, Moduli spaces of singular Yamabe metrics, J. Amer. Math. Soc. 9 (1996), 303–344. Morio Obata, Certain conditions for a Riemannian manifold to be isometric with a sphere, J. Math. Soc. Japan 14 (1962), 333–340. , The conjectures on conformal transformations of Riemannian manifolds, J. Differential Geometry 6 (1971/72), 247–258. Richard S. Palais, The principle of symmetric criticality, Comm. Math. Phys. 69 (1979), 19–30. Robert C. Reilly, On the Hessian of a function and the curvatures of its graph, Michigan Math. J. 20 (1973), 373–383. Richard M. Schoen, “Variational theory for the total scalar curvature functional for Riemannian metrics and related topics” in Topics in Calculus of Variations (Montecatini Terme, 1987) Lecture Notes in Math. 1365, Springer-Verlag, Berlin, 1989, 120–154. Jeff A. Viaclovsky, Conformally invariant Monge-Ampère equations: Global solutions, to appear in Trans. Amer. Math. Soc. Xu Jia Wang, A class of fully nonlinear elliptic equations and related functionals, Indiana Univ. Math. J. 43 (1994), 25–54.

Department of Mathematics, Princeton University, Princeton, New Jersey 08544, USA; [email protected] Current: Department of Mathematics, The University of Texas at Austin, Austin, Texas 78712, USA

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

SYMPLECTIC-ORTHOGONAL THETA LIFTS OF GENERIC DISCRETE SERIES GORAN MUIC´ and GORDAN SAVIN 0. Introduction. Let F be a non-Archimedean local field of characteristic zero. In this paper we study a correspondence between representations of symplectic groups Sp(n, F ) and special even-orthogonal split groups SO(2r, F ), where r ≥ 2. Let ωn,r be the Weil representation of Sp(2nr, F ) attached to a nontrivial additive character ψF of F . We show that the correspondence arising by restricting the Weil representation ωn,r to Sp(n, F ) × SO(2r, F ) is functorial for generic square integrable representations. More precisely, let T be a smooth, irreducible representation of Sp(n, F ). Let 2(T, r) be the maximal T-isotypic quotient of ωn,r . The smallest r such that 2(T, r) 6 = 0 is called the first occurrence index of T. Now assume that T is a ψ-generic discrete series. (See (1.1) for the definition of ψ.) Let L(s, T) be the standard Lfunction attached to T as in [Sh1]. Then we have the following results. If L(0, T) = ∞, then the first occurrence index is n. Let τ 0 be an irreducible quotient of 2(T, n). Then τ 0 is a ψ 0 -generic discrete series representation of SO(2n, F ), and for any discrete series representation δ of GL(m, F ) (m arbitrary), we have ¡
L(s, δ × T) = L(s, δ)L s, δ × τ 0 . If L(0, T) 6 = ∞, then the first occurrence index is n + 1. Then 2(T, n + 1) has the unique irreducible ψ 0 -generic quotient τ 0 . Furthermore, τ 0 is a discrete series representation of SO(2n + 2, F ), and for any discrete series representation δ of GL(m, F ) (m arbitrary), we have L(s, δ × τ 0 ) = L(s, δ)L(s, δ × T). We also have analogous results for ψ 0 -generic discrete series of SO(2n, F ). We refer the reader to Section 2 for precise statements. Our results have a conjectural interpretation as follows. Consider inclusions of dual groups SO(2n, C) ⊂ SO(2n + 1, C) ⊂ SO(2n + 2, C). Let W 0 (F ) be the Weil-Deligne group of F . The conjectural Langlands parameter of T is an admissible homomorphism (see [Bo]) ϕ : W 0 (F ) −→ SO(2n + 1, C). Received 22 January 1999. Revised 1 April 1999. 1991 Mathematics Subject Classification. Primary 22E35. Authors’ work partially supported by National Science Foundation grant number DMS-9970689. 317

318

´ AND SAVIN MUIC

Since ϕ is a discrete series parameter, each irreducible summand of ϕ is nondegenerate with respect to the SO(2n + 1, C)-invariant orthogonal form, and appears with multiplicity one. Note that the Artin L-function L(s, ϕ) has a pole at s = 0 if and only if ϕ has a trivial summand. In that case, ϕ = ϕ 0 ⊕ 1, where ϕ 0 is a discrete series parameter ϕ 0 : W 0 (F ) −→ SO(2n, C). Otherwise, ϕ 0 = ϕ ⊕ 1 defines a discrete series parameter ϕ 0 : W 0 (F ) −→ SO(2n + 2, C). Since, conjecturally, L(s, ϕ) = L(s, T), this discussion gives an explanation of our results on the first occurrence index. Next, an irreducible representation φ of W 0 (F ) appears as a direct summand of ϕ if and only if the tensor product Artin L-function L(s, φ ⊗ ϕ) has a pole at s = 0. Let m be the degree of φ, and let δ be the discrete series representation of GL(m, F ) corresponding, again conjecturally, to φ. Then, conjecturally, L(s, φ ⊗ ϕ) = L(s, δ × T). Therefore, our functional equations imply that the irreducible summands of ϕ are the same as the irreducible summands of ϕ 0 , with the exception of the trivial summand, which we have to remove or add to ϕ, depending on whether L(s, T) has a pole at s = 0 or not. In [P], Prasad has given a conjectural description of the correspondence for tempered representations in the case of groups of close rank. Our results confirm his conjecture for generic discrete series representations. Also, for representations with Iwahori-fixed vectors, our results agree with those of Aubert [A]. In fact, we slightly improve her results, since we settle the problem of nonvanishing (the first occurrence index). Our main tools are the theory of L-functions for generic representations developed by Shahidi (see [Sh1]), and the square-integrability criterion for generic representations from [M1] and [M3] (which itself is in terms of the L-functions). Acknowledgements. We would like to thank Jeff Adams for helping us understand his work with Dan Barbasch. 1. Preliminaries. Let F be a non-Archimedean field of characteristic zero. Let q be the number of elements in the residue field of F . We fix a nontrivial additive character ψF of F . Let Z+ , R, and C be the set of nonnegative rational integers, the field of real numbers, and the field of complex numbers, respectively. We consider the standard matrix realization of split classical groups Sp(n, F ) (symplectic group of rank n) and SO(2n, F ) (special even-orthogonal group of rank n). More precisely, if n ≥ 1, then we have the following. Let Jn be the n × n matrix

319

THETA LIFTS

having 1’s on the second diagonal, and all other entries 0. The group Sp(n, F ) is the group of all 2n × 2n F -matrices g that satisfy     0 −Jn 0 −Jn ·g = . gt · Jn 0 Jn 0 We take Sp(0, F ) to be the trivial group. The even-orthogonal group O(2n, F ) is the group of all 2n×2n F -matrices g over F that satisfy g t · J2n · g = J2n . The group SO(2n, F ) consists of all matrices in O(2n, F ) with determinant 1. We take SO(0, F ) = O(0, F ) to be the trivial group. (Here g t denotes the transpose of the matrix g.) Put G = Sp(n, F ) (n ≥ 1) or G = SO(2n, F ) (n ≥ 2), and fix the standard Borel subgroup consisting of upper triangular matrices. This is used to fix a set of standard parabolic subgroups of G. We write Un and Un0 for the maximal unipotent radicals of G = Sp(n, F ) (n ≥ 1) and G = SO(2n, F ) (n ≥ 2), respectively. A character of the standard maximal unipotent radical of G, fixed above, is nondegenerate if it is nontrivial on each root subgroup corresponding to a simple root. Also, the maximal (split) torus acts by conjugaction on the set of all nondegenerate characters. The orbits are parametrized by F × /(F × )2 . More precisely, for each µ ∈ F × , we define the nondegenerate characters ψ = ψµ and ψ 0 = ψµ0 in the following way. If G = Sp(n, F ) (n ≥ 1), then (1.1)

ψ(u) = ψµ (u) = ψF (u12 + u23 + · · · + un−1n + µ · unn+1 ),

u ∈ Un .

If G = SO(2n, F ) (n ≥ 2), then (1.2)

ψ 0 (u) = ψµ0 (u) = ψF (u12 + u23 + · · · + un−1n + µ · un−1n+1 ),

u ∈ Un0 .

Finally, every nondegenerate character Un (resp., Un0 ) is conjugate to some ψµ (resp., ψµ0 ), and ψµ (resp., ψµ0 ) is equivalent to ψµ0 (resp., ψµ0 0 ) if and only if µ0 = a 2 µ, for some a ∈ F × . Next, let χ be either ψ or ψ 0 depending on the type of G. Then an admissible representation (π, Vπ ) of G is χ -generic if there is a nontrivial linear functional lπ : Vπ → C, such that ¡
lπ π(u)v = χ (u)lπ (v), for all v ∈ Vπ , and u in the standard maximal unipotent radical of G. Further, we have the following easy consequence of [Ro] and [Sh1, page 282]. (See also [M1, page 707].) Lemma 1.1. Let P = MN be a standard parabolic subgroup of G. Assume that π is an irreducible representation of M. Then π is χ|M -generic if and only if IndG MN (π ) is χ-generic.

320

´ AND SAVIN MUIC

We need local factors associated to pairs of generic representations δ ⊗ T ∈ Irr(GL(m, F ) × G). (Recall, if G = SO(2n, F ), we are assuming n ≥ 2.) These are defined by Shahidi [Sh1]. (See also [Sh2].) We follow the notation [Sh2]. Let G0 be the group of the same type as G with a standard parabolic subgroup P = MN. ˆ 0 (C) be the dual complex group of G0 . Note that Let M ∼ = GL(m, F ) × G. Let G ( ˆ Sp(n) = SO(2n + 1, C) ˆ SO(2n) = SO(2n, C). ∼ ˆ ˆ Also, we can consider the dual complex group M(C) of M = GL(m, C) × G(C) 0 ˆ (C). Now the adjoint as a Levi factor of a standard parabolic subgroup Pˆ (C) of G ˆ representation r of M(C), on the Lie algebra nˆ (C) of the unipotent radical of Pˆ (C), decomposes as r = r1 ⊕ r2 where

r1 = ρm ⊗ ρ G

and r2 = ∧2 ρm . ˆ Here ρ G is the standard representation of G(C), and ρm is the standard representation of GL(m, C). Shahidi has defined the γ -factor [Sh1, Section 7] ¡
γ (s, δ × T, ψF ) = γ s, δ ⊗ T, ρm ⊗ ρ G , ψF as a rational function in q −s , which satisfies a number of properties [Sh1, Theorem 3.5]. We also have L and -factors. We recall the definition only for tempered representations. The general case can be found in [Sh1]. If δ and T are tempered representations, then L(s, δ × T) = P (q −s )−1 , where P is a polynomial, such that P (q −s ) has the same zeroes as γ (s, δ × T, ψF ), normalized by P (0) = 1. Finally, the -factor is a unit in the ring C[q s , q −s ] and is defined by the following equation:
¡
¡
¡
¡ e L s, δ × T −1 . δ ×T γ s, δ × T, ψF =  s, δ × T, ψF L 1 − s,e Definition 1.1. If m = 1 and δ = 1F × , we write L(s, T) instead of L(s, 1F × × T), and we call this L-function the principal L-function. 2. Main results. In this section we formulate our main results. To explain the results, we need to introduce more notation. The pair (Sp(n, F ), O(2r, F )) is a dual pair in Sp(2nr, F ) (see [MVW], [Ku]). We write ωn,r for the oscillator representation associated to that pair and a fixed additive character ψF of F . (Here ω0,r is the trivial representation of O(2r, F ), and ωn,0 is the trivial representation of Sp(n, F ).) For each T ∈ Irr(Sp(n, F )) and r ≥ 0, write 2(T, r) for a smooth representation of O(2r, F ) such that ωn,r T ⊗ 2(T, r) ∼ = ∩T ker(T )

THETA LIFTS

321

where T runs over HomSp(n,F ) (ωn,r , T) (see [MVW, Lemma 3.4]). Similarly, if T0 ∈ Irr(O(2r, F )), we write 2(T0 , n) for the analogously defined smooth representation of Sp(n, F ), n ≥ 0. It is known from [MVW, Chapitre III] that if r is large enough, then 2(T, r) 6= 0, and the smallest such r we call the first occurrence index of T. We have 2(T, j ) 6= 0, for j ≥ r [MVW, Chapitre III]. The analogous discussion is also valid for T0 and the symplectic tower. Definition 2.1. Let τ 0 ∈ Irr(SO(2r, F )) (r ≥ 1). Then we write τ0 for a representation of SO(2r, F ) obtained from τ 0 conjugating by an element of O(2r, F ) with the determinant −1. Now we have the following simple fact (see [MVW, Chapitre III]). Proposition 2.1. Let τ 0 ∈ Irr(SO(2r, F )) (r ≥ 1). Then we have the following: O(2r,F ) (i) If τ 0 6 ∼ = τ0 , then IndSO(2r,F ) (τ 0 ) is irreducible. We denote this induced representation by T0 . We have T0 ∼ = det ⊗T0 . O(2r,F ) 0 0 ∼ (ii) If τ = τ , then IndSO(2r,F ) (τ 0 ) is reducible, and it is a direct sum of two nonequivalent representations T0 and det ⊗T0 . We can restrict the oscillator representation ωn,r to Sp(n, F ) × SO(2r, F ). If τ 0 ∈ Irr(SO(2r, F )), then we have the following: ( if τ 0 6∼ 2(τ 0 , n) ∼ = 2(T0 , n) = τ0 (2.1) 2(τ 0 , n) ∼ = 2(T0 , n) ⊕ 2(det ⊗T0 , n) if τ 0 ∼ = τ0 . If τ 0 is a ψ 0 -generic representation of SO(2r, F ) (r ≥ 2), it follows directly from the definition of the local factors using the local coefficients [Sh1, Theorem 3.5] that ( L(s, δ × τ0 ) = L(s, δ × τ 0 ) (2.2) (s, δ × τ0 , ψF ) = (s, δ × τ 0 , ψF ). Now we are ready to formulate our main results. Theorem 2.1. Assume that T is a ψ-generic discrete series representation of Sp(n, F ) (n ≥ 1). Then we have the following: (i) If L(s, T) has a pole at s = 0, then the first occurrence index is n. Let τ 0 be an irreducible quotient of 2(T, n). Then τ 0 is a ψ 0 -generic discrete series, and for each discrete series δ of GL(m, F ), we have ( L(s, δ × T) = L(s, δ)L(s, δ × τ 0 ) (s, δ × T, ψF ) = (s, δ, ψF )(s, δ × τ 0 , ψF ). (ii) If L(s, T) does not have a pole at s = 0, then the first occurrence index is n+1. Moreover, 2(T, n+1) has the unique irreducible ψ 0 -generic quotient τ 0 . Then

322

´ AND SAVIN MUIC

τ 0 is a discrete series, and for each discrete series δ of GL(m, F ), we have ( L(s, δ × T)L(s, δ) = L(s, δ × τ 0 ) (s, δ × T, ψF )(s, δ, ψF ) = (s, δ × τ 0 , ψF ). Note that in part (i), 2(T, n) has one or two irreducible quotients, depending on whether τ 0 is isomorphic to τ0 or not. In any case, all irreducible quotients are given by the set {τ 0 , τ0 }. Also, if we assume that the Howe duality principle holds, then τ 0 in part (ii) is the unique quotient of 2(T, n + 1). Finally, note that if n = 1, then all discrete series of Sp(1, F ) = SL(2, F ) have the first occurrence n = 2. Thus (i) never occurs if n = 1. This fact is well known, but it also follows from Theorem 2.1 and the following result of Shahidi. Proposition 2.2. Assume that T is a ψ-generic discrete series of Sp(1, F ) = SL(2, F ). Then the principal L-function L(s, T) does not have a pole for s = 0. Proof. This is an application of the well-known Gelbart-Jacquet lift GL(2) → GL(3). In [Sh1, Section 8, Example 2], Shahidi has computed the L-function L(s, T) if T is supercuspidal. In particular, L(s, T) does not have a pole for s = 0. If T is the Steinberg representation, then its lift is the Steinberg representation StGL(3) of GL(3, F ). In particular, ¡
L(s, T) = L s, StGL(3) . Since L(s, St GL(3) ) = 1/(1 − q −(s+1) ), by [J], the proposition follows. As is usual with the theta correspondence, we have a converse theorem. Theorem 2.2. Assume that τ 0 is a ψ 0 -generic discrete series representation of SO(2n, F ) (n ≥ 2). Then we have the following: (i) If L(s, τ 0 ) has a pole at s = 0, then the first occurrence index of τ 0 is n − 1. Moreover, 2(τ 0 , n−1) has the unique irreducible quotient T. Then T is a ψ-generic discrete series, and for each discrete series δ of GL(m, F ), we have ( L(s, δ × T)L(s, δ) = L(s, δ × τ 0 ) (s, δ × T, ψF )(s, δ, ψF ) = (s, δ × τ 0 , ψF ). (ii) If L(s, τ 0 ) does not have a pole at s = 0, then the first occurrence index of is n. Moreover, 2(τ 0 , n) has a unique ψ-generic irreducible quotient T. Furthermore, T is a discrete series, and for each discrete series δ of GL(m, F ), we have ( L(s, δ × T) = L(s, δ)L(s, δ × τ 0 ) (s, δ × T, ψF )(s, δ, ψF ) = (s, δ × τ 0 , ψF ). τ0

Now assume that the Howe duality principle holds (i.e., true if the residue characteristic of F is odd [W]). If τ 0 6∼ = τ0 , then (2.1) implies that in part (ii) T is the

323

THETA LIFTS

unique quotient of 2(τ 0 , n). Otherwise, 2(τ 0 , n) should have another irreducible, but not ψ-generic, quotient (see [P]). Our methods say nothing about that quotient. Finally, discrete series of SO(2, F ) = F × are unitary characters of F × . Except the trivial character, they first occur for n = 1, and their lifts are tempered nonsquare integrable representations. The trivial character has the first occurrence index n = 0, and its lift to Sp(0, F ) = {1} is the trivial representation. The reason why Theorem 2.2 fails only for n = 1 is that GL(m, F ) × SO(2n, F ) is a Levi factor of a maximal parabolic subgroup of SO(2(n + m), F ) only if n = 0 or n ≥ 2, and thus a characterization of discrete series in terms of L-functions does not hold (see Theorem 3.1). The remainder of the paper is devoted to the proof of Theorems 2.1 and 2.2. 3. A characterization of generic discrete series. The goal of this section is to introduce certain results from [M1] and [M3] needed in the proof of our main results. Assume that π is an irreducible generic representation of G = SO(2n, F ) (n ≥ 2) or G = Sp(n, F ) (n ≥ 1). Choose a standard parabolic subgroup P = MN, with M∼ = GL(m1 , F )×· · ·×GL(mk , F )×G0 (G0 is a classical group of the same type but of a smaller rank), and a (generic) supercuspidal representation ρ = ρ1 ⊗· · ·⊗ρk ⊗σ of M such that π is an subquotient of IndG P (ρ). Each ρi can be uniquely written as ρi = | det |e(ρi ) ρiu , where e(ρi ) ∈ R and ρiu is unitary. Then we say that π satisfies property (∗) if (∗)

ρ eiu ∼ = ρiu

and

2e(ρi ) ∈ Z,

for all i = 1, . . . , k.

Theorem 3.1. The representation π is a discrete series representation if and only if (∗) holds, and for every m and every square integrable representation δ of GL(m, F ), the function γ (s, δ × π, ψF )γ (2s, δ, ψF , ∧2 ρm ) is holomorphic at s = 0, with at most a simple zero. Proof. This was proved in [M1] for G = Sp(n, F ). It is generalized to all quasisplit classical groups in [M3, Theorem 3.1]. The factors γ (s, δ × π, ψF ) can be calculated using supercuspidal support (i.e., parabolic inducing data), as specified in the following proposition that is a mild generalization of a result of Shahidi [Sh2]. Proposition 3.1. Let π be a generic representation. Assume that π is a subquotient of IndG P (ρ) where P and ρ are as above. Let δ be a generic representation of GL(m, F ). Then Y (3.1) ei , ψF )γ (s, δ × ρi , ψF ). γ (s, δ × π, ψF ) = γ (s, δ × σ, ψF ) γ (s, δ × ρ i

(If G0 = SO(0, F ) = {1}, then γ (s, δ × σ, ψF ) = 1, and if G0 = Sp(0, F ) = {1}, then σ = 1, and γ (s, δ × σ, ψF ) is a principal γ -factor for GL(m) [J].) Also, if

´ AND SAVIN MUIC

324 δ ,→ δ1 × · · · × δl , then

γ (s, δ × π, ψF ) =

(3.2)

Y

γ (s, δi × σ, ψF ).

i

Proof. The first formula follows from [Sh2, Corollary 5.6] if π is a subrepresentation of IndG P (ρ). In general, there exists w ∈ NG (M)/M such that π is a subrep0 0 0 resentation of IndG P (w(ρ)) (see [C]). Note that if w(ρ) = ρ1 ⊗ · · · ⊗ ρl ⊗ σ , then 0 the ρi ’s can be obtained from the ρi ’s by a permutation of the factors and taking contragredients of some of them. If G = Sp(2n, F ), then σ ∼ = σ 0 . If G = SO(2n, F ), then σ ∼ = σ or σ (see [G]). Since γ (s, δ × σ, ψF ) = γ (s, δ × σ , ψF ) by (2.2), the first formula follows. Finally, the second formula follows from [Sh2, Corollary 5.6].

4. On the first occurrence. The goal of this section is to obtain lower bounds on the first occurrence index. Recall that if j ≥ 1, Uj is the maximal unipotent radical of Sp(j, F ) and ψ = ψµ is its nondegenerate character. Also, for r ≥ 2, Ur0 is the maximal unipotent radical of SO(2r, F ), and ψ 0 = ψµ0 is its nondegenerate character. Let U10 be the trivial group and let ψ 0 be its trivial character. Then we have the following proposition (see [GRS]). Proposition 4.1. Let c-Ind denote the induction with compact support (see [C]). (i) The ψ-twisted Jacquet Un -module of ωn,r is zero if r < n. If r = n, it is SO(2n,F ) (ψ 0 ) as a Un × SO(2n, F )-module. isomorphic to ψ ⊗ c-IndU 0 n 0 -module of ω 0 (ii) The ψ -twisted Jacquet Un+1 j,n+1 is zero if j < n. If j = n, it is Sp(n,F )

isomorphic to c-IndUn

0 -module. (ψ) ⊗ ψ 0 as an Sp(n, F ) × Un+1

Proof. (i) follows from computations in the proofs of Proposition 2.4 and Corollary 2.5 in [GRS]. (ii) has a completely analogous proof. Corollary 4.1. Assume n ≥ 1. Then we have the following: (i) Assume that T ∈ Irr(Sp(n, F )) is ψ-generic. Then 2(T, r) = 0, for r < n. Moreover, if 2(T, n) 6 = 0, then all irreducible SO(2n, F )-quotients are ψ 0 -generic. (ii) Assume that τ 0 ∈ Irr(SO(2(n + 1), F )) is ψ 0 -generic. Then 2(τ 0 , j ) = 0, for j < n. Moreover, if 2(τ 0 , n) 6 = 0, then all its irreducible quotients are ψ-generic. If τ 0 is a ψ 0 -generic representation of SO(2n, F ) (n ≥ 1), then Proposition 4.1(i) implies that precisely one irreducible subquotient of 2(τ, n) is ψ-generic. In view of (2.1), we now make the following definition. T0

Definition 4.1. Assume that τ 0 is a ψ 0 -generic representation of SO(2n, F ). Let be the unique extension of τ 0 to O(2n, F ) such that 2(T0 , n) is ψ-generic.

Let Qn be the standard parabolic subgroup of Sp(n, F ) with a Levi factor isomorphic to F × × Sp(n − 1, F ), and let Pn be the standard parabolic subgroup of O(2n, F ) with a Levi factor isomorphic to F × × O(2n − 2, F ). Finally, put Pn0 =

325

THETA LIFTS

Pn ∩ SO(2n, F ). Then Pn0 is a standard parabolic subgroup of SO(2n, F ) with a Levi factor isomorphic to F × × SO(2n − 2, F ). Proposition 4.2. Assume that T and τ 0 are discrete series, as in Theorems 2.1 and 2.2. Then we have the following: (i) If L(0, τ 0 ) 6 = ∞, then 2(τ 0 , n − 1) = 0. (ii) If L(0, T) 6 = ∞, then 2(T, n) = 0. Proof. We prove (i). The proof of (ii) is analogous. If 2(τ 0 , n − 1) 6= 0, then all its irreducible quotients are ψ-generic by Corollary 4.1. Let π be one of them. Note that the normalized Jacquet module of ωn−1,n+1 with respect to Pn+1 has a quotient | |−1 ⊗ ωn−1,n (see [Ku]). The Frobenius reciprocity implies that 
 SO(2n+2,F ) ¡ −1 (4.1) HomSp(n−1,F )×SO(2n+2,F ) ωn−1,n+1 , π ⊗ Ind 0 | | ⊗ τ 0 6= 0. Pn+1

Lemma 4.1. The induced representation Ind

SO(2n+2,F ) (| |−1 ⊗ τ 0 ) 0 Pn+1

is irreducible.

Proof. Let e τ 0 be the contragredient representation of τ 0 . Then the contragredient SO(2n+2,F ) (| |−1 ⊗ τ 0 ) is of Ind 0 Pn+1

Ind

SO(2n+2,F ) ¡ | 0 Pn+1


| ⊗e τ0 .

It is a standard module. Next, [M3, Theorem 1.1] implies that a generic standard module is irreducible if and only if the corresponding Langlands quotient is generic. On the other hand, [CSh, Proposition 5.3] implies that the Langlands quotient of SO(2n+2,F ) (| | ⊗ e τ 0 ) is generic if and only if L(1 − s,e τ 0 ) has no pole at s = 1. Ind 0 Pn+1 Since e τ0 ∼ = τ 0 or τ0 , (2.2) implies that the L-function condition is satisfied for τ˜ 0 . In particular,

SO(2n+2,F ) ¡ SO(2n+2,F ) ¡ −1 | | ⊗e τ0 and Ind 0 | | ⊗τ0 Ind 0 Pn+1

Pn+1

are irreducible representations. Now (4.1) and Lemma 4.1 imply that π ∈ Irr(Sp(n−1, F )) is a lift of a ψ 0 -generic irreducible representation of SO(2n + 2, F ). This contradicts Corollary 4.1(ii). 5. Supercuspidal support. Note that by [MVW, Chapitre III], a lift of a supercuspidal representation of Sp(n, F ) or O(2n, F ) is irreducible whenever it is not zero and it is a supercuspidal representation at the first occurrence. The next proposition summarizes some properties of the lifts of supercuspidal representations. Proposition 5.1. Assume that T and τ 0 are supercuspidal representations of Sp(n, F ) (n ≥ 0) and SO(2n, F ) (n ≥ 0, n 6= 1), respectively. Assume that T is

326

´ AND SAVIN MUIC

ψ-generic if n ≥ 1, and τ 0 is ψ 0 -generic if n ≥ 2. Let T0 be fixed as in Definition 4.1. Then we have the following: (i) Either 2(T, n) is an irreducible supercuspidal representation of O(2n, F ), and
O(2n+2,F ) ¡ 1F × ⊗ 2(T, n) , 2(T, n + 1) ,→ IndPn+1 or 2(T, n) = 0, and 2(T, n + 1) is an irreducible supercuspidal representation of O(2n + 2, F ). In any case, 2(T, n + 1) is always nonzero, and its restriction to SO(2n + 2, F ) has all irreducible constituents ψ 0 -generic. The same is true for 2(T, n) if it is nonzero. (ii) Either 2(T0 , n − 1) is an irreducible supercuspidal representation of Sp(n − 1, F ), and
Sp(n,F ) ¡ 1F × ⊗ 2(T0 , n − 1) , 2(T0 , n) ,→ IndQn or 2(T0 , n − 1) = 0, and 2(T0 , n − 1) is an irreducible supercuspidal representation of Sp(n, F ). In any case, 2(T0 , n) is always nonzero and ψ 0 -generic. The same is true for 2(T0 , n − 1) if it is nonzero. Proof. The proposition is a direct consequence of [MVW, Théoreme principal, page 69], combined with Corollary 4.1. Next we consider the general case. Thus let τ 0 ∈ Irr(SO(2n, F )) (n ≥ 2) be ψ 0 generic, and let T0 be given by Definition 4.1. In particular, 2(T0 , n) is not zero, and all its irreducible quotients have the same supercuspidal support (see [Ku]). More precisely, using the notation of [Ku], if [ρ1 , . . . , ρm , σ 0 ] is a supercuspidal support of T0 where σ 0 is a supercuspidal representation of O(2n01 , F ), then by Proposition 4.1 and [Ku, Theorem 2.5], we have the following proposition. Proposition 5.2. Under the above assumptions, we have the following. (i) If 2(σ 0 , n01 ) is supercuspidal, then all irreducible quotients of 2(T0 , n) have the support [ρ1 , . . . , ρm , 2(σ 0 , n01 )]. (ii) If 2(σ 0 , n01 − 1) is nonzero and supercuspidal, then all irreducible quotients of 2(T0 , n) have the support [1F × , ρ1 , . . . , ρm , 2(σ 0 , n01 − 1)]. In either case, the corresponding induced representation contains a unique ψgeneric irreducible subquotient. Write Tψ for that subquotient. Next, let T ∈ Irr(Sp(n, F )) (n ≥ 1) be ψ-generic. Then by Proposition 4.1, 2(T, n + 1) is nontrivial, and all its irreducible quotients have the same supercuspidal support on O(2n + 2, F ). More precisely, if [ρ1 , . . . , ρm , σ ] is a supercuspidal support of T, where σ is a supercuspidal representation of Sp(n1 , F ), then by Proposition 4.1 and [Ku, Corollary 2.6], we have the following proposition. Proposition 5.3. Under the above assumptions, we have the following. (i) If 2(σ, n1 + 1) is supercuspidal, then all irreducible quotients of 2(T, n + 1) have the support [ρ1 , . . . , ρm , 2(σ, n1 + 1)].

THETA LIFTS

327

(ii) If 2(σ, n1 ) is nonzero and supercuspidal, then all irreducible quotients of 2(T, n + 1) have the support [1F × , ρ1 , . . . , ρm , 2(σ, n1 )]. In either case, restricting this support to SO(2n + 2, F ), we can have at most two different irreducible ψ 0 -generic irreducible subquotients: τψ0 0 and τψ0 0 , . The next proposition is the most important result of this section. Proposition 5.4. For any discrete series δ of GL(m, F ) (m ≥ 1 arbitrary), we have (  ª γ (s, δ × T, ψF )γ (s, δ, ψF ) = γ (s, δ × τ0 , ψF ), where τ0 ∈ τψ0 0 , τψ0 0 , γ (s, δ × τ 0 , ψF )γ (s, δ, ψF ) = γ (s, δ × Tψ , ψF ).

Proof. The equations follow from the above-mentioned results of Kudla for supercuspidal supports and the multiplicative properties of γ -factors (see Proposition 3.1) as soon as we prove the following lemma. Lemma 5.1. If T and τ 0 are supercuspidal, then the functional equations in Proposition 5.4 hold. Proof. First, using the multiplicative properties of γ -factors (3.2), we can assume that δ is supercuspidal. We prove the first equation. The proof of the second is analogous. There is no loss of generality in assuming ψ = ψ1 (since γ -factors are invariant under twisting described in [Sh1, page 283]). Let k be an algebraic number field. For each place v of k, let kv denote its completion at v. Let A be the ring of adeles of k. We need the following lemma. Lemma 5.2. For every p-adic field F , there exists a totally imaginary number field k such that kv0 ∼ = F for a finite place v0 . Proof. Let d be a negative integer such that d is a square mod p if p 6= 2 or d ≡ 1 mod 8 if p =√2. Then the prime p splits completely in the imaginary quadratic field K = Q( d). Let p(x) be an irreducible polynomial with coefficients in Qp such that F is isomorphic to Qp [x]/(p) where (p) is the maximal ideal in Qp [x] generated by p(x). We can assume that the coefficients of p(x) are in Q. It follows that k = K[x]/(p) is a totally imaginary field with precisely two finite places v such that kv ∼ = F. Let k and v0 be as in Lemma 5.2. We fix an additive character ψA = ⊗v ψv of k \A, such that ψv0 = ψF , as fixed before. Let χ = ⊗χv be a nondegenerate character of Un (A), trivial on Un (k), defined using ψA , exactly as ψ1 using ψF in (1.1). In particular, we have χv0 = ψ1 . Let 5 = ⊗v 5v be the globally χ-generic automorphic cuspidal representation constructed in [Sh1, Proposition 5.1] of Sp(n, A) such that each 5v is class one with respect to some special maximal compact subgroup, for all finite places v 6= v0 , and 5 v0 ∼ = T.

328

´ AND SAVIN MUIC

Similarly, let 6 = ⊗v 6v be a cuspidal representation of GL(m, A), such that 6 v0 ∼ = δ and 6v is unramified for all finite places v 6= v0 . Let S be a finite set of places, including all infinite places, such that 5v , 6v , and ψv are all unramified if v 6 ∈ S. e in the global As it was proved in [GRS, page 110], if the first occurrence of 5 theta correspondence is n + 1, then the corresponding space of automorphic forms is cuspidal and globally χ 0 -generic when restricted to the corresponding special orthogonal group. (The character χ 0 is defined using ψA , exactly as ψ10 using ψF in (1.2).) Take one irreducible constituent 50 = ⊗v 50v of that space that is globally χ 0 -generic. Then 5v and 50v are paired by the local correspondence for all v. By Proposition 4.1, the first occurrence of 5v0 = T is n or n + 1. First we assume e is also that the first occurrence of 5v0 = T is n + 1. Then the first occurrence of 5 n + 1. Now, if v is an infinite place or v 6∈ S, then using the results of Rallis on unramified theta correspondence (see [Ra1]), and using the results of Adams and Barbasch on archimedean theta correspondence [AB], as well as the definition of local factors in that two situations, we obtain ( ¡
L(s, 6v × 5v )L(s, 6v ) = L s, 6v × 50v
¡  s, 6v × 5v , ψv (s, 6v , ψv ) = (s, 6v × 50v , ψv ). Next, for v ∈ S, v finite, and v 6= v0 , applying the multiplicative properties of γ -factors as well as the results of Kudla about supports (analogous to that of Proposition 5.3, but simpler since the corresponding supercuspidal representation is the trivial representation of Sp(0, F )), we get γ (s, 6v × 5v , ψv )γ (s, 6v , ψv ) = γ (s, 6v × 50v , ψv ). Q Let LS (s, 6) = v6∈S L(s, 6v ). Define LS (s, 6 × 5) and LS (s, 6 × 50 ) similarly. Then we have a global functional equation Y ¡
e , γ (s, 6v , ψv ) · LS 1 − s, 6 LS (s, 6) = v∈S

as well as (see [Sh1, Theorem 3.5 (3.14)]) ( ¡
Q e ×5 e LS (s, 6 × 5) = v∈S γ (s, 6v × 5v , ψv ) · LS 1 − s, 6 ¡
Q f0 . e ×5 LS (s, 6 × 50 ) = v∈S γ (s, 6v × 50v , ψv ) · LS 1 − s, 6 The global functional equations, combined with the local functional equations, imply e is n + 1. the lemma if the first occurrence of 5 If the first occurrence of 5v0 = T is n, let τ 0 be an irreducible constituent of the lift of T to SO(2n, F ). It is ψ 0 -generic and supercuspidal. Let 50 = ⊗v 50v be an automorphic, globally χ 0 -generic cuspidal representation of SO(2n, A) such that

329

THETA LIFTS

50v0 = τ 0 , as constructed in [Sh1, Proposition 5.1]. Then the first occurrence index of f0 is n, and its lift is globally χ-generic (see [GRS, Proposition 3.5]). Let 500 = ⊗500v 5 be an irreducible constituent of that space that is globally χ-generic. Then 50v and 500v are paired by the local correspondence for all v. In particular, 500v0 ∼ = T. As in the first part of the proof, we see that γ (s, δ × τ 0 , ψF )γ (s, δ, ψF ) = γ (s, δ × T, ψF ). Since 2(T, n + 1), restricted to SO(2n + 2, F ), must share a ψ 0 -generic component with

SO(2n+2,F ) ¡ SO(2n+2,F ) ¡ 1F × ⊗ 50v0 or Ind 0 1F × ⊗ 50v0 , , Ind 0 Pn+1

Pn+1

the lemma follows from Proposition 3.1 (multiplicative properties of γ -factors). The proof of Proposition 5.4 is also complete. Corollary 5.1. If Tψ , τψ0 0 , and τψ0 0 , are tempered, then for any discrete series δ of GL(m, F ), we have (  ª Ł(s, δ × T)L(s, δ) = L(s, δ × τ0 ), where τ0 ∈ τψ0 0 , τψ0 0 , L(s, δ × τ 0 )L(s, δ) = L(s, δ × Tψ ).

Proof. This is an immediate consequence of Proposition 5.4, since L-functions of pairs are holomorphic in the right-half plane Re(s) > 0. (See [CSh, Theorem 4.1] or [M1, Corollary 4.1].) 6. Proofs Proposition 6.1. Let T ∈ Irr(Sp(n, F )) (n ≥ 2) be a ψ-generic discrete series, and let τ 0 ∈ Irr(SO(2n, F )) (n ≥ 2) be a ψ 0 -generic discrete series. Then we have the following. (i) If L(0, τ 0 ) = ∞, then Tψ , defined in Proposition 5.2, is the unique irreducible quotient of 2(T0 , n) (see Definition 4.1). 0 (ii) If L(0, T) = ∞, then τψ0 0 and τψ0 , defined in Proposition 5.3, are the irreducible quotients of 2(T, n + 1). (Proposition 2.2 forces n ≥ 2.) Proof. First we need a lemma that is an application of the theory of R-groups in [G] and [Sh1]. Lemma 6.1. Let T ∈ Irr(Sp(n, F )) (n ≥ 1) be a ψ-generic discrete series, and let τ 0 ∈ Irr(SO(2n, F )) (n ≥ 2) be a ψ 0 -generic discrete series. Then we have the following. SO(2n+2,F ) (1F × ⊗τ 0 ) is irreducible, and τ0 ∼ (i) L(0, τ 0 ) = ∞ if and only if IndPn+1 = τ 0. Sp(n+1,F )

(ii) L(0, T) = ∞ if and only if IndQn+1

(1F × ⊗ T) is irreducible.

We now proceed with the proof of Proposition 6.1. We prove (i). The proof of

´ AND SAVIN MUIC

330

(ii) is analogous. Let Pn+1 be a standard parabolic subgroup of O(2n + 2, F ) with a Levi factor F × × O(2n, F ), defined above. Then [Ku] implies that the normalized Jacquet module with respect to Pn+1 of ωn,n+1 has a quotient 1F × ⊗ ωn,n . This is a GL(1, F ) × Sp(n, F ) × O(2n, F )-module. Let π be any irreducible quotient of 2(T0 , n). The above discussion and the Frobenius reciprocity imply 
 SO(2n+2,F ) ¡ 1 ⊗ τ 0 6= 0. HomSp(n,F )×SO(2n+2,F ) ωn,n+1 , π ⊗ Ind 0 Pn+1

By Lemma 6.1, Ind

SO(2n+2,F ) (1 ⊗ τ 0 ) 0 Pn+1

is irreducible and ψ 0 -generic. By Corol-

lary 4.1, π is ψ-generic. Hence π ∼ = Tψ , as they have the same supercuspidal support. Proposition 6.2. Assume that T and τ 0 are discrete series as in Proposition 6.1. Then we have the following. (i) If L(0, τ 0 ) = ∞, then the statement of Theorem 2.2(i) holds. (ii) If L(0, T) = ∞, then the statement of Theorem 2.1(i) holds. Proof. We prove (i). The proof of (ii) is analogous. First, by Proposition 6.1, 2(T0 , n) contains Tψ as the unique irreducible quotient. Now we show the following lemma. Lemma 6.2. There exists a ψ-generic discrete series π of Sp(n − 1, F ), such that Sp(n,F )

Tψ ,→ IndQn

(1F × ⊗ π).

Proof. First, by Proposition 5.4, we have (6.1)

γ (s, δ × Tψ , ψF ) = γ (s, δ, ψF )γ (s, δ × τ 0 , ψF ),

for any discrete series δ of GL(m, F ). Next, the assumption L(0, τ 0 ) = ∞ implies that γ (s, τ 0 , ψF ) has a simple zero at s = 0. Further, since L(s, 1F × ) = (1 − q −s )−1 has a pole at s = 0, we see that γ (s, 1F × , ψF ) has a simple zero at s = 0. Hence, by (6.1), γ (s, Tψ , ψF ) has a double zero at s = 0. Now choose a standard parabolic subgroup P = MN, with M ∼ = GL(m1 , F ) × · · · × GL(mk , F ) × G0 (G0 is a symplectic group of a smaller rank), and a (generic) supercuspidal representation ρ = ρ1 ⊗· · ·⊗ρk ⊗π 0 of M such that Tψ is a subquotient of IndG P (ρ). Then we have Y ei , ψF )γ (s, ρi , ψF ). γ (s, Tψ , ψF ) = γ (s, π 0 , ψF ) γ (s, ρ i

Note that γ (s, ρi , ψF ) is a monomial in q −s unless m1 = 1 and ρi = | |si . The results of [Ku] and Theorem 3.1 force that 2si ∈ Z. Now γ (s, ρi , ψF ) is up to monomial in q −s , equal to

331

THETA LIFTS

L(1 − s − si , 1F × ) . L(s + si , 1F × ) Hence γ (s, ρi , ψF ) can have a zero at s = 0 if and only if si = 0. Since, by Theorem 3.1, γ (s, π 0 , ψF ) has at most a simple zero at s = 0, we conclude that there exists i such that ρi = 1F × . We fix such i, and we consider the supercuspidal support of Sp(n − 1, F ) formed using the remaining supercuspidal representations of the supercuspidal support of Tψ . Let π be the unique ψ-generic subquotient of that Sp(n,F ) (1F × ⊗ π) contains Tψ as an irreducible subquotient. support. Clearly IndQn Now, using Theorem 3.1, we show that π is a discrete series. This ends the proof of the lemma. First, Proposition 5.2 shows that the property (∗) of Section 3 holds for the supercuspidal support of π, since it holds for the supercuspidal support of τ 0 . Next, the formula (3.1) of Proposition 3.1 can be applied to both γ (s, δ × Tψ , ψF ) and γ (s, δ × π, ψF ) to obtain γ (s, δ × Tψ , ψF ) = γ (s, δ, ψF )2 γ (s, δ × π, ψF ).

(6.2)

Now, combining (6.1) and (6.2), we obtain γ (s, δ × π, ψF )γ (s, δ, ψF ) = γ (s, δ × τ 0 , ψF ).

(6.3)

Using (6.3), it is not difficult to check the remainder of Theorem 3.1. Now Proposition 6.1(i) and Lemma 6.2 imply ¡
Sp(n,F ) (1F × ⊗ π) ⊗ T0 6= 0. (6.4) HomSp(n,F )×O(2n,F ) ωn,n , IndQn Lemma 6.3. The normalized Jacquet module of ωn,n , with respect to Qn , has two subquotients:  J1 = 1F × ⊗ ωn−1,n (quotient) × ¡
J2 = IndSp(n−1,F )×F × ×O(2n,F ) 6 ⊗ ωn−1,n−1 (subrepresentation) Sp(n−1,F )×F ×Pn where 6 is an F × × F × -module. It is the space of all smooth, compactly supported, locally constant complex functions on F × , and the action is given by ¡
(x1 , x2 )f (h) = |x1 |n |x2 |−n f x1−1 hx2 . Assume that 2(T0 , n−1) = 0. Then (6.4) and the Frobenius reciprocity imply that 1F × ⊗ π ⊗ T0 is a quotient of J2 . Now we have (by an easy extension of a result of Bernstein [M1, Lemma 3.3]) ¡
HomSp(n−1,F )×F × ×O(2n,F ) J2 , π ⊗ 1F × ⊗ T0 ¡
∼ = HomSp(n−1,F )×F × ×F × ×O(2n−2,F ) 6 ⊗ ωn−1,n−1 , π ⊗ 1F × ⊗ R (T0 ) . Pn

(T0 )

T0 ,

(Here RP n denotes the normalized Jacquet module of with respect to the opposite parabolic of Pn .) It follows that RP n (T0 ) contains a subquotient of the form 1F × ⊗ T0 . Restricting to SO(2n, F ), this contradicts the square integrable criterion (see [C]).

332

´ AND SAVIN MUIC

So we have proven that 2(T0 , n − 1) 6= 0, and it has π as its irreducible quotient. Since, by Corollary 4.1, all irreducible quotients of 2(T0 , n − 1) are ψ-generic and all have the same supercuspidal support, there is only one irreducible quotient, and it comes with multiplicity one. Thus, to prove the first part of Theorem 2.2(i), we need to check 2(det ⊗T0 , n − 1) = 0 (see (2.1)). But this follows from the well-known inequality of Rallis (which follows from [Ra2, Appendix]):
¡ 2n ≤ n(T0 ) + n det ⊗T0 , where n(T0 ) and n(det ⊗T0 ) denote the first occurrence indices of the corresponding representations. Finally, the functional equations follow from Corollary 5.1 and (6.3). Proposition 6.3. Assume that T ∈ Irr(Sp(n, F )) (n ≥ 1) is a ψ-generic discrete series, and assume that τ 0 ∈ Irr(SO(2n, F )) (n ≥ 2) is a ψ 0 -generic discrete series. Then we have the following. (i) If L(0, τ 0 ) 6 = ∞, then Tψ is a discrete series representation. Moreover, Tψ is the unique irreducible ψ-generic quotient of 2(τ 0 , n). (ii) If L(0, T) 6 = ∞, then τψ0 0 ∼ = τψ0 0 , is a discrete series of SO(2n + 2, F ). Moreover, 2(T, n + 1) has τψ0 0 ∼ = τψ0 0 , as the unique irreducible ψ 0 -generic quotient. Proof. We prove (i). First, combining Proposition 5.2, Proposition 5.4, and Theorem 3.1, we see that Tψ is discrete series. Now, by Corollary 5.1, L(s, Tψ ) = L(s, 1F × )L(s, τ 0 ). Thus, L(s, Tψ ) has a pole at s = 0. Next, by Proposition 6.2(ii), 2(Tψ , n) is nontrivial, and all its irreducible SO(2n, F )-quotients are ψ 0 -generic. It follows from the supercuspidal support that T0 or det ⊗T0 is the unique irreducible O(2n, F )-quotient of 2(Tψ , n). Going back, we see that Tψ is a quotient of 2(τ 0 , n). There cannot be any other irreducible ψ-generic quotients by Corollary 4.1. Part (i) follows. The proof of (ii) is analogous. We only remark that, by Lemma 6.1, L(0, τψ0 0 ) = ∞ implies that τψ0 0 ∼ = τψ0 0 , . The functional equations now follow from Corollary 5.1. The proposition is proved. Finally, Propositions 4.2 and 6.3 complete the proof of Theorems 2.1(ii) and 2.2(ii). References [AB] [A] [Ba] [Bo]

J. Adams and D. Barbasch, Reductive dual pair correspondence for complex groups, J. Funct. Anal. 132 (1995), 1–42. A.-M. Aubert, Description de la correspondance de Howe en termes de classification de Kazhdan-Lusztig, Invent. Math. 103 (1991), 379–415. D. Ban, Selfduality in the case of SO(2n, F ), to appear in Glas. Mat. Ser. III. A. Borel, “Automorphic L-functions” in Automorphic Forms, Representations and LFunctions (Oregon State Univ., Corvallis, Ore., 1977), Part 2, Proc. Sympos. Pure Math. 33, Amer. Math. Soc., Providence, 1979, 27–61.

THETA LIFTS [C] [CSh] [GRS] [G] [J]

[Ku] [MVW] [M1] [M2] [M3] [MSh] [P] [Ra1] [Ra2] [Ro]

[Sh1] [Sh2]

[T] [W]

333

W. Casselman, Introduction to the theory of admissible representations of p-adic reductive groups, preprint. W. Casselman and F. Shahidi, On irreducibility of standard modules for generic representations, Ann. Sci. École Norm. Sup. (4) 31 (1998), 561–589. D. Ginzburg, S. Rallis, and D. Soudry, Periods, poles of L-functions and symplecticorthogonal theta lifts, J. Reine Angew. Math. 487 (1997), 85–114. D. Goldberg, Reducibility of induced representations for Sp(2n) and SO(n), Amer. J. Math. 116 (1994), 1101–1151. H. Jacquet, “Principal L-functions of the linear group” in Automorphic Forms, Representations and L-Functions (Oregon State Univ., Corvallis, Ore., 1977), Part 2, Proc. Sympos. Pure Math. 33, Amer. Math. Soc., Providence, 1979, 63–86. S. S. Kudla, On the local theta-correspondence, Invent. Math. 83 (1986), 229–255. C. Moeglin, M.-F. Vignéras, and J.-L. Waldspurger, Correspondances de Howe sur un corps p-adique, Lecture Notes in Math. 1291, Springer-Verlag, Berlin, 1987. G. Mui´c, Some results on square integrable representations; irreducibility of standard representations, Internat. Math. Res. Notices 1998, 705–726. , The Howe correspondence and reducibility of induced representations for Sp(n) and O(V ), preprint, 1998. , A proof of Casselman-Shahidi’s conjecture for quasi-split classical groups, to appear in Canad. Math. Bull. G. Mui´c and F. Shahidi, Irreducibility of standard representations for Iwahori-spherical representations, Math. Ann. 312 (1998), 151–165. D. Prasad, On the local Howe duality correspondence, Internat. Math. Res. Notices 1993, 279–287. S. Rallis, Langlands’ functoriality and the Weil representation, Amer. J. Math. 104 (1982), 469–515. , On the Howe duality conjecture, Compositio Math. 51 (1984), 333–399. F. Rodier, “Whittaker models for admissible representations of reductive p-adic split groups” in Harmonic Analysis on Homogeneous Spaces (Williams Coll., Williamstown, Mass., 1972), Proc. Sympos. Pure Math. 26, Amer. Math. Soc., Providence, 1973, 425–430. F. Shahidi, A proof of Langlands’ conjecture on Plancherel measures; complementary series for p-adic groups, Ann. of Math. (2) 132 (1990), 273–330. , “On multiplicativity of local factors” in Festschrift in Honor of I. I. PiatetskiShapiro (Ramat Aviv, 1989), Part 2, Israel Math. Conf. Proc. 3, Weizmann, Jerusalem, 1990, 279–289. M. Tadi´c, On regular square integrable representations of p-adic groups, Amer. J. Math. 120 (1998), 159–210. J.-L. Waldspurger, “Démonstration d’une conjecture de dualité de Howe dans le cas padique, p 6 = 2” in Festschrift in Honor of I. I. Piatetski-Shapiro (Ramat Aviv, 1989), Part 1, Israel Math. Conf. Proc. 2, Weizmann, Jerusalem, 1990, 267–324.

Department of Mathematics, University of Utah, Salt Lake City, Utah 84112, USA; [email protected]; [email protected]

Vol. 101, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p]) JENNIFER TABACK

1. Introduction. Combining the work of many people yields a complete quasiisometry classification of irreducible lattices in semisimple Lie groups (see [F] for an overview of these results). One of the first general results in this classification is the complete description, up to quasi-isometry, of all nonuniform lattices 3 in semisimple Lie groups of rank 1, proved by R. Schwartz [S1]. He shows that every quasi-isometry of such a lattice 3 is equivalent to a unique commensurator of 3. (A commensurator of 3 ⊂ G is an element g ∈ G so that g3g −1 ∩ 3 has finite index in 3.) We call this result commensurator rigidity, although it is a different notion than the commensurator rigidity of Margulis. In [FS] it was conjectured that commensurator rigidity, or at least a slightly weaker statement, “quasi-isometric if and only if commensurable,” should apply to nonuniform lattices in a wide class of Lie groups. Here we prove that both of these statements are true for PSL2 (Z[1/p]). In a different direction, B. Farb and L. Mosher proved analogous quasi-isometric rigidity results for the solvable Baumslag-Solitar groups. These groups are given by the presentation BS(1, n) = ha, b | aba −1 = bn i and are not lattices in any Lie group. The group PSL2 (Z[1/p]) is a nonuniform (i.e., noncocompact) lattice in the group PSL2 (R)×PSL2 (Qp ), analogous to the classical Hilbert modular group PSL2 (Od ) in PSL2 (R)×PSL2 (R). It is also a basic example of an S-arithmetic group. The proofs of Theorems A, B, and C (stated below) combine techniques from the two types of quasiisometric rigidity results mentioned above. When we construct a space p on which PSL2 (Z[1/p]) acts properly, discontinuously, and cocompactly by isometries, we see that the horospheres forming the boundary components of p carry the geometry of the group BS(1, p). In this way the results of [FM] play a role in the quasi-isometric rigidity of PSL2 (Z[1/p]). 1.1. Statement of results. In this paper we prove the following quasi-isometric rigidity results for the finitely generated groups PSL2 (Z[1/p]), where p is a prime. Theorem A may be viewed as a strengthening of strong (Mostow) rigidity for PSL2 (Z[1/p]). [M]

Received 17 August 1998. Revision received 21 April 1999. 1991 Mathematics Subject Classification. Primary 20F32; Secondary 20E99. 335

336

JENNIFER TABACK

Theorem A (Main theorem). Every quasi-isometry of PSL2 (Z[1/p]) is equivalent to a commensurator of PSL2 (Z[1/p]). Hence the natural map         1 1 −→ QI PSL2 Z Comm PSL2 Z p p is an isomorphism. Since for any prime p, the commensurator group of the group PSL2 (Z[1/p]) in PSL2 (R) × PSL2 (Qp ) is PSL2 (Q), the quasi-isometry group is also PSL2 (Q). (See §2.2 for a model of PSL2 as an algebraic group.) Thus we cannot distinguish the quasi-isometry classes of these groups via their quasi-isometry groups. However, using a result of Farb and Mosher [FM] (Theorem 2.1 below), we are able to prove the following. Theorem B (Quasi-isometric if and only if commensurable). Let p and q be primes. Then PSL2 (Z[1/p]) and PSL2 (Z[1/q]) are quasi-isometric if and only if they are commensurable, which occurs only when p = q. Theorems A and B together give the first example of groups that have the same quasi-isometry group but are not quasi-isometric. The following theorem characterizes PSL2 (Z[1/p]) uniquely among all finitely generated groups by its quasi-isometry type. Theorem C (Quasi-isometry characterization). Let 0 be any finitely generated group. If 0 is quasi-isometric to PSL2 (Z[1/p]), then there is a short exact sequence 1 −→ N −→ 0 −→ 3 −→ 1 where N is a finite group and 3 is abstractly commensurable to PSL2 (Z[1/p]). Two groups are abstractly commensurable if they have isomorphic finite index subgroups. 1.2. An outline of the proofs of Theorems A and B. The group PSL2 (Z[1/p]) is a nonuniform (i.e., noncocompact) lattice in G = PSL2 (R) × PSL2 (Qp ) under the diagonal embedding sending a matrix M to the pair (M, M). The group G acts on H2 × Tp , where H2 is the hyperbolic plane and Tp is the Bruhat-Tits-Serre tree associated to PGL2 (Qp ). Let f : PSL2 (Z[1/p]) → PSL2 (Z[1/q]) be a quasi-isometry. The proofs of Theorems A and B both begin as follows. Step 1 (The geometric model). We construct a space p on which PSL2 (Z[1/p]) acts properly, discontinuously, and cocompactly by isometries (hence by a result of Milnor and Svarc [Mi], PSL2 (Z[1/p]) and p are quasi-isometric). The space p has a boundary consisting of horospheres of H2×Tp , each of which is a quasi-isometrically embedded copy of the group BS(1, p). The quasi-isometry f : PSL2 (Z[1/p]) →

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

337

PSL2 (Z[1/q]) then induces a quasi-isometry, also denoted f , from p to q . Step 2 (The boundary detection theorem). This theorem shows that for every horosphere boundary component σ of p , there is a corresponding horosphere boundary component τ of q so that f restricts to a quasi-isometry of horospheres σ → τ . The proof of this theorem uses the coarse separation theorem of [FS], and the geometry of p . Remark. We are now able to prove Theorem B. The initial quasi-isometry f : PSL2 (Z[1/p]) → PSL2 (Z[1/q]) induces a quasi-isometry f : p → q . From the boundary detection theorem, we obtain a quasi-isometry fˆ : σ → τ by restriction, where σ and τ are horosphere boundary components of p and q , respectively. By Step 1, the map fˆ can be considered as a quasi-isometry of Baumslag-Solitar groups, namely, fˆ : BS(1, p) → BS(1, q). From [FM], we conclude that p = q. The proof of Theorem A continues with the following steps. We are now considering a quasi-isometry f : p → p . Step 3 (The geometry of p ). For any two horosphere boundary components σ1 and σ2 of p , there is a unique line l ⊂ Tp so that σ1 and σ2 are a specified fixed distance apart in H2 × t, for all vertices t of l. This line is called the closeness line of σ1 and σ2 . The set of closeness lines of all horospheres is preserved under quasiisometry. This set of lines is analogous to the lamination of shadows in [FS]. This geometric result replaces the usual group-equivariance assumed in Mostow-Prasad rigidity, and provides the structure necessary for Step 4. Step 4 (S-arithmetic action rigidity). Here we prove a 2-dimensional S-arithmetic version of the action rigidity theorem of Schwartz [S2]. The action rigidity theorem concludes that the map induced by f on the set of horospheres of ∂p (which is indexed by Q ∪ {∞}) is given by an affine map of Q ∪ {∞}. Step 5 (Conclusion of the proof). From Step 4, we are able to choose a specific commensurator g of PSL2 (Z[1/p]) so that the composite map f ◦ g is a bounded distance from the identity map. This finishes the proof of Theorem A. Acknowledgments. I would like to thank B. Farb, R. Schwartz, and A. Eskin for many useful mathematical conversations, and David Fisher for corrections. This paper was submitted as a thesis to the University of Chicago. 2. Preliminary material 2.1. Quasi-isometries Definition. Let K ≥ 1 and C ≥ 0. A (K, C)-quasi-isometry between metric spaces (X, dX ) and (Y, dY ) is a map f : X → Y satisfying: (1) (1/K)dX (x1 , x2 )−C ≤ dY (f (x1 ), f (x2 )) ≤ KdX (x1 , x2 )+C for all x1 , x2 ∈ X.

338

JENNIFER TABACK

(2) For some constant C 0 , the C 0 neighborhood of f (X) is all of Y . We often omit the constants K and C and simply refer to f as a quasi-isometry. A quasi-isometry f can always be changed by a bounded amount using the standard “connect-the-dots” procedure so that it is continuous (see, e.g., [FS]). A quasiisometry also has a coarse inverse; that is, there is a quasi-isometry g : Y → X so that f ◦ g and g ◦ f are a bounded distance from the appropriate identity map in the sup norm. A map satisfying (1) but not (2) in the definition above is called a quasi-isometric embedding. We define the quasi-isometry group of a space X, denoted QI(X), to be the set of all self-quasi-isometries of X, modulo those a bounded distance from the identity in the sup norm, under composition of quasi-isometries. Inverses exist in QI(X) since every quasi-isometry has a coarse inverse. A quasi-isometry between two metric spaces X and Y induces an isomorphism between QI(X) and QI(Y ). 2.2. PSL2 as an algebraic group. We use the following model of PSL2 as an algebraic group. Consider the map Ad : SL2 (C) → GL(sl2 ), where we view the Lie algebra sl 2 as a vector space. Let G0 = Ad(SL2 (C)). Then G0 is a model for PSL2 as an algebraic group, since the center of SL2 (C) vanishes under the map Ad. By PSL2 (Q) we mean the Q-points of G0 , denoted G0Q . 2.3. The geometry of BS(1, n). The Baumslag-Solitar group BS(1, n) = ha, b | aba −1 = bn i acts properly discontinuously and cocompactly by isometries on a metric 2-complex Xn defined explicitly in [FM]. This complex Xn is topologically Tn × R where Tn is a regular (n + 1)-valent tree, directed so that each vertex has 1 incoming edge and n outgoing edges. (See Figure 1.) A height function on Tn is a continuous function h : Tn → R that maps each oriented edge of Tn homeomorphically onto an oriented interval of a given length d. A vertex of Tn whose height under h is kd is said to have combinatorial height k. ∂u

π

∂l Figure 1. The complex Xn associated to BS(1, n) where the map π denotes projection onto the tree Tn . The upper and lower boundaries of Xn are also marked.

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

339

Fix a basepoint for Tn with height 0. This determines a height function h on Tn . Let π : Tn × R → Tn denote projection. Then h ◦ π is a height function on Xn . A proper line in Tn is the image of a proper embedding R → Tn . A coherently oriented proper line is one on which the height function is strictly monotone. We use the term “line” to mean a proper line in Tn . The metric on Xn is defined so that for each infinite, coherently oriented line l ⊂ T , the plane l × R is isometric to a hyperbolic plane. When studying the geometry of Xn , there are two boundaries of the complex that play an important role (see Figure 1). The lower boundary, denoted ∂l Xn , is homeomorphic to R and is the common lower boundary of all hyperbolic planes in Xn . The upper boundary, denoted ∂ u Xn , is defined to be the space of hyperbolic planes in Xn , with the following metric. If Q1 and Q2 are hyperbolic planes in Xn that agree below combinatorial height k, define the distance between them to be n−k . With this metric, ∂ u Xn is isometric to the set of n-adic rational numbers, Qn , with the metric defined by the n-adic absolute value. Farb and Mosher [FM] obtain the following quasi-isometric rigidity results for the groups BS(1, n). Theorem 2.1 [FM]. For integers m, n ≥ 2, the groups BS(1, m) and BS(1, n) are quasi-isometric if and only if they are commensurable. This happens if and only if there exist integers r, j, k > 0 such that m = r j and n = r k . Theorem 2.2 [FM]. The quasi-isometry group of BS(1, n) is given by the following isomorphism: ¡
QI BS(1, n) ∼ = Bilip(R) × Bilip(Qn ). A quasi-isometry f ∈ QI(BS(1, n)) induces bilipschitz maps f u and fl on the upper and lower boundaries of Xn , respectively (see [FM]). From Theorem 2.2 we see that the map QI(BS(1, n)) → Bilip(R) × Bilip(Qp ) given by f → (fl , f u ) is an isomorphism. It is perhaps surprising that PSL2 (Z[1/p]) should have such a small quasi-isometry group while BS(1, p) has such a large quasi-isometry group. 3. The geometry of PSL2 (Z[1/p]) 3.1. The action of PSL2 (R) × PSL2 (Qp ) on H2 × Tp . We consider the group PSL2 (Z[1/p]) ⊂ PSL2 (R) × PSL2 (Qp ) as the image of the diagonal map η : PSL2 (Z[1/p]) → PSL2 (R) × PSL2 (Qp ) given by η(M) = (M, M). Viewed in this way, PSL2 (Z[1/p]) is a lattice in the group PSL2 (R) × PSL2 (Qp ), for any prime p. We now define the Bruhat-Tits tree Tp associated to PGL2 (Qp ). We consider a tree T to be a set of vertices Vert(T ) together with a set of adjacency relations among the vertices. Let Vert(Tp ) be the set of equivalence classes of Zp -lattices in Qp × Qp . Two lattices L1 and L2 are equivalent if L2 = αL1 , where α ∈ Qp −{0}. Two vertices [L1 ] and [L2 ] are adjacent if there exist representatives L1 ∈ [L1 ] and L2 ∈ [L2 ] with

340

JENNIFER TABACK

L1 ⊂ L2 and [L2 : L1 ] = p. An example of two adjacent vertices is [Zp × Zp ] and [pZp × Zp ]. The tree Tp is a regular (p + 1)-valent tree. We fix [L0 ] = [Zp × Zp ] as the basepoint of Tp as well as a height function h giving [L0 ] height 0. An element g ∈ PGL2 (Qp ) acts on [L] ∈ Vert(Tp ) by matrix multiplication on the basis vectors of a representative lattice in the equivalence class [L]. Note that PSL2 (Qp ) also acts on Tp in this way. Let H2 denote 2-dimensional hyperbolic space in the upper half-plane model, that is, H2 = {(x, y) | x ∈ R, y > 0} with the metric (dx 2 + dy 2 )/y 2 . We define the action of an element (g1 , g2 ) ∈ PSL2 (R) × PSL2 (Qp ) on a point (x, [L]) ∈ H2 × Tp . The element g1 ∈ PSL2 (R) acts on x ∈ H2 by fractional linear transformations. The element g2 ∈ PSL2 (Qp ) acts on [L] ∈ Vert(Tp ) by matrix multiplication on the basis vectors of a representative lattice in the equivalence class [L]. When we are considering the action of PSL2 (Z[1/p]) on H2 × Tp , we have g1 = g2 . We use only one coordinate to represent the elements of PSL2 (Z[1/p]) ⊂ PSL2 (R) × PSL2 (Qp ). So g ∈ PSL2 (Z[1/p]) corresponds to (g, g) ∈ PSL2 (R) × PSL2 (Qp ). Hence we can refer to g ∈ PSL2 (Z[1/p]) acting on either H2 or Tp or H2 ×Tp in the appropriate manner. 3.2. Constructing the space p . We want to construct a space p ⊂ H2 × Tp on which PSL2 (Z[1/p]) acts properly discontinuously and cocompactly by isometries. The Milnor-Svarc criterion [M] states that if a finitely generated group 0 acts properly discontinuously and cocompactly by isometries on a space X, then 0 is quasi-isometric to X. We then refer to the geometry of X as the large scale geometry of 0. This additional geometric information associated to 0 is often useful in determining rigidity properties of 0. Although PSL2 (Z[1/p]) acts by isometries on H2 ×Tp , it does not act cocompactly, because the fundamental domain for the action of PSL2 (Z[1/p]) on a fixed H2 is the same as the fundamental domain for the action of PSL2 (Z) on H2 , which is unbounded in one direction. Let w be the segment of the horocircle based at ∞ at height h0 in this fundamental domain (in the upper half-space model of hyperbolic space) for h0 sufficiently large. Fix H > 1. Lift the segment w to H2 × [L0 ] to obtain a horocyclic segment at height H whose orbit under PSL2 (Z) = StabPSL2 (Z[1/p]) ([L0 ]) is a disjoint collection of horocircles of H2 , centered at points of Q ∪ {∞} ⊂ ∂∞ H2 (where [L0 ] is defined in §3.1 as the basepoint of Tp ). The orbit of the lift of w under the entire group PSL2 (Z[1/p]) gives a PSL2 (Z[1/p])-equivariant collection of horocircles in H2 ×Tp , based at Q ∪ {∞} in each copy of H2 ⊂ H2 × Tp . We now define a horosphere of H2 ×Tp based at α ∈ Q∪{∞} to be the collection of horocircles, one in each H2 ×[L] for every [L] ∈ Vert(Tp ), all based at α. According to the above construction, there is exactly one such horocircle in each H2 × [L]. We denote this horosphere of H2 × Tp by σα . In order to have a connected picture of a horosphere, we can put an edge e between

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

341

Figure 2. This is a piece of the horosphere σ∞ in H2 × T3 , the space quasiisometric to PSL2 (Z[1/3]). Notice how the height of the horosphere increases in each successive copy of H2 . Viewing the bold black lines as part of the tree T3 helps one to see that this horosphere is topologically T3 × R.

any two adjacent vertices of Tp and extend the horosphere linearly in H2 × e. So we can think of a horosphere as a hollow tube, or in the case of σ∞ , as a flat sheet, whose image under the projection π : H2 × Tp → Tp is all of Tp . (See Figure 2.) We define the space p , where PSL2 (Z[1/p]) acts properly discontinuously and cocompactly by isometries, to be H2 × Tp with the interiors of all the horospheres removed. The interior of a horosphere is the union of the interiors of the component horocircles. 3.3. The metric. The following theorem allows us to use the product metric dH2 × dT on p . Although it is stated for semisimple Lie groups, it is proven in the more general context of S-arithmetic groups. Theorem 3.1 [LMR]. If G is a semisimple Lie group of rank at least 2 and 0 is an irreducible lattice in G, then dR restricted to 0 is Lipschitz equivalent to dW where dW is the word metric on 0 and dR is the left invariant Riemannian metric on 0. By construction, PSL2 (Z[1/p]) acts properly discontinuously and cocompactly by isometries on p . 3.4. The closeness line. We are now interested in the packing of the horospheres of H2 ×Tp , that is, how the horosphere “tubes” fit together. We consider two horospheres, without loss of generality σ0 and σ∞ , and the distance between them when restricted to H2 × [L], for any [L] ∈ Vert(Tp ). If σ is a horosphere boundary component of p , we use the notation σ |[L] for the horocircle σ ∩ (H2 × [L]). We show that there is a unique line l ⊂ Tp so that σ0 and σ∞ remain a constant distance apart when restricted to H2 × [L], for all vertices [L] lying on l. In addition, we show that all other lines l 0 ⊂ Tp have the property that dH2 (σ0 |[L] , σ∞ |[L] ) increases without bound

342

JENNIFER TABACK

as the vertices [L] lying on l 0 increase in height. We call l the closeness line of σ∞ and σ0 . By symmetry, any two horospheres σα and σβ have a unique closeness line. The closeness lines are analogous to the “shadows” of [FS]. Note that a matrix g that translates σα |[L1 ] to σα |[L2 ] lies in StabPSL2 (Z[1/p]) (α) and satisfies [gL1 ] = [L2 ]. However, to move between the horocircles σβ |[L1 ] and σβ |[L2 ] , for β 6 = α, we need to use a different element of PSL2 (Z[1/p]), that is, h ∈ StabPSL2 (Z[1/p]) (β) such that [hL1 ] = [L2 ]. ¡
¡ 0
and B = 01 11 give the PSL2 (Qp ) action Products of the matrices A = p0 1/p on Tp . (See, e.g., [Se].) Since A ∈ StabPSL2 (Z[1/p]) (0) ∩ StabPSL2 (Z[1/p]) (∞), letting Ai act on H2 × [L0 ] moves the horocircle σ0 |[L0 ] (resp., σ∞ |[L0 ] ) to the horocircle σ0 |[Ai L0 ] (resp., σ∞ |[Ai L0 ] ). Moreover, A ∈ PSL2 (R) = Isom+ (H2 ), so we have ¡
¡
dH2 σ0 |[L0 ] , σ∞ |[L0 ] = dH2 σ0 |[Ai L0 ] , σ∞ |[Ai L0 ] = 2 log H where H was chosen in §3.2. Let l be the line in Tp that is the orbit of [L0 ] under the cyclic group generated by the matrix A. We call l the diagonal line. Usually we need two different matrices to move two different horocircles in H2 × [L0 ] to their corresponding horocircles in H2 × [L] for [L] ∈ Vert(Tp ); here we need only one matrix since the matrix A lies in the intersection of the stabilizers of 0 and ∞. We view a ray of Tp based at [L0 ] as an infinite sequence of products {5N i=1 Ci }N∈N where either Ci = A or Ci = B j A for j ∈ {0, 1, . . . , p −1}. The diagonal ray (i.e., the part of the diagonal line beginning at [L0 ] and moving upwards in height) is described , by {5N i=1 A}N ∈N , and we know from the previous paragraph that dH2 (σ0 |[5N i=1 A] σ∞ |[5N A ]) = 2 log H for all N ∈ N, where H was chosen in §3.2. i=1 Now suppose that r ⊂ Tp is the ray based at [L0 ] given by the sequence j {5N i=1 Ci }N ∈N where Ci = B A for j ∈ {0, 1, . . . , p −1}. For any N0 ∈ N, the product N

0 5i=1 Ci has the form (5Ti=1 A)(5Si=1 Ci ) where Ci is as above. Such a product is a   S+T s/(p S−T ) , where S or T may be 0. We may assume matrix of the form M = p 0 1/(p S+T ) that S > T , meaning that in the tree factor, we are considering vertices sufficiently far away from the diagonal line l. A computation shows that σ0 |[ML0 ] has height 1/(p2m H ) in H2 . The height of σ∞ |[ML0 ] is p2(S+T ) H . Hence

¡
¡
dH2 σ0 |[ML0 ] , σ∞ |[ML0 ] = log p2S H 2 . When S = 0, the vertex 5T1 A lies on the diagonal line, and the above formula gives a

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

343

Figure 3. The two horospheres shown are (pieces of) σα and σ∞ , for α 6= ∞. The dark line in σ∞ represents the closest points in σ∞ to σα . We can view this line as the closeness line of σ∞ and σα drawn in σ∞ .

constant distance of 2 log H between σ∞ and σ0 in H2 × l. When S 6= 0, we see that S increases by a factor of 2 for each unit of height in Tp ; hence the distance between σ∞ and σ0 increases by a factor of p4 . So we see that for any line l 0 ⊂ T that is not the diagonal line, dH2 (σ0 |[L] , σ∞ |[L] ) increases without bound as we consider vertices [L] of l 0 of increasing height. So the diagonal line l is the closeness line of σ0 and σ∞ . It is clear by construction that the closeness line of σα and σβ and the closeness line of σα and σγ are distinct. ¡ 0
and 3.5. The geometry of the horospheres. Consider the matrices A = p0 1/p ¡1 1
2 −1 p B = 0 1 . Then A, B ∈ PSL2 (Z[1/p]) ⊂ PSL2 (Qp ) and ABA = B . The map 2 8 sending the generator a of BS(1, p 2 ) = ha, b | aba −1 = bp i to the matrix A and the generator b to B is a homomorphism of BS(1, p 2 ) into the group PSL2 (Qp ). Products of the matrices A and B form the elements of PGL2 (Qp ) needed to move between vertices of Tp . If we take the orbit of the segment w from H i to 1+H i in H2 ×[L0 ] under the group 8(BS(1, p 2 )) ⊂ PSL2 (Qp ), we obtain the horosphere σ∞ . The width of a horostrip in σ∞ between vertices whose combinatorial heights differ by 1, in the metric on H2 ×Tp , is 1+2 log p, while in Xp2 it is 2 log p. Since these distances are comparable, the horosphere σ∞ (and hence any horosphere σα ) is quasi-isometric to the complex Xp2 associated to BS(1, p2 ). Since BS(1, p2 ) and BS(1, p) are commensurable and hence quasi-isometric, we may assume that the horospheres are quasi-isometrically embedded copies of the complex Xp associated to BS(1, p). 3.6. Another view of the closeness lines. The following discussion of closeness lines provides some geometric intuition for understanding them. We can consider the collection of closeness lines of all horospheres of ∂p with the horosphere σ∞ . This is a collection of distinct lines in the tree Tp . These closeness lines can also be viewed as lying in the horosphere σ∞ , which is thought of as the

344

JENNIFER TABACK

Figure 4. A subset of the collection of closeness lines of all horospheres with σ∞ , drawn in the complex Xp of σ∞

complex Xp associated to BS(1, p). In this case, the closeness line of σ∞ and σα can be viewed as the set of points in σ∞ closest to σα that project under π : H2 ×Tp → Tp to the closeness line of σ∞ and σα as described in §3.4 (see Figures 3 and 4). Then each hyperbolic plane H2 ⊂ Xp = σ∞ contains at most one complete closeness line. 4. The boundary detection theorem. The first goal of this section is to prove Theorem 4.5 (boundary detection theorem), which states that a quasi-isometry f : p → q maps a horosphere boundary component of p to within a bounded Hausdorff distance of a horosphere boundary component of q . Two subsets X and Y of a metric space W have bounded Hausdorff distance if there exists an  > 0 so that X ⊂ N bhd (Y ) and Y ⊂ N bhd (X). The infimum of such constants  is called the Hausdorff distance between X and Y . The boundary detection theorem is analogous to Theorem 4.1 of [FS] and Theorem 3.1 of [S2]. The second goal is to use Theorem 4.5, combined with Theorem 2.1, to prove Theorem B, that is, that PSL2 (Z[1/p]) and PSL2 (Z[1/q]) are quasi-isometric if and only if p = q. We use the notation N bhdr (S) to be the r-neighborhood in p of a subset S of p . We say that a subset S of a metric space X has the strong separation property in X if there is a fixed r > 0 with the following property. For every k > 0, there are at least two connected components of X − Nbhdr (S) that contain metric balls of radius k. We say that S separates X if S has the strong separation property in X. The constant r is called the separation constant. A metric space X is called uniformly contractible if there is a function α : R+ → + R with the following property. If f : 1 → X is a map of a finite simplicial complex, and f (1) ⊂ Br , where Br ⊂ X is a metric r-ball, then f (1) is contractible in Bα(r) , where α is independent of dim(1). Any contractible space admitting a cocompact group of isometries is uniformly contractible. We need the following coarse topology results, which we state as special cases of Theorem 5.2 and Corollary 5.3 of [FS] and Theorem 4.1 of [S2]. We are using the

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

345

bounded geometry metric d on R3 described below, which comes from choosing a proper embedding of the tree Tp → R2 . This allows us to consider Tp ×R as a subset of (R3 , d). In applying these results, we use the fact that R3 in this bounded geometry metric is uniformly contractible and contains spheres of arbitrarily large radius (the “expanding spheres” condition of [FS] and [S2]). Theorem 4.1 (coarse separation [FS], [S2]). Suppose φ : (R3 , d) → Y is a (K, C)-quasi-isometric embedding of (R3 , d) into a uniformly contractible Riemannian manifold Y diffeomorphic to R4 . Then φ(R3 ) separates Y , where the separation constant depends on (K, C). Corollary 4.2 (packing theorem [FS]). Suppose φ : (R3 , d) → (R3 , d) is a (K, C)-quasi-isometric embedding. Then φ is a (K 0 , C 0 )-quasi-isometry, for some constants (K 0 , C 0 ) depending on (K, C). 4.1. Separation. Let σ be any horosphere boundary component of p and let f : p → q be a quasi-isometry. In this section we show that the image f (σ ) separates H2 × Tq . To prove this, we extend the quasi-isometric embedding f |σ : σ → H2 × Tq to a quasi-isometric embedding fb : (R3 , d) → H2 × R2 to which we can apply Theorem 4.1. As above, d is the bounded geometry metric on R3 , which comes from choosing a proper embedding of the tree Tp → R2 . Consider σ as the complex Xp associated to BS(1, p) and choose a homeomorphism β : Xp → Tp × R. Let αp : Tp → R2 be any proper embedding. Then we can consider Xp as a subset of R3 via the map (αp × Id) ◦ β. It is shown in [FM] that R3 can be given a bounded geometry metric d for which this map is an isometric embedding, as follows. The boundary of each connected component C of R3 −Xp is topologically a plane. We use two coordinates (t, r) on this plane ∂C, where t ∈ Tp and r ∈ R. Choose a homeomorphism that identifies ∂C ∪ C with ∂C × [0, ∞). Then a point in C has three coordinates: (t, r, s) where t and r as above give a point in ∂C, and s ∈ [0, ∞). We use the product metric on each component. The quasi-isometric embedding f |σ has image in H2 ×Tq . Analogous to the above situation, we choose a proper embedding αq : Tq → R2 and view H2 × Tq as a subset of H2 × R2 via the map Id ×αq . As above, we obtain a metric d 0 on H2 × R2 for which this map is an isometric embedding. To apply Theorem 4.1, we use the fact that (H2 × R2 , d 0 ) is diffeomorphic to R4 , uniformly contractible, and contains arbitrary large metric balls. We can now extend the map f |σ to a quasi-isometric embedding fb: (R3 , d) −→ (H2 × R2 , d 0 ) by ¡
fb(t, r, s) = f (t, r), s . Note that f (t, r) is a point in H2 ×Tq , and so provides two coordinates. Also, fb|σ = f .

346

JENNIFER TABACK

Proposition 4.3 (Separation). Let f : p → q ⊂ H2 × Tq be a (K, C)-quasiisometry, and let σ ⊂ ∂p be any horosphere boundary component. Then f (σ ) separates H2 × Tq , where the separation constant depends on (K, C). Proof. Extend f |σ as above to a quasi-isometric embedding
¡ ¡
fˆ : R3 , d −→ H2 × R2 , d 0 . It is understood that a path avoiding f (σ ) or fb(R3 ) stays outside the neighborhood N bhdr (f (σ )) or N bhdr 0 (fb(R3 )) for a constant r or r 0 . Also note that f (σ ) = fb(σ ). The map fb : (R3 , d) → (H2 × R2 , d 0 ) satisfies the conditions of Theorem 4.1, so fb(R3 ) separates H2 × R2 . Suppose that fb(R3 ) separates H2 × R2 , and fb(σ ) does not separate H2 × Tq . This means that any two points in (H2 × Tq ) − fb(σ ) can be joined by a path avoiding fb(σ ). We show that if fb(σ ) does not separate H2 × Tq , any two points in (H2 × R2 ) − fb(R3 ) can be connected by a path avoiding fb(R3 ), contradicting the fact that fb(R3 ) separates H2 × R2 . Let x1 and x2 be any two points in (H2 × R2 ) − fb(R3 ). Each xi , i = 1, 2, has coordinates (αi , ti , si ), where αi ∈ H2 and ti and si are as above. / fb(R3 ), each line {(αi , ti , s) | s ∈ [0, ∞)} is not contained in fb(R3 ) by Since xi ∈ construction. When s = 0, each line gives a point in H2 × Tq not contained in fb(σ ). Call these points βi . Since fb(σ ) does not separate H2 × Tq , we can connect β1 to β2 by a path γ lying in H2 × Tq , which avoids fb(σ ) and hence fb(R3 ). So x1 and x2 are connected by the path
¡
−1 α1 , t1 , [0, s2 ] ∗ γ ∗ α2 , t2 , [0, s1 ] ,

¡

which avoids fb(R3 ). Thus, if fb(R3 ) separates R2 ×H2 , then f (σ ) = fb(σ ) separates H2 × Tq . We now prove a lemma that shows that the space p with the neighborhood of any horosphere removed is path connected. This lemma is needed in the proof of Theorem 4.5 and is analogous to Lemma 4.4 of [FS]. Lemma 4.4. Let r be any positive real number. For any horosphere boundary component σ of p , the space p − Nbhdr (σ ) is path connected. Proof. We show that p − Nbhdr (σ ) is path connected for any σ ⊂ ∂ and for any r. For convenience we may assume that σ = σ∞ . For any horosphere τ ⊂ ∂p , consider the set of vertices of Tp where τ intersects N bhdr (σ∞ ):  ¡
ª Sr,τ = [L] ∈ Vert Tp | τ |[L] ∩ Nbhdr (σ∞ ) 6= ∅ . Since each τ has a closeness line with σ∞ , for large enough r, the set Sr,τ is the neighborhood in Tp of a line lτ ⊂ Tp . There is one line lτ for each horosphere

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

347

τ 6 = σ∞ . Since the horosphere boundary components of p are indexed by rational numbers and the set of lines of Tp is indexed by elements of Qp , there are lines l ⊂ Tp such that no horosphere intersects Nbhdr (σ∞ ) over every point of l. Let l 0 be such a line. Choose two points x and y in p − Nbhdr (σ ). If x and y lie in H2 × l 0 , then there is a path connecting them. If this is not the case, let α be any path from x to a point x 0 ∈ H2 × l 0 , and β any path from y to a point y 0 ∈ H2 × l 0 . Let γ be a path in H2 × l 0 connecting x 0 and y 0 . The composite path β −1 ◦ γ ◦ α connects x to y and lies in p − N bhdr (σ ), proving the theorem. 4.2. Proof of the boundary detection theorem. We now state and prove Theorem 4.5, which plays a major role in the proofs of Theorems A and B. Theorem 4.5 (Boundary detection theorem). Let f : p → q be a (K, C)quasi-isometry. There exist constants (K 0 , C 0 ) depending on (K, C) and the spaces p and q , with the following property. For every horosphere boundary component σ of p , there is a horosphere boundary component τ of q so that f |σ : σ → τ is a (K 0 , C 0 )-quasi-isometry. As in [FS] and [S2], the following lemmas form the two major components of the proof of the boundary detection theorem. In these lemmas, f : p → q is a (K, C)-quasi-isometry. Lemma 4.6. There exists a constant  depending on (K, C) and the spaces p and q , with the following property. For every horosphere boundary component σ of p , there is a horosphere boundary component τ of q so that τ ⊂ Nbhd (f (σ )). Proof. The proof follows verbatim from the proof of Lemma 4.5 of [FS], with Lemma 4.4 above replacing Lemma 4.4 of [FS]. Lemma 4.7. There exists a constant  0 depending on (K, C) and the spaces p and q , with the following property. For every horosphere boundary component σ of p , there is a horosphere boundary component τ of q so that f (σ ) ⊂ Nbhd 0 (τ ). Proof. Consider a horosphere boundary component σ of p . From Lemma 4.6, there is a horosphere boundary component τ of q and a constant  so that τ ⊂ N bhd (f (σ )). Define a map ψ : τ → σ by ψ(y) = x ∈ σ , where x is any point so that f (x) is metrically closest to y. If there is more than one such point, choose randomly. From Lemma 4.6, we see that ψ differs from the coarse inverse f −1 of f by at most a constant. From Proposition 4.3, we know that ψ(τ ) separates H2 ×Tq . So for some constant δ 0 , the horosphere σ is contained in Nbhdδ 0 (ψ(τ )), that is, every point of σ is within a constant δ 0 of some point x ∈ ψ(τ ) that maps to a point f (x) within  of a point of τ . By enlarging δ 0 to a constant  0 , we see that f (σ ) ⊂ Nbhd 0 (τ ). Since all the horospheres are isometric, the constant  0 is independent of the choice of σ .

348

JENNIFER TABACK

Proof of the boundary detection theorem. Apply Lemma 4.7 to both f and f −1 . Compose f with a nearest point projection to obtain a quasi-isometric embedding f 0 = f |σ : σ → τ . As in §4.1, embed σ and τ isometrically in (R3 , d) and extend f 0 to a quasi-isometric embedding fˆ : (R3 , d) → (R3 , d). (Recall that d was the bounded geometry metric on R3 described in §4.1.) From the packing theorem, this map is a (K 0 , C 0 )-quasi-isometry, where the pair (K 0 , C 0 ) depends on (K, C). Then by the construction of fˆ, the map f 0 must also be a (K 0 , C 0 )-quasi-isometry. It is now clear that closeness lines are preserved under quasi-isometry. 4.3. Proof of Theorem B. Using Theorem 4.5, we prove Theorem B. Proof of Theorem B. If PSL2 (Z[1/p]) and PSL2 (Z[1/q]) are commensurable, then they are automatically quasi-isometric. Let f : PSL2 (Z[1/p]) → PSL2 (Z[1/q]) be a (K, C)-quasi-isometry. Construct the spaces p and q corresponding to PSL2 (Z[1/p]) and PSL2 (Z[1/q]), respectively. Then f induces a quasi-isometry, also denoted f , from p to q . Theorem 4.5 allows us to restrict f to a quasi-isometry f 0 on horospheres. In §2.3 we showed that a horosphere of H2 × Tp has the geometry of the group BS(1, p). Hence f 0 is a quasi-isometry from BS(1, p) to BS(1, q). According to Theorem 2.1, we must have p = q for such a quasi-isometry to exist. 5. Theorems A and C. Every commensurator of PSL2 (Z[1/p]) acts as a quasiisometry of p . To prove Theorem A, we must show that by composing an element f ∈ QI(PSL2 (Z[1/p])) with a specific commensurator, we obtain a map that is a bounded distance from the identity map. It is Theorem 5.3 (stated below) that tells us which commensurator to choose for this purpose. In the appendix, we show that the commensurator group of PSL2 (Z[1/p]) in PSL2 (R) × PSL2 (Qp ) is PSL2 (Q), where PSL2 is viewed as an algebraic group as in §2.2. In all that follows, we assume that f : p → p is a (K, C)-quasi-isometry that has been changed by a bounded amount using the “connect the dots” procedure so that it is continuous. (See, e.g., [FS].) Since Comm(PSL2 (Z[1/p])) = PSL2 (Q) acts transitively on pairs of distinct points of R ∪ {∞}, we can assume that f has been composed with a commensurator so that f (σ∞ ) = σ∞ and f (σ0 ) = σ0 . (Note that these horospheres are not necessarily fixed pointwise.) 5.1. Action rigidity. The following notation and lemmas parallel [S2]. In [S2] the action rigidity theorem is proved for lattices in Rn . A geometric intuition is provided for lattices in R×R, which we follow here, although in R×Qp the analogous results must be proved algebraically rather than geometrically. The arguments below have been adapted from those in [S2] to account for the p-adic factor. We now define a boundary of the space H2 ×Tp . Let t be the common endpoint of any two lines in Tp and consider ∂∞ (H2 × t) − {∞} ∼ = R. Recall from §2.3 that we can consider σ∞ ⊂ ∂p as a quasi-isometrically embedded copy of the 2-complex

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

349

Xp associated to BS(1, p). So we can refer to the lower boundary ∂l (σ∞ ) of σ∞ (resp., the upper boundary ∂ u (σ∞ )) as the lower (resp., upper) boundary of Xp . The inclusion i : σ∞ → H2 × Tp induces an identification between ∂l (σ∞ ) and the copy of R described above. Since f (σ∞ ) = σ∞ , the quasi-isometry f induces bilipschitz maps fl and f u on ∂l (σ∞ ) and ∂ u (σ∞ ) of σ∞ , respectively. In addition, f induces a map on R = ∂∞ (H2 ×t)−{∞}, which determines the permutation of the horospheres of H2 × Tp under f . Hence, from the identification induced by the inclusion map i, the map fl is exactly the map that determines the permutation of the horospheres under f . Let 1Q denote the lattice in R × Qp given by the diagonal {(a, a) | a ∈ Q} ⊂ R × Qp , and 1 the sublattice {(b, b) | b ∈ Z[1/p]}. Clearly 1 ⊂ 1Q . We view the first coordinate of the pair (b, b) ∈ 1Q as b ∈ Q ⊂ R denoting the basepoint of a horosphere of H2 × Tp , and the second coordinate b ∈ Q ⊂ Qp as the point in Qp determined by the closeness line of σb and σ∞ . Using the identification described above, we see that the map induced by f on Q ⊂ R is exactly fl |Q . We can view Q ⊂ Qp as a subset of ∂ u (σ∞ ). Since f (σ∞ ) = σ∞ , the map induced by f on Q ⊂ Qp is exactly f u |Q . The maps f u |Q and fl |Q are identical because closeness lines are preserved under quasi-isometry and f (σ∞ ) = σ∞ . Hence f induces a map of 1Q given by (fl |Q , f u |Q ), and we use a single coordinate for points of 1Q . Let φ denote the common restriction of fl and f u to Q. Then φ is K0 -bilipschitz for some constant K0 depending on the pair (K, C). ¡ 0
. We make the following Let H be the cyclic group generated by the matrix p0 1/p definitions relating to a group-invariant diameter function that allows us to state Theorem 5.3. Definitions 1. For any subset S of R × Qp and H as above, define the H -invariant diameter of S by ¡
δH (S) = inf diam T (S) . T ∈H

For the remainder of this paper, we write δ for δH . 2. For subsets S1 , S2 of R×Qp , we say that the map φ : S1 → S2 is quasi-adapted to δ if there exists a map α : N → N such that for any compact set V , ¡
δ(V ) ≤ k ⇒ δ φ(V ) ≤ α(k) and

¡
δ φ(V ) ≤ k ⇒ δ(V ) ≤ α(k).

3. Let S be a subset of 1Q . We say that S has bounded height if S ⊂ (1/M)1 for some M ∈ Z+ with (M, p) = 1. 4. A bijection φ : 1Q → 1Q is said to be quasi-integral if both φ and φ −1 take sets of bounded height to sets of bounded height.

350

JENNIFER TABACK

5. A bijection φ : 1Q → 1Q is said to be quasicompatible with H if φ is quasiintegral and, when restricted to sets of bounded height, both φ and φ −1 are quasiadapted to δ. In a sequence of lemmas, we show that the bilipschitz map φ : 1Q → 1Q obtained from the original quasi-isometry f is quasicompatible with H . Associate to each point of 1Q the horosphere boundary component of p based at that point. Then the action of 0∞ on ∂p induces an action on 1Q . In particular, this action preserves denominators, that is, 0∞ · (1/M)1 ⊂ (1/M)1. Consider the diameter function on subsets S of 1Q defined by δ0 (S) = inf diam T (S). T ∈0∞

Also consider the following diameter function, with S as above. Let δB (S) be the diameter in p of the smallest metric ball that intersects all horosphere boundary components based at points of S. We now show that these two diameter functions are quasi-identical when restricted to (1/M)1; that is, if δ0 (S) is small for a subset S of 1Q , then δB (S) is bounded and vice versa. Lemma 5.1. The restrictions of δ0 and δB to (1/M)1 are quasi-identical. Proof. Let S be a subset of 1Q so that δB (S) is small, say, δB (S) = . Then there exists a point x ∈ p that is within  of all horosphere boundary components based at points of S. Since p / PSL2 (Z[1/p]) is compact by construction, there are only a finite number of choices for the set S (modulo Aut(1)). Thus δ0 (S) must be bounded. Now suppose δ0 (S) is small. By compactness, there are only finitely many choices for S (modulo Aut(1)). Hence δB (S) is bounded. Lemma 5.2. For every M ∈ Z+ , there exists M 0 ∈ Z + depending on the bilipschitz constant K0 of φ and the space p , so that φ((1/M)1) ⊂ (1/M 0 )1. Proof. Consider the action of 0∞ = StabPSL2 (Z[1/p]) (∞) on (1/M)1. This action preserves denominators, that is, 0∞ · (1/M)1 ⊂ (1/M)1. In particular, under this action, (1/M)1/0∞ consists of a finite set of points, which we view as a finite collection of horospheres σ1 , . . . , σn ⊂ ∂p . Choose a point x ∈ σ∞ and consider the smallest metric ball in p based at x that intersects all of the σi . Let  be the radius of this ball. There is an  0 (depending on ) so that the  0 ball around f (x) must intersect all of the f (σi ). Thus there are only a finite number of choices of horospheres in ∂p for the images f (σi ). It follows that there is a number M 0 ∈ N so that the collection of horospheres {f (σi )} is based at points in (1/M 0 )1. Since 0∞ preserves denominators, the image of σα (for any α ∈ (1/c)1) must be a horosphere based at a point of (1/M 0 )1. This is equivalent to saying that φ((1/M)1) ⊂ (1/M 0 )1. Lemma 5.2 shows that φ is quasi-integral, and since φ is a bilipschitz map of both R and Qp , it is quasi-adapted to δ. Hence φ is quasicompatible with H .

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

351

We can now state Theorem 5.3. We say that a map ψ : R×Qp → R×Qp is affine if its restriction to each factor is affine. This theorem is a 2-dimensional S-arithmetic version of the action rigidity theorem of Schwartz [S2]. Theorem 5.3 (Action rigidity). Let 1Q ⊂ R סQp be
the lattice {(a, a) | a ∈ Q} 0 and let H be the group generated by the matrix p0 1/p . Then any bilipschitz map φ : 1Q → 1Q that is quasicompatible with H is the restriction of an affine map of R × Qp . 5.2. The parallelogram lemma. The key lemma in the proof of Theorem 5.3 is Lemma 5.4 (the parallelogram lemma). It is this lemma that is proved geometrically in [S2, Section 6.4] for lattices in R × R. We follow the same outline, proving each step algebraically rather than graphically. Let 1Q be as in §5.1 and let φ : 1Q → 1Q be quasicompatible with H and K0 -bilipschitz. We now make some preliminary definitions. Let M ∈ Z+ . A parallelogram P in (1/M)1 is a quadruple of points   a Definition. b with a, b, c, d ∈ (1/M)1 satisfying a − c = b − d. c d  φ(b)  We say that φ(P ) is the quadruple φ(a) φ(c) φ(d) . The goal of Lemma 5.4 is to determine when φ(P ) is again a parallelogram. We first define two quantities associated to a parallelogram P that are invariant under the group action and translation.   Definition. Let P = ac db be a parallelogram. The H -invariant perimeter of P is given by per(P ) = δ(a ∪ b) + δ(a ∪ c) where δ is the H -invariant diameter function defined in §5.1. The shape of P is given by shape(P ) = |ν(b − a) − ν(c − a)| where the p-adic valuation ν is defined for pn x/y ∈ Qp , where (x, p) = 1 and (y, p) = 1, by ν(p n x/y) = n. For any T ∈ H and x ∈ 1Q , we have per(P ) = per(T (P ) + x) and shape(P ) = shape(T (P ) + x).  0 0   Let P = ac db be a parallelogram in (1/M)1 such that φ(P ) = ac0 db 0 is again a parallelogram. Since φ is quasicompatible with H , there exists a constant D (depending on per(P ) and K) such that δ(a 0 ∪ b0 ) + δ(a 0 ∪ c0 ) ≤ D. By symmetry, we also have δ(c0 ∪ d 0 ) + δ(b0 ∪ d 0 ) ≤ D. From Lemma 5.2 we obtain a constant k such that φ((1/M)1) ⊂ (1/k)1. We now describe the points x ∈ (1/k)1, with (k, p) = 1, satisfying δ(0 ∪ x) ≤ D: o n 1 Sk,D = x ∈ 1 | x 6= 0 and δ(0 ∪ x) ≤ D . k

352

JENNIFER TABACK

The set Sk,D is the orbit under H of a finite set of points of (1/k)1, denoted {ai /kp ri }i∈I , where (ai , p) = 1. Since δ is invariant under translation, the points of (1/k)1 within D of some point y ∈ (1/k)1 are given by Sk,D + y. We use the following notation in the statement of Lemma 5.4. Let B2 = max{ai }, B1 = max {ai − aj }, i,j ∈I i∈I 
¡
ª B3 = max ν(ai − aj , ν ai + aj , i,j ∈I

and  ¡
ª B4 = max ν ai + aj − ak i,j,k∈I

where the ai are as above. Lemma 5.4 (Parallelogram lemma). Let φ : 1Q → 1Q be quasicompatible with H and K0 -bilipschitz. Let logp (K0 ) = R and L ∈ N with (L, p) = 1. If P is a parallelogram in (1/L)1 with per(P ) ≤ L where

and

shape(P ) > s0

 ª s0 = max 2 logp (B1 + B2 ) + 2R, 3B32 + 2R, 2B4 + 2R ,

then φ(P ) is also a parallelogram.  0 0   Proof. Let P = ac db and φ(P ) = ac0 db 0 . We are assuming that φ(0) = 0, so without loss of generality translate P so that a = 0 and hence a 0 = 0. As stated above, there is a constant D so that δ(0 ∪ b0 ) + δ(0 ∪ c0 ) ≤ D and δ(b0 ∪ d 0 ) + δ(c0 ∪ d 0 ) ≤ D. We can write b0 = pn x/k and c0 = p m y/k where x = ai and y = aj for some ai , aj as above. Then d 0 can be expressed in one of two ways. It is δ-close to both b0 and c0 , and hence of the form pn x/k + p N z/k and also pm y/k + p M w/k where z = ai , w = aj for some i, j ∈ I . If φ(P ) is to be a parallelogram, we must show that d 0 = b0 + c0 . Note that x, y, z, w above are all relatively prime to p. We first obtain a lower bound on n − m, with n, m as above. We can write b = p n0 x0 /L and c = pm0 y0 /L, with (x0 , p) = 1 and (y0 , p) = 1, chosen so that shape(P ) = n0 − m0 . We know by assumption that n0 − m0 > s0 . Since φ is a K0 bilipschitz map on Q ⊂ Qp and φ(0) = 0, letting R = logp (K0 ), we obtain  x x0  p −n0 −R ≤ φ pn0 = pn ≤ p−n0 +R . p L k p Hence n0 − R ≤ n ≤ n0 + R; similarly, m0 − R ≤ m ≤ m0 + R. Thus, (n0 − m0 ) − 2R ≤ n − m ≤ (n0 − m0 ) + 2R. Using the fact that n0 − m0 > s0 , we see that

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

353

ª  n − m ≥ s = max 2 logp (B1 + B2 ), 3B32 , 2B4 . Considering all possible equalities among the exponents n, N, m, and M, we compare the p-norms of both sides of the equation p n x + pN z = p m y + pM w. Suppose first that n > N . Since the p-norms of both sides must be equal, we know that N = min{m, M}. We immediately rule out n = m because n − m > 0. The possible equalities divide into five cases, the first of which (below) gives the conclusion N = m, y = z, and x = w, which implies that φ(P ) is again a parallelogram. In all other cases we obtain a contradiction. In comparing these p-norms, we use the bound on the difference of exponents n − m given by the choice of s0 . Case 1. Suppose N = m and M 6= m. Then the above equation simplifies to p n−m x = pM−m w + (y − z). We know that n − m > s. Suppose y − z 6= 0. The highest power of p dividing the right-hand side of the equation is bounded by B3 . But by the choice of s, we have n − m > B3 ; hence this situation cannot occur. If by chance M − m = B3 , and y − z = pB3 t for some integer t, then note that ν(w + t) is also bounded. If this is the case, change the initial value of s0 by the quantity ν(w + t), and we may assume that we are not in this case. So we must have y −z = 0, that is, z = y and p n−m x = pM−m w. Thus n = M and x = w, and φ(P ) is a parallelogram. Case 2. Suppose N = M and M 6= m. Then the above equation simplifies to
¡ p m−M pn−m x − y = w − z. Since m−M > 0 and w −z ≤ B1 , our choice of s0 ensures that pm−M (pn−m x −y) > B1 , which is a contradiction. The remaining three cases are analogous to this one. 5.3. Proof of the action rigidity theorem. We now prove Theorem 5.3. This proof parallels the proof of the action rigidity theorem of [S2] but uses the definitions of per(P ) and shape(P ) given above. Proof of Theorem 5.3. Fix q ∈ N, with (q, p) = 1, and let S be a generating set for (1/q)1 containing 1/q. Let s0 be as in Lemma 5.4, and let H (S) denote the orbit of S under H . Given x, y ∈ (1/q)1, we say that (x, y) is a distinguished pair if x − y ∈ H (S). Let C = maxa∈H (S) {q, 2δ(0  a)}.  ∪ We write P ((x, y), (z, w)) if P = xz wy is a parallelogram with P ⊂ (1/q)1, per(P ) ≤ C, and shape(P ) ≥ s0 . Then by the parallelogram lemma, φ(P ) is also a parallelogram, that is, φ(x) − φ(z) = φ(y) − φ(w). We write P ((x, y), (z, w)) if there is a finite sequence of pairs (ai , bi ) for i = 0, . . . , n with (a0 , b0 ) = (x, y) and (an , bn ) = (z, w) such that P ((ai , bi ), (ai+1 , bi+1 )).

354

JENNIFER TABACK

This means that we have a sequence of parallelograms between the two given pairs of points, each satisfying the conclusions of the parallelogram lemma, and with any two consecutive parallelograms in this sequence sharing a common side. Concatenating these intermediate parallelograms allows us to conclude that the parallelogram given by the original pairs also satisfies the conclusions of the Lemma 5.4. We first need the following lemma. Lemma 5.5. Let a ∈ (1/q)1 be arbitrary, and let (u, v) be a distinguished pair. Then P ((u, v), (u + a, v + a)).  u v  Proof. Consider Py = u+y v+y for any y ∈ H (S). By construction, we have per(Py ) ≤ C. Let Y = Y (u, v) ⊂ H (S) denote those y ∈ H (S) such that shape(Py ) ≥ s0 . Let 60 Y be the sublattice in 1Q generated by Y and 6Y = 60 Y ∩ (1/q)1. We first show that P ((u, v), (u + x, v + x)) for any x ∈ 6Y and then show that 6Y = (1/q)1. We prove this first assertion by induction. If x ∈ Y , it is certainly true that P ((u, v), (u + x, v + x)). If x ∈ 6Y , write x = x 0 + y for y ∈ Y , and we have P ((u, v), (u + x 0 , v + x 0 )) by the induction hypothesis. Consider the parallelogram     u + x0 v + x0 v + x0 u + x0 . = P= u + x0 + y v + x0 + y u+x v +x We see that shape(P ) = shape(Py ) ≥ s0 . Also, δ(u + x 0 ∪ u + x) = δ(0 ∪ y) ≤ C/2 and δ(u + x 0 ∪ v + x 0 ) = δ(u ∪ v) ≤ C/2. So we have per(P ) ≤ C. Therefore, we know that P ((u + x 0 , v + x 0 ), (u + x, v + x)). Now we show that 6Y = (1/q)1. We know that shape(Py ) = |ν(u − v) − ν(y)|. Let ω = ν(u−v). Then Y = {y ∈ H (S) | |ω −ν(y)| ≥ s0 }. Hence for large enough n, we have (1/p n )(1/q) ∈ Y , so 6Y = (1/q)1, proving the lemma. Given x ∈ (1/q)1, we can create a distinguished pair by taking (x, x + a) for any a ∈ S. So for any x, y ∈ (1/q)1, we know that P ((x, x + a), (y, y + a)), or φ(x +a)−φ(x) = φ(y +a)−φ(y). Since S generates (1/q)1, we know that for any x, y, z ∈ (1/q)1 we have P ((x, x + z), (y, y + z)) or φ(x + z) − φ(x) = φ(y + z) − φ(y). In particular, since φ(0) = 0, we know that φ|(1/q)1 is multiplication by a constant Cq . Let q1 , q2 ∈ N, with (q1 , p) = (q2 , p) = 1. Then we must have Cq1 q2 = Cq1 = Cq2 . Hence there is a constant α such that the map φ|1Q is multiplication by α. It follows that φ is the restriction of an affine map of R × Qp . 5.4. Proof of Theorem A. We now use Theorem 5.3 to prove Theorem A. Proof of Theorem A. It is clear that every commensurator of PSL2 (Z[1/p]) gives rise to a unique quasi-isometry of p . Given f ∈ QI(PSL2 (Z[1/p])), we now choose a commensurator g ∈ Comm(PSL2 (Z[1/p])) so that the composition g ◦ f is a bounded distance from the identity map.

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

355

Let f ∈ QI(PSL2 (Z[1/p])). Then f induces a quasi-isometry from p to itself. Compose f with a commensurator so that f (σ∞ ) = σ∞ and f (σ0 ) = σ0 . Combining Theorems 2.2 and 5.3, we know that f induces a map fl on the lower boundary of σ∞ , which is multiplication by a constant α and determines the permutation of the horospheres under f . Recall from §2.2 that we view PSL2 (Q) as the Q-points G0Q , where G0 = ¡ 1/√α 0
Ad(SL2 (C)). Compose f with the commensurator g = Ad 0 √α . Then the permutation of the horospheres under g ◦ f is the identity permutation. We must now show that for any m ∈ p , the image f (m) lies in an -ball around m, where  is independent of the choice of m. The set of points within n units of any three distinct horospheres has bounded diameter, independent of the choice of horospheres. Also, any quasi-isometry f has the property that ¡ ¡
¡

hd f Nbhd(A) ∩ N hbd(B) , Nbhd f (A) ∩ N bhd(B) < ∞ where hd denotes Hausdorff distance. So if x ∈ p lies in the intersection of the n-neighborhoods of three horospheres, then there is a constant n0 so that f (x) ∈ Bn0 (x). Thus f ◦ g is a bounded distance from the identity map, so the natural map 9 : Comm(PSL2 (Z[1/p])) → QI(PSL2 (Z[1/p])) is an isomorphism. 5.5. Proof of Theorem C. We now prove Theorem C. The proof uses some standard techniques from the study of quasi-isometric rigidity for lattices in semisimple Lie groups, as well as Theorem A and Margulis’s S-arithmetic superrigidity theorem. It is similar to the proof of Corollary 1 in [FS]. Proof of Theorem C. Since PSL2 (Z[1/p]) is quasi-isometric to the space p , we get a quasi-isometry f : 0 → p . To obtain the exact sequence of the theorem, we find a representation ρ : 0 → PSL2 (R) × PSL2 (Qp ) with finite kernel so that ρ(0) is a nonuniform lattice in PSL2 (R) × PSL2 (Qp ) and, hence, is commensurable to PSL2 (Z[1/p]) by Theorem 5.6 (see [M]). Since 0 acts on itself by isometries via left multiplication Lγ , for all γ ∈ 0, we obtain a uniform family of quasi-isometries fγ = f ◦ Lγ ◦ f −1 : p −→ p . From Theorem A, we can think of fγ as a commensurator and, hence, a bounded distance from an isometry. For each fγ , compose with a bounded alteration Bγ to obtain a map ¡
ρ : 0 −→ Isom H2 × Tp . First we show that ρ is a homomorphism. Suppose ρ(γ ) is a bounded distance from the identity isometry, that is, dH2 ×Tp (x, ρ(γ ) · x) <  for all x ∈ H2 × Tp . But then it follows that ρ(γ ) is the identity; hence ρ is a homomorphism. We must show that ρ(γ ) is a lattice and that ρ has finite kernel.

356

JENNIFER TABACK

Since f (0) is a net in p , the image ρ(0) = {fγ } = {Bγ ◦ f ◦ Lγ ◦ f −1 | γ ∈ 0} acts cocompactly on p . It follows that ρ(0) acts on H2 × Tp with cofinite volume. Choose a basepoint x ∈ p and consider dH2 ×Tp (x, ρ(γ ) · x). For finitely many γ ∈ 0, ρ(γ ) moves x a bounded amount; that is, there exists a constant C 0 > 0 so that dH2 ×Tp (x, ρ(γ ) · x) ≤ C 0 for some finite set {γi } ⊂ 0. (To find C 0 , we are using the fact that {fγ } is a uniform family of quasi-isometries.) In particular, the distance dH2 ×Tp (x, ρ(γ ) · x) = 0 for only finitely many γ ∈ 0. But if dH2 ×Tp (x, ρ(γ ) · x) > 0, then ρ(γ ) is not the identity isometry. Hence ρ has finite kernel. The previous paragraph showed that ρ(0) is discrete, so we can now construct the short exact sequence 1 −→ N −→ 0 −→ 3 −→ 1 where N = ker(ρ) and 3 = ρ(0) is a lattice in PSL2 (R) × PSL2 (Qp ). To complete the proof, we apply the following theorem of Margulis [M], stated in the case n = 2, from which we conclude that the lattice 3 obtained above is commensurable to PSL2 (Z[1/p]). Theorem 5.6 [M]. Let p be a prime and let n ∈ N + , with n ≥ 2. If a subgroup 0 of SL2 (Q) under the diagonal embedding in SL2 (R) × SL2 (Qp ) is a lattice in SL2 (R) × SL2 (Qp ), then the subgroups 0 and SL2 (Z[1/p]) are commensurable. The proof of Theorem 5.6 uses Margulis’s superrigidity theorem for S-arithmetic groups. Theorem 5.6 completes the proof of Theorem C. Appendix We now compute the commensurator group of PSL2 (Z[1/p]) in PSL2 (R) × PSL2 (Qp ), denoted Comm(PSL2 (Z[1/p])). Proposition. The commensurator subgroup for PSL2 (Z[1/p]) is given by    1 = PSL2 (Q) Comm PSL2 Z p ¡
 ª = (a, a) | a ∈ PSL2 (Q) ⊂ PSL2 (R) × PSL2 Qp . 

Proof. Since any commensurator g must preserve p and, hence, preserve the horospheres, that is, map Q ∪ {∞} to Q ∪ {∞}, we must have g ∈ PSL2 (Q). Let (a, b) ∈ Comm(PSL2 (Z[1/p])) act by conjugation by a on H2 and by b on Tp . Then (Id, b) ∈ / Comm(PSL2 (Z[1/p])), for b 6= Id, because a trivial action on the H2 factor induces a trivial action on Tp . By bounding denominators, one shows that (a, a) ∈ Comm(PSL2 (Z[1/p])), for a ∈ PSL2 (Q). If (a, b) ∈ Comm(PSL2 (Z[1/p])), with a, b ∈ PSL2 (Q), composition with (a −1 , a −1 ) ∈ Comm(PSL2 (Z[1/p])) implies that

QUASI-ISOMETRIC RIGIDITY FOR PSL2 (Z[1/p])

357

(Id, ba −1 ) ∈ Comm(PSL2 (Z[1/p])), which is a contradiction. Thus    1 = PSL2 (Q) Comm PSL2 Z p ª ¡
 = (a, a) | a ∈ PSL2 (Q) ⊂ PSL2 (R) × PSL2 Qp . 

References [F] [FM] [FS] [G]

[LMR]

[M] [Mi] [S1] [S2] [Se]

B. Farb, The quasi-isometry classification of lattices in semisimple Lie groups, Math. Res. Lett. 4 (1997), 705–717. B. Farb and L. Mosher, A rigidity theorem for the solvable Baumslag-Solitar groups, with an appendix by D. Cooper, Invent. Math. 131 (1998), 419–451. B. Farb and R. Schwartz, The large-scale geometry of Hilbert modular groups, J. Differential Geom. 44 (1996), 435–478. M. Gromov, “Infinite groups as geometric objects” in Proceedings of the International Congress of Mathematicians, Vol. 1, 2 (Warsaw, 1983), PWN, Warsaw, 1984, 385– 392. A. Lubotzky, S. Mozes, and M. S. Raghunathan, Cyclic subgroups of exponential growth and metrics on discrete groups, C. R. Acad. Sci. Paris Sér. I Math. 317 (1993), 735–740. G. A. Margulis, Discrete Subgroups of Semisimple Lie Groups, Ergeb. Math. Grenzgeb. (3) 17, Springer-Verlag, Berlin, 1991. J. Milnor, A note on curvature and fundamental group, J. Differential Geom. 2 (1968), 1–7. R. Schwartz, The quasi-isometry classification of rank one lattices, Inst. Hautes Études Sci. Publ. Math. 82 (1996), 133–168. , Quasi-isometric rigidity and Diophantine approximation, Acta Math. 177 (1996), 75–112. J.-P. Serre, Trees, Springer-Verlag, Berlin, 1980.

Department of Mathematics, University of California at Berkeley, Berkeley, California 94720, USA; [email protected] Current: Department of Mathematics and Statistics, University at Albany, State University of New York, Albany, New York 12222, USA; [email protected]

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

A CATEGORIFICATION OF THE JONES POLYNOMIAL MIKHAIL KHOVANOV

To my teacher Igor Frenkel

Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360 2. Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 2.1. The ring R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 2.2. The algebra A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 2.3. Algebra A and (1 + 1)-dimensional cobordisms . . . . . . . . . . . . . . . . . . . . . . 367 2.4. Kauffman bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 3. Cubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 3.1. Complexes of R-modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 3.2. Commutative cubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 3.3. Skew-commutative cubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 3.4. Skew-commutative cubes and complexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375 4. Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 4.1. Reidemeister moves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 4.2. Constructing cubes and complexes from plane diagrams . . . . . . . . . . . . . . 377 4.3. Surfaces and cube morphisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 5. Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385 5.1. Left-twisted curl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385 5.2. Right-twisted curl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 5.3. The tangency move . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 5.4. Triple point move . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 6. Properties of cohomology groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 6.1. Some elementary properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 6.2. Computational shortcuts and cohomology of (2, n) torus links . . . . . . . . . 405 6.3. Link cobordisms and maps of cohomology groups . . . . . . . . . . . . . . . . . . . 409 7. Setting c to zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 7.1. Cohomology groups Ᏼi,j . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 7.2. Properties of Ᏼi,j : Euler characteristic, change of orientation . . . . . . . . . . 413 7.3. Cohomology of the mirror image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 Received 4 March 1998. Revision received 1 April 1999. 1991 Mathematics Subject Classification. Primary 57M25. Author’s work supported by National Science Foundation grant number DMS 9304580. 359

360

MIKHAIL KHOVANOV

7.4. Cohomology of the disjoint union and connected sum of knots . . . . . . . . 415 7.5. A spectral sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 7.6. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 7.7. An application to the crossing number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 8. Invariants of (1, 1)-tangles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 8.1. Graded A-modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 8.2. Nonclosed (1 + 1)-cobordisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421 8.3. (1, 1)-tangles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422 8.4. Invariants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 1. Introduction. During the past fifteen years many new structures have arisen in the topology of low-dimensional manifolds: the Jones and HOMFLY polynomials of links, Witten-Reshetikhin-Turaev invariants of 3-manifolds, Floer homology groups of homology 3-spheres, and Donaldson and Seiberg-Witten invariants of 4manifolds. These invariants of 3- and 4-manifolds naturally split into two groups. Members of the first group are combinatorially defined invariants of knots and 3manifolds, such as various link polynomials, finite-type invariants, and quantum invariants of 3-manifolds. Floer and Seiberg-Witten homology groups of 3-manifolds and Donaldson-Seiberg-Witten invariants of 4-manifolds constitute the second group. While invariants from the first group have a combinatorial description and in each instance can be computed algorithmically, invariants from the second group are understood through moduli spaces of solutions of suitable differential-geometric equations and the infinite-dimensional Morse theory and have evaded all attempts at a finite combinatorial definition. These invariants have been computed for many 3and 4-manifolds, yet the methods of computation use some extra structure on these manifolds, such as Seifert fibering or complex structure. The problem of finding an algorithmic construction of these invariants remains open. It is probably due to this striking difference in the origins and computational complexity that so far not many direct relations have been found between invariants from different groups. The most notable connection is the Casson invariant of homology 3-spheres (see [AM]), which is equal to the Euler characteristic of Floer homology (refer to [F]). Yet the Casson invariant is computable and intimately related to the Alexander polynomial of knots and to Witten-Reshetikhin-Turaev invariants (see [M]), which are examples of invariants from the first group. A similar relation has recently been discovered between Seiberg-Witten invariants and Milnor torsion of 3manifolds (see [MT]). In summary, Euler characteristics of Floer and Seiberg-Witten homology groups bear an algorithmic description, while no such procedure is known for finding the groups themselves. A speculative question now comes to mind: Quantum invariants of knots and 3manifolds tend to have good integrality properties. Can these invariants be interpreted as Euler characteristics of some homology theories of 3-manifolds? Our results suggest that such an interpretation exists for the Jones polynomial of

A CATEGORIFICATION OF THE JONES POLYNOMIAL

361

links in 3-space (see [Jo]). We give an algorithmic procedure that to a generic plane projection D of an oriented link L in R3 associates cohomology groups Ᏼi,j (D) that depend on two integers i, j. If two diagrams D1 and D2 of the same link L are related by a Reidemeister move, a canonical isomorphism of groups Ᏼi,j (D1 ) and Ᏼi,j (D2 ) is constructed. Thus, isomorphism classes of these groups are invariants of the link L. These groups are finitely generated and may have nontrivial torsion. Tensoring these groups with Q, we get a 2-parameter family {dimQ (Ᏼi,j (D) ⊗ Q)}i,j ∈Z of integervalued link invariants. From our construction of groups Ᏼi,j (D), we immediately conclude that the graded Euler characteristic
  (−1)i q j dimQ Ᏼi,j (D) ⊗ Q (1) i,j

is equal, up to a simple change of variables, to the Jones polynomial of L, multiplied by q + q −1 . We conjecture that not just the isomorphism classes of Ᏼi,j (D) but the groups themselves are invariants of links. We will consider this conjecture in a subsequent paper. To define cohomology groups Ᏼi,j (D), we start with the Kauffman state sum model (see[Ka]) for the Jones polynomial and then, roughly speaking, turn all integers into complexes of abelian groups. In the Kauffman model a link is projected generically onto the plane so that the projection has a finite number of double transversal intersections. There are two ways to “smooth” the projection near the double point, that is, erase the intersection of the projection with a small neighbourhood of the double point and connect the four resulting ends by a pair of simple, nonintersecting arcs. See Figure 1. A diagram D with n double points admits 2n resolutions of these double points. Each of the resulting diagrams is a collection of disjoint simple closed curves on the plane. In [Ka] Kauffman associates the Laurent polynomial (−q − q −1 )k to a collection of k simple curves and then forms a weighted sum of these numbers over all 2n resolutions. After normalization, Kauffman obtains the Jones polynomial of the link L. The principal constant in this construction is −q −q −1 , the number associated

Figure 1

362

MIKHAIL KHOVANOV

to a simple closed curve. In our approach q + q −1 becomes a certain module A over the base ring Z[c]. In detail, we work over the graded ring Z[c] of polynomials in c, where c has degree 2, and we define A to be a free Z[c]-module of rank 2 with generators in degrees 1 and −1. This is the object we associate to a simple closed curve in the plane. Given a diagram D, to each resolution of all double points of D we associate the graded Z[c]-module A⊗k , where k is the number of curves in the resolution. Then we glue these modules over all 2n resolutions into a complex C(D) of graded Z[c]-modules. The gluing maps come from commutative algebra and cocommutative coalgebra structure on A. When two diagrams D1 and D2 are related by a Reidemeister move, we construct a quasi-isomorphism between the complexes C(D1 ) and C(D2 ). The cohomology groups H i (D) of the complex C(D) are graded R-modules, and we prove that isomorphism classes of H i (D) do not depend on the choice of a diagram of the link. We then look at some elementary properties of these groups and introduce several cousins of H i (D). Outline of the paper. In Section 2 we define an algebra A over the ring R = Z[c] and use A to construct a 2-dimensional topological quantum field theory. In our case this topological quantum field theory is a functor from the category of 2dimensional cobordisms between 1-dimensional manifolds to the category of graded Z[c]-modules. In Section 2.4 we review the Kauffman state sum model of [Ka] for the Jones polynomial of oriented links. In Section 3 we review the notions of a commutative cube and a map between commutative cubes. In Section 4.1 we review Reidemeister moves. In Section 4.2 we associate a complex of Z[c]-modules to a plane diagram of a link. As an intermediate step, to a diagram D we associate a commutative cube VD of Z[c]-modules and maps between them. That is, we consider an n-dimensional cube with its edges standardly oriented, and, given a plane projection with n double points of a link, to each vertex of the cube we associate a Z[c]-module and to each oriented edge a map of modules so that all square facets of this diagram are commutative squares. This is done in Section 4.2. In the same section we pass from commutative cubes to complexes of Z[c]-modules and to a diagram D we associate a complex C(D) of graded Z[c]-modules. In Section 5, which is the technical core of the paper, to a Reidemeister move between diagrams D1 and D2 we associate a quasi-isomorphism between the complexes C(D1 ) and C(D2 ). These isomorphisms seem to be canonical. We conjecture that the quasi-isomorphisms are coherent, which would naturally associate cohomology groups to links. Our quasi-isomorphism result shows that the isomorphism classes of the cohomology groups are invariants, but not necessarily that the groups are functorial under link isotopy. We define H i (D) to the be ith cohomology group of the complex C(D). These cohomology groups are graded Z[c]-modules, and the isomorphism class of each H i (D) is a link invariant. If we split these groups into the direct sum of their graded components,

A CATEGORIFICATION OF THE JONES POLYNOMIAL

H i (D) =



H i,j (D),

363 (2)

j ∈Z

we get a two-parameter family of “abelian-group-valued” link invariants. These results are stated at the end of Section 4.2, as Theorems 1 and 2. For a diagram D, the groups H i,j (D) are trivial for j 0. Moreover, for each j only finitely many of the groups H i,j (D) are nonzero. Consequently, the graded Euler characteristic of C(D), defined as    
(−1)i q j dimQ H i,j (D) ⊗ Q , χ  C(D) =

(3)

i,j ∈Z

is well defined as a Laurent series in q. Since our construction of C(D) lifts Kauffman’s construction of the Jones polynomial, it is not surprising that the graded Euler characteristic of C(D) is related to the Jones polynomial. Namely, χ (C(D)), multiplied by (1 − q 2 )/(q + q −1 ), is equal, after a simple change of variables, to the Jones polynomial of L. If a link in R3 has cohomology groups, then cobordisms between links, that is, surfaces embedded in R3 ×[0, 1], should provide maps between the associated groups. A surface embedded in the 4-space can be visualized as a sequence of plane projections of its 3-dimensional sections (see [CS]). Given such a presentation J of a compact oriented surface S properly embedded in R3 × [0, 1] with the boundary of S being the union of two links L0 ⊂ R3 × {0} and L1 ⊂ R3 × {1}, we explain in Section 6.3 how to associate to J a map of cohomology groups θJ : H i,j (D0 ) −→ H i,j +χ (S) (D1 ),

i, j ∈ Z,

(4)

χ (S) being the Euler characteristic of the surface S and D0 and D1 being diagrams of L0 and L1 induced by J. We conjecture that, up to an overall minus sign, this map does not depend on the choice of J ; in other words, ±θJ behaves invariantly under isotopies of S. If this conjecture is true, we get a 4-dimensional topological quantum field theory, restricted to links in R3 and R3 × [0, 1]-cobordisms between them. Because the theory has a combinatorial definition, all cohomology groups and maps between them are algorithmically computable. If successful, this program can realize the Jones polynomial as the Euler characteristic of a cohomology theory of link cobordisms. In Section 7 we explain how a version of our construction, when the base algebra Z[c] is reduced to Z by taking c = 0, produces graded cohomology groups Ᏼi,j (D). The complex that is used to define Ᏼi,j (D) is given by tensoring C(D) with Z over Z[c]. As before, the isomorphism classes of these groups are invariants of links. These groups are “smaller” than the groups H i,j (D). In particular, for each D, these groups are nonzero for only finitely many pairs (i, j ) of integers. As with the groups

364

MIKHAIL KHOVANOV

H i,j (D), the graded Euler characteristic
  (−1)i q j dimQ Ᏼi,j (D) ⊗ Q ,

(5)

i,j

divided by q + q −1 , is equal to the Jones polynomial of the link represented by the diagram D. In Section 7.5 we exhibit a spectral sequence whose E1 -term is made of Ᏼ(D) and which converges to H (D). Apparently, H (D) is a kind of S 1 -equivariant version of the groups Ᏼi,j (D). In Section 7.7 we use these cohomology groups to reprove a result of Thistlethwaite on the crossing number of adequate links. Section 8 presents mild variations on cohomology groups H i (D) and Ᏼi,j (D). There we switch from links to (1, 1)-tangles. We consider the category A-mod0 of graded A-modules and grading-preserving homomorphisms between them. Given a plane diagram D of a (1, 1)-tangle L and a graded A-module M, in Section 8.3 we define cohomology groups H i (D, M), which are graded A-modules. The arguments of Sections 4–5 go through without a single alteration and show that isomorphism classes of H i (D, M) do not depend on the choice of D and are invariants of the underlying (1, 1)-tangle L. In fact, to every (1, 1)-tangle and an integer i we associate an isomorphism class of functors from the category of graded A-modules to itself. Motivations for this work and its relations to representation theory. What is the representation-theoretical meaning of the cohomology groups H i,j (D)? The Jones polynomial of links is encoded in the finite-dimensional representation theory of the quantum group Uq (sl2 ). It is shown in [FK] and [K] that the integrality and positivity properties of the Penrose-Kauffman q-spin networks calculus, of which the Jones polynomial is a special instance, are related to Lusztig canonical bases in tensor products of finite-dimensional Uq (sl2 )-representations. Lusztig’s theory [L], among other things, says that various structure coefficients of quantum groups can be obtained as dimensions of cohomology groups of sheaves on quiver varieties. This suggests a “categorification” of quantum groups and their representations; that is, there exist certain categories and 2-categories whose Grothendieck groups produce quantum groups and their representations. Crane and Frenkel [CF] conjecture that quantum sl2 invariants of 3-manifolds can be lifted to a 4-dimensional topological quantum field theory via canonical bases of Lusztig. They also introduce a notion of Hopf category and associated to it 4dimensional invariants. Representations of a Hopf category form a 2-category, and a relation between 2-categories and invariants of 2-knots in R4 are established in [Fs]. In a joint work with Bernstein and Frenkel [BFK], we propose a categorification of the representation theory of Uq (sl2 ) via categories of highest-weight representations for Lie algebras gln for all natural n. This approach can be viewed as an algebraic counterpart of Lusztig’s original geometric approach to canonical bases. Motivated by the geometric constructions of [BLM] and [GrL], we obtain a categorification of the Temperley-Lieb algebra and Schur quotients of U (sl2 ) via projective and Zuckerman

A CATEGORIFICATION OF THE JONES POLYNOMIAL

365

functors. We consider categories ᏻn that are direct sums of certain singular blocks of the category ᏻ for gln . Given a tangle L in the 3-space with n bottom and m top ends and a plane projection P of L, we associate to P a functor between derived categories D b (ᏻn ) and D b (ᏻm ). Properties of these functors suggest that their isomorphism classes, up to shifts in the derived category, are invariants of tangles. When the tangle is a link L, we expect to get cohomology groups Hi (L) as invariants of links. These groups are a special case of the cohomology groups constructed in this paper: conjecturally   Hi (L) = Ᏼi,j (L) ⊗ C . (6) j

Acknowledgements. This work was started during a visit to the Institut des Hautes Etudes Scientifiques and finished at the Institute for Advanced Study. I am grateful to these institutions for creating a wonderful working atmosphere. I am indebted to Joseph Bernstein for interesting discussions, to Greg Kuperberg for reading and correcting the first version of the manuscript and explaining to me a natural way to hide minus signs, and to Oliver Dasbach and Arkady Vaintrob for pointing out that Corollary 13 was proved by Thistlethwaite in [T]. On numerous occasions Igor Frenkel, who was my supervisor at Yale University, advised me to look for a lift of the Penrose-Kauffman quantum spin networks calculus to a calculus of surfaces in S4 . This work can be seen as a partial answer to his questions. It is a pleasure to dedicate this paper to my teacher. 2. Preliminaries 2.1. The ring R. Let R = Z[c] denote the ring of polynomials with integral coefficients. Introduce a Z-grading on R by deg(1) = 0,

deg(c) = 2.

(7)

Denote by R- mod0 the abelian category of graded R-modules. Denote the ith graded component of an object M of R- mod0 by Mi . Morphisms in the category R- mod0 are grading-preserving homomorphisms of modules. For n ∈ Z denote by {n} the automorphism of R- mod0 given by shifting the grading of a module down by n. Thus for a graded R-module N = ⊕i Ni , the shifted module N{n} has graded components N {n}i = Ni+n . In this paper we sometimes consider graded, rather than just grading-preserving, maps. A map α : M → N of graded R-modules is called graded of degree i if α(Mj ) ⊂ Ni+j for all j ∈ Z. Let R- mod be the category of graded R-modules and graded maps between them. This category has the same objects as the category R- mod0 but more morphisms. It is not an abelian category. A graded map α is a morphism in the category R- mod0 if and only if the degree of α is zero. At the end we favor grading-preserving maps, and when at some point we

366

MIKHAIL KHOVANOV

look at a graded map α : M → N of degree i, later we will make it grading-preserving by appropriately shifting the degree of one of the modules. For example, α gives rise to a grading-preserving map M → N{i}, also denoted α. Let M be a finitely generated graded R-module. As an abelian group, M is the direct sum of its graded components: M = ⊕j ∈Z Mj , where each Mj is a finitely generated abelian group. Define the graded Euler characteristic χ (M) of M by
  dimQ Mi ⊗Z Q q j . (8) χ (M) = j ∈Z

Since χ (R) = 1 + q 2 + q 4 + · · · =

1 , 1 − q2

(9)

χ (M) is not, in general, a Laurent polynomial in q, but an element of the Laurent series ring. Moreover, for any M as above, there are Laurent polynomials a, b ∈ Z[q, q −1 ] such that χ (M) = a +

b . 1 − q2

(10)

2.2. The algebra A. Let A be a free graded R-module of rank 2 spanned by 1 and X with deg(1) = 1,

deg(X) = −1.

(11)

We equip A with a commutative algebra structure with the unit 1 and multiplication 1X = X1 = X,

X 2 = 0.

(12)

We denote by ι the unit map R → A that sends 1 to 1. This map is a graded map of graded R-modules and it increases the degree by 1. We equip A with a coalgebra structure with a coassociative cocommutative comultiplication "(1) = 1 ⊗ X + X ⊗ 1 + cX ⊗ X,

(13)

"(X) = X ⊗ X

(14)

and a counit #(1) = −c,

#(X) = 1.

(15)

A, equipped with these structures, is not a Hopf algebra. Instead, the identity " ◦ m = (m ⊗ Id) ◦ (Id ⊗") holds.

(16)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

367

Grading deg, given by (7), (11), induces a grading, also denoted deg, on tensor powers of A by deg(a1 ⊗ · · · ⊗ an ) = deg(a1 ) + · · · + deg(an ) for a1 , . . . , an ∈ A.

(17)

Hereafter all tensor products are taken over the ring R unless specified otherwise. We next describe the effect of the structure maps ι, m, #, " on the gradings. We say that a map f between two graded R-modules V = ⊕Vn and W = ⊕Wn has degree k if f (x) ∈ Wn+k whenever x ∈ Vn . Proposition 1. Each of the structure maps ι, m, #, " is graded relative to the grading deg. Namely, deg(ι) = 1,

deg(m) = −1,

deg(#) = 1,

deg(") = −1.

(18)

Proposition 2. We have an R-module decomposition A ⊗ A = (A ⊗ 1) ⊕ "(A),

(19)

which respects the grading deg. 2.3. Algebra A and (1 + 1)-dimensional cobordisms. Consider the surfaces S21 , and S11 depicted in Figure 2. Each of the surfaces Sab defines a cobordism from a union of a circles to a union of b circles. We denote by ᏹ the category whose objects are closed 1-dimensional manifolds and whose morphisms are 2-dimensional cobordisms between these manifolds generated by the above cobordisms. Specifically, objects of ᏹ are enumerated by nonnegative integers Ob(ᏹ) = {n | n ∈ Z+ }. A morphism between n and m is a compact oriented surface S with boundary being the union of n + m circles. The boundary circles are split into two sets ∂0 S and ∂1 S with ∂0 S containing n and ∂1 S containing m circles. An ordering of elements of each of these two sets is fixed. The surface S is presented as a concatenation of disjoint unions of elementary surfaces, depicted in Figure 2. Morphisms are composed in the usual way by gluing boundary circles. Two morphisms are equal if the surfaces S, T representing these morphisms are diffeomorphic via a diffeomorphism that extends the identification ∂0 S ∼ = ∂ 0 T , ∂1 S ∼ = ∂1 T of their boundaries. ᏹ is a monoidal category with tensor product of morphisms defined by taking the disjoint union of surfaces. Let us construct a monoidal functor from ᏹ to the category R- mod of graded Rmodules and graded module maps. Assign graded R-module A⊗n to the object n, and to the elementary surfaces S21 , S12 , S01 , S10 , S22 , S11 assign morphisms m, ", ι, #, Perm, Id, respectively:       F S12 = ", F S01 = ι, F S21 = m, (20)       F S22 = Perm, F S11 = Id, F S10 = #, S12 , S01 , S10 , S22 ,

368

MIKHAIL KHOVANOV

S21

S12

S01

S10

S22

S11

Figure 2

where Perm : A ⊗ A → A ⊗ A is the permutation map, Perm(u ⊗ v) = v ⊗ u, and Id is the identity map Id : A → A. To check that F is well defined, one must verify that for any two ways to glue an arbitrary surface S in Mor(ᏹ) from copies of these six elementary surfaces, the two maps of R-modules, defined by these two decompositions of S, coincide. This follows from the commutative algebra and cocommutative coalgebra axioms of A and the identity (16). Remark. Suppose that a surface S ∈ ᏹ contains a punctured genus-2 surface as a subsurface. Then F (S) is the zero map. Indeed, we only need to check this when S has genus 2 and one boundary component. Then F (S) = 0 follows from m◦"◦m◦" = 0. Maps F (S21 ), F (S12 ), F (S01 ), F (S10 ), F (S22 ), and F (S11 ) between tensor powers of A are graded relative to deg with degrees       deg F S21 = deg F S12 = −1,       deg F S01 = deg F S10 = 1,       deg F S22 = deg F S11 = 0.

369

A CATEGORIFICATION OF THE JONES POLYNOMIAL





= q + q −1

Figure 3







=



−q





Figure 4

Proposition 3. For a surface S ∈ Mor(ᏹ), the degree of the map F (S) of graded R-modules is equal to the Euler characteristic of S. 2.4. Kauffman bracket. In this section we review the Kauffman bracket and its relation to the Jones polynomial, following Kauffman [Ka]. Fix an orientation of the 3-space R3 . A plane projection D of an oriented link L in R3 is called generic if it has no triple intersections, no tangencies, and no cusps. In this paper, a plane projection means a generic plane projection. Given a plane projection D, we assign a Laurent polynomial D ∈ Z[q, q −1 ] to D by the following rules: (1) A simple closed loop evaluates to q + q −1 : see Figure 3. (2) Each over- and undercrossing is a linear combination of two simple resolutions of this  crossing: see Figure 4.  (3) D1 D2  = D1 D2  where D1 D2  stands for the disjoint union of the diagrams D1 and D2 . From these rules we deduce what is depicted in Figure 5. 



= −q 2









,









= −q





=

Figure 5



= q −1





370

MIKHAIL KHOVANOV

Figure 6

Figure 7

L1

L2

L3

Figure 8

Curves of the diagram D inherit orientations from that of L. Let x(D) be the number of double points in the diagram D that look like those in Figure 6 and y(D) the number of double points that look like those in Figure 7. Then the quantity K(D) = (−1)x(D) q y(D)−2x(D) D

(21)

does not depend on the choice of a diagram D of the oriented link L and is an invariant of L. We denote this invariant by K(L). Up to a simple normalization, K(L) is the Kauffman bracket of link L and is equal to the Jones polynomial of L. The Kauffman bracket, f [L], as defined in [Ka], is a Laurent polynomial in an indeterminate A. (This A has no relation to the algebra A in Section 2.2 of this paper.) One easily sees that, by setting our q to −A−2 and dividing by (−A2 − A−2 ), we get f [L]:   (22) K(L)(q=−A−2 ) = − A2 − A−2 f [L]. In this paper we call K(L) the scaled Kauffman bracket. Let L1 , L2 , and L3 be three oriented links that differ as shown in Figure 8. The rules for computing the Kauffman bracket imply   (23) q −2 K(L1 ) − q 2 K(L2 ) = q −1 − q K(L3 ). Moreover, K(L) = q + q −1 if L is the unknot. The Jones polynomial V (L) of an oriented link L is determined by two properties: (1) The Jones polynomial of the unknot is 1.

371

A CATEGORIFICATION OF THE JONES POLYNOMIAL

(2) For oriented links L1 , L2 , L3 as above,

 √ 1 t − √ V (L3 ). t −1 V (L1 ) − tV (L2 ) = t

(24)

Therefore, the scaled Kauffman bracket and the Jones polynomial are related by V (L)√t=−q =

K(L) . q + q −1

(25)

3. Cubes 3.1. Complexes of R-modules. Denote by Kom(Ꮾ) the category of complexes of an abelian category Ꮾ. An object N of Kom(Ꮾ) is a collection of objects N i ∈ Ꮾ, i ∈ Z, together with morphisms d i : N i −→ N i+1 , i ∈ Z, such that d i+1 d i = 0. A morphism f : M → N of complexes is a collection of morphisms f i : M i → N i such that f i+1 d i = d i f i , i ∈ Z. A morphism f : M → N is called a quasi-isomorphism if the induced map of the cohomology groups H i (f ) : H i (M) → H i (N) is an isomorphism for all i ∈ Z. For n ∈ Z denote by [n] the automorphism of Kom(Ꮾ) that is defined on objects by N[n]i = N i+n , d[n]i = (−1)n d i+n and continued to morphisms in the obvious way. The cone of a morphism f : M → N of complexes is a complex C(f ) with       dC(f ) mi+1 , ni = − dM mi+1 , f mi+1 + dN ni . (26) C(f )i = M[1]i ⊕ N i , The grading shift automorphism {n}, introduced in Section 2.1, can be extended naturally to an automorphism of the category Kom(R- mod0 ) of complexes of graded R-modules. This automorphism of Kom(R- mod0 ) is also denoted {n}. To a complex M of graded R-modules we associate a graded R-module ⊕i∈Z M i . Each M i is a graded R-module, M i = ⊕j ∈Z Mji , and thus ⊕i∈Z M i is a bigraded R-module when we extend our usual grading of R to a bigrading with c ∈ R having degree (0, 2). From this viewpoint the differential dM of a complex M is a homogeneous map of degree (1, 0) of bigraded R-modules. 3.2. Commutative cubes. Let Ᏽ be a finite set. Denote by |Ᏽ| the cardinality of Ᏽ and by r(Ᏽ) the set of all pairs (ᏸ, a) where ᏸ is a subset of Ᏽ and a an element of Ᏽ that does not belong to ᏸ. To simplify notation we often (a) denote a one-element set {a} by a, (b) denote a finite set {a, b, . . . , d} by ab · · · d, (c) denote the disjoint union ᏸ1  ᏸ2 of two sets ᏸ1 , ᏸ2 by ᏸ1 ᏸ2 ; in particular, we denote by ᏸa the disjoint union of a set ᏸ and a one-element set {a}; similarly, ᏸab means ᏸ  {a}  {b}, and so on. Definition 1. Let Ᏽ be a finite set and Ꮾ a category. A commutative Ᏽ-cube V over Ꮾ is a collection of objects V (ᏸ) ∈ Ob(Ꮾ) for each subset ᏸ of Ᏽ and morphisms

372

MIKHAIL KHOVANOV

ξaV (ᏸ) : V (ᏸ) −→ V (ᏸa)

(27)

for each (ᏸ, a) ∈ r(Ᏽ), such that for each triple (ᏸ, a, b), where ᏸ is a subset of Ᏽ and a, b, a  = b are two elements of Ᏽ that do not lie in ᏸ, there is an equality of morphisms ξbV (ᏸa)ξaV (ᏸ) = ξaV (ᏸb)ξbV (ᏸ),

(28)

that is, the following diagram is commutative: V (ᏸ) ξbV (ᏸ)

 V (ᏸb)V

ξaV (ᏸ)

/ V (ᏸa) ξbV (ᏸa)

 / V (ᏸab).

ξaV (ᏸb)

We say a commutative Ᏽ-cube is an Ᏽ-cube or, sometimes, a cube when Ᏽ is clear. Maps ξaV are called the structure maps of V . Example. If Ᏽ is the empty set, an Ᏽ-cube is an object in Ꮾ. If Ᏽ consists of one element, an Ᏽ-cube is a morphism in Ꮾ. If Ᏽ consists of two elements, Ᏽ = {a, b}, an Ᏽ-cube is a commutative square of objects and morphisms in Ꮾ: V (∅)

/ V (a)

 V (b)

 / V (ab).

In general, an Ᏽ-cube can be visualized in the following manner. Let n be the cardinality of Ᏽ. We take an n-dimensional cube in a standard position in the Euclidean n-dimensional space, that is, each vertex has coordinates (a1 , . . . , an ) where ai ∈ {0, 1}. We orient each edge in the direction of the vertex with the bigger sum of the coordinates. Then the edges of any 2-dimensional facet of this cube are oriented as shown in Figure 9.

Figure 9

A CATEGORIFICATION OF THE JONES POLYNOMIAL

373

Choose a bijection between elements of Ᏽ and coordinates of Rn . This bijection defines a bijection between vertices of the n-cube and subsets of Ᏽ, with the (0, . . . , 0) vertex associated to the empty set. Oriented edges of the n-cube correspond to pairs (ᏸ, a) ∈ r(Ᏽ). Given an Ᏽ-cube V , put an object V (ᏸ) into the vertex associated to the set ᏸ and assign a morphism ξaV (ᏸ) to the arrow going from the vertex associated to ᏸ to the vertex associated to ᏸa. Equation (28) is equivalent to the commutativity of diagrams in all 2-dimensional faces of the n-cube. Given two Ᏽ-cubes V , W over a category Ꮾ, an Ᏽ-cube map ψ : V −→ W is a collection of maps ψ(ᏸ) : V (ᏸ) −→ W (ᏸ),

for all ᏸ ⊂ Ᏽ,

that make diagrams V (ᏸ)

ψ(ᏸ)

ξaV (ᏸ)

 V (ᏸa)

/ W (ᏸ )

(29)

ξaW (ᏸ)

ψ(ᏸa)

 / W (ᏸa)

commutative for all (ᏸ, a) ∈ r(Ᏽ). The map ψ is called an isomorphism if ψ(ᏸ) is an isomorphism for all ᏸ ⊂ Ᏽ. The map ψ of Ᏽ-cubes over an abelian category Ꮾ is called injective/surjective if ψ(ᏸ) is injective/surjective for all ᏸ ⊂ Ᏽ. The class of Ᏽ-cubes over an abelian category Ꮾ and maps between Ᏽ-cubes constitute an abelian category in the obvious way. In particular, direct sums of Ᏽ-cubes are defined. For a finite set Ᏽ and a ∈ Ᏽ, let ᏶ be the complement, Ᏽ = ᏶ {a}. Given an Ᏽ-cube V , let Va (∗0), Va (∗1) be ᏶-cubes defined as follows: Va (∗0)(ᏸ) = V (ᏸ),

Va (∗1)(ᏸ) = V (ᏸa),

for ᏸ ⊂ ᏶.

(30)

The structure maps of Va (∗0), Va (∗1) are determined by the structure maps ξbV , b ∈ ᏶ of V in the obvious fashion. Sometimes we write V (∗0) for Va (∗0), and so forth. The structure map ξaV of V defines an ᏶-cube map ξaV : Va (∗0) → Va (∗1). This provides a one-to-one correspondence between Ᏽ-cubes and maps of ᏶-cubes. We say that a map ψ : V → W of Ᏽ-cubes over the category R-mod of graded R-modules and graded maps is graded of degree i if the map ψ(ᏸ) : V (ᏸ) → W (ᏸ) has degree i for all ᏸ ⊂ Ᏽ. For a cube V over R- mod0 , denote by V {i} the cube V with the grading shifted by i: V {i}(ᏸ) = V (ᏸ){i}

for all ᏸ ⊂ Ᏽ;

(31)

the structure maps are the appropriate shifts of the structure maps of V . A degree i map ψ : V → W of Ᏽ-cubes over R- mod0 induces a grading-preserving map V → W {i}, also denoted ψ.

374

MIKHAIL KHOVANOV

3.3. Skew-commutative cubes. We next define skew-commutative Ᏽ-cubes over an additive category Ꮾ. A skew-commutative Ᏽ-cube is almost the same as a commutative Ᏽ-cube, but now we require that for every square facet of the cube the associated diagram of objects and morphisms of Ꮾ anticommutes. Definition 2. Let Ᏽ be a finite set and Ꮾ an additive category. A skew-commutative Ᏽ-cube V over Ꮾ is a collection of objects V (ᏸ) ∈ Ob(Ꮾ) for ᏸ ⊂ Ᏽ, and morphisms ξaV (ᏸ) : V (ᏸ) −→ V (ᏸa), such that for each triple (ᏸ, a, b), where ᏸ is a subset of Ᏽ and a, b, a  = b, are two elements of Ᏽ that do not lie in ᏸ, there is an equality ξbV (ᏸa)ξaV (ᏸ) + ξaV (ᏸb)ξbV (ᏸ) = 0. We call a skew-commutative Ᏽ-cube over Ꮾ a skew Ᏽ-cube or, without specifying Ᏽ, a skew cube. Given Ᏽ-cubes or skew Ᏽ-cubes V and W over R- mod0 , their tensor product is defined to be an Ᏽ-cube (if V and W are both cubes or both skew cubes) or a skew Ᏽ-cube (if one of V , W is a cube and the other is a skew cube), denoted V ⊗ W, given by

(V ⊗ W )(ᏸ) = V (ᏸ) ⊗ W (ᏸ),

ᏸ ⊂ Ᏽ,

ξaV ⊗W (ᏸ) = ξaV (ᏸ) ⊗ ξaW (ᏸ),

(ᏸ, a) ∈ r(Ᏽ),

where, we recall, the tensor products are taken over R. For a finite set ᏸ denote by o(ᏸ) the set of complete orderings or elements of ᏸ. For x, y ∈ o(ᏸ) let p(x, y) be the parity function. p(x, y) = 0 if y can be obtained from x via an even number of transpositions of two neighboring elements in the ordering. Otherwise, p(x, y) = 1. To a finite set ᏸ associate a graded R-module E(ᏸ) defined as the quotient of the graded R-module, freely generated by elements x for all x ∈ o(ᏸ), by relations x = (−1)p(x,y) y for all pairs x, y ∈ o(ᏸ). Module E(ᏸ) is a free graded R-module of rank 1. For a  ∈ ᏸ there is a canonical isomorphism of graded R-modules E(ᏸ) → E(ᏸa) induced by the map o(L) → o(La) that takes x ∈ o(L) to xa ∈ o(La). Moreover, for a, b, a  = b, the diagram below anticommutes: E(ᏸ)

/ E(ᏸa)

 E(ᏸb)

 / E(ᏸab).

(32)

Denote by EᏵ the skew Ᏽ-cube with EᏵ (ᏸ) = E(ᏸ) for ᏸ ⊂ Ᏽ and the structure map EᏵ (ᏸ) → EᏵ (ᏸa) being canonical isomorphism E(ᏸ) → E(ᏸa). We use EᏵ to pass from Ᏽ-cubes over R- mod0 to skew Ᏽ-cubes over R- mod0 by tensoring an Ᏽ-cube with EᏵ .

A CATEGORIFICATION OF THE JONES POLYNOMIAL

375

3.4. Skew-commutative cubes and complexes. Let V be a skew Ᏽ-cube over an i abelian category Ꮾ. To V we associate a complex C(V ) = ( C (V ), d i ), i ∈ Z of objects of Ꮾ by  i C (V ) = V (ᏸ). (33) ᏸ⊂Ᏽ, |ᏸ|=i

i

The differential d i : C (V ) → C

i+1

(V ) is given on an element x ∈ V (ᏸ), |ᏸ| = i by
d i (x) = ξaV (ᏸ)x. (34) a∈Ᏽ\ᏸ

Examples. (1) If |Ᏽ| = 1, Ᏽ = {a},   V (∅) if i = 0, i C (V ) = V (a) if i = 1,   0 otherwise. The differential d 0 = ξaV (∅) and d i = 0 if i  = 0, so C(V ) is the complex ξaV (∅)

· · · −→ 0 −→ V (∅) −−−−→ V (Ᏽ) −→ 0 −→ · · · .

(35)

(2) If Ᏽ contains two elements, say, Ᏽ = {a, b}, then  V (∅) if i = 0,    V (a) ⊕ V (b) if i = 1, i C (V ) =  V (ab) if i = 2,    0 otherwise, and the differentials are as follows: d 0 : V (∅) −→ V (a) ⊕ V (b), d 0 = ξbV (∅) + ξaV (∅), d 1 : V (b) ⊕ V (a) −→ V (ab),   d 1 = ξaV (b), ξbV (a) . Proposition 4. Let V be a skew Ᏽ-cube over an abelian category Ꮾ and suppose that for some a ∈ Ᏽ and any ᏸ ⊂ Ᏽ \ {a} the map ξaV : V (ᏸ) → V (ᏸa) is an isomorphism. Then the complex C(V ) is acyclic. Proof. The complex C(V ) is isomorphic to the cone of the identity map of the complex C(Va (∗1))[−1] and, therefore, is acyclic.

376

MIKHAIL KHOVANOV

Every map of Ᏽ-cubes φ : V −→ W over R- mod0 induces a map of complexes C(φ) : C(V ⊗ EᏵ ) −→ C(W ⊗ EᏵ ).

(36)

If φ is an isomorphism of commutative cubes, C(φ) is an isomorphism of complexes. Proposition 5. Let V be an Ᏽ-cube over R- mod0 and suppose that for some a ∈ Ᏽ the structure map ξaV : Va (∗0) −→ Va (∗1)

(37)

is an isomorphism. Then the complex C(V ⊗ EᏵ ) is acyclic. Proof. The proof is immediate from Proposition 4. The following proposition and its corollary are obvious. Proposition 6. We have a canonical splitting of complexes C(V ⊕ W ) = C(V ) ⊕ C(W ),

(38)

where V and W are skew-commutative Ᏽ-cubes over an abelian category and V ⊕W is the direct sum of V and W . Corollary 1. We have a canonical splitting of complexes       C (V ⊕ W ) ⊗ EᏵ = C V ⊗ EᏵ ⊕ C W ⊗ EᏵ ,

(39)

where V and W are Ᏽ-cubes over R- mod0 . 4. Diagrams 4.1. Reidemeister moves. Given a link L in R3 , we can take its generic projection on the plane. A generic projection is the one without triple points and double tangencies. An isotopy class of such projections is called a plane diagram of L or, simply, a diagram. Four types of transformations of plane diagrams, shown in Figures 10–13, preserve the isotopy type of the associated link. Proposition 7. If plane diagrams D1 and D2 represent isotopic oriented links, these diagrams can be connected by a chain of moves as depicted in Figures 10–13.

Figure 10. Addition/removal of a left-twisted curl

A CATEGORIFICATION OF THE JONES POLYNOMIAL

377

Figure 11. Addition/removal of a right-twisted curl

Figure 12. Tangency move

Figure 13. Triple point move

4.2. Constructing cubes and complexes from plane diagrams. Fix a plane diagram D with n double points of an oriented link L. Denote by Ᏽ the set of double points of D. To D we associate an Ᏽ-cube VD over the category R- mod0 of graded R-modules. This cube does not depend on the orientation of components of L. Given a double point of a diagram D, it can be resolved in two possible ways, as seen in Figure 14. Let us call the resolution on the left the 0-resolution, and the one on the right the 1-resolution. A resolution of D is a resolution of each double point of D. Thus, D admits 2n resolutions. There is a one-to-one correspondence between resolutions of D and subsets ᏸ of the set Ᏽ of double points. Namely, to ᏸ ⊂ Ᏽ we associate a resolution, denoted D(ᏸ), by taking a 1-resolution of each double point that belongs to ᏸ and a 0-resolution if the double point does not lie in ᏸ.

0-resolution

1-resolution Figure 14

378

MIKHAIL KHOVANOV

D

D(∅)

D(a)

Figure 15

D

Figure 16

A resolution of a diagram D is always a collection of simple disjoint curves on the plane and is thus a 1-manifold embedded in the plane. Now the functor F from (1 + 1) cobordisms to R-modules (see Section 2.3) comes into play. To a union of k circles it assigns the kth tensor power of A. The functor F , applied to the diagram D(ᏸ), considered as a 1-dimensional manifold, produces a graded R-module A⊗k where k is the number of components of D(ᏸ). We raise the grading of A⊗k by |ᏸ|, the cardinality of ᏸ, and assign the R-module F (D(a)){−|ᏸ|} to the vertex VD (ᏸ) of the cube VD :    (40) VD (ᏸ) = F D(ᏸ) − |ᏸ| . (Recall from Section 3.1 that the automorphism {1} of the category R- mod0 lowers the grading by 1.) Let us now define maps between vertices of VD . Choose (ᏸ, a) ∈ r(Ᏽ). We want to have a map ξaVD (ᏸ) : VD (ᏸ) −→ VD (ᏸa).

(41)

The diagrams D(ᏸ) and D(ᏸa) differ only in the neighbourhood of the double point a of D, as Figure 15 demonstrates (for n = 1, so that D has one double point, Ᏽ = {a}). Take the direct product of the plane R2 and the interval [0, 1]. We identify the diagram D(ᏸ) (respectively, D(ᏸa)) with a 1-dimensional submanifold of R2 × {0} (respectively, R2 × {1}). We can choose a small neighbourhood U of a such that D(ᏸ) and D(ᏸa) coincide outside U and inside they look as shown in Figure 16. The boundary of U is depicted by a dashed circle; the central picture shows the intersection of D(ᏸ) and U ; and the rightmost picture shows the intersection of D(ᏸa) and U. Let S be a surface properly embedded in R2 × [0, 1] such that (1) the boundary of S is the union of the diagrams D(ᏸ) and D(ᏸa);

A CATEGORIFICATION OF THE JONES POLYNOMIAL

D

a

379

b

D(a)

D(ab)

Figure 17

D(ab) S D(a) Figure 18

(2) outside of U ×[0, 1] surface S is the direct product of D(ᏸ)∩(R2 \U ) and the interval [0, 1]; (3) the connected component of S that has a nonempty intersection with U ×[0, 1] is homeomorphic to the 2-sphere with three holes; (4) the projection S −→ [0, 1] onto the second component of the product R2 × [0, 1] has only one critical point—the saddle point that lies inside U × [0, 1]. Example. Let D be a diagram with two double points, Ᏽ = {a, b}, as in Figure 17. Diagram D(a) (respectively, D(ab)) consists of two (respectively, three) simple curves. The boundary of the neighbourhood U of the double point a is depicted by the dashed circle. Then Figure 18 shows what the surface S looks like. Recall that earlier we defined VD (ᏸ) to be F (D(ᏸ)) for ᏸ ⊂ Ᏽ, with the degree raised by |ᏸ|. Now define the map ξaVD (ᏸ) : VD (ᏸ) −→ VD (ᏸa) to be given by

    F (S) : F D(ᏸ) −→ F D(ᏸa) .

Note that the degree of F (S) is equal to −1, the Euler characteristic of the surface S (Proposition 3). But |ᏸa| = |ᏸ| + 1, so, with degrees shifted,

380

MIKHAIL KHOVANOV

a b

D(∅)

D(b)

D(a)

D(ab)

Figure 19. Diagram D and four resolutions

  VD (ᏸ) = F D(ᏸ) {−|ᏸ|},   VD (ᏸa) = F D(ᏸa) {−|ᏸ| − 1},

(42) (43)

and the map ξaVD (ᏸ) is a grading-preserving map of graded R-modules. Proposition 8. VD , defined in this way, is an Ᏽ-cube over the category R- mod0 of graded R-modules and grading-preserving maps. The proof consists of verifying commutativity relations (28) for maps ξaVD (ᏸ). They follow immediately from the functoriality of F. Example. For the diagram D and its four resolutions depicted in Figure 19 we get     F D(∅) = A⊗2 , F D(a) = A,     F D(b) = A, F D(ab) = A⊗2 . The cube VD has the form A⊗2

m

/ A{−1}

"

 / A⊗2 {−2}.

m

 A{−1}

"

Let us now go back to our construction. So far, to a plane diagram D with the set

Ᏽ of double points we associated an Ᏽ-cube VD over the category R- mod0 of graded

A CATEGORIFICATION OF THE JONES POLYNOMIAL

381

R-modules. We would like to build a complex of graded R-modules out of VD . We know how to build a complex from a skew-commutative Ᏽ-cube (see Section 3.4). To make a skew-commutative Ᏽ-cube out of an Ᏽ-cube VD , we put minus sign in front of some structure maps ξVD of VD so that for any commutative square of VD an odd number of the four maps constituting the square change signs. A more intrinsic way to do this is to tensor VD with the skew-commutative Ᏽ-cube EᏵ , defined at the end of Section 3.3. To the skew-commutative Ᏽ-cube VD ⊗EᏵ there is associated the complex C(VD ⊗ EᏵ ) of graded R-modules (see Section 3.4). Denote this complex by C(D):  def  C(D) = C VD ⊗ EᏵ . (44) Thus, C(D) is a complex of graded R-modules and grading-preserving homomorphisms. It does not depend on the orientations of the components of the link L. Recall (Section 3.1) that the category Kom(R- mod0 ) has two commuting automorphisms: [1], which shifts a complex 1 step to the left, and {1}, which lowers the grading of each component of the complex by 1. To the diagram D of link L we associated (Section 2.4) two numbers, x(D) and y(D). Define a complex C(D) by    C(D) = C(D) x(D) 2x(D) − y(D) . (45) Define H i (D) as the ith cohomology group of C(D). It is a finitely generated graded R-module. Theorem 1. If D is a plane diagram of an oriented link L, then for each i ∈ Z, the isomorphism class of the graded R-module H i (D) is an invariant of L. The proof of this theorem occupies Section 5, together with some preliminary material contained in Section 4.3. Define H i,j (D) as the ith cohomology group of the degree j subcomplex of C(D). Thus, H i,j (D) is the graded component of H i (D) of degree j , and we have a decomposition of abelian groups  H i (D) = H i,j (D). (46) j ∈Z

We denote by H i (L) the isomorphism class of H i (D) in the category of graded R-modules. For an oriented link L, only finitely many of H i (L) are nonzero as i varies over all integers. Corollary 2. If D is a plane diagram of an oriented link L, then for each i, j ∈ Z, the isomorphism class of the abelian group H i,j (D) is an invariant of L. We next show that the Kauffman bracket is equal to a suitable Euler characteristic of these cohomology groups.

382

MIKHAIL KHOVANOV

D1

D2

D3

Figure 20

Proposition 9. For an oriented link L, 
   K(L) = 1 − q 2 (−1)i χ  H i (D) ,

(47)

i∈Z

where K(L) is the scaled Kauffman bracket defined in Section 2.4, χ  is the Euler characteristic of Section 2.1, and D is any diagram of L. (M) for a finitely generated graded RProof. First notice that χ (M{n}) = q −n χ module M. Given a bounded complex M : · · · −→ M i −→ M i+1 −→ · · · of finitely generated graded R-modules, define
  (−1)i χ  Mi . χ (M) =

(48)

(49)

i∈Z

Since

   
(−1)i χ  H i (D) , χ  C(D) =

(50)

i∈Z

it is enough to prove

     C(D) . K(L) = 1 − q 2 χ

(51)

For three diagrams D1 , D2 , and D3 that differ as shown in Figure 20, the complex C(D1 )[1] is isomorphic, up to a shift, to the cone of a map of complexes C(D2 ) → C(D3 ){−1}. Therefore,            C(D2 ) − χ  C(D3 ){−1} = χ  C(D2 ) − q χ C(D3 ) . (52) χ  C(D1 ) = χ On the other hand, for diagrams D1 , D2 , D3 as in Figure 20, we have D1  = D2  − qD3 

(53)

(see Section 2.4, where D is defined). If the diagram D is a disjoint union of k simple plane curves, then  k    ⊗k    q + q −1 −1 k χ  C(D) = χ A χ (R) = (54) = q +q 1 − q2

383

A CATEGORIFICATION OF THE JONES POLYNOMIAL

pm

p1

p2

Figure 21

p1 pm

p1 pm

p2

Q0

p2

Q1

Figure 22

and D = (q + q −1 )k . Therefore, for any diagram D,     D = 1 − q 2 χ  C(D) .

(55)

Since     χ  C(D) = χ  C(D) [x(D)]{2x(D) − y(D)}    C(D) , = (−1)x(D) q y(D)−2x(D) χ

(56)

and in view of (21), the proposition follows. 4.3. Surfaces and cube morphisms. Let U be a closed disk in the plane R2 and U˙ the interior of U so that U = ∂U ∪ U˙ . Let T ! be a tangle in (R2 \ U˙ ) × [0, 1] with m points (where m is even) on the boundary ∂U × [0, 1] and T a generic projection of T ! on R2 \ U˙ . The intersection of T with ∂U consists of m points. Denote them by p1 , . . . , pm (see Figure 21; ∂U is shown by a dashed circle). Let Ᏽ be the set of double points of T . Pick two systems Q0 and Q1 of m/2 simple disjoints arcs in U with ends in points p1 , . . . , pm , as depicted in Figure 22. Then Q0 ∪ T and Q1 ∪ T (here and further on we denote them by P0 and P1 , respectively) can be considered as two plane diagrams of links in R3 , shown in Figure 23. To P0 and P1 there are associated Ᏽ-cubes VP0 and VP1 .

384

MIKHAIL KHOVANOV

P0

P1

Figure 23

Let S be a compact-oriented surface in U × [0, 1] such that the boundary of S is the union of Q0 × {0}, Q1 × {1}, and (p1 ∪ · · · ∪ pm ) × [0, 1]. To S we associate an Ᏽ-cube map ψS : VP0 −→ VP1 as follows. For each ᏸ ⊂ Ᏽ we must construct a map ψS,ᏸ : VP0 (ᏸ){−|ᏸ|} −→ VP1 (ᏸ){−|ᏸ|}

(57)

and check the commutativity of diagrams (29). To ᏸ there is associated a resolution T (ᏸ) of double points of T . Thus T (ᏸ) is a ˙ with ends in p1 , . . . , pm . Then, collection of simple closed curves and arcs in R2 \ U by (40),   VP0 (ᏸ) = F T (ᏸ) ∪ Q0 {−|ᏸ|}, (58)   (59) VP1 (ᏸ) = F T (ᏸ) ∪ Q1 {−|ᏸ|}, where F is the functor described in Section 2.3. (Notice that T (ᏸ)∪Q0 and T (ᏸ)∪Q1 are collections of simple closed curves on the plane, so that we can apply functor F to them.) Let S ! be a surface in R2 × [0, 1] that is S inside U × [0, 1] and T (ᏸ) × [0, 1] outside U × [0, 1]. The map     F (S ! ) : F T (ᏸ) ∪ Q0 −→ F T (ᏸ) ∪ Q1 (60) is a graded map of R-modules of degree χ(S ! ) = χ(S) − m/2. Define ψS,ᏸ as this map, shifted by |ᏸ|: ψS,ᏸ = F (S ! ){−|ᏸ|} : VP0 (ᏸ){−|ᏸ|} −→ VP1 (ᏸ){−ᏸ|}.

(61)

The commutativity condition (29) is immediate. We sum up our result as follows. Proposition 10. The map ψS : VP0 −→ VP1 is a degree (χ (S) − m/2) map of Ᏽ-cubes.

(62)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

385

p1 pm

p2

Figure 24

Everything in this section extends to the case when the diagrams Q0 and Q1 are allowed to have simple closed circles in addition to m/2 simple disjoint acts joining points p1 , . . . , pm . For instance, Q0 may look like Figure 24. In this more general case, to each compact oriented surface S in U × [0, 1] such that the boundary of S is the union of Q0 × {0}, Q1 × {1} and (p1 ∪ · · · ∪ pm ) × [0, 1], in exactly the same fashion as before, we associate an Ᏽ-cube map ψS : VP0 −→ VP1 .

(63)

This map is a graded map of cubes over R- mod0 of degree equal to the Euler characteristic of S minus m/2. Tensoring the map ψS with the identity map of the skew-commutative n-cube EᏵ and passing to associated complexes, we obtain a map of complexes of graded R-modules ψS! : C(P0 ) −→ C(P1 ).

(64)

In general this map is not a morphism in the category Kom(R- mod0 ) of complexes of graded R-modules and grading-preserving homomorphism, as it shifts the grading by χ (S) − m/2, but ψS! becomes a morphism in Kom(R- mod0 ) when the grading of C(P0 ) or C(P1 ) is appropriately shifted. 5. Transformations. In this section we prove Theorem 1. We associate a quasiisomorphism of complexes of graded R-modules C(D) → C(D ! ) to a Reidemeister move between two plane diagrams D and D ! of an oriented link L. 5.1. Left-twisted curl. Let D be a plane diagram with n − 1 double points and let D1 be a diagram constructed from D by adding a left-twisted curl. Denote by Ᏽ! the set of double points of D1 , by a the double point in the curl, and by Ᏽ the set of double points of D. There is a natural bijection of sets Ᏽ → Ᏽ! \ {a}, coming from identifying a double point of D with the corresponding double point of D1 . We use this bijection to identify the two sets Ᏽ and Ᏽ! \ {a}.

386

MIKHAIL KHOVANOV

D

D1

D2

a Figure 25

U



D1 a Figure 26

The crossing a of D1 can be resolved in two ways; see Figure 25. The 0-resolution of a is a diagram D2 that is a disjoint union of D and a circle. The 1-resolution is a diagram isotopic to D, and we identify this diagram with D. In this section we define a quasi-isomorphism of the complexes C(D) and C(D1 ). This quasi-isomorphism arises from a splitting of the Ᏽ! -cube VD1 as a direct sum of two cubes, VD1 = V ! ⊕ V !! . This splitting induces a decomposition of the complex C(D1 ) into a direct sum of an acyclic complex and a complex isomorphic to C(D). Recall that VD , VD1 , and VD2 are the cubes associated with the diagrams D, D1 , and D2 , respectively. VD1 has index set Ᏽ! , while VD and VD2 are Ᏽ-cubes. From the decomposition of D2 as a union of D and a simple circle we get a canonical isomorphism of cubes VD2 = VD ⊗ A,

(65)

where VD ⊗ A is the Ᏽ-cube obtained from VD by tensoring graded R-modules VD (ᏸ), ᏸ ⊂ Ᏽ with A and tensoring the structure maps ξaVD (ᏸ) with the identity map of A. Let U ⊂ R2 be a small neighborhood of a that contains the curl as depicted in Figure 26. The figure shows how the diagram D1 looks inside U. The boundary of U is shown by a dashed circular line. Intersections of U with diagrams D and D2 are depicted in Figure 27. Outside of U , diagrams D, D1 , and D2 coincide. It is explained in Section 4.3 how surfaces in U ×[0, 1], satisfying certain conditions, give rise to cube maps. Using this construction we now define three cube maps between cubes VD and VD2 : ma : VD2 −→ VD ,

(66)

"a : VD −→ VD2 ,

(67)

ιa : VD −→ VD2 .

(68)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

U



D

U



387

D2

Figure 27

U × {0}

U × {0.5}

U × {1}

Figure 28

Figure 29

The map ma is associated to the surface presented in Figure 28. Hereafter we depict surfaces embedded in U × [0, 1] by a sequence of their cross sections U × {t}, t ∈ [0, 1], the leftmost one being the intersection of the surface with U × {0}, the rightmost being the intersection with U × {1}. For such a surface S ∈ U × [0, 1], we call the projection S → [0, 1] the height function of S. These surfaces have only nondegenerate critical points relative to the height function. We depict enough sections of S to make it obvious what surface we are considering, sometimes adding extra information, for example, that the surface in Figure 28 has one saddle point and no other critical points relative to the height function. In Figure 28, the intersections S ∩ U × {0}, S ∩ U × {1} of the surface S with the boundary disks U × {0}, U × {1} are isomorphic to the intersections D2 ∩ (U × {0}) (respectively, D ∩ (U × {1})). Thus, S defines a map ma from the cube VD2 to VD . The cube map "a is associated to the surface shown in Figure 29. This surface has one saddle point and no other critical points relative to the height function. The cube map ιa is associated to the surface shown in Figure 30. The only critical

388

MIKHAIL KHOVANOV

Figure 30

point of the height function is a local minimum. The cube maps ma , "a , ιa are graded maps and change the grading by −1, −1, 1, respectively. So let us keep in mind that ma , "a , ιa become grading-preserving if we appropriately shift gradings of our cubes; for example, ma : VD2 −→ VD {−1},

(69)

"a : VD −→ VD2 {−1},

(70)

ιa : VD −→ VD2 {1}

(71)

are grading-preserving maps of cubes over R- mod0 . The composition ma ιa is equal to the identity map from VD to itself. Denote by a the map def

a = "a − ιa ma "a : VD −→ VD2 .

(72)

The map a is a graded map of degree −1. Proposition 11. The Ᏽ-cube VD2 splits as a direct sum:     V D 2 = ι a VD ⊕  a VD .

(73)

Proof. It is enough to consider the case when D is a single circle. Then Ᏽ! = {a}, Ᏽ = ∅, VD = A, and ιa (VD ) = 1 ⊗ A. But   a 1 = "a − ιa ma "a 1 = X ⊗ 1 − 1 ⊗ X + cX ⊗ X,    a X = " a − ι a ma " a X = X ⊗ X and, thus, A ⊗ A is a direct sum of 1 ⊗ A and the R-submodule spanned by a 1 and a X. Note that

  m a a = m a "a − ι a ma " a = 0

(74)

because ma ιa = Id . The Ᏽ! -cube VD1 contains VD and VD2 as subcubes of codimension 1. Namely, we have canonical isomorphisms VD1 (∗0) ∼ = V D2 , VD (∗1) ∼ = VD {−1}. 1

(75) (76)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

389

Recall from Section 3.2 that VD1 (∗0) denotes the Ᏽ! \ {a}-cube (i.e., Ᏽ-cube) with VD1 (∗0)(ᏸ) = VD1 (ᏸ) for ᏸ ⊂ Ᏽ, and so on. Under these isomorphisms the structure VD1

map ξa

(denoted below by ξa ) for the Ᏽ! -cube VD1 , ξa : VD1 (∗0) −→ VD1 (∗1),

(77)

is equal to the map ma of Ᏽ-cubes; that is, the following diagram is commutative: VD1 (∗0) 

ξa

∼ =

V D2

/ VD1 (∗1) ∼ =

ma

 / VD {−1}.

Using the splitting (73) of VD2 , we can decompose the Ᏽ! -cube VD1 as a direct sum of two Ᏽ! -cubes as follows: VD1 = V ! ⊕ V !! ,

(78)

V ! (∗0) = a (VD ),

(79)

where

!

(80)

!!

(81)

!!

(82)

V (∗1) = 0, V (∗0) = ιa (VD ), V (∗1) = VD1 (∗1).

Some explanation: In the formula (79), a (VD ) is a subcube of VD2 and, due to (75), a (VD ) sits inside VD1 as a subcube of codimension 1. Equation (80) means that V ! (∗1)(ᏸ) = 0 for all ᏸ ⊂ Ᏽ! . Thus, V ! (ᏸ) = a (VD (ᏸ)) ⊂ VD1 (ᏸ) for ᏸ ⊂ Ᏽ! if ᏸ does not contain a. If ᏸ contains a, V ! (ᏸ) = 0. Tensoring (78) with EᏵ! , we get a splitting of skew-commutative Ᏽ! -cubes     (83) VD1 ⊗ EᏵ! = V ! ⊗ EᏵ! ⊕ V !! ⊗ EᏵ! . This induces a splitting of complexes associated to these skew-commutative Ᏽ! -cubes       C VD1 ⊗ EᏵ! = C V ! ⊗ EᏵ! ⊕ C V !! ⊗ EᏵ! . (84) Proposition 12. The complex C(V !! ⊗ EᏵ! ) is acyclic. Proof. The complex C(V !! ⊗ EᏵ! ) is isomorphic to the cone of the identity map of the complex C(VD ⊗ EᏵ )[−1]{−1}. Proposition 13. The complexes C(V ! ⊗EᏵ! ) and C(VD ⊗EᏵ ){1} are isomorphic.

390

MIKHAIL KHOVANOV

D

D1

a

D2

Figure 31

Proof. We have a chain of isomorphisms of complexes     C(V ! ⊗ EᏵ! ) = C V ! (∗0) ⊗ EᏵ = C VD {1} ⊗ EᏵ = C(VD ⊗ EᏵ ){1}. Corollary 3. The complexes C(D1 ) and C(D){1} are quasi-isomorphic. Proof. We have   C(D1 ) = C VD1 ⊗ EᏵ!     = C V ! ⊗ EᏵ! ⊕ C V !! ⊗ EᏵ!     = C VD ⊗ EᏵ {1} ⊕ C V !! ⊗ EᏵ! = C(D){1} ⊕ (acyclic complex). Note that x(D1 ) = x(D) and y(D1 ) = y(D) + 1. By (45), C(D) = C(D)[x(D)]{2x(D) − y(D)}

(85)

and C(D1 ) = C(D1 )[x(D1 )]{2x(D1 ) − y(D1 )} = C(D1 )[x(D)]{2x(D) − y(D) − 1}. Therefore, complexes C(D) and C(D1 ) are quasi-isomorphic. 5.2. Right-twisted curl. Let D be a diagram with n − 1 double points and let D1 be a diagram constructed from D by adding a right-twisted curl. Denote by a the new crossing that appears in the curl. Let Ᏽ be the set of crossings of D and Ᏽ! the set of crossings of D1 . We have a natural bijection of sets Ᏽ → Ᏽ! \{a} and use it to identify these two sets. Crossing a can be resolved in two ways; see Figure 31. 0-resolution gives a diagram, isotopic to D and canonically identified with D. 1-resolution produces a diagram, denoted D2 , which is a disjoint union of D and a simple circle. Note that diagrams D and D2 are the same as diagrams D and D2 from Section 5.1, and we can use cube maps ma , "a , ιa defined in that section. Also define a map #a : VD2 −→ VD ,

(86)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

391

Figure 32

where #a is associated to the surface shown in Figure 32. This surface has one critical point relative to the height function and it is a local maximum. The cube map #a changes the grading by 1 and becomes grading-preserving after an appropriate shift: #a : VD2 −→ VD {1}.

(87)

ℵ = ιa − cιa ma "a : VD −→ VD2 .

(88)

Let ℵ be the map

ℵ is graded of degree 1. Proposition 14. We have a cube splitting VD2 = ℵ(VD ) ⊕ "a (VD ).

(89)

Proof. It suffices to check this when D is a simple circle. Then ℵ(1) = 1 ⊗ 1 − 2c1 ⊗ X, ℵ(X) = 1 ⊗ X. The R-submodule of A ⊗ A generated by these two vectors complements "(A), and there is direct sum decomposition of R-modules A ⊗ A = R · ℵ(1) ⊕ R · ℵ(X) ⊕ "(A). Denote by ℘ the cube map ℘ = ma − ma "a #a : VD2 −→ VD .

(90)

Note that ℘ is a graded map of degree −1. Lemma 1. We have equalities ℘"a = 0, ℘ℵ = Id(VD ).

(91) (92)

392

MIKHAIL KHOVANOV

Proof. Map ℘"a : VD → VD is the zero map because ℘"a = ma "a − ma "a #a "a = ma "a − ma "a = 0

(93)

(the second equality uses that #a "a = Id). The equality (92) is checked similarly:    ℘ℵ = ma − ma "a #a ιa − cιa ma "a = ma ιa − cma ιa ma "a − ma "a #a ιa + cma "a #a ιa ma "a = Id −cma "a + cma "a − c2 ma "a ma "a = Id −c2 ma "a ma "a = Id . The third equality in the computation above follows from the identities ma ιa = Id,

#a ιa = −c.

(94)

The fifth equality is implied by ma "a ma "a = 0. This identity follows from the nilpotence property m"m" = 0 of the structure maps m and " of A. Using the splitting (89) of VD2 and Lemma 1, we can decompose the Ᏽ! -cube VD1 as a direct sum of two Ᏽ! -cubes as follows: VD1 = V ! ⊕ V !! ,

(95)

where V ! (∗0) = 0,

(96)

!

(97)

!!

(98)

!!

(99)

V (∗1) = ℵ(VD ){−1} ⊂ VD2 {−1} = VD1 (∗1), V (∗0) = VD = VD1 (∗0), V (∗1) = "a (VD ){−1} ⊂ VD2 {−1} = VD1 (∗1). Tensoring (95) with EᏵ! , we get a splitting of skew-commutative Ᏽ! -cubes     VD1 ⊗ EᏵ! = V ! ⊗ EᏵ! ⊕ V !! ⊗ EᏵ! .

(100)

This induces a splitting of complexes associated to these skew Ᏽ! -cubes,       C VD1 ⊗ EᏵ! = C V ! ⊗ EᏵ! ⊕ C V !! ⊗ EᏵ! .

(101)

Proposition 15. The complex C(V !! ⊗ EᏵ! ) is acyclic. Proof. The complex C(V !! ⊗ EᏵ! ) is isomorphic to the cone of the identity map of the complex C(VD ⊗ EᏵ )[−1]. Proposition 16. The complexes C(V ! ⊗EᏵ! ) and C(D)[−1]{−2} are isomorphic.

A CATEGORIFICATION OF THE JONES POLYNOMIAL

D

D1

393

b a

Figure 33

Proof. We have a chain of isomorphisms of complexes     C V ! ⊗ EᏵ! = C V ! (∗1) ⊗ EᏵ [−1]   = C VD {−2} ⊗ EᏵ [−1]   = C VD ⊗ EᏵ [−1]{−2} = C(D)[−1]{−2}. The first isomorphism here follows from (96) and is obtained by fixing an isomorphism between skew-commutative Ᏽ-cubes EᏵ! (∗1) and EᏵ . The second isomorphism comes from an isomorphism V ! (∗1) = VD {−2}, induced by ℵ. Corollary 4. The complexes C(D1 ) and C(D)[−1]{−2} are quasi-isomorphic. Proof. We have

  C(D1 ) = C VD1 ⊗ EᏵ!     = C V ! ⊗ EᏵ! ⊕ C V !! ⊗ EᏵ!   = C(D)[−1]{−2} ⊕ C V !! ⊗ EᏵ! = C(D)[−1]{−2} ⊕ (acyclic complex).

Note that x(D1 ) = x(D) + 1 and y(D1 ) = y(D). By (45),    C(D) = C(D) x(D) 2x(D) − y(D)

(102)

and C(D1 ) = C(D1 )[x(D1 )]{2x(D1 ) − y(D1 )} = C(D1 )[x(D) + 1]{2x(D) − y(D) + 2}. Therefore, complexes C(D) and C(D1 ) are quasi-isomorphic. 5.3. The tangency move. Let D and D1 be two diagrams that differ as depicted in Figure 33. In this section we construct a quasi-isomorphism of complexes C(D) and C(D1 ). We assume that D has n − 2 double points. Consequently, D1 has n double points. Let Ᏽ! be the set of double points of D1 , let Ᏽ be Ᏽ! \{a, b}, where a and b are double points of D1 depicted in Figure 33. We identify Ᏽ with the double points set of D.

394

MIKHAIL KHOVANOV

D1 (∗00)

D1 (∗01)

D1 (∗10)

D1 (∗11)

Figure 34

Denote by d the differential of the complex C(D1 ). Consider diagrams D1 (∗00), D1 (∗01), D1 (∗10), D1 (∗11) obtained by resolving double points a and b of D1 , as in Figure 34. (E.g., D1 (∗01) is constructed from D1 by taking the 0-resolution of a and 1-resolution of b, etc.) Each of these four diagrams has Ᏽ as the set of its double points. To each diagram D1 (∗uv), where u, v ∈ {0, 1}, there is associated the complex C(D1 (∗uv)) of graded R-modules. Denote by duv the differential in this complex:     duv : C D1 (∗uv) −→ C D1 (∗uv) .

(103)

(i)

We denote by duv the differential in shifted complexes,     (i) duv : C D1 (∗uv) [i]{i} −→ C D1 (∗uv) [i]{i}

for i ∈ Z.

(104)

The commutative Ᏽ-cube VD1 can be viewed as a commutative square of Ᏽ! -cubes VD1 (∗00) φ1



VD1 (∗10) {−1}

φ2

φ4

/ VD1 (∗01) {−1} 

φ3

/ VD1 (∗11) {−2},

where φi , 1 ≤ i ≤ 4, denote the corresponding cube maps. Recall that these cube maps are associated to certain elementary surfaces (see Sections 4.2, 4.3) that have one saddle point relative to the height function and no other critical points. For example, φ1 is associated to the surface shown in Figure 35. The maps φi induce maps ψi between complexes:     ψ1 : C D1 (∗00) −→ C D1 (∗10) {−1},     ψ2 : C D1 (∗00) −→ C D1 (∗01) {−1},     ψ3 : C D1 (∗01) [−1]{−1} −→ C D1 (∗11) [−1]{−2},     ψ4 : C D1 (∗10) [−1]{−1} −→ C D1 (∗11) [−1]{−2}.

A CATEGORIFICATION OF THE JONES POLYNOMIAL

395

Figure 35

We can decompose C(D1 ), considered as a Z ⊕ Z-graded R-module (see the end of Section 3.1), into the following direct sum of Z ⊕ Z-graded R-modules:     C(D1 ) = C D1 (∗00) ⊕ C D1 (∗01) [−1]{−1}     ⊕ C D1 (∗10) [−1]{−1} ⊕ C D1 (∗11) [−2]{−2}. Let us say a few words about this decomposition: C(D1 ) is the direct sum of Rmodules VD1 (ᏸ), which sit in the vertices of the Ᏽ! -cube VD1 . Since we presented this cube as a commutative square of Ᏽ-cubes VD1 (∗uv) {−u − v} for u, v ∈ {0, 1}, the above decomposition results. Well, almost. Indeed, when we pass from ᏶-cubes to complexes, we tensor with the fixed skew cube E᏶ . To define the left-hand side of the above formula, we tensor VD1 with the skew Ᏽ! -cube EᏵ! , while for the right-hand side similar tensor products are formed with the skew Ᏽ-cube EᏵ . Therefore, we must say how we identify R-modules sitting in the vertices of EᏵ! with R-modules sitting in the vertices of EᏵ . For D1 (∗00), we map EᏵ (ᏸ), where ᏸ ⊂ Ᏽ, to EᏵ! (ᏸ) by sending z ∈ o(ᏸ) to z ∈ o(ᏸ). For D1 (∗10), we map EᏵ (ᏸ), where ᏸ ⊂ Ᏽ, to EᏵ! (ᏸa) by sending z ∈ o(ᏸ) to za ∈ o(ᏸa). We proceed similarly for D1 (∗01). For D1 (∗11), we map EᏵ (ᏸ) to EᏵ! (ᏸab) by sending z ∈ o(ᏸ) to zab ∈ o(ᏸab). This is not a canonical choice, since we could have sent z to zba and would have gotten minus the original map. So, to define the latter map, we implicitly fix an ordering of a and b. Note that the above decomposition is not a direct sum of complexes, as the differ(−u−v) ential duv of C(D1 (∗uv)) differs from d restricted to C(D1 (∗uv))[−u−v]{−u− v} ⊂ C(D1 ), except when u = v = 1. Exactly, we have   dx = d00 x + [−1]ψ1 x + [−1]ψ2 x for x ∈ C D1 (∗00) ,   (−1) dx = −d01 x − [−1]ψ3 x for x ∈ C D1 (∗01) [−1]{−1},   (−1) dx = −d10 x + [−1]ψ4 x for x ∈ C D1 (∗10) [−1]{−1},   (−2) dx = d11 x for x ∈ C D1 (∗11) [−2]{−2}. Some explanation. Applying ψ1 to x ∈ C(D1 (∗00)), we get an element of the complex C(D1 (∗10)){−1}, so that we shift ψ1 x by [−1] to land it in C(D1 (∗10)) [−1]{−1} ⊂ C(D1 ), and so forth. Various signs in the above formulas come from our previous four identifications of the skew cube EᏵ with codimension 2 faces of EᏵ! .

396

MIKHAIL KHOVANOV

Figure 36

Figure 37

Let α be the map of complexes     α : C D1 (∗01) [−1]{−1} −→ C D1 (∗10) [−1]{−1}

(105)

associated to the surface shown in Figure 36. Considered as a map of Z ⊕ Z-graded R-modules, α is grading-preserving. Let β be the map of complexes     β : C D1 (∗11) [−2]{−2} −→ C D1 (∗10) [−1]{−1} (106) associated to the surface shown in Figure 37. Note that β is a graded map of degree (−1, 0). Let X1 , X2 , X3 be R-submodules of C(D1 ) given by     (107) X1 = z + α(z) | z ∈ C D1 (∗01) [−1]{−1} ,    (108) X2 = z + dw | z, w ∈ C D1 (∗00) ,     (109) X3 = z + β(w) | z, w ∈ C D1 (∗11) [−2]{−2} . Proposition 17. These submodules are stable under d: dXi ⊂ Xi

(110)

and respect the Z ⊕ Z-grading of C(D1 ). Proof. Let us first check that X1 , X2 , and X3 are direct sums of their graded components. For X2 it follows from the fact that C(D1 (∗00)) is a direct sum of its graded components and d is graded of degree (1, 0). Submodule X3 is graded

397

A CATEGORIFICATION OF THE JONES POLYNOMIAL

Figure 38

Figure 39

because C(D1 (∗11))[−2]{−2} is a direct sum of its graded components and β is a graded map. Finally, X1 is graded since α is grading-preserving. We now verify that these three submodules are stable under d. For X2 this is obvious. To see it for X3 , notice that dz ∈ C(D1 (∗11))[−2]{−2} whenever z ∈ C(D1 (∗11))[−2]{−2}. Moreover, for such a z, (−1)

(−1)

(−2)

dβ(z) = −d10 β(z) + [−1]ψ4 β(z) = −d10 β(z) + z = βd11 (z) + z.

(111)

The second equality is implied by [−1]ψ4 β = Id . Map ψ4 β is associated to the surface in Figure 38, which is obtained by composing surfaces to which ψ4 and β are associated. This surface is isotopic, through an isotopy fixing the boundary, to the surface shown in Figure 39, which represents the identity map. Hence [−1]ψ4 β = Id. (−2) Formula (111) implies that X3 is stable under d, since the rightmost term βd11 (z)+z lies in X3 . Finally, to check the d-stability of X1 , we compute, for z ∈ C(D1 (∗01))[−1]{−1},   d z + α(z) = dz + dα(z) (−1)

(−1)

= −d01 z − [−1]ψ3 z − d10 α(z) + [−1]ψ4 α(z)  (−1)    (−1) = − d01 z + d10 α(z) + [−1] − ψ3 z + ψ4 α(z)  (−1)  (−1) = − d01 z + d10 α(z)  (−1) (−1)  = − d01 z + αd01 z ∈ X1 . (−1)

In the fourth equality we use that ψ4 α = ψ3 , and in the fifth that αd01 since α is a grading-preserving map of complexes.

(−1)

= d10 α,

398

MIKHAIL KHOVANOV

Corollary 5. Submodules X1 , X2 , X3 are graded subcomplexes of the complex C(D1 ). Proposition 18. (1) We have a direct sum decomposition C(D1 ) = X1 ⊕ X2 ⊕ X3

(112)

in the category Kom(R- mod0 ) of complexes of graded R-modules. (2) The complexes X2 and X3 are acyclic. (3) The complex X1 is isomorphic to the complex C(D)[−1]{−1}. Proof. Since we already know that X1 , X2 , and X3 are graded subcomplexes of C(D1 ), it suffices to check (112) on the level of underlying abelian groups. We have α = βψ3 and, therefore, for z ∈ C(D1 (∗01))[−1]{−1}, αz = βψ3 z ∈ X3 .

(113)

Subcomplex X1 consists of elements z + αz and we know that αz ∈ X3 . We are thus reduced to proving the following direct sum splitting of abelian groups:   C(D1 ) = C D1 (∗01) [−1]{−1} ⊕ X2 ⊕ X3 . (114) Next recall that X2 consists of elements z + dw for z, w ∈ C(D1 (∗00)). The differential dw reads dw = d00 w + [−1]ψ1 w + [−1]ψ2 w.

(115)

Note that [−1]ψ2 (w) ∈ C(D1 (∗01))[−1]{−1} and d00 w ∈ C(D1 (∗00)). Let X2! be the subgroup of C(D1 ) given by    X2! = z + [−1]ψ1 w | z, w ∈ C D1 (∗00) . (116) Then it is enough to verify that C(D1 ) is a direct sum of its subgroups C(D1 (∗01))[−1] {−1}, X2! and X3 :   C(D1 ) = C D1 (∗01) [−1]{−1} ⊕ X2! ⊕ X3 . (117) Note that X3 contains C(D1 (∗11))[−2]{−2} and X2! contains C(D1 (∗00)). Recall the direct sum decomposition     C(D1 ) = C D1 (∗00) ⊕ C D1 (∗01) [−1]{−1}     ⊕ C D1 (∗10) [−1]{−1} ⊕ C D1 (∗11) [−2]{−2} of C(D1 ). Let X2!! and X3! be the following abelian subgroups of C(D1 (∗10))[−1]{−1}:    X2!! = [−1]ψ1 (w) | w ∈ C D1 (∗00) ,     X3! = β(w) | w ∈ C D1 (∗11) [−2]{−2} .

A CATEGORIFICATION OF THE JONES POLYNOMIAL

D1

399

D1

Figure 40

Now we are reduced to proving the direct sum decomposition   C D1 (∗10) [−1]{−1} = X2!! ⊕ X3!

(118)

in the category of abelian groups. As an abelian group, C(D1 (∗10))[−1]{−1} is a direct sum of F (D1 (ᏸa)) over all possible resolutions of the (n−2)-double points of D1 . Similar direct sum splittings can be formed for X2!! and X3! , and one sees then that it suffices to check (118) when D1 has only two double points. There are two such D1 ’s, as seen in Figure 40. In each of these two cases decomposition (118) follows from the splitting (2). That proves part 1 of the proposition. We next prove part 2. The complex C(X2 ) is isomorphic to the cone of the identity map of C(D1 (∗00))[−1] and, therefore, acyclic. Similarly, C(X3 ) is acyclic, being isomorphic to the cone of the identity map of C(D1 (∗11)){−2}[−2]. To prove part 3 of the proposition, notice that the diagrams D and D1 (∗01) are isomorphic. This induces an isomorphism between the complexes   C(D) = C D1 (∗01) . (119) An isomorphism

is given by

  ∼ = γ : C D1 (∗01) [−1]{−1} −−→ X1

(120)

  γ (z) = (−1)i z + α(z)

(121)

i

for z ∈ C (D1 (∗01))[−1]{−1}. We need (−1)i in the above formula to match the differentials in these two complexes. Corollary 6. The complexes C(D)[−1]{−1} and C(D1 ) are quasi-isomorphic. Note that x(D1 ) = x(D) + 1 and y(D1 ) = y(D) + 1. From (45) we get    C(D) = C(D) x(D) 2y(D) − x(D) ,    C(D1 ) = C(D1 ) x(D) + 1 2y(D) − x(D) + 1 , which, together with Corollary 6, implies that C(D) is quasi-isomorphic to C(D1 ).

400

MIKHAIL KHOVANOV

r1

q2 q1

p2

p1

r2 D1

D2 Figure 41

D1 (∗0)

D1 (∗1)

D2 (∗0)

D2 (∗1)

Figure 42

5.4. Triple point move. We are given two diagrams with n double points each, D1 and D2 , that differ as depicted in Figure 41. In this section we construct a quasiisomorphism of complexes C(D1 ) and C(D2 ). Let Ᏽ! be the set of double points of D1 . We have Ᏽ! = Ᏽ {p1 , q1 , r1 }, where Ᏽ are all double points not shown on Figure 41. In particular, we can identify Ᏽ {p2 , q2 , r2 } with the set of double points of D2 . For starters, consider Figure 42. The diagrams D1 (∗0), D1 (∗1), D2 (∗0), D2 (∗1) are obtained by resolving double points r1 of D1 and r2 of D2 . Note that diagrams D1 (∗1) and D2 (∗1) are isomorphic and that diagrams D1 (∗0) and D2 (∗0) represent isotopic links. We decompose C(D1 ) and C(D2 ) into following direct sums:      C(Di ) = C Di (∗1) [−1]{−1} ⊕ C Di (∗uv0) [−u − v]{−u − v}. (122) u,v∈{0,1}

These are direct sum decompositions of Z ⊕ Z-graded R-modules, not complexes. The diagrams Di (∗uv0) for i = 1, 2 and u, v ∈ {0, 1} are depicted in Figures 43 and 44. For all i, u, v, we identify the set of double points of Di (∗uv0) with Ᏽ. To fix the direct decomposition (122), we need identifications between the skew cube EᏵ and codimension 3 facets of EᏵ! . From the discussion in the previous section it should be clear how these identifications are chosen. For instance, for D1 (∗110), we map EᏵ to a codimension 3 facet of EᏵ! via maps EᏵ (ᏸ) → EᏵ! (ᏸp1 q1 ) given by o(ᏸ) $ z % −→ zp1 q1 ∈ o(ᏸ  {p1 , q1 }). Let τ1 be the map of complexes     (123) τ1 : C D1 (∗100) [−1]{−1} −→ C D1 (∗010) [−1]{−1}

A CATEGORIFICATION OF THE JONES POLYNOMIAL

D1 (∗000)

D1 (∗010)

D1 (∗100)

401

D1 (∗110)

Figure 43

D2 (∗000)

D2 (∗010)

D2 (∗100)

D2 (∗110)

Figure 44

Figure 45

associated to the surface shown in Figure 45. Relative to the height function, this surface has two critical points, one of which is a saddle point and the other a local minimum. Considered as a map of Z⊕Z-graded R-modules, τ1 is grading-preserving. Let δ1 be the map of complexes     δ1 : C D1 (∗110) [−2]{−2} −→ C D1 (∗010) [−1]{−1} (124) associated to the surface depicted in Figure 46. Let X1 , X2 , X3 be R-submodules of C(D1 ) given by       X1 = x + τ1 (x) + y | x ∈ C D1 (∗100) [−1]{−1}, y ∈ C D1 (∗1) [−1]{−1} ,    X2 = x + d1 y | x, y ∈ C D1 (∗000) ,     X3 = δ1 (x) + d1 δ1 (y) | x, y ∈ C D1 (∗110) [−2]{−2} , (125) where d1 denotes the differential of C(D1 ). Warning: These X1 , X2 , X3 have no relation to the complexes X1 , X2 , X3 considered in Section 5.3.

402

MIKHAIL KHOVANOV

Figure 46

Figure 47

Propositions 19–21 can be proved in the same fashion as Propositions 17 and 18 of the previous section. For this reason and to keep this paper concise, the proofs are omitted. Proposition 19. Submodules X1 , X2 , X3 are stable under d1 and respect the Z⊕ Z-grading of C(D1 ). Corollary 7. Submodules X1 , X2 , X3 are graded subcomplexes of the complex C(D1 ). Let τ2 be the map of complexes     τ2 : C D2 (∗010) [−1]{−1} −→ C D1 (∗100) [−1]{−1}

(126)

associated to the surface shown in Figure 47. Considered as a map of Z ⊕ Z-graded R-modules, τ2 is grading-preserving. Let δ2 be the map of complexes     δ2 : C D2 (∗110) [−2]{−2} −→ C D2 (∗100) [−1]{−1} (127) associated to the surface in Figure 48. Let Y1 , Y2 , Y3 be R-submodules of C(D2 ) given by       Y1 = x + τ2 (x) + y | x ∈ C D2 (∗010) [−1]{−1}, y ∈ C D2 (∗1) [−1]{−1} ,    Y2 = x + d2 y | x, y ∈ C D2 (∗000) ,     Y3 = δ2 (x) + d2 δ2 (y) | x, y ∈ C D2 (∗110) [−2]{−2} , (128) where d2 stands for the differential of C(D2 ).

A CATEGORIFICATION OF THE JONES POLYNOMIAL

403

Figure 48

Proposition 20. These submodules are stable under d2 and respect the Z ⊕ Zgrading of C(D2 ). Corollary 8. Subcomplexes Y1 , Y2 , Y3 are graded subcomplexes of the complex C(D2 ). Proposition 21. (1) We have direct sum decompositions C(D1 ) = X1 ⊕ X2 ⊕ X3 ,

(129)

C(D2 ) = Y1 ⊕ Y2 ⊕ Y3 .

(130)

(2) The complexes X2 , X3 , Y2 , and Y3 are acyclic. (3) The complexes X1 and Y1 are isomorphic. Proof. Parts 1 and 2 of this proposition are proved similarly to Proposition 18. The isomorphism X1 ∼ = Y1 comes from the diagram isomorphisms D1 (∗100) = D2 (∗010), D1 (∗1) = D2 (∗1). These diagram isomorphisms induce isomorphisms of complexes     C D1 (∗100) = C D2 (∗010) ,     C D1 (∗1) = C D2 (∗1) ,

(131)

(132)

which allow us to identify x in the definition (125) of X1 with x in the definition (128) of Y1 and, similarly, identify y’s. An isomorphism X1 ∼ = Y1 of complexes is then given by X1 $ x + τ1 (x) + y % −→ x + τ2 (x) + y ∈ Y1 .

(133)

Corollary 9. Complexes C(D1 ) and C(D2 ) are quasi-isomorphic. The above isomorphism of complexes X1 and Y1 induces a quasi-isomorphism of C(D1 ) and C(D2 ). Note that x(D1 ) = x(D2 ) and y(D1 ) = y(D2 ). Therefore, the complexes C(D1 ) and C(D2 ) are quasi-isomorphic, and the cohomology groups H i (D1 ) and H i (D2 ) are isomorphic as graded R-modules. This completes the proof of Theorem 1.

404

MIKHAIL KHOVANOV

+1

−1 Figure 49

6. Properties of cohomology groups 6.1. Some elementary properties. Pick an oriented link L and a component L! of L. Let L0 be L with the orientation of L! reversed, and let l be the linking number of L! and L \ L! . Fixing a plane diagram D of L, we count l as half the number of double intersection points in D of L! with L \ L! with weights +1 or −1 according to the convention depicted in Figure 49. Denote by D0 the diagram D with the reversed orientation of L! . Since D0 and D are the same as unoriented diagrams, C(D0 ) = C(D). Also x(D0 ) = x(D) − 2l,

y(D0 ) = y(D) + 2l.

(134)

Proposition 22. For L, L0 as above, there is an equality H i (L0 ) = H i+2l (L){2l}

(135)

of isomorphism classes of graded R-modules. Let K, K1 be oriented knots and (−K) be K with its orientation reversed. In a similar fashion we deduce the following. Proposition 23. There is an equality     H i K#K1 = H i (−K)#K1

(136)

of isomorphism classes of graded R-modules. Let D be a diagram of an oriented link L and denote by cm(L) the number of connected components of L. Then it is easy to see that Cji (D) = 0 if parities of j and cm(L) differ. This observation implies the next proposition. Proposition 24. For an oriented link L, H i,j (L) = 0 if j + 1 ≡ cm(L) (mod 2).

(137)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

405

D

D(∗00)

D(∗01)

D(∗10)

D(∗11)

Figure 50. Diagram D and its four resolutions

6.2. Computational shortcuts and cohomology of (2, n) torus links. Given a plane diagram D, a straighforward computation of cohomology groups H i (D) is daunting. These groups are cohomology groups of the graded complex C(D), and the ranks of the abelian groups Cji (D) grow exponentially in the complexity of D. Probably there is no fast algorithm for computing H i (D), since these groups carry full information about the Jones polynomial, computing which is #P -hard (see [JVW]). Yet, one can try to reduce C(D) to a much smaller complex, albeit still exponentially large, but more practical for a computation. In this section we provide an example by simplifying C(D) in the case when D contains a chain of positive halftwists, and we apply our result by computing cohomology groups of (2, n) torus links. Let D be a plane diagram with n crossings and suppose that D contains a subdiagram as pictured in Figure 50. Four possible resolutions of these two double points of D produce diagrams D(∗00), D(∗01), D(∗10), and D(∗11). Note that diagrams D(∗01) and D(∗10) are isomorphic and D(∗00) is isomorphic to a union of D(∗01) and a simple circle. The complex C(D) is isomorphic to the total complex of the bicomplex   ∂0     · · · −→ 0 −→ C D(∗00) −−→ C D(∗01) {−1} ⊕ C D(∗10) {−1}   ∂1 −−→ C D(∗11) {−2} −→ 0 −→ · · · , where the differentials ∂ 0 and ∂ 1 are determined by the structure maps of the skew Ᏽ-cube VD ⊗EᏵ (where Ᏽ is the set of crossings of D). Denote this bicomplex by C. To simplify notation we denote the diagram D(∗01) by D0 and D(∗11) by D1 (see

406

MIKHAIL KHOVANOV

D0

D1

Figure 51

Figure 51). Then the bicomplex C becomes ∂0

· · · −→ 0 −→ C(D0 ) ⊗ A −−→ C(D0 ){−1} ⊕ C(D0 ){−1} ∂1

−−→ C(D1 ){−2} −→ 0 −→ · · · . Clearly, the differential ∂ 0 , if restricted to the subcomplex C(D0 )⊗1 of C(D0 )⊗A, is injective, and so the total complex of the subbicomplex ∂0

0 −→ C(D0 ) ⊗ 1 −−→ C(D0 ){−1} −→ 0

(138)

of C is acyclic. Denote this subbicomplex by Cs and the quotient bicomplex by C/Cs . The total complexes Tot(C) and Tot(C/Cs ) of C and C/Cs are quasi-isomorphic; so to compute the cohomology of C(D) = Tot(C) it suffices to find the cohomology of Tot(C/Cs ). We next give a precise description of the bicomplex Tot(C/Cs ). Let u, l, w be maps of complexes   (139) u : C D(∗00) −→ C(D0 ),   (140) l : C D(∗00) −→ C(D0 ), w : C(D0 ) −→ C(D1 )

(141)

induced by the surfaces shown in Figure 52, respectively. Note that each of these maps has degree −1, and to make them homogeneous we need to shift gradings of our complexes appropriately. We use the same notation for shifted maps, since the shifts are always clear. Let   (142) v : C(D0 ) −→ C(D0 ) ⊗ A = C D(∗00) be the map of complexes v(t) = t ⊗X, t ∈ C(D0 ). The map v has degree −1. Denote by uX and lX the compositions uX = u ◦ v,

lX = l ◦ v.

(143)

These are degree (−2) maps of complexes, and for each i they induce degree 0 maps C(D0 ){i} → C(D0 ){i − 2}, also denoted uX and lX .

A CATEGORIFICATION OF THE JONES POLYNOMIAL

407

Figure 52

Lemma 2. The bicomplex C/Cs is isomorphic to the bicomplex uX −lX

w

0 −→ C(D0 ){1} −−−−−→ C(D0 ){−1} −−→ C(D1 ){−2} −→ 0.

(144)

We skip the proof, which is a simple linear algebra. i

Corollary 10. Cohomology groups H (D) are isomorphic to the cohomology of the total complex of the bicomplex (144). i

We thus see that the cohomology H (D) of the diagram D can be computed via the quotient complex Tot(C/Cs ) of C(D). The quotient complex is smaller than the original one, and computing its cohomology requires less work. This reduction is not drastic since ranks of homogeneous components of complexes C(D) and Tot(C/Cs ) have the same order of magnitude; but a similar reduction (described next, when D contains a long chain of positive twists) leads to an effective computation of H i (D) for certain diagrams D. Suppose that a diagram D contains a chain of k positive half-twists, as in Figure 53. As before, denote by D0 and D1 diagrams that are suitable resolutions of the k-chain of D.

408

MIKHAIL KHOVANOV

D

k crossings

D0

D1

Figure 53. Diagram D and two resolutions

From our previous discussion we retain degree (−2) maps uX , lX and a degree (−1) map w between (appropriately shifted) complexes C(D0 ) and C(D1 ). Let C ! be the bicomplex ∂0

∂1

0 −→ C(D0 ){k − 1} −−→ C(D0 ){k − 3} −−→ · · · ∂ k−3

∂ k−2

∂ k−1

−−−→ C(D0 ){3 − k} −−−→ C(D0 ){1 − k} −−−→ C(D1 ){−k} −→ 0, where ∂ k−1 = w, ∂ k−2 = uX − lX , ∂ k−3 = uX + lX , ∂ k−4 = uX − lX , ··· ∂ = uX − (−1)k lX , 0

that is, ∂ k−i = uX − (−1)i lX ,

for 2 ≤ i ≤ k.

(145)

Proposition 25. The complex C(D) is quasi-isomorphic to the total complex i Tot(C ! ) of the bicomplex C ! . Cohomology groups H (D) are isomorphic to the co! homology groups of Tot(C ). The proof goes by induction on k, induction base k = 2 being given by Corollary 10, and consists of finding a suitable acyclic subcomplex by which to quotient. We omit the details. We conclude this section by applying this proposition to compute cohomology groups of (2, k) torus links. Fix k > 0 and denote by D the diagram of the (2, k) torus link T2,k depicted in Figure 54. The diagram D0 is isomorphic to a simple circle and D1 to a disjoint union of two simple circles. Then uX = lX is the operator A → A of multiplication by X and the

A CATEGORIFICATION OF THE JONES POLYNOMIAL

D

409

k crossings

Figure 54

bicomplex C ! becomes a complex 0

0 −→ A{k − 1} −→ A{k − 3} −→ · · · −→ A{5 − k} 2X

"

0

−−−→ A{3 − k} −→ A{1 − k} −−→ A ⊗ A{−k} −→ 0. Recalling that x(D) = k and y(D) = 0, we get the following. Proposition 26. The isomorphism classes of the graded R-modules H i (T2,k ) are given by H i (T2,k ) = 0

for i < −k and i > 0,

H 0 (T2,k ) = R{k} ⊕ R{k − 2}, H −1 (T2,k ) = 0,   H −2j (T2,k ) = R/2R {4j + k} ⊕ R{4j − 2 + k} H −2j −1 (T2,k ) = R{4j + 2 + k}

for 1 ≤ j ≤

k −1 , j ∈ Z, 2 for even k.

k −1 , j ∈ Z, 2

for 1 ≤ j ≤

H −k (T2,k ) = R{3k} ⊕ R{3k − 2}

6.3. Link cobordisms and maps of cohomology groups. In this section, by a surface S in R4 we mean an oriented, compact surface S, possibly with boundary, properly embedded in R3 × [0, 1]. The boundary of S is then a disjoint union ∂S = ∂0 S  −∂1 S of the intersections of S with two boundary components of R3 × [0, 1]:   ∂0 S = S ∩ R3 × {0} ,   −∂1 S = S ∩ R3 × {1} . Note that ∂0 S and ∂1 S are oriented links in R3 .

(146)

410

MIKHAIL KHOVANOV

birth

death

fusion

fusion

Figure 55

The surface S can be represented by a sequence J of plane diagrams of oriented links where every two consecutive diagrams in J are related either by one of the Reidemeister moves (see Figures 10–13 of Section 4.1) or by one of the four moves depicted in Figure 55. (See [CS] where such representations by sequences of plane diagrams are studied in detail.) Following Carter-Saito [CS] and Fisher [Fs], we call these moves birth, death, and fusion. ([CS] and [Fs] deal with the nonoriented version of these moves.) We call J a representation of S. The first diagram in the sequence J is necessarily a diagram of the oriented link ∂0 S, and the last diagram is a diagram of ∂1 S. The birth move consists of adding a simple closed curve to a diagram D. Denote the new diagram by D1 . Then C(D1 ) = C(D) ⊗R A and the unit map ι : R → A of the algebra A induces a map of complexes C(D) → C(D1 ). This is the map we associate to the birth move. The death move consists of removing a simple circle from a diagram D1 to get a diagram D. In this case the counit # : A → R induces a map of complexes C(D1 ) → C(D). Finally, to a fusion move between diagrams D0 and D1 we associate a map C(D0 ) → C(D1 ) corresponding to the elementary surface with one saddle point in the manner discussed in Section 4.3. In Section 5, to each Reidemeister move between diagrams D0 and D1 we associated a quasi-isomorphism map of complexes C(D0 ) → C(D1 ). Given a representation J of a surface S by a sequence of diagrams, denote the first and last diagrams of J by J0 and J1 , respectively. Then to J we can associate a map of complexes ϕJ : C(J0 ) −→ C(J1 ),

(147)

which is the composition of maps associated to elementary transformations between consecutive diagrams of J. The map ϕJ induces a map of cohomology groups θJ : H i,j (J0 ) −→ H i,j +χ (S) (J1 ), We are now ready to state our main conjecture.

i, j ∈ Z.

(148)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

411

Conjecture 1. If two representations J, J of a surface S have the property that (a) diagrams J0 and J0 are isomorphic, (b) diagrams J1 and J1 are isomorphic, then the maps θJ and θJ are equal up to an overall minus sign: θJ = ±θJ. In other words, we conjecture that, after a suitable Z2 extension of the link cobordism category, our construction associates honest cohomology groups H i (L) to oriented links L in R3 (and not just isomorphism classes of groups) and associates homomorphisms between these groups to isotopy classes of oriented surfaces embedded in R3 × [0, 1]. In the categorical language, we expect to get a functor from the category of (Z2 -extended) oriented link cobordisms to the category of bigraded R-modules and module homomorphisms. Suppose that the above conjecture is true. Then, in the case of a closed oriented surface S embedded in R4 , the map θS of cohomology groups is a homomorphism from R to itself (since ∂S = ∅ and the cohomology of the empty link is equal to the ground ring R). This homomorphism has degree χ(S) and is automatically zero when χ (S) < 0. Thus, the conjectural invariants are zero whenever S has empty boundary and the Euler characteristic of S is negative. If ∂S = ∅ and the Euler characteristic of S is nonnegative (when S is connected, S is then necessarily a 2-sphere or a 2-torus), the homomorphism θS : R → R is determined by θS (1) = kc(χ(S)/2) and amounts to an integer number k. Hence, we expect to have integer-valued invariants of closed oriented surfaces with nonnegative Euler characteristic, embedded in R4 . 7. Setting c to zero 7.1. Cohomology groups Ᏼi,j . Setting c = 0 and taking Z instead of R = Z[c] as the base ring, everything from Sections 2, 4, and 5 goes through in exactly the same manner. The role of the ring A is played by the free graded abelian group Ꮽ of rank 2 with generators 1 and X in degrees 1 and −1, correspondingly. Ꮽ has commutative algebra and cocommutative coalgebra structures,

12 = 1,

1X = X1 = X,

"(1) = 1 ⊗ X + X ⊗ 1,

X 2 = 0,

"(X) = X ⊗ X,

(149) (150)

and the identity (16) holds. By abuse of notation, we use m and " to denote multiplication and comultiplication in Ꮽ. Earlier, m and " were used to denote multiplication and comultiplication in A. As in Section 2.3, we construct a functor Ᏺ from the category ᏹ of closed 1-manifolds and cobordisms between them to the category of graded abelian groups and graded homomorphisms. To a disjoint union of k circles, the functor Ᏺ assigns the group Ꮽ⊗k . To elementary surfaces S21 , S12 , S01 , S10 , S22 , and S11 (see Section 2.3), the functor Ᏺ assigns maps m, ", ι, #, Perm, and Id between

412

MIKHAIL KHOVANOV

suitable tensor powers of Ꮽ. The maps ι : Z → Ꮽ and # : Ꮽ → Z are given by ι(1) = 1,

#(1) = 0,

#(X) = 1,

(151)

while Perm is just the permutation map Ꮽ ⊗ Ꮽ → Ꮽ ⊗ Ꮽ. To a diagram D of an oriented link L, we can then associate a commutative Ᏽ-cube ᐂD of graded abelian groups and grading-preserving homomorphism, by the same procedure as the one described in Section 4.2, using the functor Ᏺ instead of F . In particular, for ᏸ ⊂ Ᏽ we have ᐂD (ᏸ) = Ᏺ(D(ᏸ)){−|ᏸ|} where {k} shifts the grading down by k. Let ᏭᏮ be the category of graded abelian groups and grading-preserving homomorphisms. Let ᏱᏵ be the skew-commutative Ᏽ-cube EᏵ ⊗R Z over ᏭᏮ. Tensoring ᐂD with ᏱᏵ over Z, we get a skew-commutative Ᏽ-cube ᐂD ⊗ ᏱᏵ over the category ᏭᏮ. From this skew-commutative Ᏽ-cube, we get a complex Ꮿ(ᐂD ⊗ ᏱᏵ ) of graded abelian groups with a grading-preserving differential (see Section 3.4). Denote by Ꮿ(D) this complex and by Ꮿ(D) the shifted complex 





Ꮿ(D) = Ꮿ(D) x(D) 2x(D) − y(D) .

(152)

If we consider Z as a graded R-module, concentrated in degree zero so that cZ = 0, then Ꮿ(D) = C(D) ⊗R Z

and

Ꮿ(D) = C(D) ⊗R Z.

(153)

To a plane diagram D of an oriented link L, we thus associate a complex of graded abelian groups Ꮿ(D). Denote the ith cohomology group of the j th graded summand of Ꮿ(D) by Ᏼi,j (D). These cohomology groups are finitely generated abelian groups. For each diagram D as i and j vary over all integers, only a finite number of these groups are nonzero. Theorem 2. For an oriented link L, isomorphism classes of abelian groups Ᏼi,j (D) do not depend on the choice of a diagram D of L and are invariants of L. Proof. Set c = 0 in the proof of Theorem 1. For a diagram D of the link L, denote the isomorphism classes of Ᏼi,j (D) by i Ᏼi,j (L). Denote by Ꮿ (D) (respectively, Ꮿi (D)) the ith group of the complex Ꮿ(D) i (respectively, Ꮿ(D)) and by Ꮿj (D) (respectively, Ꮿij (D)) the j th graded compoi

i

i

nent of Ꮿ (D) (respectively, Ꮿi (D)), so that Ꮿ (D) = ⊕j ∈Z Ꮿj (D) (respectively, Ꮿi (D) = ⊕j ∈Z Ꮿij (D)). For a diagram D denote by Ᏼi (D) the graded abelian group ⊕j ∈Z Ᏼi,j (D). In other words, Ᏼi (D) is the ith cohomology group of Ꮿ(D). Denote i

i,j

by Ᏼ (D) the ith cohomology group of the complex Ꮿ(D) and by Ᏼ (D) the j th i i i,j graded component of Ᏼ (D), so that Ᏼ (D) = ⊕j ∈Z Ᏼ (D).

A CATEGORIFICATION OF THE JONES POLYNOMIAL

413

7.2. Properties of Ᏼi,j : Euler characteristic, change of orientation. The Kauffman bracket of an oriented link L is equal to the graded Euler characteristic of the cohomology groups Ᏼi,j (L), as stated in the following proposition. Proposition 27. For an oriented link L,
  (−1)i q j dimQ Ᏼi,j (L) ⊗ Q , K(L) =

(154)

i,j ∈Z

where K(L) is the scaled Kauffman bracket (see Section 2.4). The proof is completely analogous to that of formula (47). The statements and proofs of Propositions 22–24 transfer without change to the case of cohomology groups Ᏼi,j , as indicated below. Let L be an oriented link and L! a component of L. Denote by l the linking number of L! with its complement L \ L! in L. Let L0 be the link L with the orientation of L! reversed. Proposition 28. For i, j ∈ Z there is an equality of isomorphism classes of abelian groups Ᏼi,j (L0 ) = Ᏼi+2l,j +2l (L).

(155)

Proposition 29. Let K and K1 be oriented knots and (−K) be K with the reversed orientation. Then   Ᏼi,j (K#K1 ) = Ᏼi,j (−K)#K1 . (156) Similarly to Proposition 24 we can prove the following. Proposition 30. For an oriented link L, Ᏼi,j (L) = 0

(157)

if j + 1 ≡ cm(L) (mod 2). 7.3. Cohomology of the mirror image. Let L be an oriented link and denote by L! the mirror image of L. Let D be a diagram of L with n crossings, Ᏽ the set of these crossings, and D ! the corresponding diagram of L! , as shown in Figure 56. If M is a graded abelian group, M = ⊕j ∈Z Mj , define the dual graded abelian group M ∗ by (M ∗ )j = Hom(M−j , Z). The dual map f ∗ : N ∗ → M ∗ of a map f : M → N is defined as the dual of f in the sense of linear algebra. For ᏸ ⊂ Ᏽ denote by  ᏸ the complement Ᏽ \ ᏸ. Let ᐂ be a commutative Ᏽ-cube over the category ᏭᏮ of graded abelian groups and grading-preserving homomorphisms. Define the dual cube ᐂ∗ by   ∗ ᐂ ∗ (ᏸ ) = ᐂ  (158) ᏸ .

414

MIKHAIL KHOVANOV

D!

D

Figure 56

The structure map ∗

ξaᐂ : ᐂ∗ (ᏸ) −→ ᐂ∗ (ᏸa)

(159)

of ᐂ∗ is the dual of the structure map     ξᐂ : ᐂ  ᏸ \ a −→ ᐂ  ᏸ

(160)

a

of ᐂ. Denote by {s} the automorphism of the category ᏭᏮ that shifts the grading down by s. If ᐂ is a commutative Ᏽ-cube over ᏭᏮ, denote by ᐂ{s} the commutative Ᏽ-cube ᐂ with the grading of each group ᐂ(ᏸ) shifted down by s. Proposition 31. Let D be a diagram with n crossings and D ! the dual diagram. Then the commutative Ᏽ-cube ᐂD ! {−n} is isomorphic to the dual (ᐂD )∗ of the Ᏽcube ᐂD . Proof. Introduce a basis {1∗ , X ∗ } in the abelian group Ꮽ∗ = Hom(Ꮽ, Z) by 1∗ (1) = 0,

1∗ (X) = 1,

X ∗ (1) = 1,

X ∗ (X) = 0.

(161)

Denote by m∗ , "∗ maps dual to " and m, respectively: m∗ : Ꮽ∗ ⊗ Ꮽ∗ −→ Ꮽ∗ , "∗ : Ꮽ∗ −→ Ꮽ∗ ⊗ Ꮽ∗ . Then, in the basis {1∗ , X ∗ }, these maps are     m∗ 1∗ ⊗ X ∗ = m∗ X ∗ ⊗ 1∗ = X ∗ ,   m∗ 1∗ ⊗ 1∗ = 1∗ ,   m∗ X ∗ ⊗ X ∗ = 0,   " ∗ 1∗ = 1 ∗ ⊗ X ∗ + X ∗ ⊗ 1 ∗ ,   "∗ X ∗ = X ∗ ⊗ X ∗ . Hence, under the isomorphism µ : Ꮽ → Ꮽ∗ of graded abelian groups, given by

415

A CATEGORIFICATION OF THE JONES POLYNOMIAL

µ(1) = 1∗ and µ(X) = X ∗ , maps m, " become m∗ , "∗ . Note that the  ᏸ-resolution D( ᏸ) of diagram D and the ᏸ-resolution D ! (ᏸ) of D ! are isomorphic. Let k be the number of circles in D( ᏸ). Then      ! Ᏺ D  ᏸ = Ᏺ D (ᏸ) = Ꮽ⊗k (162) and, via µ, we can identify   ∗    Ᏺ D  ᏸ = (Ꮽ∗ )⊗k = Ᏺ D ! (ᏸ) .

(163)

Since µ maps m, " to m∗ , "∗ , we see that after suitable shifts (recall that ᐂD (ᏸ) is equal to Ᏺ(D(ᏸ)) shifted up by |ᏸ|), the identification (163) extends to an isomorphism of Ᏽ-cubes ᐂD ! {−n} and (ᐂD )∗ . Given a complex C of graded abelian groups and grading-preserving homomorphisms di

· · · −→ C i −−→ C i+1 −→ · · · ,

(164)

define the dual complex C ∗ by (C ∗ )i = (C −i )∗ , the differential (d ∗ )i being the dual of the differential d −i−1 of C. From the last proposition we easily obtain the following. Proposition 32. The complex Ꮿ(D ! ) is isomorphic to the dual of the complex Ꮿ(D). Corollary 11. For an oriented link L and integers i, j , there are equalities of isomorphism classes of abelian groups   Ᏼi,j L! ⊗ Q = Ᏼ−i,−j (L) ⊗ Q, (165)   1−i,−j  i,j  !  (L) , (166) Tor Ᏼ L = Tor Ᏼ where Tor stands for the torsion subgroup. Note that this corollary provides a necessary condition for a link to be amphicheiral. 7.4. Cohomology of the disjoint union and connected sum of knots. Pick diagrams D1 , D2 of oriented links L1 , L2 and consider a diagram D1 D2 of the disjoint union L1  L2 . We then have an isomorphism of cochain complexes Ꮿ(D1  D2 ) = Ꮿ(D1 ) ⊗ Ꮿ(D2 )

(167)

of free graded abelian groups. From the Künneth formula we derive the following. Proposition 33. There is a short split exact sequence of cohomology groups    Ᏼi,j (D1 ) ⊗ Ᏼk−i,m−j (D2 ) −→ Ᏼk,m D1  D2 ) 0 −→ i,j ∈Z

−→



i,j ∈Z

  Tor Z1 Ᏼi,j (D1 ), Ᏼk−i+1,m−j (D2 ) −→ 0.

416

MIKHAIL KHOVANOV

1

1

2

D1

D2

2

1

D3

2 D4

1

2 D5 Figure 57

Corollary 12. For each k, m ∈ Z, there is an equality of isomorphism classes of abelian groups   Ᏼk,m (L1  L2 ) = Ᏼi,j (L1 ) ⊗ Ᏼk−i,m−j (L2 ) i,j ∈Z



  Tor Z1 Ᏼi,j (L1 ), Ᏼk−i+1,m−j (L2 ) .

i,j ∈Z

Let D1 , D2 be diagrams of oriented knots K1 , K2 , as in Figure 57. Consider diagrams D3 , D4 , and D5 of oriented links K1  K2 , K1 #K2 , and K1 #(−K2 ). By resolving the central double point of D5 , we get a short exact sequence of complexes of graded abelian groups 0 −→ Ꮿ(D3 )[−1]{−1} −→ Ꮿ(D5 ) −→ Ꮿ(D4 ) −→ 0.

(168)

After shifts, we obtain an exact sequence 0 −→ Ꮿ(D3 )[−1]{−1} −→ Ꮿ(D5 )[−1]{−2} −→ Ꮿ(D4 ) −→ 0,

(169)

which induces, for every integer j , a long exact sequence of cohomology groups · · · −→ Ᏼi−1,j −1 (D3 ) −→ Ᏼi−1,j −2 (D5 ) −→ Ᏼi,j (D4 ) −→ −→ Ᏼi,j −1 (D3 ) −→ Ᏼi,j −2 (D5 ) −→ Ᏼi+1,j (D4 ) −→ · · · .

(170)

A CATEGORIFICATION OF THE JONES POLYNOMIAL

417

Since the diagrams D3 , D4 , and D5 represent oriented links K1  K2 , K1 #K2 , and K1 #(−K2 ), respectively, then in view of Proposition 29, we obtain the next proposition. Proposition 34. For oriented knots K1 , K2 , the isomorphism classes of the abelian groups Ᏼi,j (K1  K2 ), Ᏼi,j (K1 #K2 ) can be arranged into long exact sequences −→ Ᏼi−1,j −1 (K1  K2 ) −→ Ᏼi−1,j −2 (K1 #K2 ) −→ Ᏼi,j (K1 #K2 ) −→ −→ Ᏼi,j −1 (K1  K2 ) −→ Ᏼi,j −2 (K1 #K2 ) −→ Ᏼi+1,j (K1 #K2 ) −→ .

(171)

7.5. A spectral sequence. Let D be an plane diagram of a link. In this section we construct a spectral sequence whose E1 term is made of groups Ᏼi,j (D) and which converges to cohomology groups H i,j (D). Due to the direct sum decomposition R = ⊕k≥0 ck Z of abelian groups, we have an abelian group decomposition  Cji (D) = Ꮿij −2k (D), (172) k≥0

where Cji (D) and Ꮿij (D) are defined as in Sections 4.2 and 7.1, respectively. Let us fix a j ∈ Z. Denote by d the differential in the weight-j subcomplex of the complex C(D): d

d

d

d

· · · −−→ Cji−1 (D) −−→ Cji (D) −−→ Cji+1 (D) −−→ · · · .

(173)

Denote by ∂ the differential in the complex ∂

∂

∂

∂

i+1 i · · · −−→ Ꮿi−1 j −2k (D) −−→ Ꮿj −2k (D) −−→ Ꮿj −2k (D) −−→ · · · ,

(174)

where we suppress the dependence of ∂ on k. Under the identification (172), the differential d becomes a differential of the complex d  i d  i+1 d · · · −−→ Ꮿj −2k (D) −−→ Ꮿj −2k (D) −−→ · · · . (175) k≥0

k≥0

Consider a bigraded abelian group C=

 k≥0,i∈Z

Ꮿij −2k (D),

(176)

where we set the grading of Ꮿij −2k (D) to (i, −k). We thus have a bigraded abelian group C and two maps, d and ∂, from C to C. Map ∂ is bigraded of degree (1, 0) while d is only graded relative to the first grading. However, we can decompose d = ∂ + ∂,

(177)

418

MIKHAIL KHOVANOV

where  ∂ has grading (1, −1) and satisfies  ∂ ∂ = 0, ∂ ∂ + ∂∂ = 0.

(178) (179)

Besides, since ∂ is a differential, ∂∂ = 0. Therefore, H i,j (D) is equal to the ith cohomology group of the total complex of the bicomplex (C, ∂, ∂). Group Ᏼs,j −2k (D) is equal to the sth cohomology group of the subcomplex (relative to the differential ∂) ∂

∂

∂

∂

· · · −−→ Ꮿi−1,j −2k (D) −−→ Ꮿi,j −2k (D) −−→ C i+1,j −2k (D) −−→ · · ·

(180)

of C. Therefore, for each j ∈ Z we get a spectral sequence whose E1 term is given by cohomology groups Ᏼs,j −2k (D), s ∈ Z, k ≥ 0, and which converges to cohomology groups H i,j (D). In the few cases where we managed to compute cohomology groups, we have H i,j (D) = ⊕k≥0 Ᏼi,j −2k (D) and, consequently, the spectral sequence degenerates at E1 . We have no idea whether this is true for any diagram D. 7.6. Examples. Perhaps the graded groups Ᏼi (D) are easier to compute than H i (D). The latter are computed via the complex C(D) of free graded R-modules, and a complex for Ᏼi (D) is obtained by tensoring C(D) with Z over R, so that free graded R-modules become free abelian groups of the same rank. In practice, the computation of Ᏼi (D) faces the problem of effectively simplifying complexes of abelian groups of exponentially high rank. The shortcut for the computation of H i (D) described in Section 6.2 works equally well for groups Ᏼi (D), with Proposition 25 generalized to complexes Ꮿ(D). This easily leads to a computation of cohomology groups Ᏼi,j (T2,k ) of the (2, k) torus link T2,k , oriented as in Section 6.2. Proposition 35. Cohomology groups Ᏼi,j (T2,k ), k > 1 are isomorphic to Ᏼ0,−k (T2,k ) = Z, Ᏼ0,2−k (T2,k ) = Z, Ᏼ−2j −1,−4j −2−k (T2,k ) = Z Ᏼ−2j,−4j −k (T2,k ) = Z2 Ᏼ−2j,−4j +2−k (T2,k ) = Z Ᏼ−k,−3k (T2,k ) = Z Ᏼ−k,2−3k (T2,k ) = Z Ᏼi,j (T2,k ) = 0

k −1 , j ∈ Z, 2 k −1 , j ∈ Z, for 1 ≤ j ≤ 2 k −1 , j ∈ Z, for 1 ≤ j ≤ 2 for even k, for 1 ≤ j ≤

for even k, for all other values of i and j .

419

A CATEGORIFICATION OF THE JONES POLYNOMIAL

7.7. An application to the crossing number Definition 3. A plane diagram D with the set Ᏽ of double points is called + adequate if for each double point a the diagram D(Ᏽ \ {a}) has one circle less than D(Ᏽ). Definition 4. A plane diagram D with the set Ᏽ of double points is called − adequate if for each double point a the diagram D({a}) has one circle less than D(∅). Definition 5. A plane diagram D is called adequate if it is both + and −adequate. These definitions are from [LT] and [T]. 0

Proposition 36. Let D be a diagram with n crossings. Then Ᏼ (D)  = 0 if and n only if D is −adequate and Ᏼ (D)  = 0 if and only if D is +adequate. 0

1

0

Proof. The differential ∂ 0 : Ꮿ (D) → Ꮿ (D) is not injective and, hence, Ᏼ (D)  = n 0 if and only if D is −adequate. We can proceed similarly for Ᏼ (D) and +adequate i diagrams. (Groups Ᏼ (D) are defined at the end of Section 7.1.) Definition 6. Homological length hl(L) of an oriented link L is the difference between the maximal i such that Ᏼi (L)  = 0 and the minimal i such that Ᏼi (L)  = 0. Denote by c(L) the crossing number of L. It is the minimal number of crossings in a plane diagram of L. Proposition 37. For an oriented link L, c(L) ≥ hl(L).

(181) i

Proof. Let D be a diagram of L with c(L) crossings. Then Ꮿ (D) = 0 for i < 0 i and for i > hl(L). Consequently, Ᏼ (D) = 0 for i < 0 and i > hl(L). Corollary 13. Let D be an adequate diagram with n crossings of a link L. Then c(L) = n. 0

n

Proof. By Proposition 36, Ᏼ (D)  = 0 and Ᏼ (D)  = 0. Therefore, c(L) ≥ hl(L) ≥ n. But since D is an n-crossing diagram of L, the crossing number of L is n. Corollary 13 was originally obtained by Thistlethwaite (see [T, Corollary 3.4]) through the analysis of the 2-variable Kauffman polynomial (not to be confused with the Kauffman bracket). 8. Invariants of (1, 1)-tangles 8.1. Graded A-modules. In Section 2.2 we define algebra A as a free module of rank 2 over the ring R = Z[c], generated by 1 and X, with the multiplication rules 11 = 1,

1X = X1 = X,

X 2 = 0.

(182)

420

MIKHAIL KHOVANOV

Gradings of 1 and X are equal to 1 and −1, respectively, so that the multiplication in A is a graded map of degree −1. Definition 7. A graded A-module M is a Z-graded abelian group M = ⊕i∈Z Mi , together with group homomorphisms X : Mi −→ Mi−2 ,

i ∈ Z,

(183)

c : Mi −→ Mi+2 ,

i ∈ Z,

(184)

X 2 = 0.

(185)

that satisfy relations Xc = cX

and

Definition 8. A homomorphism of graded A-modules M and N is a grading-preserving homomorphism of abelian groups f : M → N that intertwines the action of X and c in M and N: Xf = f X,

cf = f c.

(186)

Denote by A-mod0 the category whose objects are graded A-modules and whose morphisms are grading-preserving homomorphisms of graded A-modules. Note that A-mod0 is an abelian category. Denote by {n} the automorphism of A-mod that shifts the grading down by n. Let A-mod be the category of graded A-modules and graded maps. A-mod has the same objects as A-mod0 but more morphisms. Given a graded A-module M, define the multiplication map mM : A ⊗R M −→ M

(187)

by mM (1 ⊗ t) = t,

mM (X ⊗ t) = Xt,

t ∈ M.

(188)

The multiplication map mM is a degree (−1) map with the grading on A⊗R M defined as the product grading of gradings of A and M. To a graded A-module M associate a map "M : M −→ A ⊗ M

(189)

(recall that all tensor products are over R = Z[c]) by "M (t) = X ⊗ t + 1 ⊗ Xt + cX ⊗ Xt,

t ∈ M.

(190)

Then "M equips M with the structure of a cocommutative comodule over A. Map "M has degree −1. The following relation between "M and mM is straightforward to check:       (191) "M mM = IdA ⊗mM " ⊗ IdM = m ⊗ IdM IdA ⊗"M .

A CATEGORIFICATION OF THE JONES POLYNOMIAL

421

Denote by ιM the map M → A ⊗ M given by ιM (t) = 1 ⊗ t for t ∈ M. Introduce an A-module structure on A⊗n ⊗ M by a(x ⊗ y) = x ⊗ ay

for a ∈ A, x ∈ A⊗n , y ∈ M,

(192)

where ay means mM (a ⊗ y). Then mM , "M , and ιM , after appropriate shifts by {1} or {−1}, are maps of graded A-modules. Proposition 38. For any graded A-module M we have direct sum decompositions of A ⊗ M, considered as a graded A-module: A ⊗ M = "M M ⊕ ιM M,   A ⊗ M = ιM M ⊕ "M − ιM mM "M M,   A ⊗ M = "M M ⊕ ιM − cιM mM "M M.

(193) (194) (195)

Proof. Let us check (194), for instance. We have a decomposition A ⊗ M = (1 ⊗ M) ⊕ (X ⊗ M).

(196)

Denote by p the projection A ⊗ M → X ⊗ M, orthogonal to 1 ⊗ M. Since ιM M = 1 ⊗ M, it suffices to check that   (197) p "M − ιM mM "M : M −→ X ⊗ M is an A-module isomorphism. This map is given by t % −→ X ⊗ (1 + cX)t,

where t ∈ M.

(198)

The inverse map is X ⊗ t % −→ (1 − cX)t.

(199)

Decompositions (193) and (195) can be verified analogously. 8.2. Nonclosed (1 + 1)-cobordisms. Let ᏹ1 be the category whose objects are 1dimensional manifolds that are unions of a finite number of circles and one interval. An ordering of the ends of this interval is fixed. Morphisms between objects α and β of ᏹ1 are oriented surfaces whose boundary is the union of α, β and two intervals that join corresponding ends of the intervals of α and β. An example is depicted in Figure 58. This surface represents a morphism from an interval to a union of a circle and an interval. We require that a surface can be presented as a composition of disjoint unions of surfaces S21 , S12 , S01 , S10 , S22 , S11 , defined in Section 2.3, and surfaces T1 , T2 , depicted in Figure 59. We compose morphisms in this category by concatenating surfaces. Category ᏹ1 is a module category over ᏹ, defined in Section 2.3. The bifunctor ᏹ ⊗ ᏹ1 → ᏹ1 is defined on objects and morphisms by taking disjoint unions.

422

MIKHAIL KHOVANOV

Figure 58

T2 T1

Figure 59

Recall from the previous section that A-mod denotes the category of graded A-modules and graded homomorphisms. Given a graded A-module M, define a monoidal functor FM : ᏹ1 −→ A- mod by assigning the graded A-module A⊗n ⊗ M to a union of n circles and one interval, maps "M and mM (defined in the previous section) to the elementary surfaces T1 and T2 : FM (T1 ) = "M ,

FM (T2 ) = mM .

(200)

To the other six elementary surfaces S21 , S12 , S01 , S10 , S22 , S11 (in Section 2.3) associate the same maps as for the functor F :       FM S12 = ", FM S01 = ι, FM S21 = m, (201)       FM S22 = Perm, FM S11 = Id . FM S10 = #, 8.3. (1, 1)-tangles. A (1, 1)-tangle is a proper smooth embedding e : T G→ R2 × [0, 1] of a finite collection T of circles and one interval [0, 1] into R2 × [0, 1] such that the boundary points of [0, 1] go to the corresponding boundary component of R2 × [0, 1]: e(0) ∈ R2 × {0},

e(1) ∈ R2 × {1}.

(202)

Two (1, 1)-tangles are called equivalent if they are isotopic via an isotopy that fixes the boundary. Oriented (1, 1)-tangles are (1, 1)-tangles with a chosen orientation of each component, the orientation of [0, 1] always chosen in the direction from 0 to 1.

A CATEGORIFICATION OF THE JONES POLYNOMIAL

423

Figure 60. Diagram D and two resolutions

Define a marked oriented link (in R3 ) as an oriented link with a marked component. Obviously, there is a natural one-to-one correspondence between marked oriented links and oriented (1, 1)-tangles: the closure of a (1, 1)-tangle is a marked oriented link. Denote this map from oriented (1, 1)-tangles to oriented marked links by cl. 8.4. Invariants. A plane diagram D of an oriented (1, 1)-tangle L is a generic projection of L onto R × [0, 1]. If D is a plane diagram of an oriented (1, 1)-tangle, define x(D) and y(D) in the same way as for plane diagrams of oriented links (see Section 2.4). Fix a graded A-module M. Let n be the number of double points of D, so that n = x(D) + y(D) and Ᏽ is the set of double points of D. To M and D associate a commutative Ᏽ-cube VDM over the category A-mod0 of graded A-modules as follows. For ᏸ ⊂ Ᏽ the ᏸ-resolution D(ᏸ) of D consists of a disjoint union of circles and an interval. The functor FM (see Section 8.2) assigns a graded A-module to D(ᏸ). Define   VDM (ᏸ) = FM D(ᏸ) {−|ᏸ|}.

(203)

Maps between VDM (ᏸ) for various subsets ᏸ are defined by the procedure completely analogous to the one described in Section 4.2. Due to shifts {−|ᏸ|}, these maps of graded A-modules are grading-preserving, rather than just graded maps, so that VDM is a commutative cube over A-mod0 . Example. See the diagram D and its resolutions depicted in Figure 60. We have VDM (∅) = A ⊗ M, VDM (Ᏽ) = M{−1}, and the structure map VDM (∅) −→ VDM (Ᏽ) is the multiplication map mM : A ⊗ M → M{−1}. Next we transform the commutative Ᏽ-cube VDM into a skew-commutative Ᏽ-cube by putting minus signs in front of some structure maps of VDM or, equivalently, by

424

MIKHAIL KHOVANOV

tensoring it with EᏵ . Denote by C M (D) the complex C(VDM ⊗ EᏵ ) of graded Amodules. Define CM (D) = C M (D)[x(D)]{y(D) − 2x(D)}.

(204)

Denote the ith cohomology group of the complex CM (D) by H i (D, M). These cohomology groups are graded A-modules. Denote the j th graded component of H i (D, M) by H i,j (D, M) Theorem 3. For a graded A-module M, an oriented (1, 1)-tangle L, and a diagram D of L, isomorphism classes of graded A-modules H i (D, M) do not depend on the choice of D and are invariants of L. Our proof of Theorem 1 immediately generalizes without essential modifications to a proof of Theorem 3. Proposition 38 is used to establish direct sum decompositions of CM (D), analogous to decompositions of C(D), for suitable D, given by Propositions 11, 14, 18(1), and 21(1). Cohomology groups H i (D), defined in Section 4.2, are a special case of groups i H (D, M), as the next proposition explains. Proposition 39. Let D be a diagram of an oriented (1, 1)-tangle L and denote by cl(D) the associated diagram of the marked oriented link cl(L). Considering A as a graded A-module, we have a canonical isomorphism of cohomology groups (as graded R-modules)   H i (D, A) ∼ (205) = H i cl(D) , i ∈ Z. Given a finitely generated graded A-module M, define the graded Euler characteristic χ (M) by
  dimQ Mj ⊗Z Q . (206) χ (M) = j ∈Z

Proposition 40. Let M be a finitely generated graded A-module, L an oriented (1, 1)-tangle, and D a diagram of L. Then  
  K cl(L) χ (M) = (−1)i q j dimQ H i,j (D, M) ⊗Z Q , (207) −1 q +q i,j ∈Z

that is, the Kauffman bracket of cl(L) is proportional to the Euler characteristic of groups H i,j (D, M). Given two graded A-modules M, N and a grading-preserving homomorphism f : M → N, it induces a map of commutative cubes VDM → VDN , which, in turn, induces a map of complexes CM (D) → CN (D) and a map of cohomology groups H i (D, M) → H i (D, N ). So, in fact, each diagram D of an oriented long link defines functors HDi

A CATEGORIFICATION OF THE JONES POLYNOMIAL

425

from the category A-mod0 of graded A-modules to itself, HDi (M) = H i (D, M). If two diagrams D1 , D2 are related by a Reidemeister move, constructions of Section 5 ∼ =

extend to the functor isomorphism HDi 1 −→ HDi 2 . Let us frame this observation into a proposition. Proposition 41. For an oriented (1,1)-tangle L and a diagram D of L isomorphism classes of functors, HDi : A- mod0 −→ A- mod0

(208)

do not depend on the choice of D and are invariants of L. Oriented long links with one component correspond one-to-one to oriented knots in R3 . Thus, Proposition 41 gives invariants of oriented knots in R3 . Moreover, if a diagram D represents an oriented knot and D ! is the diagram obtained from D by reversing the orientation of the underlying curve, there is a natural in M isomorphism H i (D, M) = H i (D ! , M). Consequently, for knots, isomorphism classes of functors HDi do not depend on the orientation, and HDi provide “functor-valued” invariants of nonoriented knots. Of course, these invariants depend on how the ambient 3-space is oriented. Let D be the 3-crossing diagram of the left-hand trefoil (knot T2,3 in the notation of Section 6.2). The functors HDi are written as HD−3 (M) = ker 2X(M){8}, HD−2 (M) = (M/2XM){6}, HD0 (M) = M{2}, HDi (M) = 0

for all other values of i,

where ker 2X(M) = {t ∈ M | 2Xt = 0}. References [AM] [BLM] [BFK]

[CS] [CF] [Fs] [F]

S. Akbulut and J. McCarthy, Casson’s Invariant for Oriented Homology 3-spheres—An Exposition, Math. Notes 36, Princeton Univ. Press, Princeton, 1990. A. A. Beilinson, G. Lusztig, and R. MacPherson, A geometric setting for the quantum deformation of GLn , Duke Math. J. 61 (1990), 655–677. J. Bernstein, I. Frenkel, and M. Khovanov, A categorification of the Temperley-Lieb algebra and Schur quotients of U (sl2 ) by projective and Zuckerman functors, to appear in Selecta Math. (N.S.). J. S. Carter and M. Saito, Reidemeister moves for surface isotopies and their interpretation as moves to movies, J. Knot Theory Ramifications 2 (1993), 251–284. L. Crane and I. B. Frenkel, Four-dimensional topological quantum field theory, Hopf categories, and the canonical bases, J. Math. Phys. 35 (1994), 5136–5154. J. Fischer, 2-categories and 2-knots, Duke Math. J. 75 (1994), 493–526. A. Floer, An instanton-invariant for 3-manifolds, Comm. Math. Phys. 118 (1988), 215–240.

426 [FK] [GrL]

[JVW]

[Jo] [Ka] [K] [LT] [L] [M] [MT] [T]

MIKHAIL KHOVANOV I. B. Frenkel and M. Khovanov, Canonical bases in tensor products and graphical calculus for Uq (sl2 ), Duke Math. J. 87 (1997), 409–480. I. Grojnowski and G. Lusztig, “On bases of irreducible representations of quantum GLn ” in Kazhdan-Lusztig Theory and Related Topics (Chicago, Ill., 1989), Contemp. Math. 139, Amer. Math. Soc., Providence, 1992, 167–174. F. Jaeger, D. L. Vertigan, and D. J. A. Welsh, On the computational complexity of the Jones and Tutte polynomials, Math. Proc. Cambridge. Philos. Soc. 108 (1990), 35– 53. V. F. R. Jones, A polynomial invariant for knots via von Neumann algebras, Bull. Amer. Math. Soc. (N.S.) 12 (1985), 103–111. L. H. Kauffman, State models and the Jones polynomial, Topology 26 (1987), 395–407. M. Khovanov, Graphical calculus, canonical bases and Kazhdan-Lusztig theory, Ph.D. thesis, Yale University, 1997. W. B. R. Lickorish and M. B. Thistlethwaite, Some links with nontrivial polynomials and their crossing-numbers, Comment. Math. Helv. 63 (1988), 527–539. G. Lusztig, Introduction to Quantum Groups, Progr. Math. 110, Birkhäuser, Boston, 1993. H. Murakami, Quantum SU(2)-invariants dominate Casson’s SU(2)-invariant, Math. Proc. Cambridge Philos. Soc. 115 (1994), 253–281. G. Meng and C. H. Taubes, SW = Milnor torsion, Math. Res. Lett. 3 (1996), 661–674. M. B. Thistlethwaite, On the Kauffman polynomial of an adequate link, Invent. Math. 93 (1988), 285–296.

Department of Mathematics, University of California, Davis, California 95616, USA; [email protected]

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

A LEFSCHETZ (1,1) THEOREM FOR NORMAL PROJECTIVE COMPLEX VARIETIES J. BISWAS and V. SRINIVAS 1. Introduction. Let X be a projective variety over C. Let Xan be the analytic space associated to X. Let c1 : Pic(X) → H 2 (Xan , Z) be the map that associates to a line bundle (or equivalently a Cartier divisor) on X its cohomology class. We may identify the Néron-Severi group NS(X) with the image of Pic(X) in H 2 (Xan , Z) under the above map. If X is smooth, then by the Hodge decomposition theorem, we know that 



 H 2 Xan , C = H 2,0 Xan ⊕ H 1,1 Xan ⊕ H 0,2 Xan . Let F 1 H 2 (Xan , C) = H 2,0 (Xan ) ⊕ H 1,1 (Xan ). The Lefschetz theorem on (1, 1) classes (see [GH], [L]) states that if X is a smooth, projective variety, then  
NS(X) = α ∈ H 2 (Xan , Z) | αC ∈ F 1 H 2 Xan , C . If X is an arbitrary singular variety, then by [D, Theorem 8.2.2], the cohomology groups of X with Z-coefficients carry mixed Hodge structures. Hence it makes sense to talk of F 1 H 2 (Xan , C) for such a variety X. Spencer Bloch, in a letter to Jannsen [J, Appendix A], asks whether the “obvious” extension of the Lefschetz (1, 1) theorem is true for singular projective varieties, that is, is it true that   

NS(X) = α ∈ H 2 Xan , Z | αC ∈ F 1 H 2 Xan , C ? Barbieri Viale and Srinivas [BS2] give a counterexample to this question. Let X be a surface defined by the homogenous equation w(x 3 − y 2 z) + f (x, y, z) = 0 in P3C , where x, y, z, w are homogenous coordinates in P3C and f is a “general” homogenous polynomial over C of degree 4. They show that for such an X,   

NS(X)⊆/ α ∈ H 2 Xan , Z | αC ∈ F 1 H 2 Xan , C . In the same paper [BS2], the authors ask the following question. Let X be a complete 1 ) be the subgroup of H 2 (X , Z) consisting of Zariski– variety over C. Let H 1 (X, ᏴX an 1 ) if and locally trivial cohomology classes, that is, η ∈ H 2 (Xan , Z) lies in H 1 (X, ᏴX only if there exists a finite open cover {Ui } of X by Zariski open sets such that η → 0 under the restriction maps H 2 (Xan , Z) → H 2 ((Ui )an , Z) for all i. Is  

1 NS(X) = α ∈ H 1 X, ᏴX | αC ∈ F 1 H 2 Xan , C ? Received 13 April 1999. 1991 Mathematics Subject Classification. Primary 14C25, 14C30; Secondary 32J25, 32S35. 427

428

BISWAS AND SRINIVAS

We remark that for a smooth, projective variety X, if a cohomology class η ∈ H 2 (Xan , Z) is zero when restricted to a nonempty Zariski open set U ⊂ X, then η is 1 ) = NS(X), and the above question has a positive the class of a divisor. So H 1 (X, ᏴX answer for a smooth, projective variety X. For any projective variety X, there is an 1 ); Barbieri Viale and Srinivas also give an example in inclusion NS(X) ⊂ H 1 (X, ᏴX [BS2] of a singular variety for which this inclusion is strict. In general, for any projective variety X over C, 

 1 | αC ∈ F 1 H 2 Xan , C . NS(X) ⊂ α ∈ H 1 X, ᏴX 1 ), combined with This follows from the inclusion NS(X) ⊂ H 1 (X, ᏴX  



ker H 2 Xan , C −→ H 2 Xan , ᏻXan ⊂ F 1 H 2 Xan , C ,

which is a consequence of results of Du Bois [Du] (and is also implicit in [D]). When X is normal, we prove the reverse inclusion, thereby answering the question in the affirmative for the normal case. The statement of our main theorem is the following. Theorem 1.1. Let X be a normal, projective variety over C. Then  

1 NS(X) = α ∈ H 1 X, ᏴX | αC ∈ F 1 H 2 Xan , C . We also describe a counterexample of a nonnormal irreducible projective 3-fold with smooth normalization (isomorphic to P1 × P1 × P1 ) for which the question has a negative answer. However, it seems likely that the conclusion of the theorem holds for any seminormal projective variety X over C. 1) ⊂ Our proof of Theorem 1.1 is in two steps. First we show that H 1 (X, ᏴX 2 H (Xan , Z) is a sub-MHS (mixed Hodge structure) of level 1; hence by [D], it determines a 1-motive, which we show to be an extension of a direct sum of Tate structures Z(−1) by that of (H 1 of) an abelian variety. In the second part of the proof, we give a direct construction of a certain 1-motive using the Zariski topology on X, and we show that it is isogenous to the earlier one. The theorem is an immediate corollary. In a future work, we hope to use the second, algebraically defined 1-motive to also obtain the analogue of the Tate conjecture in our situation, which would similarly characterize the Z -span of the classes of Cartier divisors in Het2 (XK¯ , Z (1)), for a normal projective variety X over a number field K. In another direction, our result suggests a question analogous to the Hodge conjecture. Let X be a normal projective variety over C, and let α : Xan → X be the obvious continuous map leading to a Leray spectral sequence

 p,q E2 = H p X, R q α∗ QXan ⇒ H p+q Xan , Q , with an induced decreasing Leray filtration {Lp H n (Xan , Q)}p≥0 on each cohomology group H n (Xan , Q). Let  

Hgp (X) = Lp H 2p Xan , Q ∩ F p H 2p Xan , C .

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

429

Is Hgp (X) the image of the pth Chern class map cp : K0 (X) ⊗ Q → H 2p (X, Q)? Note that this does not hold without some hypothesis like (at least) normality; for example, Bloch’s letter to Jannsen [J, Appendix A] gives a counterexample. On the other hand, results of Collino [Co] imply it when X has a unique singular point. Note also that, unlike the standard Hodge conjecture, the positive answer (our Theorem 1.1) for divisors does not automatically imply a positive answer for the case of 1-cycles since we do not have Poincaré duality. This work formed part of the first author’s Ph.D. thesis, written at the Tata Institute of Fundamental Research, Mumbai, submitted to the Mumbai University in March 1997. 2. Some preliminaries 2.1. Constructible sheaves. We need some technical results on constructible sheaves on complex algebraic varieties. We begin by recalling the appropriate definitions, the first from [V] and the second from [BS1]. Definition 2.1. Let X be a complex algebraic variety. We say that a sheaf Ᏺ of abelian groups on the analytic space Xan is (algebraically) Z-constructible if there is a finite decomposition X = ∪i∈I Xi , where each Xi is irreducible and Zariski closed in X, such that if Ui = Xi − ∪Xj ⊆/ Xi Xj , then each Ui is nonsingular, X is the disjoint union of the Ui , and Ᏺ|(Ui )an is a locally constant sheaf whose fibre is a finitely generated group. We call any such collection of subsets {Xi }i∈I an admissible family of subsets for Ᏺ. Definition 2.2. A sheaf Ᏻ on a scheme X (over an algebraically closed field k, say) is said to be Z-constructible for the Zariski topology if we can express X as a finite union X = ∪Xi , where Xi ⊂ X are Zariski closed, such that if Ui = Xi −∪Xj ⊆/ Xi Xj , then each Ui is nonsingular, X is the disjoint union of the Ui , and Ᏻ|Ui is a constant sheaf associated to a finitely generated abelian group. We call any such collection of subsets {Xi }i∈I an admissible family for Ᏻ. Remark 2.3. We note that in the cited works, it is not required that the open strata Ui are nonsingular, but this may clearly be assumed as well, without loss of generality, by refining any given stratification that has all the remaining properties. Note that if {Xi }i∈I is an admissible family of subsets for a Z-constructible sheaf in either of the senses above, then there is a natural partial order on the index set I given by j ≤ i ⇐⇒ Xj ⊂ Xi . Then we clearly have Ui = Xi − ∪j
430

BISWAS AND SRINIVAS

but for which we do not know a reference. Let X be a topological space, and let {Ui }i∈I be any finite collection of locally closed subsets of X that stratify X (i.e., X is the disjoint union of the Ui , and for each i, the closure Xi := U¯i is again a union of some Uj ). Let ≤ denote the obvious partial order on I , given by i ≤ j ⇐⇒ Xi ⊂ Xj . Let fi : Ui → X be the inclusion. If Ᏺ is a sheaf of abelian groups on X and if {i0 ≤ i1 ≤ · · · ≤ ip } is a p-chain in I , let












Ᏺi0 i1 ···ip := fi0 ∗ fi−1 · · · f ip fi1 ∗ fi−1 0 1



f −1 Ᏺ. ∗ ip

Note that the sheaves Ᏺi0 ···ip define a cosimplicial sheaf on the simplicial space X × N(I ), where N(I ) denotes the nerve of I (regarded as a discrete simplicial space). The augmentation X × N(I ) → X gives rise to a complex of sheaves on X: 0 −→ Ᏺ −→

 i∈I

Ᏺi −→

 {i0 ≤i1 }∈N1 (I )

−→ · · · −→

Ᏺi0 i1



Ᏺi0 ···ip −→ · · ·

··· .

(∗)

{i0 ≤···≤ip }∈Np (I )

Lemma 2.5. The above complex (∗) is a resolution of Ᏺ. Proof. If x ∈ Ui , then taking the stalks at x, we have an associated cosimplicial abelian group (Ᏺi0 i1 ···ip )x and a corresponding augmented complex. Clearly, (Ᏺi0 i1 ···ip )x = 0 unless i ≤ i0 . Since the partially ordered subset (I ≥ i) = {j ∈ I | i ≤ j } has a minimal element, we see easily that the stalk complex at x is contractible. (Note that if x ∈ Ui and if σ = {i0 ≤ · · · ≤ ip } is a p-simplex in the nerve of (I ≥ i), the stalks at x of Ᏺi0 ···ip and Ᏺii0 ···ip are naturally isomorphic where {i ≤ i0 ≤ · · · ≤ ip } is the cone over σ with vertex i.) Remark 2.6. If Ᏺ is Z-constructible for the Zariski topology on a scheme X, and {Xi } is an admissible family for Ᏺ such that Ᏺ |Ui is the constant sheaf associated to Ai , then Ᏺi0 i1 ···ip is just the constant sheaf (Aip )Xi0 . In particular, for a Z-constructible sheaf in the Zariski topology, we obtain a flasque resolution. The key technical result of this section is the following lemma. Lemma 2.7. Let Ꮽ = AXan be a constant sheaf on a complex algebraic variety X, and let Ᏻ be a Z-constructible sheaf on Xan . Let f : Ꮽ → Ᏻ be a sheaf homomorphism, and take Ᏺ = image f . Let a : Xan → X be the natural continuous map from the analytic space Xan to X, which is the identity on points. Then we have the following. (i) a∗ Ᏺ is a constructible sheaf on X for the Zariski topology. (ii) The natural map a −1 a∗ Ᏺ → Ᏺ is an isomorphism, and the natural map a∗ Ꮽ → a∗ Ᏺ is surjective, that is, a∗ Ᏺ is a quotient of the constant sheaf on X associated to the abelian group A.

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

431

(iii) Let {Xi }i∈I be an admissible family of subsets for Ᏻ. Then it is also admissible for Ᏺ and for a∗ Ᏺ. There is an exact sequence 

 

  H 0 (Ui )an , Ᏺ |(Ui )an −→ H 0 Uj an , Ᏺ |(Uj )an . 0 −→ H 0 Xan , Ᏺ −→ i∈I

i≤j i,j ∈I

Proof. We first claim that if U ⊂ X is an irreducible (Zariski) locally closed subset such that Ᏻ |Uan is locally free, then Ᏺ |Uan is a constant sheaf associated to a finitely generated abelian group that is a quotient of A. Indeed, Ᏻ |Uan corresponds to a representation of the fundamental group of Uan (with respect to any convenient base point), while AUan corresponds to the trivial representation. The sheaf map f |Uan is then a morphism of local systems whose image Ᏺ |Uan is clearly a trivial (i.e., constant) local subsystem of Ᏻ |Uan . Now let {Xi }i∈I be admissible for Ᏻ. As observed above, Ᏺ |(Ui )an is constant for each i, and so {Xi }i∈I is also admissible for Ᏺ. From Lemma 2 of [BS1], it follows that a∗ Ᏺ is Z-constructible for the Zariski topology. From the beginning of the exact sequence (∗) of Lemma 2.5 (for Ᏺ on Xan ), we have inclusions    Ᏺ $→ Ᏺi , a∗ Ᏺ $→ a∗ Ᏺi , a −1 a∗ Ᏺ $→ a −1 a∗ Ᏺi . i∈I

i∈I

i∈I

We see at once from the definitions that a∗ Ᏺi is (the direct image on X of) a constant sheaf on Xi , for each i, and the natural sheaf map a −1 a∗ Ᏺi → Ᏺi is injective. Since Ꮽ is a constant sheaf, we also have that a −1 a∗ Ꮽ → Ꮽ is an isomorphism. Now from the commutative diagram a −1 a∗ Ꮽ ∼ =

 Ꮽ

/ a −1 a∗ Ᏺ   _   //Ᏺ

/

/

i∈I

a −1  _ a∗ Ᏺi

 

i∈I

Ᏺi ,

we deduce that a −1 a∗ Ᏺ → Ᏺ is an isomorphism, and that the natural map a −1 a∗ Ꮽ → a −1 a∗ Ᏺ is surjective. This implies that a∗ Ꮽ → a∗ Ᏺ is surjective as well, and that {Xi } is admissible for a∗ Ᏺ. The exact sequence in (iii) of the lemma is obtained from the resolution of Lemma 2.5 for a∗ Ᏺ. 2.2. A homological lemma. We prove here an abstract homological lemma (Lemma 2.8) that is a variant of a lemma in [PS], which we will need later. The lemma is formulated and proved with abelian groups, but a similar argument yields it in an arbitrary abelian category. Suppose we have the following 9-diagram {Cij }, in the category of complexes of abelian groups, with exact rows and columns:

432

BISWAS AND SRINIVAS

0

0

0

0

 / C• 11

 / C• 12

 / C• 13

/0

0

 / C• 21

 / C• 22

 / C• 23

/0

0

 / C• 31

 / C• 32

 / C• 33

/0

 0

 0

 0.

Applying the cohomology functor, we get an infinite double sequence with exact rows and columns as shown below:
•



 / H i−2 C • / H i−1 C • / H i−1 C • / H i−1 C • / H i−2 C22 23 21 22 23
 •  H i−2 C32


  / H i−2 C • 33


  / H i−1 C • 31


  / H i−1 C • 32


  / H i−1 C • 33 /


 •  H i−1 C12


  / H i−1 C • 13


  / H i C• 11


  / H i C• 12


  / H i C• 13 /


 •  H i−1 C22


  / H i−1 C • 23


  / H i C• 21


  / H i C• 22


  / H i C• 23 /


 •  H i−1 C32


  / H i−1 C • 33


  / H i C• 31


  / H i C• 32


  / H i C• 33 /









.

• ) (say, for example, α ∈ H i (C • ) Now suppose that we have an element α ∈ H i (Crs 22 • ). in the above diagram) such that α → 0 under both the maps with domain H i (Crs We can then do a diagram chase in the above cohomology diagram in the following • ); arbitrarily choose lifts β ∈ H i (C • ) and β ∈ H i (C • ) way. Suppose α ∈ H i (C22 1 2 21 12 • ) and let β → γ ∈ H i (C • ). Then since γ → 0 ∈ lifting α. Let β1 → γ1 ∈ H i (C31 2 2 1 13 • ) and γ → 0 ∈ H i (C • ), there exist δ and δ , both in H i−1 (C • ), lifting γ H i (C32 2 1 2 1 23 33 and γ2 , respectively.

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

433

• ), for We can do a similar diagram chase beginning with an element α ∈ H i (Crs • j arbitrary i, r, s, and end up with two elements δ1 , δ2 in the same group H (Cr+1 s+1 ), where we read the subscripts modulo 3, and j is either i − 1, i, or i + 1, depending on (r, s). (We end up at the two places in the diagram that have the same entry and are each one “knight’s move” away from the starting point.) • j j (C • Let H r+1 s+1 ) denote the quotient of H (Cr+1 s+1 ) by the subgroup generated by the images of the two maps in the large commutative cohomology diagram with • range H j (Cr+1 s+1 ). For example,
•
•  H i−1 C33
i−1
•
• . H C3,3 = image H i−1 C23 + image H i−1 C32

Lemma 2.8. With the notation as above, we have
  j
C • δ1 − δ2 −→ 0 ∈ H r+1 s+1 . Proof. We first note that, by an argument with mapping cones and cylinders (ro• ) tating the distinguished triangles in the 9-diagram), we may assume that α ∈ H i (C22 without loss of generality. For such an α, the analogous result for the cohomology diagram arising from a particular 9-diagram in the category of étale sheaves on a scheme was proved in [PS, Section 3]. The proof of this lemma is similar: regarding the given 9-diagram as a (bounded) double complex of complexes, one considers the total complex, which is a 5-term exact sequence of complexes, say, 0 −→ Ꮿ0 −→ Ꮿ1 −→ Ꮿ2 −→ Ꮿ3 −→ Ꮿ4 −→ 0. Regarding this again as a double complex, there is a spectral sequence 

E1r,s = H s Ꮿr ⇒ H r+s Tot(Ꮿ• ) (the limit is in fact zero). Then the conclusion of the lemma is interpreted as giving two (equivalent) ways of computing the differential E22,i → E24,i−1 . Remark 2.9. An analogue of Lemma 2.8 can be formulated for a 9-diagram {Cij } in the derived category of abelian groups similar to the one considered above, where the rows and columns are distinguished triangles, and where instead of the cohomology diagram considered earlier, we have the diagram obtained by applying any abelian group valued cohomological functor (of course, a still more general formulation is also possible). This is false; O. Gabber has kindly shown us a counterexample. Remark 2.10. A version of Lemma 2.8 also appears in a letter from U. Jannsen to B. Gross. 3. A short exact sequence of mixed Hodge structures. In this section, we make an analysis of the mixed Hodge structure on 

1 H 1 X, ᏴX = subgroup of Zariski locally trivial elements in H 2 Xan , Z .

434

BISWAS AND SRINIVAS

Our goal is to describe it as an extension of a direct sum of Tate Hodge structures Z(−1) by a polarizable pure Hodge structure of weight 1. Let X be our given normal projective variety over C. Let Y be a resolution of singularities of X, and let Yan be the associated analytic space of Y . We have the following commutative diagram: Yan 

aY

π

π an

Xan

/Y

aX

 / X.

The Leray spectral sequence for the constant sheaf Z = ZYan and the map π an : Yan → Xan leads to an exact sequence


   0 −→ H 1 Xan , Z −→ H 1 Yan , Z −→ H 0 Xan , R 1 π∗an Z  

(3.1) −→ H 2 Xan , Z −→ H 2 Yan , Z . Note that since X is normal, we have π∗an Z ∼ = Z. Define a new sheaf ᏲZ on Xan by

  (3.2) ᏲZ = image H 1 Yan , Z X −→ R 1 π∗an Z . an

H 1 (Y

Here by an , Z)Xan we mean the constant sheaf on Xan associated to the group H 1 (Yan , Z), and the map on sheaves is induced at the level of presheaves by the restriction map on cohomology H 1 (Yan , Z) → H 1 ((π an )−1 (Uan ), Z), where Uan ⊂ Xan is open. By taking global sections, we have the following commutative diagram:
 
/ H 0 Xan , ᏲZ H 1 Yan , Z _ PPP PPP PPP PPP  
( H 0 Xan , R 1 π∗an Z . Hence we have an inclusion 



 
H 0 Xan , ᏲZ H 0 Xan , R 1 π∗an Z  −→

 = ker H 2 Xan , Z −→ H 2 Yan , Z , 0 −→
1
1 Im H Yan , Z Im H Yan , Z where the last equality is due to the exact sequence (3.1) of low-degree terms of the Leray spectral sequence. Note that ᏲZ satisfies the hypotheses of Lemma 2.7, with A = H 1 (Yan , Z) and Ᏻ = R 1 π∗an Z (the latter is algebraically Z-constructible by Theorem 2.4). Hence the following properties hold: (i) ᏲZ is algebraically Z-constructible.

435

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

(ii) a∗X ᏲZ := ᏳZ is Z-constructible for the Zariski topology, and (a X )∗ ᏳZ ∼ = ᏲZ . (iii) The natural sheaf map 
(3.3) H 1 Yan , Z X −→ a∗X ᏲZ is surjective. 1 ). Lemma 3.1. We have (H 0 (Xan , ᏲZ )/Im(H 1 (Yan , Z))) ⊂ H 1 (X, ᏴX

Proof. Let α ∈ H 0 (Xan , ᏲZ ) = H 0 (X, a∗X ᏲZ ). Then by (3.3), there exists a Zariski open cover {Ui } of X such that α|(Ui )an = Im(βi ), where βi ∈ H 1 (Yan , Z). Therefore, α|(Ui )an → 0 ∈ H 2 ((Ui )an , Z) as shown in the following commutative diagram (where U stands for any of the Ui ):
 H 1 Yan , Z

H1







−1
π an U

 an



,Z


 / H 0 Xan , ᏲZ


 / H 2 Xan , Z


  / H 0 Uan , ᏲZ


 / H 2 Uan Z .

This finishes the proof of the lemma. If {Xi }i∈I is an admissible family of subsets for the constructible sheaf R 1 π∗an Z on Xan , then (by Lemma 2.7) it is also an admissible family for ᏲZ and for a∗X ᏲZ = ᏳZ . We fix such an admissible family once and for all, and fix base points xi ∈ Ui with corresponding reduced fibers Fi = π −1 (xi )red . Let F = ∪i Fi = π −1 ({xi | i ∈ I }). By the proper base change theorem, the stalk (R 1 π∗an Z)xi is naturally identified with H 1 ((Fi )an , Z); thus R 1 π∗an Z |(Ui )an is a local system with fiber H 1 ((Fi )an , Z). Note that the stalk (ᏲZ )xi has the resulting description:

 


 (3.4) ᏲZ x = image H 1 Yan , Z −→ H 1 Fi an , Z . i

By mixed Hodge theory [D], we deduce that (ᏲZ )xi naturally supports a pure Hodge structure of weight 1, which is a quotient Hodge structure of H 1 (Yan , Z) (depending only on i ∈ I and not on the chosen base point xi ∈ Ui ), as well as a Hodge substructure of H 1 ((Fi )an , Z). Finally, note also that ᏲZ |(Ui )an is a constant sheaf whose fiber supports this pure Hodge structure (i.e., is the underlying lattice). Lemma 3.2. H 0 (Xan , ᏲZ ) carries a pure Hodge structure of weight 1, such that 0 an , Z) → H (Xan , ᏲZ ) is a morphism of Hodge structures.

H 1 (Y

Proof. From Lemma 2.7(iii), there exists an exact sequence of abelian groups  


 ᏲZ x −→ ᏲZ x . 0 −→ H 0 Xan , ᏲZ −→ i∈I

i

i0 ,i1 ∈I, i0 ≤i1

i1

The natural surjective maps (ᏲZ )xi → (ᏲZ )xj (for i ≤ j ) are maps of pure Hodge

436

BISWAS AND SRINIVAS

structures of weight 1, which are quotients of H 1 (Yan , Z). Hence H 0 (Xan , ᏲZ ) is identified with the kernel of a morphism of pure Hodge structures of weight 1 and, hence, itself supports a pure Hodge structure of weight 1. It is also clear from the construction that the composition 
 

 H 1 Yan , Z −→ H 0 Xan , ᏲZ $→ ᏲZ x i∈I

i

is a direct sum of the natural quotient maps H 1 (Yan , Z) → (ᏲZ )xi and, hence, is a morphism of Hodge structures. Hence H 1 (Yan , Z) → H 0 (Xan , ᏲZ ) is one as well. Proposition 3.3. H 0 (Xan , ᏲZ ) → H 2 (Xan , Z) is a morphism of Hodge structures; that is, the Hodge structures on H 0 (Xan , ᏲZ ) and H 2 (Xan , Z) are compatible. Proof. Let Fi = π −1 (xi ) as above, and let F = ∪i∈I Fi . We note that the natural map H 0 (Xan , R 1 π∗an Z) → H 1 (Fan , Z) is an injection (any section in the kernel must vanish in all stalks). This implies that the map H 2 (Xan , Z) → H 2 (Yan , Fan , Z) (which is a morphism of mixed Hodge structures) is injective in the following commutative diagram (here Gi = (ᏲZ )xi ):
 

H 0 Xan , ᏲZ   H 0 Xan , R 1 π∗an Z   / / H 2 Xan , Z



 1  1 _ Im H Yan , Z Im H Yan , Z _ pP







⊕i G i






Im H 1 Yan , Z
_









 
 
 ⊕i H 1 Fi an , Z   / H 2 Yan , Fan , Z .

 1 Im H Yan , Z We are done, because all the arrows in the above diagram are injections, and the vertical arrows (on the left and right borders), as well as the lower horizontal arrow, are morphisms of mixed Hodge structures. Let A ⊂ B be an inclusion of abelian groups. Let A ⊂ As ⊂ B denote the saturation of A in B, that is, As is the smallest subgroup of B containing A such that B/As is torsion-free. Let H 0 (Xan , ᏲZ )s be the saturation of H 0 (Xan , ᏲZ ) in H 0 (Xan , R 1 π∗an Z). Lemma 3.4. We have


ker H

1




1 X, ᏴX








−→ H Yan , Z 2


s H 0 Xan , ᏲZ

 . = Im H 1 Yan , Z

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

437

Proof. It is easy to see from Lemma 3.1 that
s
1
 
H 0 Xan , ᏲZ 1 2

 . ker H X, ᏴX −→ H Yan , Z ⊃ Im H 1 Yan , Z We prove, using Lemma 2.8, that given any element 


1 −→ H 2 Yan , Z α ∈ ker H 1 X, ᏴX and any preimage β1 ∈ H 0 (Xan , R 1 π∗an Z), some nonzero (integral) multiple of β1 lies in H 0 (Xan , ᏲZ ) ⊂ H 0 (Xan , R 1 π∗an Z). This proves the assertion of the lemma. 1 ), there exists a finite Zariski open cover {U } of X such that Since α ∈ H 1 (X, ᏴX i α → 0 in H 2 ((Ui )an , Z) for all i. Let U denote any one of these Ui ’s and consider again the above commutative diagram with exact rows and columns: 
H 1 Yan , Z 
H 1 Uan , Z



   / H 1 π −1 Uan , Z

 


   H 2 Xan , Uan , Z −→ H 2 Yan , π −1 Uan , Z


 H 1 Yan , Z


 / H 0 Xan , R 1 π an Z ∗


  / H 2 Xan , Z


  / H 2 Yan , Z



   H 1 π −1 Uan , Z


 / . Uan , R 1 π an Z ∗


  / H 2 Uan , Z



   / H 2 π −1 Uan , Z .

We wish to apply Lemma 2.8 to this diagram. For this, we need to know that the diagram arises by applying the cohomology functor to a suitable 9-diagram in the category of complexes of abelian groups (we thank the referee for useful suggestions for cleaning up our exposition at this point). We do this using Cartan-Eilenberg resolutions; we briefly recall what these are. If 0 −→ A0 −→ A1 −→ · · · is a complex in an abelian category that has enough injectives, a Cartan-Eilenberg resolution of A• is a double complex of injective objects I •,• (indexed to lie in the first quadrant), together with an augmentation A• → I •,• , given by morphisms Aj → I j,0 , such that for each j ≥ 0, the following augmented complexes are injective

438

BISWAS AND SRINIVAS

resolutions: 0 −→ Aj −→ I j,0 −→ I j,1 −→ I j,2 · · · 

0 −→ ker Aj −→ Aj +1 −→ ker I j,0 −→ I j +1,0 

−→ ker I j,1 −→ I j +1,1 −→ ker I j,2 −→ I j +1,2 −→ · · ·

 0 −→ image Aj −→ Aj +1 −→ image I j,0 −→ I j +1,0

 −→ image I j,1 −→ I j +1,1 −→ image I j,2 −→ I j +1,2 −→ · · ·



 0 −→ H j A• −→ H j I •,0 −→ H j I •,1 −→ H j I •,2 −→ · · · . It is a standard fact that any such complex A• has a Cartan-Eilenberg resolution, and that for any such resolution, the map to the total complex A• → Tot I •,• is a quasi isomorphism. Now consider an injective resolution 0 → ZY → Ᏽ• . Then by definition, Rπ∗ ZY = π∗ Ᏽ• . Let π∗ Ᏽ• → ᏶•,• be a Cartan-Eilenberg resolution. In particular, ᏶i,• is an injective resolution of π∗ Ᏽi for each i. Then π∗ Ᏽ• → Tot(᏶•,• ) is a quasi isomorphism. Define ᏸ• = ker(᏶0,• → ᏶1,• ). Then ᏸ• is an injective resolution of 
ker π∗ Ᏽ0 −→ Ᏽ1 ∼ = ZX . Let ᏽ• be defined via the exact sequence of complexes
 0 −→ ᏸ• −→ Tot ᏶•,• −→ ᏽ• −→ 0. Since the first two terms are complexes of injectives, so is the third complex ᏽ• , and the exact sequence of complexes is termwise split exact. Let j : U → X and i : X − U → X be the inclusions. Then, since j ∗ , j∗ , i ! and i∗ have exact left adjoints and hence preserve injectives, we have an exact 9-diagram of complexes of injectives: 0

0

0

0

 / i i ! ᏸ• ∗


 / i∗ i ∗ Tot ᏶•,•

 / i i ! ᏽ• ∗

/0

0

 / ᏸ•


  / Tot ᏶•,•

 / ᏽ•

/0

0

 / j∗ j ∗ ᏸ •


 / j∗ j ∗ Tot ᏶•,•

 / j∗ j ∗ ᏽ•

/0

 0

 0

 0.

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

439

Applying the global section functor .(X, −) yields a 9-diagram of complexes of abelian groups, and it is clear from the construction that the corresponding cohomology diagram is the one we began with. Returning to our cohomology diagram, note that the relative cohomology sequences    



−→ H 1 Uan , Z −→ H 2 Xan , Uan , Z −→ H 2 Xan , Z −→ H 2 Uan , Z −→ and

−1
 

−1
  −→ H 1 π an Uan , Z −→ H 2 Yan , π an Uan , Z


−1
   2 Uan , Z −→ H Yan , Z −→ H 2 π an are sequences in the category of mixed Hodge structures by [D, (8.3.9)]. Since α → 0 ∈ H 2 (Yan , Z), αQ ∈ W1 H 2 (Xan , Q) by [D, Proposition 8.2.5]. This implies, by [D, Theorem 2.3.5] (i.e., strictness of morphisms of mixed Hodge structures with respect to W ), that we can choose β2 ∈ H 2 (Xan , Uan , Z) such that 

 β2 Q ∈ W1 H 2 Xan , Uan , Q ,

β2 −→ nα, n ∈ Z>0 .

Let 
β2 −→ γ2 ∈ H 2 Yan , (π an )−1 (Uan ), Z ∼ = Z(−1)k , for some k ≥ 0, where the last isomorphism is because Y is nonsingular; then

−1
 
γ2 Q ∈ W1 H 2 Yan , π an Uan , Q = 0, that is, γ2 = 0. So we can choose a preimage δ2 ∈ H 1 ((π an )−1 (Uan ), Z) of γ2 to be zero. On the other hand, chasing the diagram the other way, we get nβ1 ∈ H 0 (Xan , R 1 π∗an Z), which lifts nα, and nβ → nγ1 ∈ H 0 (Uan , R 1 π∗an Z). Now take a lift nδ1 ∈ H 1 ((π an )−1 (Uan , Z) of nγ1 . By Lemma 2.8, we know that nδ1 ≡ δ2 = 0 modulo the images of H 1 (Yan , Z) and 1 H (Uan , Z). Therefore,

 
nγ1 ∈ Im H 1 Yan , Z −→ H 0 Uan , R 1 π∗an Z . This proves that nβ1 |Uan comes from H 1 (Yan , Z). Since X has a finite cover by such open sets U, we see that β1 ∈ H 0 (Xan , ᏲZ )s . Corollary 3.5. There exists a short exact sequence of mixed Hodge structures
s 



H 0 Xan , ᏲZ 1 1  −→ H 1 X, ᏴX 0 −→
1
−→ Im H 1 X, ᏴX −→ H 2 Yan , Z −→ 0. Im H Yan , Z

440

BISWAS AND SRINIVAS

Let Im(H 1 (Yan , Z))s denote the saturation of H 1 (Yan , Z) in H 0 (Xan , ᏲZ )s . Then

s  0
s H Xan , ᏲZ Im H 1 Yan , Z

 =

 . Im H 1 Yan , Z Im H 1 Yan , Z torsion

F 1 H 2 (X

Since NS(X) ⊂ an , Z), it follows that it has finite intersection with (H 0 (Xan , ᏲZ )s )/(Im(H 1 (Yan , Z))), which is a pure Hodge structure of weight 1. On the other hand, H 2 (Xan , Z)torsion ⊂ NS(X) from the exponential sequence. Thus 

s s H 0 Xan , ᏲZ H 0 Xan , ᏲZ

  . = NS(X) ∩
1
Im H 1 Yan , Z Im H Yan , Z torsion

Hence we get the exact sequence
s
 1 H 0 Xan , ᏲZ H 1 X, ᏴX 0 −→

s −→ NS(X) Im H 1 Yan , Z
1
 
1 2 f Im H X, ᏴX −→ H Yan , Z
 −→ 0. −→ Im NS(X)

(+)

It is clear that the third term is pure of type (1, 1) as it lies inside NS(Y )/ Im(NS(X)). Let

 
1 −→ H 2 Y , Z Im H 1 X, ᏴX an
 A= , Im NS(X) and let


 M = f −1 Atorsion .

We then have a short exact sequence of mixed Hodge structures
 1 H 1 X, ᏴX A ∼ 0 −→ M −→ −→ = Z(−1)r −→ 0, NS(X) Atorsion

(++)

where the third term is free of rank r and pure of type (1, 1), and M is a pure Hodge structure of weight 1. Further, all of the underlying abelian groups are free. We recall some facts about extensions of mixed Hodge structures (see [C], for example). Let H be a finitely generated abelian group that supports a pure Hodge structure of weight 1, and let G be a finitely generated abelian group, regarded as a pure Hodge structure of type (0, 0). Then there is a natural identification of the abelian group Ext 1MHS (G(−1), H ) of extensions in the category MHS of mixed Hodge structures with the abelian group Hom(G, J (H )), where J (H ) = J 1 (H ) =

HC ; F 1 HC + Im(H )

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

441

here HC = H ⊗Z C, and F gives the Hodge filtration. In particular, we have
 Ext 1MHS Z(−1), H = J (H ). If 0 −→ H −→ E −→ G(−1) −→ 0 is an extension of mixed Hodge structures, let ψE : G → J (H ) be the corresponding homomorphism (which we call the extension class map of E). This may be described as follows: There is an identification α:

∼ HC EC = −→ 1 , 1 F HC F EC

giving β : J (H ) =

∼ HC EC = , −→ 1 F EC + H C +H

F 1H

and ψE is the composition G∼ =

E EC β −1 −→ 1 −−−→ J (H ). H F EC + H

In the case when G = Z⊕r is free abelian, we have that HomMHS (Z(−1), E) is naturally identified with ker ψE . Also, if G is free abelian and H is polarizable, then J (H ) is an abelian variety, and for an extension E, the homomorphism ψE : G → J (H ) is the 1-motive over C associated to the mixed Hodge structure E by the procedure in [D, (10.1.3)]. In particular, the sequence of mixed Hodge structures (+) is an extension of a pure Hodge structure of type (1, 1) by a pure Hodge structure of weight 1 and, hence, gives rise to an extension class homomorphism, 

 

s 1 −→ H 2 Y , Z Im H 1 X, ᏴX H 0 Xan , ᏲZ an ψ: −→ J


s . NS(X) Im H 1 Yan , Z Similarly, the sequence of mixed Hodge structures (++) gives rise to a related homomorphism ∼ A −→ J (M), ψ1 : Z⊕r = Ators which is in fact a 1-motive. It is clear that J (M) is isogenous to J (H 0 (Xan , ᏲZ )s / (Im(H 1 (Yan , Z)))s ) and thus to J (H 0 (Xan , ᏲZ )s / Im(H 1 (Yan , Z))). By the above remarks, our main result, Theorem 1.1, is equivalent to proving ψ1 is injective.

442

BISWAS AND SRINIVAS

4. Proof of the main theorem 4.1. Construction of a 1-motive. The aim of this section is to directly construct a certain 1-motive over C. The proof of the main theorem is achieved by showing that 1 )/ NS(X)) ⊗ Z(1). it is isogenous to that associated to (H 1 (X, ᏴX Let π : Y → X be a desingularization of X as before, and let U ⊂ X be a Zariski open subset. We have an exact sequence of groups

 Pic0 (Y ) −→ Pic π −1 (U ) −→ H 1 π −1 (U ), ᏴY1 −→ 0. We sheafify this on X = XZar to get an exact sequence of sheaves Pic0 (Y )X −→ R 1 π∗ ᏻ∗Y −→ R 1 π∗ ᏴY1 −→ 0, where Pic0 (Y )X is the constant sheaf on X associated to the group Pic0 (Y ). Define Ᏺ to be the Zariski sheaf
 Ᏺ := Im Pic0 (Y )X −→ R 1 π∗ ᏻ∗Y on X. Hence we have short exact sequence of sheaves (4.1)

0 −→ Ᏺ −→ R 1 π∗ ᏻ∗Y −→ R 1 π∗ ᏴY1 −→ 0.

Lemma 4.1. (1) There is an injective map s 

µ : J H 0 Xan , ᏲZ −→ H 0 (X, Ᏺ), whose image H 0 (X, Ᏺ)0 is a subgroup of finite index, such that the natural map Pic0 (Y ) → H 0 (X, Ᏺ) factors through µ. The induced map Pic0 (Y ) → H 0 (X, Ᏺ)0 is that determined by the map on Hodge structures H 1 (Yan , Z) → H 0 (Xan , ᏲZ )s . Thus H 0 (X, Ᏺ)0 is the group of C-points of an abelian variety, such that Pic0 (Y ) → H 0 (X, Ᏺ)0 is a homomorphism of abelian varieties. (2) We have that 
s
0 H 0 Xan , ᏲZ H 0 X, Ᏺ

J is isogenous to
, Im H 1 Yan , Z Im Pic0 (Y ) and hence the latter is isogenous to J (M). Proof. Let x ∈ X be any point, and let Fx = π −1 (x)red . Let Yx = Spec(ᏻX,x )×X Y , and let Fxn = Spec(ᏻX,x /ᏹxn ) ×X Y , where ᏹ ⊂ ᏻX,x is the maximal ideal. For each n, we have the restriction maps hn : Pic(Fxn ) → Pic(Fx ). We claim that the kernel of hn , for each n, is a C-vector space. To see this, consider the short exact sequence of Zariski sheaves exp

hn

0 −→ ᏵFxn /Fx −−→ ᏻ∗Fxn −−→ ᏻ∗Fx −→ 0,

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

443

where exp denotes the exponential map, which makes sense as ᏵFxn /Fx is nilpotent. Considering the associated cohomology sequence, we get


 hn 0 −→ H 1 Fx , ᏵFxn |Fx −→ Pic Fxn −−→ Pic Fx , which proves the kernel of hn , for each n, is a C-vector space. For each n and for each x ∈ X, we have a commutative diagram fn

54
 76  / Pic(Yx ) / Pic F n Pic0 (Y )J x JJ t JJg hn ttt JJ t t JJ t $
 ytt Pic Fx .

Let fn be the composition Pic0 (Y ) → Pic(Yx ) → Pic(Fxn ). Then, ker(fn ) and ker(g) are both closed subgroups of Pic0 (Y ) and, hence, are compact (topological) groups. Since ker(hn ) is a C-vector space, it follows that ker(fn ) = ker(g), as any continuous homomorphism from a compact group to a C-vector space is zero. (Note that Pic0 (Y ), Pic0 (Fxn ), and Pic0 (Fx ) are isomorphic to the corresponding analytic groups, by GAGA, and hence from the exponential sequence carry natural topologies, such that the restriction homomorphisms are continuous.) Passing to the inverse limit, we have a commutative diagram
 f / Pic Yx   / Pic(Yˆ ) Pic0 (Y )J x JJ t t JJg h tt JJ t JJ tt %
 ztt Pic Fx , where Yˆx stands for the completion of Yx along Fx . By Grothendieck’s formal func(Pic(Fxn )) is an tion theorem [H, Theorem 11.1] and the fact that Pic(Yˆx ) → lim ← n− isomorphism [H, Example 9.6], we have that Pic(Yx ) → Pic(Yˆx ) is an injection. Thus it follows that ker(f ) = ker(g). We have from the definition of Ᏺ that the stalk of Ᏺ at x, Ᏺx = Im(Pic0 (Y ) → Pic(Yx )). By our analysis so far, we have proved that the natural map Ᏺx → Pic(Fx ) is an inclusion, and it clearly factors through the subgroup Pic0 (Fx ). By the results of Du Bois [Du], there exists a commutative triangle 

 / H 1 Fx , ᏻF H 1 Fx , C x LLL LLL α LLL L%  
H 1 Fx , C
. F 1 H 1 Fx , C

444

BISWAS AND SRINIVAS

Note that ker(α) is a C-vector space. Now α induces a map



 H 1 Fx , ᏻ 0  −→ J H 1 Fx , Z . β : Pic Fx ∼ = 1
H Fx , Z Thus we have a diagram H1




Fx , ᏻFx


  Pic0 Fx



α


 1 F ,C H x /
 F 1 H 1 Fx , C

β



  / J H 1 Fx , Z .

Since H 1 (Fx , Z) is a mixed Hodge structure with weights 0 and 1 (by [D2], as Fx is a projective variety), H 1 (Fx , Z) injects into (H 1 (Fx , C))/(F 1 H 1 (Fx , C)). Thus it is clear from the above diagram that ker(β) = ker(α) and so ker(β) is also a C-vector space. Hence the composite Ᏺx → Pic0 (Fx ) → J (H 1 (Fx , Z)) is injective, as Ᏺx is a compact group, from its definition. s Let ᏲZ,x = Im(H 1 (Y, Z) → H 1 (Fx , Z)) be the stalk of ᏲZ at x. Let ᏲZ,x be the 1 1 saturation of ᏲZ,x in H (Fx , Z). The inclusion ᏲZ,x → H (Fx , Z) induces a natural map with finite kernel J (ᏲZ,x ) → J (H 1 (Fx , Z)). In fact this map factors as 
J ᏲZ,x


 / / J Ᏺs Z,x



 / J H 1 Fx , Z .

s ) is the image of J (Ᏺ 1 The second map is an inclusion, and J (ᏲZ,x Z,x ) in J (H (Fx ,Z)). We thus have a commutative diagram with surjective and injective maps as follows (where we identify Pic0 (Y ) with J (H 1 (Yan , Z))):

Pic0 (Y )   Ᏺx


 / J Ᏺs Z,x qk qqq K q q q  qqq xq
1
 / J H Fx , Z .
 / / J ᏲZ,x

s ) and Ᏺ are both the image of Since it is clear from the diagram that J (ᏲZ,x x s ) ∼ Ᏺ . Therefore, there exists a Pic0 (Y ) in J (H 1 (Fx , Z)), it follows that J (ᏲZ,x = x map J (ᏲZ,x ) → Ᏺx that is an isogeny. We had proved that the sheaf ᏲZ was constructible, that is, constant with groups Gi over locally closed sets (Ui )an , and this data gives rise to a flasque resolution of a∗ ᏲZ in the Zariski site (by Lemmas 2.7 and 2.5). It is then clear that analogous results also hold for the sheaf ᏲZs , where ᏲZs denotes the saturation of the sheaf ᏲZ

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

445

s ) are in R 1 π∗an Z (which is a torsion-free sheaf). Thus the abelian varieties J (ᏲZ,x 0 constant quotients of Pic (Y ) over the strata Ui , and hence so are Ᏺx . This proves that the sheaf Ᏺ is a constructible sheaf on X for the Zariski topology, with admissible family {Xi }, and further (by Lemma 2.5), Ᏺ has a flasque resolution similar to a∗ ᏲZ . Taking global sections of the flasque resolution of a∗ ᏲZs , we get an exact sequence  
 s s ᏲZ,x −→ ᏲZ,x . 0 −→ H 0 Xan , ᏲZs −→ i j i

i<j

It is also clear from the definitions that H 0 (Xan , ᏲZs ) = H 0 (Xan , ᏲZ )s . Applying J on all the terms, we obtain a complex
s 
s  

−→ ⊕i<j J ᏲZ,x . 0 −→ J H 0 Xan , ᏲZs −→ ⊕i J ᏲZ,x i j This complex is exact on the left and has finite homology in the middle, since s ) is torsion-free. Similarly, taking global sections of the flasque resoH 0 (⊕i<j ᏲZ,x j lution of Ᏺ, we get an exact sequence 0 −→ H 0 (X, Ᏺ) −→ ⊕i Ᏺxi −→ ⊕i<j Ᏺxj . There exists a commutative diagram

s  J H 0 X, ᏲZ


 / ⊕i J Ᏺ s Z,xi


s  / ⊕i<j J ᏲZ,x j

/ ⊕i Ᏺxi

/ ⊕i<j Ᏺxj ,

µ

0


  / H 0 X, Ᏺ

where the two vertical arrows are isomorphisms. Hence the dotted arrow µ exists and is an inclusion with finite cokernel. Define

s  H 0 (X, Ᏺ)0 = Im J H 0 X, ᏲZ . Clearly this is an abelian variety, and there is an isogeny J (H 0 (X, ᏲZ )) → H 0 (X, Ᏺ)0 . Also, by construction, the natural map Pic0 (Y ) → H 0 (X, Ᏺ) clearly factors through the map



 Pic0 (Y ) = J H 1 Yan , Z −→ J H 0 X, ᏲZ . Thus this proves that we have an isogeny J (H 0 (Xan , ᏲZ ))/ Im(Pic0 (Y )) → (H 0 (X, Ᏺ)0 / Im(Pic0 (Y ))). We finally note that there exists an isogeny  


H 0 Xan , ᏲZ J H 0 Xan , ᏲZ

 ,
 −→ J Im H 1 Yan , Z Im Pic0 (Y ) since J (H 1 (Yan , Z)) ∼ = Pic0 (Y ).

446

BISWAS AND SRINIVAS

This finishes the proof that H 0 (X, Ᏺ)0
 Im Pic0 (Y )

 and

J


 H 0 Xan , ᏲZ

 Im H 1 Yan , Z

are isogenous. We can now construct a 1-motive, as follows. Since X is normal, we have that π∗ ᏻY = ᏻX , and so we have an exact sequence
 0 −→ Pic(X) −→ Pic(Y ) −→ H 0 X, R 1 π∗ ᏻ∗Y . This induces another exact sequence


 H 0 X, R 1 π∗ ᏻ∗Y NS(X) −→ NS(Y ) −→
 . Im Pic0 (Y )

We thus have an injective map (4.2)


 H 0 X, R 1 π∗ ᏻ∗Y NS(Y )
 −→
 . Im NS(X) Im Pic0 (Y )

Lemma 4.2. The map (4.2) induces an (injective) map  


1 −→ H 2 Y , Z Im H 1 X, ᏴX .(X, Ᏺ) an
 −→ φ:
. Im NS(X) Im Pic0 (Y ) Proof. Using the short exact sequence of sheaves (4.1), we get the following commutative diagram, whose right column is exact,  


1 −→ H 2 Y , Z .(X, Ᏺ) Im H 1 X, ᏴX φ an /

 Im Pic0 (Y ) Im NS(X) _  NS(Y )  
 Im NS(X)


 . X, R 1 π∗ ᏻ∗ /
 Im Pic0 (Y ) 
 . X, R 1 π∗ ᏴY1 .

Here we claim the dotted arrow φ exists (and is also injective) because the composition

 
1 −→ H 2 Y , Z
 Im H 1 X, ᏴX an −→ H 0 X, R 1 π∗ ᏴY1 NS(X) is zero. This is obvious as this map can be described in the following way: Given the image in H 1 (Y, ᏴY1 ) = NS(Y ) of a Zariski locally trivial cohomology class η ∈ 1 ), consider a line bundle L on Y that represents it. Then consider the line H 1 (X, ᏴX η

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

447

bundle restricted to open sets π −1 (U ) ⊂ Y, Lη |π −1 (U ) (where U ⊂ X open), and take the Chern classes of these restrictions. These give a global section of R 1 π∗ ᏴY1 that is zero as the line bundle came from a locally trivial cohomology class on X. Since (.(X, Ᏺ))/(Im(Pic0 (Y ))) has a subgroup of finite index that is an abelian variety, φ determines a 1-motive in an obvious way: B −→

.(X, Ᏺ)0
, Im Pic0 (Y )

where B is the inverse image under φ of the abelian variety. Remark 4.3. We do not know if .(X, Ᏺ) is itself an abelian variety, that is, if .(X, Ᏺ)0 = .(X, Ᏺ). 4.2. Comparison of the two 1-motives. We now finish the proof of the theorem by comparing the 1-motive constructed above using φ with that constructed earlier 1 ). using the extension class map ψ for the mixed Hodge structure on H 1 (X, ᏴX 1 an Recall that {Xi }i∈I is the chosen admissible family of subsets for R π∗ Z and hence for ᏲZ and Ᏺ as well; recall also the corresponding (irreducible, nonsingular) locally closed strata {Ui }i∈I . Also recall the choice of points xi ∈ Ui , and Fi = π −1 (xi ), F = ∪i∈I Fi .

→ Y with Y

nonsingular projective, so that the Suitably blow up Y to get f : Y

. Note reduced strict transform of F is a smooth, possibly disconnected subvariety F that there exists the following diagram:  /Y



F   F

f

 / Y.

Now consider the following commutative diagram that is a diagram in the category of mixed Hodge structures:
s


  
H 0 Xan , ᏲZ / / H 1 X, Ᏼ1 / Im H 1 X, Ᏼ1 −→ H 2 Yan , Z /0

 0 X X 1 Im H Yan , Z

0

0


 1 F ,Z H an /

 Im H 1 Yan , Z


 / H 2 Yan , Fan , Z




/ Im H 2 Yan , Fan , Z −→ H 2 Yan , Z

/0


  H1 F an , Z /

 Im H 1 Y an , Z


 / H 2 Y  an , F an , Z




2 ,Z / Im H 2 Y  an , F an , Z −→ H Y an

/ 0.

448

BISWAS AND SRINIVAS

s 1  , Z)) in H 1 (F  Let Im(H 1 (Y an , Z)) be the saturation of Im(H (Y an an , Z). Then, arguing as before,

 s

 Im H 1 Y H1 F an , Z an , Z  =

 . Im NS(X) ∩
1
Im H Y Im H 1 Y an , Z an , Z

So we have a short exact sequence of mixed Hodge structures  

  H1 F H 2 Y an , Z an , F an , Z
 0 −→

s −→ Im NS(X) Im H 1 Y an , Z  


2 ,Z  Im H 2 Y an , F an , Z −→ H Y an
 −→ 0 −→ Im NS(X) and a commutative diagram
s H 0 Xan , ᏲZ

s Im H 1 Yan , Z

0 /

0


 1 1 / H X, ᏴX NS(X) /



 
1 −→ H 2 Y , Z Im H 1 X, ᏴX an
 Im NS(X)

/0

 

2

  
2 Y 2 Y       H1 F , Z) H Im H , F , Z , F Y , Z −→H , Z an an an an an an / / / 0.



s / Im NS(X) Im NS(X) Im H 1 Y an , Z

The following diagram commutes by functoriality of the extension class maps: 


1 −→ H 2 Y , Z Im H 1 X, ᏴX an

 
 Im H Y , F , Z −→ H 2 Y an an an , Z





2



where J

ψ

 /J

 /J


s H 0 Xan , ᏲZ

 Im H 1 Yan , Z


  H1 F an , Z

s , Im H 1 Y an , Z

 
s
 H1 F H 0 Xan , ᏲZ an , Z



s −→ J s Im H 1 Yan , Z Im H 1 Y an , Z

is induced from the map on the underlying Hodge structures. Let  

1
 
H1 F " an , Z 1 2

 ψ : Im H X, ᏴX −→ H Yan , Z −→ J Im H 1 Y an , Z

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

449

be the composite in the diagram above. We also have a natural “sheaf-theoretic” map 

  

1 −→ H 2 Y , Z  H1 F Im H 1 X, ᏴX " an an , Z

 , φ : −→ J NS(X) Im H 1 Y an , Z 1 ) → H 2 (Y , Z)). Consider a which is defined as follows. Let η ∈ Im(H 1 (X, ᏴX an line bundle L on Y such that c1 (L) = η. Then, L|F (where F = ∪i Fi ) gives an element of Pic0 (F ) and hence an element of J (H 1 (F, Z)) via the mapping Pic0 (F ) → J (H 1 (F, Z)). Under this mapping, NS(X) goes to zero, and hence we get a welldefined mapping 

 

 1 −→ H 2 Y , Z Im H 1 X, ᏴX H 1 Fan , Z an

 . −→ J NS(X) Im H 1 Yan , Z

Now compose with the map (induced by the morphism of the underlying Hodge structures)    

 H1 F H 1 Fan , Z an , Z

 −→ J

 J Im H 1 Yan , Z Im H 1 Y an , Z to get




  
1 −→ H 2 Y , Z  H1 F Im H 1 X, ᏴX an an , Z

 . −→ J φ : NS(X) Im H 1 Y an , Z "

It is easy to see the following diagram commutes:  


1 −→ H 2 Y , Z H 0 (X, Ᏺ) Im H 1 X, ᏴX φ an /

 Im Pic0 (Y ) Im NS(X) RRR RRR φ " RRR RRR R)  
 1  H F an , Z

 , J Im H 1 Y an , Z 1  , Z)))) is  where the map (H 0 (X, Ᏺ))/(Im(Pic0 (Y ))) → J ((H 1 (F an , Z))/(Im(H (Y an the composition  




 H 0 X, Ᏺ Pic0 F H1 F Pic0 (F ) an , Z

 .
 −→
 −→

 −→ J

Im H 1 Y Im Pic0 (Y ) Im Pic0 (Y ) Im Pic0 Y an , Z

We now note that the map H 0 (Xan , ᏲZ )s → H 1 (Fan , Z) is an injective map. Since 0 s  W0 H 1 (Fan , Q) = ker(H 1 (Fan , Q) → H 1 (F an , Q)) and H (Xan , ᏲZ ) is pure of weight 1, it follows that

1
 



 ker H Fan , Q −→ H 1 F ∩ Im H 0 Xan , ᏲZ −→ H 1 Fan , Q = 0. an , Q

450

BISWAS AND SRINIVAS

Hence the composite
  

 H 0 Xan , ᏲZ −→ H 1 Fan , Z −→ H 1 F an , Z has finite kernel and is hence injective (as ᏲZ ⊂ R 1 π∗an Z is torsion-free). It follows that  J

 
s
 H 0 Xan , ᏲZ H1 F an , Z

s −→ J

s Im H 1 Yan , Z Im H 1 Y an , Z

(+ + +)

has finite kernel. We have the following diagram, which shows all the maps we have constructed so far (the outer border is not yet known to commute):  J


s H 0 Xan , ᏲZ

s o Im H 1 Yan , Z



 
1 −→ H 2 Y , Z Im H 1 X, ᏴX an
 Im NS(X) _

ψ

φ

 J


   H1 F an , Z

s o Im H 1 Y an , Z



s  J H 0 Xan , ᏲZ 
0  Im Pic (Y )

 0 (X, Ᏺ) H /
 Im Pic0 (Y )



  J H1 F an , Z

 o

Im Pic0 Y


 

Pic0 F
0
 .

Im Pic Y

We prove that the following subdiagram commutes: 


s H 0 Xan , ᏲZ

s o Im H 1 Yan , Z




  H1 F an , Z

s o Im H 1 Y an , Z

J

J

ψ



 
1 −→ H 2 Y , Z Im H 1 X, ᏴX an
 Im NS(X) _ φ


  H 0 X, Ᏺ
. Im Pic0 (Y )

(∗)

Note that the composite map 

  

1 −→ H 2 Y , Z  Im H 1 X, ᏴX H1 F H 0 (X, Ᏺ) an an , Z


 −→
 −→ J Im NS(Y ) Im H 1 Y Im Pic0 (Y ) an , Z

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

451

"

is the previously defined map φ , and the map   

s

1 −→ H 2 Y , Z H 0 Xan , ᏲZ Im H 1 X, ᏴX an
 −→ J

s Im NS(Y ) Im H 1 Yan , Z 
  H1 F an , Z

 −→ J Im H 1 Y an , Z "

is our previously defined map ψ . Thus the commutativity of the above diagram is equivalent to proving " " ψ =φ . Assuming this diagram commutes, we finish the proof of Theorem 1.1 as follows. We claim that the composite map    



1 −→ H 2 Y , Z  H1 F Im H 1 X, ᏴX an an , Z
 −→ J

s Im NS(X) Im H 1 Y an , Z has finite kernel, as φ is injective and the map 
  H1 F H 0 (X, Ᏺ) an , Z

s
 −→ J Im H 1 Y Im Pic0 (Y ) an , Z has finite kernel (combining Lemma 4.1 with the fact that the map in (+ + +) has finite kernel). Thus ψ has finite kernel. Now recall the map A −→ J (M), ψ1 : Zr ∼ = Ators 1 ) → H 2 (Y , Z)))/(Im(NS(X))). It follows immediately where A = (Im(H 1 (X, ᏴX an that ψ1 has finite kernel. Since A/Ators is a free abelian group, it follows that ψ1 is injective. This is equivalent to proving our main result, Theorem 1.1, as has been remarked before. We now finish the final part of the proof by showing the commutativity of the diagram (∗). Let Z be a smooth projective variety over C and let W ⊂ Z be a smooth subvariety. Let η ∈ H 2 (Zan , Z) be an algebraic class (i.e., let η ∈ NS(Z)), such that η → 0 ∈ H 2 (Wan , Z). Then η gives rise to the following pullback diagram: 


 

H 1 Wan , Z / / H 2 Zan , Wan , Z / ker H 2 Zan , Z −→ H 2 Wan , Z /0 
0 1 O O H Zan , Z

0

/


H 1 Wan , Z 
H 1 Zan , Z

/B

/ Z[η]

/ 0.

452

BISWAS AND SRINIVAS

Thus we have an extension class map Z∼ = Z[η]

ψ

""

 /J


 H 1 Wan , Z  .
H 1 Zan , Z

Again, given η as above, consider Lη , a line bundle on Z that has Chern class η. Restrict this line bundle on W to get Lη |W ∈ Pic0 (Wan ) ∼ = J (H 1 (Wan , Z)). This gives us a well-defined mapping Z∼ = Z[η]

φ

""

 /J


H 1 Wan , Z
 . H 1 Zan , Z

We now have the following lemma. ""

Lemma 4.4. With the above notation ψ = φ "" , that is, the extension class map and the restriction map corresponding to the class η are the same. Proof. Consider the following diagram with exact rows and columns: 
/ H 0 Wan , Z


 / H 1 Zan , Wan , Z


/ H 1 Zan , Z


/ H 1 Wan , Z /


 0 W ,C H an / 
F 1 H 0 Wan , C

 
1 Z ,W ,C H an an / 
F 1 H 1 Zan , Wan , C


 1 Z ,C H an / 
F 1 H 1 Zan , C


 1 W ,C H an / 
F 1 H 1 Wan , C /


  / H 0 Wan , ᏻ∗ Wan


  / Pic Zan , Wan


  / Pic Zan


  / Pic Wan /


 / H 1 Wan , Z

 
/ H 2 Zan , Wan , Z


 / H 2 Zan , Z


 / H 2 Wan , Z /


 1 W ,C H an / 
F 1 H 1 Wan , C

 
2 Z ,W ,C H an an / 
F 1 H 2 Zan , Wan , C


 2 Z ,C H an / 
F 1 H 2 Zan , C


 2 W ,C H an / 
F 1 H 2 Wan , C /







.

The above diagram comes from the following 9-diagram in the category of sheaves (where the bottom row defines ᏻZan (−Wan )∗ , and j : Zan − Wan $→ Zan is the inclusion):

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

0

0

0

0

 / j! ZZ −W an an

 / ZZ an

 / ZW an

/0

0


  / ᏻZ − Wan an

 / ᏻZan

 / ᏻWan

/0

0

 / ᏻZ (−Wan )∗ an

 / ᏻ∗Z an

 / ᏻ∗W an

/0

 0

 0

 0.

453

Let η ∈ H 2 (Zan , Z) such that η → 0 both in H 2 (Wan , Z) and (H 2 (Zan , C))/(F 1 H 2 (Zan , C)) (i.e., η ∈ NS(Z)). By a diagram chase, as before we get elements δ1 and δ2 in the group (H 1 (Wan , C))/(F 1 H 1 (Wan , C)). Both δ1 and δ2 are well defined in the quotient group (H 1 (Wan , C)/F 1 H 1 (Wan , C))/(Im(H 1 (Wan , Z)+Im(H 1 (Zan , C)/F 1 H 1 (Zan , C))). Now note that 
    


1
 H 1 Wan , Z H 1 Wan , C H 1 Zan , C ∼    .


Im H Wan , Z + Im =J F 1 H 1 Wan , C F 1 H 1 Zan , C H 1 Zan , Z Hence we get two maps η → δ¯1 and η → δ¯2 from  Z∼ = Z[η] −→ J


H 1 Wan , Z  ,
H 1 Zan , Z

where δ¯ denotes the class of δ in the quotient J ((H 1 (Wan , Z))/(H 1 (Zan , Z))). "" We claim that these two maps are nothing but our previously defined maps φ and "" "" ψ , respectively. It is clear that the map η → δ¯2 is equal to φ (η). This is because we get δ¯2 by first taking a lift of η, say, β2 in Pic(Zan ), then restricting to Pic(Wan ) to get γ2 , and finally taking the class δ¯2 ∈ J ((H 1 (Wan , Z))/(H 1 (Zan , Z))). This is "" "" "" exactly how φ (η) was defined, so δ¯2 = φ (η). It is also clear that δ¯1 = ψ (η) as the extension class map is defined exactly the same way as the map η → δ¯1 . Now by "" "" Lemma 2.8, we have that δ¯1 = δ¯2 . This implies that φ = ψ . Remark 4.5. In a similar vein, using Lemma 2.8, one can show that the cycle class map with values in Deligne-Be˘ılinson cohomology restricts to the Abel-Jacobi mapping on cycles that are homologically trivial. This is essentially the argument

454

BISWAS AND SRINIVAS

given in [EV], though the need to appeal to Lemma 2.8 is not brought out explicitly there.

and W = F

in Lemma 4.4. Let η ∈ Im(H 1 (X, Ᏼ1 ) → H 2 (Yan , Z)), Let Z = Y X 2  and let η →

η ∈ H (Yan , Z). Then

η satifies the conditions of Lemma 4.4, that is,

) and



η ∈ NS(Y η → 0 ∈ H 2 (F an , Z). Clearly, " ""
 η ψ (η) = ψ

and Hence,

" ""
 η . φ (η) = φ

"

"

ψ =φ , which proves the commutativity of diagram (∗). This finishes the proof of our main result, Theorem 1.1. 5. An example. In this section, we give an example of an integral projective variety X over C that is not normal, for which we have a strict inclusion   NS(X)⊆/ α ∈ H 2 (X, Z) | α is Zariski locally trivial and αC ∈ F 1 H 2 (X, Z) . Our variety has the property that its normalization Y is nonsingular, and the normalization map π : Y → X is bijective. Then H 2 (X, Z) ∼ = H 2 (Y, Z) as mixed Hodge structures, and the subspaces of Zariski–locally trivial classes correspond. Hence the desired property of X is equivalent to the strictness of the first inclusion NS(X)⊆/ NS(Y ) ⊂ H 2 (Y, Z). We make use of a variant of a construction in [Ha, Example 5.9] (see also [Ha, Example 5.16b]). If V is a nonsingular variety over C, then following [Ha], an infinitesimal extension of V by an invertible ᏻV -module ᏸ is a scheme W with Wred = V , such that the nilradical Ᏽ of ᏻW has square zero (so that it is an ᏻV -module), and there is an ᏻV -isomorphism Ᏽ ∼ = ᏸ. In other words, there is an exact sequence of sheaves of ᏻW -modules 0 −→ ᏸ −→ ᏻW −→ ᏻV −→ 0. There is a corresponding exact sequence of sheaves 0 −→ ᏸ −→ ᏻ∗W −→ ᏻ∗V −→ 0, where ᏸ is identified with the (multiplicative) subsheaf of units on W that restrict to 1 on V (the identification is given on sections by s → 1 + s). The latter exact sheaf sequence gives rise to an exact sequence on cohomology: δ

H 1 (V , ᏸ) −→ Pic W −→ Pic V → H 2 (V , ᏸ).

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

455

The following is an elaboration of [Ha, Example 5.9] (the proof is left as an exercise). Lemma 5.1. (i) There is a natural bijection between isomorphism classes of infinitesimal extensions of V by ᏸ and elements

 α ∈ H 1 V , Ᏼom V <1V /C , ᏸ . (ii) If W is the infinitesimal extension corresponding to α, the boundary map δ = δα : Pic V → H 2 (V , ᏸ) is expressible as the composition
 dlog
 ∪α Pic V = H 1 V , ᏻ∗V −−−→ H 1 V , <1V /C −−→ H 2 (V , ᏸ). (iii) Let f : ᏸ → ᏹ be a morphism of ᏻX -modules, and let α → f∗ (α) under the natural map



 f∗ : H 1 V , Ᏼom V <1V /C , ᏸ −→ H 1 V , Ᏼom V <1V /C , ᏹ . Let Z be the infinitesimal extension of V by ᏹ determined by f∗ (α). Then there is a unique morphism of C-schemes f : Z → W , such that the corresponding morphism of reduced schemes is the identity on V , and such that there are commutative diagrams with exact rows /ᏸ / ᏻW / ᏻV /0 0 f

0

 /ᏹ

f∗

 / ᏻZ

/ ᏻV

/0

and H 1 (V , ᏸ) f∗

 H 1 (V , ᏹ)

/ Pic W

/ Pic V

δα

/ H 2 (V , ᏸ)

δf∗(α)

 / H 2 (V , ᏹ).

f∗

 / Pic Z

f∗

/ Pic V

Example 5.2. In the above lemma, take V = P1C × P1C , and ᏸ = ωV , ᏹ = ᏻV , f : ᏸ $→ ᏹ any nonzero map (since ωV ∼ = ᏻP1 (−2)
ᏻP1 (−2), such maps f exist). Infinitesimal extensions of V by ᏸ are classified by elements of


 H 1 V , Ᏼom V <1V /C , ωV ∼ = H 1 V , <1V /C , where we have identified Ᏼom V (<1V /C , ωV ) with <1V /C using the nondegenerate bilinear form <1V /C ⊗ᏻV <1V /C −→ ωV

456

BISWAS AND SRINIVAS

given by the exterior product of 1-forms. Thus, if α ∈ H 1 (V , <1V /C ), the corresponding cup-product map
 ∪α
 H 1 V , <1V /C −→ H 2 V , ωV is just the product with α in the commutative graded ring
 ⊕n≥0 H n V ,
0 −→ Pic W −→ Pic V −−−→ Z −→ 0 ((D · L2 ) = 1, so the map to Z is surjective). Here Pic V = Pic(P1C × P1C ) = Z[L1 ] ⊕ Z[L2 ] is free abelian of rank 2, and as usual we denote a representative of the class of a[L1 ] + b[L2 ] by ᏻV (a, b). Next, note that

 f∗ (α) ∈ H 1 V , Ᏼom V <1V /C , ᏻV = 0, since <1V /C ∼ = ᏻV (−2, 0) ⊕ ᏻV (0, −2). Hence the infinitesimal extension Z of V by ᏻV determined by f∗ (α) is the trivial extension (V , ᏻV [@]), where ᏻV [@] is the sheaf of dual numbers over ᏻV . There is an obvious way in which we may regard Z = (V , ᏻV [@]) as a closed subscheme of Y = P1C × P1C × P1C = V × P1C , whose underlying reduced scheme is V × {0}. Finally, we define X to be the C-scheme that is the pushout of Y and W along the morphisms f : Z → W and the above inclusion i : Z $→ Y , so that there is a commutative pushout diagram Z

i

/ Y = V × P1 C

f

 W

π

j

 / X.

LEFSCHETZ (1,1) THEOREM FOR SINGULAR VARIETIES

457

Since f is a finite and bijective morphism, we see that for each affine open U = Spec A in Y , U ∩ Z = Spec A/I is affine and finite over the affine open subscheme f (U ∩ Z) = Spec B ⊂ W . The image of U in X is then defined as the affine scheme Spec C, where C is the inverse image of B in A under the surjection A  A/I . One shows easily that C is in fact a finitely generated C-subalgebra of A, and A is a finite C-module with conductor ideal I . Further, the construction of C localizes well. Hence the local schemes Spec C can be glued together to yield the C-scheme X. We claim that this scheme X has the desired properties, that is, X is an integral projective scheme over C with normalization π : Y → X, such that Y is nonsingular and bijective with X, while NS(X) → NS(Y ) is a strict inclusion. That X is integral with Y as its normalization is clear from the description of its affine open sets above. Next, since α = [D] and (D ·(L1 +L2 )) = 0, there is a unique Ᏼ ∈ Pic W such that Ᏼ ⊗ ᏻV = ᏻV (1, 1). Also, Pic Z → Pic V is an isomorphism. We have Pic Y = Z⊕3 , where we may regard the restriction map Pic Y → Pic V × {0} = Pic V = Z⊕2 as projection on the first two factors. Hence we see that the very ample invertible sheaf ᏻY (1, 1, 1) has the property that there is an isomorphism f ∗ Ᏼ ∼ = ᏻY (1, 1, 1) ⊗ ᏻZ . From the Mayer-Vietoris sequence of sheaves of rings 0 −→ ᏻX −→ π∗ ᏻY ⊕ j∗ ᏻW −→ (i ◦ π)∗ ᏻZ −→ 0, we have a corresponding sequence of sheaves of unit groups 0 −→ ᏻ∗X −→ π∗ ᏻ∗Y ⊕ j∗ ᏻ∗W −→ (i ◦ π)∗ ᏻ∗Z −→ 0 leading to an exact sequence


 H 0 Y, ᏻ∗Y ⊕ H 0 W, ᏻ∗W −→ H 0 Z, ᏻ∗Z −→ Pic X −→ Pic Y ⊕ Pic W −→ Pic Z. Hence there exists an invertible sheaf Ꮽ on X with π ∗ Ꮽ = ᏻY (1, 1, 1) (and j ∗ Ꮽ = Ᏼ). Since π is finite and π ∗ Ꮽ is ample on Y , we have that Ꮽ is ample on X. Hence X is projective. From the exact sequence 0 −→ ωV −→ ᏻW −→ ᏻV −→ 0, we see that H 0 (W, ᏻW ) → H 0 (V , ᏻV ) = C is an isomorphism. On the other hand, we see at once that H 0 (Z, ᏻZ ) = C[@] is the ring of dual numbers. Hence we get analogous formulas for the unit groups. Thus there is an exact sequence 0 −→ C −→ Pic X −→ Pic Y ⊕ Pic W −→ Pic Z −→ 0 (note that Pic Z = Pic V is a quotient of Pic Y ). Since Pic W $→ Pic Z is an inclusion Z $→ Z⊕2 as a direct summand, while Pic Y → Pic Z is the projection Z⊕3  Z⊕2 , we see that image (Pic X → Pic Y = NS(Y )) is a direct summand Z⊕2 $→ Z⊕3 . Thus

458

BISWAS AND SRINIVAS

NS(X) = Z⊕2 is strictly contained in NS(Y ) = Z⊕3 . References [BS1] [BS2] [C]

[Co] [D] [Du] [EV]

[GH] [H] [J] [L] [PS] [V]

L. Barbieri Viale and V. Srinivas, On the Néron-Severi group of a singular variety, J. Reine Angew. Math. 435 (1993), 65–82. , The Néron-Severi group and the mixed Hodge structure on H 2 , J. Reine Angew. Math. 450 (1994), 37–42. J. Carlson, “The geometry of the extension class of a mixed Hodge structure” in Algebraic Geometry: Bowdoin 1985 (Brunswick, Maine, 1985), Part 2, Proc. Sympos. Pure Math. 46, Amer. Math. Soc., Providence, 1987, 199–222. A. Collino, Washnitzer’s conjecture and the cohomology of a variety with a single isolated singularity, Illinois J. Math. 29 (1985), 353–364. P. Deligne, Théorie de Hodge, II, Inst. Hautes Études Sci. Publ. Math. 40 (1971), 5–57; III, Inst. Hautes Études Sci. Publ. Math. 44 (1974), 5–77. P. Du Bois, Complexe de de Rham filtré d’une variété singulière, Bull. Soc. Math. France 109 (1981), 41–81. H. Esnault and E. Viehweg, “Deligne-Be˘ılinson cohomology” in Be˘ılinson’s Conjectures on Special Values of L-Functions, Perspect. Math. 4, Academic Press, Boston, 1988, 43–91. P. A. Griffiths and J. Harris, Principles of Algebraic Geometry, Pure Appl. Math., WileyInterscience, New York, 1978. R. Hartshorne, Algebraic Geometry, Grad. Texts in Math. 52, Springer-Verlag, New York, 1977. U. Jannsen, Mixed Motives and Algebraic K-Theory, Lecture Notes in Math. 1400, SpringerVerlag, Berlin, 1990. S. Lefschetz, L’analysis situs et la géométrie algébrique, Gauthier-Villars, Paris, 1950. R. Parimala and V. Srinivas, Analogues of the Brauer group for algebras with involution, Duke Math. J. 66 (1992), 207–237. J.-L. Verdier, “Classe d’homologie associée à un cycle” in Séminaire de géométrie analytique (École Norm. Sup., Paris, 1974–75), Astérisque 36–37, Soc. Math. France, Paris, 1976, 101–151.

Biswas: Institute of Mathematical Sciences, Taramani, Chennai-600 113, India; [email protected] Srinivas: School of Mathematics, Tata Institute of Fundamental Research, Homi Bhabha Road, Bombay-400005, India; [email protected]

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

EXISTENCE AND REGULARITY FOR HIGHERDIMENSIONAL H -SYSTEMS FRANK DUZAAR and JOSEPH F. GROTOWSKI

1. Introduction. In this paper we are concerned with the existence and regularity of solutions of the degenerate nonlinear elliptic systems known as H -systems. For a given real-valued function H defined on (a subset of) Rn+1 , the associated H -system on a subdomain of Rn (we generally take the domain to be B, the unit ball) is given by
 √ Dxi |Du|n−2 Dxi u = nn (H ◦ u)ux1 × · · · × uxn (1.1) for a map u from B to Rn+1 . (Obviously for (1.1) to make sense classically, we look for u ∈ C 2 (B, Rn+1 ). As we discuss in Section 2, it also makes sense to look for a weak solution u ∈ W 1,n (B, Rn+1 ) to (1.1) under suitable restrictions on H .) Here we use the summation convention, and the cross product w1 × · · · × wn : Rn+1 ⊕ · · · ⊕ Rn+1 → Rn+1 is defined by the property that w ·w1 ×· · ·×wn = det W for all vectors w ∈ Rn+1 , where W is the (n+1)×(n+1) matrix whose first row is (w1 , . . . , w n+1 ) and whose j th row is (wj1−1 , . . . , wjn+1 −1 ) for 2 ≤ j ≤ n + 1. Equation (1.1) has a natural geometric property; namely, if u fulfills certain additional conditions, then it represents a hypersurface in Rn+1 whose mean curvature at the point u(x), for x ∈ B, is given by H ◦ u(x). Specifically, a map u : B → Rn+1 is called conformal if uxi · uxj = λ2 (x)δij

on B

(1.2)

for some real-valued function λ. If u ∈ C 2 (B, R3 ) is conformal, then it is possible to show that u defines a hypersurface in Rn+1 which has mean curvature H ◦ u(x) at every regular point u(x), meaning a point where ux1 × · · · × uxn does not vanish. For n = 2 this observation is the starting point for all existence results for parametric surfaces of prescribed mean curvature (cf. the references cited below for the Plateau problem). For n ≥ 3 a derivation can be found in [DuF4, pp. 42 ff.]. We wish to discuss boundary value problems associated with (1.1), and we first consider the case n = 2. Here the map u satisfies the Plateau boundary condition for a given rectifiable Jordan curve  in R3 if u|∂B is a homeomorphism from ∂B to . Received 2 October 1998. 1991 Mathematics Subject Classification. Primary 58E15; Secondary 35J50, 35J60, 35J70. 459

(1.3)

460

DUZAAR AND GROTOWSKI

The Plateau problem for H and , which we denote by ᏼ(H, ), consists of solving (1.1) subject to the conditions (1.2) and (1.3). The problem ᏼ(H, ) is thus a generalization of the classical Plateau problem for minimal surfaces (i.e., the case H ≡ 0) first solved by Douglas and by Radó in the early 1930s. We refer the reader to the monograph [DHKW] for details and literature concerning this case, and we assume that H does not vanish identically in the rest of this discussion. One can also consider the Dirichlet boundary condition u|∂B = ϕ

(1.4)

for a suitably regular prescribed ϕ. We denote the Dirichlet problem associated with H and ϕ (i.e., the problem of solving (1.1) subject to (1.4)) by Ᏸ(H, ϕ). Solutions of Ᏸ(H, ϕ) do not, in general, fulfill the conformality condition (1.2) and, hence, do not have the geometric interpretation as surfaces of prescribed mean curvature. We return to this point later in the discussion. The first existence results for nonzero H , both for Ᏸ(H, ϕ) and for ᏼ(H, ), were obtained by Heinz [He]. Further existence results were obtained by many authors, including Werner [Wr], Hildebrandt [Hi1], [Hi2], Wente [W], Gulliver and Spruck [GS1], [GS2], and Steffen [St1], [St2]. In particular, we note the so-called Wentetype existence theorems, such as [W, Theorem 6.2] (in the case of constant H ) and [St1, Theorem 6.2] (for H not a priori constant and under more general conditions), where smallness of H in a suitable sense (namely, when compared to an appropriate power of the minimal area of a surface spanning ) guarantees a solution of ᏼ(H, ). Similar results for the Dirichlet problem Ᏸ(H, ϕ) are given in [St1, Theorem 6.2]. In higher dimensions the formulation of the Plateau problem ᏼ(H, ) depends crucially upon the chosen generalization of the boundary condition (1.4) and in particular on the boundary . In the setting of geometric measure theory, one can take  to be a closed, integermultiplicity, rectifiable current of dimension n − 1; the Plateau problem ᏼ(H, ) is to find an n-dimensional integer-multiplicity rectifiable current T with ∂T =  such that the weak version of (1.1) is satisfied for T , that is, 
 (1.5) divM Y + H Y · νT dµT = 0 M

for all test vector fields Y ∈ Cc1 (Rn+1 , Rn+1 ) with spt(Y ) ∩ spt  = ∅. Here µT is the n-dimensional Hausdorff measure weighted by the multiplicity function of T , νT is the unit normal vector field on T , and M is the supporting set of T in Rn+1 (cf. [Si, Section 16.5]). Existence results, again in terms of Wente-type theorems, were proven by Duzaar and Fuchs [DuF2], [DuF5] and by Duzaar [Du2]. The general strategy for the solution of ᏼ(H, ) is similar in the 2-dimensional parametric setting and the geometric measure theory setting in higher dimensions. For ease of discussion, we sketch the procedure in the classical case of the 2-dimensional parametric setting. The first step is to construct a suitable energy EH (u) whose critical

EXISTENCE AND REGULARITY FOR H -SYSTEMS

461

points are (at least formally) the desired solutions of the Plateau problem ᏼ(H, ). The next step is to show that the minimum of this energy is in fact achieved, and that it is achieved by a surface in the desired class. This energy is composed of two terms, the first of which is the (2–)Dirichlet integral, denoted by D(u), the second of which is an appropriately weighted (depending on H ) volume term VH (u). The volume term is not lower-semicontinuous with respect to weak convergence in any space which is appropriate to this setting, so it is necessary to control VH (u) in terms of D(u). This is done by applying suitable isoperimetric inequalities. One also has that the volume term VH (u) is invariant under orientation-preserving diffeomorphisms from B to B, the inner variations. This yields a second Euler equation for D (the first inner variation of D; cf. [DHKW, Chapter 4.5]), which in turn yields the conformality condition (1.2); see [C, p. 112]. In the current paper we consider the Dirichlet problem Ᏸ(H, ϕ) in dimension n ≥ 3. We follow the same broad strategy discussed above to obtain existence results. In Section 3 we give a variational formulation of the problem in the space W 1,n (B, Rn+1 ); the aim is to realize the solutions of Ᏸ(H, ϕ) as minimizers of EH in an appropriate subclass of W 1,n (B, Rn+1 ). Since weak W 1,n -convergence does not preserve homology, we are unable to directly adapt the methods of [DuS3] to our situation. (In the setting of geometric measure theory, these authors obtained existence results for solutions of the Plateau problem with the image being contained in a Riemannian manifold of arbitrary dimension.) This motivates the definitions of spherical currents and of homologically n-aspherical domains (Definition 3.1), which allows a reasonable definition of the H -volume enclosed by two maps in W 1,n (B, A) for A ⊂ Rn+1 (Definition 3.4), and hence of EH , the energy functional to be minimized. In order to control the H -volume by the Dirichlet integral, we need an estimate of how much of the volume and surface area can be lost under passage to the weak limit in our chosen subclass. This is accomplished in Lemma 4.1. Such “bubbling phenomena” are an important feature of many nonlinear elliptic and parabolic problems, in particular in the area of harmonic maps. See, for example, [SU] and recent papers concerning the heat-flow for harmonic maps, such as [Q] and [DT]. Once this is accomplished, we need to adapt the notions of isoperimetric conditions from [St1] and later works to our situation. Having done this, in Section 5 we are able to prove existence results under various assumptions on H and on the support of a given extension of our Dirichlet boundary data. Our results include, as a special case (see Corollary 5.3), previous results for the constant H obtained by Duzaar and Fuchs [DuF3] and Mou and Yang [MY]. In [MY] the authors also obtain existence results for unstable solutions of higher-dimensional H -systems for a suitably restricted, constant H . As mentioned above, solutions to Ᏸ(H, ϕ), in general, fail to satisfy the conformality condition (1.2) and, hence, fail to represent surfaces of prescribed mean curvature. There are two reasons why one cannot expect (1.2) to hold for such solutions. The first, which is also true in dimension n = 2, is simply that the Dirichlet boundary

462

DUZAAR AND GROTOWSKI

condition is not invariant under the restriction to the boundary of an arbitrary inner variation (i.e., an orientation-preserving self-diffeomorphism of B). The second reason is more subtle and only occurs in dimension n ≥ 3. Even if one had boundary conditions that were invariant under all inner variations, for dimension n ≥ 3 it is far from clear that the second Euler equation for D (which can be derived in a manner analogous to dimension n = 2; see [DuF1, p. 212]) yields the conformality condition (1.2). In other words, in dimension n = 2 the conformality condition (1.2) is equivalent to the second Euler equation for D; in higher dimensions the former implies the latter, but equivalence is far from clear. In Section 6 we consider the regularity of the solutions whose existence is guaranteed by the theorems of Section 5. In the geometric measure theory setting for the Plateau problem ᏼ(H, ) discussed above, optimal regularity results were obtained by Duzaar [Du2] and by Duzaar and Steffen [DuS1], [DuS2]. Duzaar and Steffen established that the (energy-minimizing) solutions of ᏼ(H, ) are classical hypersurfaces smooth up to the boundary for n ≤ 6, and these solutions have a singular set that is closed, disjoint from the support of the boundary, and of Hausdorff dimension at most n − 7 for n ≥ 7. Due to our setting in this paper, we are able to obtain more satisfactory results (Theorem 6.1). In particular, our solutions to Ᏸ(H, ϕ) are Hölder continuous and are C 1,α under reasonable additional smoothness assumptions on H . We close this introduction with a few remarks on notation. We denote p-dimensional Lebesgue measure by ᏸp . The symbol αp is used to denote ᏸp (B p ), where B p is the unit ball in Rp . We denote by γp the optimal isoperimetric constant in Rp , that is, the smallest constant such that (cf. [Fe, 4.5.9 (31)]) M(Q) ≤ γp M(∂Q)p/(p−1)

(1.6)

holds for all integer-multiplicity rectifiable p-currents in Rp (note that γp = p −p/(p−1) −1/(p−1) αp ). We denote the standard volume form on Rn+1 by ". 2. The variational problem. We begin by giving a variational formulation of the H -system (1.1). We wish to consider, for u ∈ W 1,n (B, Rn+1 ), an energy of the form (2.1) EH (u) := D(u) + nVH (u) √  with D(u) = (1/ nn ) B |Du|n dx and VH a functional that is precisely specified in Section 3.4 below and that is seen to be a signed volume weighted by H , in an appropriate sense. For the moment, the only requirement we make of VH is that the following homotopy formula is valid:   VH (ut ) − VH (u) =

B 0

t

(H ◦ U )" ◦ U, Ut ∧ Ux1 ∧ · · · ∧ Uxn  dt dx

for variations U (t, x) = ut (x) of u(x) = u0 (x).

(2.2)

EXISTENCE AND REGULARITY FOR H -SYSTEMS

463

n+1 if u ∈ W 1,n (B, Rn+1 ) for sufA variation U is termed sufficiently regular in R t  d  ficiently small t, the initial velocity field ζ = ds s=0 us belongs to W 1,n (B, Rn+1 ) ∩ L∞ (B, Rn+1 ), and differentiation under the integral with respect to t is valid at t = 0 for D(ut ) and VH (ut ) − VH (u).

Lemma 2.1 (first variation). For sufficiently regular variations ut in W 1,n (B, Rn+1 ) with initial velocity field ζ in W 1,n (B, Rn+1 ) ∩ L∞ , we have    1 d  n−2 √ |Du| Du · Dζ + (H ◦ u)ζ · ux1 × ux2 × · · · × uxn dx.  EH (ut ) = n dt t=0 nn B Proof. Formal differentiation of D(ut ) yields the integrand √n n |Du|n−2 Du · Dζ , n and formal differentiation of (2.2) gives the integrand (H ◦ u)" ◦ u, ζ ∧ ux1 ∧ · · · ∧ uxn  = (H ◦ u)ζ · ux1 × · · · × uxn . This integral, denoted δ EH (u; ζ ), is termed the first variation of the energy EH in the direction ζ . As a direct consequence, we have the following corollary. Corollary 2.2. A map u ∈ W 1,n (B, Rn+1 ) is a weak solution of the H -surface equation if and only if δ EH (u; ζ ) = 0 for all vector fields ζ ∈ W01,n (B, Rn+1 ) ∩ L∞ . This means that the weak H -surface equation, that is,  √
Dxi |Du|n−2 Dxi u = nn (H ◦ u)ux1 × · · · × uxn

in B ,

(2.3)

is precisely the Euler equation associated to the energy functional EH . An important class of variations for our purposes are those of the form
 ut (x) = (Y tη(x), u(x)

(2.4)

for Y ∈ Cc1 (Rn+1 , Rn+1 ) a smooth vector field in Rn+1 , (Y the flow associated to Y , and η a sufficiently smooth function defined on B (generally η ∈ C 1 (B, R)). The initial field is then η(Y ◦ u) (cf. [Du1, Section 2], [DuS3, Lemma 1.3], and [DuS4, Section 2]). The following variational equality and inequality follow in direct analogy to the proof of [DuS4, Proposition 2.3(ii)]. Lemma 2.3. (i) Assume that u ∈ W 1,n (B, Rn+1 ) is EH -minimizing with respect to the variation ut given by (2.4) for each Y ∈ Cc1 (Rn+1 , Rn+1 ) and each η ∈ Cc1 (B, R). Then u is a solution to the weak H -surface equation (2.3). (ii) Let A ⊂ Rn+1 be the closure of a domain with C 2 -boundary. Suppose further that u is EH -minimizing for one-sided variations ut , 0 ≤ t  1, for η ≥ 0 and Y (a) = 0 or Y (a) directed strictly inwards at each a ∈ ∂A. Then u satisfies the inequality

464

DUZAAR AND GROTOWSKI

  δ EH (x; ζ ) = n

B

 1 n−2 √ n |Du| Du · Dζ + (H ◦ u)ζ · ux1 × · · · × uxn dx ≥ 0 (2.5) n

for all vector fields ζ ∈ W01,n (B, Rn+1 ) ∩ L∞ (B, Rn+1 ) with ζ · ( ν ◦ u) ≥ 0 almost everywhere on u−1 V for some neighbourhood V of ∂A in Rn+1 and some C 1 -extension  ν of the (inwardly pointing) unit normal vector field ν on ∂A to Rn+1 . Proposition 2.4. Let A ⊂ Rn+1 be the closure of a domain with C 2 -boundary, ν be the (inwardly pointing) unit normal on ∂A, and ᏷∂A (a) be the minimum of the principal curvatures of ∂A at the point a (with respect to ν). Let u ∈ W 1,n (B, A) satisfy the inequality (2.5). Then we have the following. (i) There exists a nonnegative Radon measure λ on B which is absolutely continuous with respect to ᏸn and which is concentrated on the coincidence set u−1 ∂A, such that  δEH (u; ζ ) = ζ · (ν ◦ u) dλ (2.6) u−1 ∂A

for each ζ ∈ W01,n (B, Rn+1 ) ∩ L∞ (B, Rn+1 ). (ii) If |H | ≤ ᏷∂A on ∂A, we have λ = 0; more generally, 
n λ ≤ ᏸn
√ |Du|n |H ◦ u| − ᏷∂A ◦ u + n n

on u−1 ∂A.

(2.7)

(iii) If |H (a)| < ᏷∂A (a) for some a ∈ ∂A and if u|∂B omits some neighbourhood of a, then there exists a neighbourhood V of a in Rn+1 such that u(B) ∩ V = ∅. Proof. We write d(p) = dist(p, ∂A) for p ∈ Rn+1 , and we extend the (inwardly pointing) unit normal vector field ν to a C 1 -vector field, again denoted by ν, such that ν coincides with grad d on a neighbourhood of ∂A. We first consider the case where A is compact. In this case ζ = η(ν ◦u) is admissible in (2.5) if 0 ≤ η ∈ Cc1 (B, R). Applying the Riesz representation theorem, we deduce the existence of a nonnegative Radon measure λ on B such that 
 δ EH u, η(ν ◦ u) = η dλ (2.8) B

holds for all η ∈ Cc1 (B, R). We now choose ϑ ∈ C ∞ (R, R) nonincreasing with ϑ ≡ 1 on (−∞, 1/2] and ϑ ≡ 0 on (1, ∞), and we define ϑε (t) = ϑ(t/ε) for ε > 0. We consider ζε = η(ϑε ◦ d ◦ u)(ν ◦ u) with η ≥ 0 as before. Then ζ = ζε on the preimage under u of a neighbourhood of ∂A, so that ζ − ζε and ζε − ζ are both admissible in the variational inequality. This means δ EH (u; ζε ) = δEH (u; ζ ) ≥ 0.

(2.9)

EXISTENCE AND REGULARITY FOR H -SYSTEMS

465

For ε sufficiently small we estimate


uxi · (ζε )xi ≤ (ϑε ◦ d ◦ u) ηxi uxi · (ν ◦ u) + ηuxi · (Dν) ◦ u uxi . Applying this in (2.5), noting that uxi · (ν ◦ u) = 0 almost everywhere on u−1 ∂A, and letting ε approach zero, we have  
 1 1 0 ≤ δ EH (u; ζ ) ≤ √ |Du|n−2 uxi · (Dν) ◦ u uxi n −1 n n u ∂A  + (H ◦ u)(ν ◦ u) · ux1 × · · · × uxn η dx. Since uxi · ((Dν) ◦ u uxi ) = −b∂A ◦ u(uxi , uxi ) almost everywhere on u−1 ∂A, where b∂A denotes the second fundamental form of ∂A in Rn+1 relative to the outwardly pointing normal on ∂A, we have    n 
 1 1 n−2 2 δ EH (u, ζ ) ≤ b∂A ◦ u uxi , uxi η dx |H ◦ u| |Du| − √ |Du| n u−1 ∂A nn i=1  
1 ≤ √ |Du|n |H ◦ u| − ᏷∂A ◦ u η dx. n −1 u ∂A n Combining this with (2.9) and (2.8) shows   
n η dλ ≤ √ |Du|n |H ◦ u| − ᏷∂A ◦ u η dx, n n u−1 ∂A B which yields the claimed estimate on the Radon measure λ, that is, 
n λ ≤ ᏸn
√ |Du|n |H ◦ u| − ᏷∂A ◦ u + nn

on u−1 ∂A.

This completes the proof of (ii). To show (i) we begin by noting that (ii) immediately yields the absolute continuity of λ with respect to ᏸn and, further, that λ(B \ u−1 ∂A) = 0. It is easy to see by approximation that (2.8) holds for all η ∈ W01,n (B, R)∩L∞ (B, Rn+1 ). In the case of a general vector field ζ ∈ W01,n (B, Rn+1 )∩L∞ (B, Rn+1 ), we decompose ζ = ζ ⊥ +ζ  , where ζ ⊥ = η(ν ◦ u) with η = ζ · (ν ◦ u) ∈ W01,n (B, Rn+1 ) ∩ L∞ (B, R). We apply (2.8) to conclude 

 ⊥ ζ · (ν ◦ u) dλ. (2.10) δ EH u; ζ = δ EH u, (ζ · ν ◦ u)ν ◦ u = u−1 ∂A

Further we have that ζ  · (ν ◦ u) = 0 almost everywhere on the preimage of a neighbourhood of ∂A under u (i.e., ζ  and −ζ  are both admissible in (2.5)), and hence δ EH (u; ζ  ) = 0. Combining this with (2.10), we have shown (i).

466

DUZAAR AND GROTOWSKI

In the case of arbitrary A, one replaces ν ◦u in the above discussion by (ψk ◦u)(ν ◦u) with ψk ∈ Cc1 (Rn+1 , [0, 1]), such that the ψk ’s tend to the identity on Rn+1 . One then argues directly and analogously to the case n = 2 (see [DuS4, Proposition 2.4]) to show that the associated Radon measures λk approach a limit measure λ, which satisfies (i) and (ii). In the same way, (iii) can be proven by direct analogy with the case n = 2. We refer the reader to [DuS4, Proposition 2.4]. Remark 2.5. If we assume that u is a conformal solution of the variational inequality (i.e., (1.3) holds), then ᏷∂A can be replaced by the mean curvature H∂A in the assumption. 3. The volume functional. Given u ∈ W 1,n (B, Rn+1 ) we can define the associated n-current Ju in Rn+1 via integration of n-forms over u, that is,  


 # Ju (β) = u β = β ◦ u, ux1 ∧ · · · ∧ uxn dx for β ∈ Ᏸn Rn+1 . (3.1) B

B

Here Ᏸk (Rn+1 ) denotes the space of smooth, compactly supported k-forms on Rn+1 . It is straightforward to see that Ju is an n-current of finite mass (where the mass of a k-current T on Rn+1 is defined by M(T ) := sup {T (β) : β ∈ Ᏸk (Rn+1 ), β∞ ≤ 1}), since   1 |Du|n dx = D(u). (3.2) M(Ju ) ≤ |ux1 ∧ · · · ∧ uxn | dx ≤ √ n n B B Using a Lusin-type approximation argument for mappings in W 1,n (cf. [EG, 6.6.3]) we can argue similarly for the case n = 2 (cf. [DuS4, Section 3]) to see that Ju is a (locally) rectifiable n-current in Rn+1 . If v is another surface in W 1,n (B, Rn+1 ), then (Ju − Jv )(β) is determined by integration of u# β − v # β over G = {x ∈ B : u(x) = v(x)}, as Du and Dv coincide ᏸn –almost everywhere on B \ G. Thus we can refine (3.2) to 
M Ju − Jv ≤ DG (u) + DG (v) if u = v on B \ G,

(3.3)

where 1 DU (u) = √ nn

 U

|Du|n dx

(3.4)

for ᏸn -measurable U ⊂ B. In general the boundary ∂T of a k-current T , k ≥ 1, is defined by ∂T (α) = T (dα) for α ∈ Ᏸk−1 (Rn+1 ). For u, v ∈ W 1,n (B, Rn+1 ) with u − v ∈ W01,n (B, Rn+1 ) we calculate directly that Ju − Jv is a closed n-current, that is, ∂(Ju − Jv ) = 0. First we

EXISTENCE AND REGULARITY FOR H -SYSTEMS

467

see that for u, v ∈ C 2 (B, Rn+1 ) with u = v on ∂B we have  
 # ∂Ju (α) = Ju (dα) = u dα = d u# α B  B # # = u α= v α = ∂Jv (α). ∂B

∂B

In the general case we approximate u by ui ∈ C 2 (B, Rn+1 ) and v − u by wi ∈ Cc∞ (B, Rn+1 ), the approximations being in the W 1,n -norm. We see that ui + wi approaches v in W 1,n , and since ui = ui + wi on ∂B, we have ∂(Jui − Jui +wi ) = 0. Letting i tend to infinity, we see ∂(Ju − Jv ) = 0, which is the desired conclusion. In the following we take A to be a closed subset of Rn+1 —the obstacle—and u0 ∈ W 1,n (B, A) to be a fixed reference surface. We let   1,n
᏿(u0 , A) = u ∈ W 1,n (B, A) : u − u0 ∈ W0 B, Rn+1 (3.5) denote the class of admissible surfaces. The idea behind the geometric definition of the H -volume VH (u, v) enclosed by two surfaces u, v ∈ ᏿(u0 , A) is to consider an (n + 1)-current Q in Rn+1 with ∂Q = Ju − Jv and to integrate H " over Q. Such currents have a relatively simple structure; they are representable by an L1 (Rn+1 , Z)function iQ , such that for all γ ∈ Ᏸn+1 (Rn+1 ) there holds  iQ γ . Q(γ ) = Rn+1

One can consider iQ to be a set with integer multiplicities and finite absolute volume. In this context the condition ∂Q = Ju − Jv means that u and v parameterize the boundary of this set with multiplicities in the dual sense of Stokes’s theorem, that is,   
 # iQ dβ = u β − v # β for all β ∈ Ᏸn Rn+1 . Rn+1

B

B

Since ∂Q is finite we can conclude that iQ is a BV -function on Rn+1 , which is a strong motivation for defining the H -volume by  VH (u, v) = iQ H ". (3.6) Rn+1

In order to make this a well-defined functional, we need to clarify the questions of existence and uniqueness for Q. One could try to finesse the question of existence by considering the variational problem restricted to those u ∈ ᏿(u0 , A) for which Ju − Ju0 is homologically trivial in A; that is, Ju − Ju0 is the boundary of an (n + 1)current Q with support in A. However, simple examples show that such a homological property is not preserved a priori under passage to a weak limit; see [DuS4, Section 1]. It is thus reasonable to impose the restriction that Ju − Jv be homologically trivial in A for all u, v ∈ ᏿(u0 , A). This amounts to the condition that certain n-currents are boundaries in A, as made precise in the following definition.

468

DUZAAR AND GROTOWSKI

Definition 3.1. An n-current T on Rn+1 with support in A is called: (i) spherical in A when it can be written in the form T = f# [[S n ]] for a map f ∈ W 1,n (S n , A), that is, 
 T (β) = f # β for β ∈ Ᏸn Rn+1 ; Sn

(ii) homologically trivial in A when it is the boundary of a rectifiable (n + 1)current with support in A. If (ii) holds for every spherical n-current with support in A, we say that A is homologically n-aspherical in Rn+1 . If T = f# [[S n ]] is homologically trivial in A, then there is an (n + 1)-current Q in Rn+1 with ∂Q = T , M(Q) < ∞, and spt Q ⊂ A. By the constancy theorem [Fe, 4.1.7 and 4.1.31], we have that Q is uniquely determined up to real multiples of [[Rn+1 ]]; that is, Q is unique. Further, it follows from the general theory of rectifiable currents [Fe, Chapter 4] that we can take Q to be an integer-multiplicity rectifiable current. The following lemma shows that, under mild regularity assumptions on A, every spherical n-current T in A can be approximated by smooth maps from S n to A and that if the approximating maps are all homologically trivial (when viewed as spherical n-currents), then so is T . Lemma 3.2. Let A be a uniform Lipschitz (respectively, C 1 ) neighbourhood retract in Rn+1 , and let f ∈ W 1,n (S n , A). (i) Given ε > 0 there exists g ∈ W 1,n (S n , A) such that g − f W 1,n < ε, g = f outside a subset of S n of measure less than ε, and g is Lipschitz continuous (respectively, C 1 ). (ii) For given s and r with 0 < s ≤ ∞, 0 < r < ∞, let M(f# [[S n ]]) < s, and let g# [[S n ]] be the boundary of a rectifiable (n + 1)-current with mass not greater than r and with support in A for all Lipschitz continuous (respectively, C 1 ) g : S n → A with M(g# [[S n ]]) < s. Then f# [[S n ]] is homologically trivial in A. Proof. (i) By following the proof of [EG, Theorem 6.6.3, Step 2], we can find, for a given λ > 0, Lipschitz maps gλ : S n → Rn+1 , such that gλ −f W 1,n → 0 as λ → ∞ and gλ = f outside a set Eλ ⊂ S n with λn |Eλ | → 0 as λ → ∞. Further, from Step 4 of the same proof we see that Lip(gλ ) ≤ Cλ for C depending √only on n. An elementary calculation shows that, for |Eλ | < |S n |, no ball of radius π n |Eλ |/|S n | can be enclosed in Eλ . Hence,√given w ∈ Eλ we can find w ! ∈ S n \ Eλ with gλ (w ! ) = f (w ! ), and |w − w ! | ≤ π n |Eλ |/|S n |. We thus have  |Eλ | ! |gλ (w) − gλ (w )| ≤ Cλπ n n . |S | Since limλ→∞ λn |Eλ | = 0, we see that, for λ sufficiently large, gλ (S n ) is contained in a uniform neighbourhood Vρ (A) that admits a Lipschitz retraction π : Vρ (A) → A.

EXISTENCE AND REGULARITY FOR H -SYSTEMS

469

We set g = π ◦ gλ for such λ. Then g ∈ Lip(S n , A), g = f on S n \ Eλ , and g − f W 1,n ,S n ≤ gW 1,n ,Eλ + f W 1,n ,Eλ . The last term vanishes as λ → ∞ (due to absolute continuity of the integral and since |Eλ | → 0). Further, we have (again, as λ → ∞) DgnLn ,Eλ ≤ (Lip g)n |Eλ | ≤ (Lip π)n C n λn |Eλ | −→ 0 and gLn ,Eλ ≤ π ◦ gλ − π ◦ f Ln ,Eλ + f Ln ,Eλ

≤ (Lip π)n gλ − f Ln ,Eλ + f Ln ,Eλ ,

which also converge to 0 as λ tends to ∞. Hence, for λ sufficiently large we have |Eλ | < ε and also g − f W 1,n ,S n < ε, which completes the proof in the Lipschitz case. In the C 1 -case we can argue completely analogously to the situation for n = 2 (see [DuS4, Lemma 3.2]). (ii) Consider f ∈ W 1,n (S n , A) with M(f# [[S n ]]) < s. Then, given ε = 1/k, there exist Lipschitz maps gk : S n → A with gk = f on S n \ Ek , |Ek | < 1/k, and f − gk W 1,n ,S n < 1/k. The strong convergence of gk to f means, in particular, that M(gk# [[S n ]]) → M(f# [[S n ]]) as k → ∞; that is, M(gk# [[S n ]]) < s for k sufficiently large. The assumptions then guarantee the existence of rectifiable (n + 1)currents Qk with support in A, mass not greater than r, and ∂Qk = gk# [[S n ]]. The BV -compactness theorem (see, e.g., [EG, Theorem 5.2.4]) then ensures (after passage to a subsequence) the existence of a rectifiable (n + 1)-current Q such that Qk → Q (weakly). The lower-semicontinuity of M then implies M(Q) ≤ r, and further ∂Q = limk→∞ ∂Qk = limk→∞ gk# [[S n ]] = f# [[S n ]]. (The last step is due to the strong convergence of gk to f .) Corollary 3.3. For all u, v ∈ W 1,n (B, A) with u − v ∈ W01,n (B, Rn+1 ), Ju − Jv is a spherical n-current. Proof. We compose u with stereographic projection from the south pole of S n and v with that from the north pole in order to obtain a map f ∈ W 1,n (S n , A) with f# [[S n ]] = Ju − Jv . Definition 3.4. Let u, v ∈ W 1,n (B, A) with u − v ∈ W01,n (B, Rn+1 ). If Ju − Jv is homologically trivial in A, we define the H -volume enclosed by u and v by  iu,v H ". VH (u, v) = Iu,v (H ") = Rn+1

Here Iu,v is the (unique) rectifiable (n + 1)-current Q in Rn+1 which is associated to the n-current T = Ju − Jv (i.e., spt Q ⊂ A, M(Q) < ∞, and ∂Q = T ), and iu,v denotes the multiplicity function of Iu,v .

470

DUZAAR AND GROTOWSKI

We now need to show that the H -volume has the properties that we require in order to be able to apply the results of Section 2 concerning our variational equalities and inequalities. This is accomplished in the following lemma. Lemma 3.5. Let u, v ∈ W 1,n (B, A) be as in Definition 3.4, so that VH (u, v) is defined. (i) Assume that A ⊂ Rn+1 has a uniform Lipschitz neighbourhood retraction π,  u ∈ W 1,n (B, A), u − u ∈ W01,n (B, Rn+1 ), and u − uL∞ is smaller than a certain positive constant that only depends on A. Then VH ( u, v) and VH ( u, u) are also well defined and satisfy

 VH  u, u + VH (u, v) = VH  u, v , 



VH  uL∞ (Lip π)n+1 DG (u) + DG  u, u  ≤ sup |H | u − u , Rn+1

where G = {x ∈ B :  u(x) = u(x)}. (ii) Let (Yt be the flow of a vector field Y ∈ Cc1 (Rn+1 , Rn+1 ) with (Yt (A) ⊂ A for small t > 0, 0 ≤ η ∈ Cc1 (B, R), and ut (x) = U (t, x), where U (s, x) = (Y (sη(x), u(x)). Then VH (ut , v) and VH (ut , u) are defined for sufficiently small t > 0, and we have VH (ut , v) − VH (u, v) = VH (ut , u)   t

(H ◦ U ) " ◦ U, Us ∧ Ux1 ∧ · · · ∧ Uxn ds dx. = B 0

Proof. (i) Using the affine homotopy U (s, x) = (1−s)u(x)+s u(x), we can define the (n + 1)-current Q in Rn+1 by   1

Q(γ ) = γ ◦ U, Us ∧ Ux1 ∧ · · · ∧ Uxn ds dx (3.7) B 0

Ᏸn+1 (Rn+1 ).

The homotopy formula (see [Fe, 4.1.9]) and the constraint for γ ∈ 1,n n+1 u − u ∈ W0 (B, R ) then imply ∂Q = Ju˜ − Ju . From (3.7) we see


1 uL∞ DG (u) + DG  u . M(Q) = u − 2 For u −  uL∞ sufficiently small, π# Q is thus an integer-multiplicity rectifiable (n + 1)-current with support in A, boundary ∂(π# Q) = π# ∂Q = Ju˜ − Ju , and mass M(π# Q) ≤ (Lip π)n+1 M(Q),

(3.8)

which allows us to conclude π# Q = Iu,u ˜ and Iu,v ˜ = Iu,u ˜ + Iu,v . This means that the H -volume satisfies the identity

 VH  u, v − VH (u, v) = π# Q(H ") = VH  u, u .

EXISTENCE AND REGULARITY FOR H -SYSTEMS

471

The conclusions of (i) now follow from (3.7) and (3.8) after approximating H " by smooth γ ∈ Ᏸn+1 (Rn+1 ) with |γ | ≤ |H |. The proof of part (ii) involves only minor modifications of the case n = 2. We omit the details and refer the reader to [DuS4, Lemma 3.6(ii)]. Part (ii) of the above lemma shows that the homotopy formula (2.2) is valid for the variations considered in (ii) for the H -volume as defined by VH (u) = VH (u, u0 ), where u0 ∈ W 1,n (B, A) is a fixed reference surface and u and u0 satisfy the conditions of Definition 3.4. Thus all the conclusions of Section 2 are valid for the H -volume as defined in (3.6). 4. A general existence theorem. In this section we apply the direct method of the calculus of variations to prove a general existence theorem for weak solutions of the Dirichlet problem Ᏸ(H, u0 ). We minimize the energy functional EH (u) = D(u) + nVH (u, u0 ) in a suitable subclass of ᏿(u0 , A). The n–Dirichlet integral D(·) is lower-semicontinuous in the topology of weak convergence for ᏿(u0 , A) in W 1,n (B, Rn+1 ); however, the H -volume VH ( · , u0 ) is not. This is because a sequence {ui } in ᏿(u0 , A) converging weakly to u may involve a large part of the volume and the surface area of ui being parameterized over a small subset of B in such a manner that the ᏸn -measure converges to 0 as i → ∞. Geometrically this can be viewed as the bubbling off of a certain amount of the volume and the surface area in the limit. This bubbling phenomenon also means that the homology type is not preserved a priori in the weak limit. The following lemma (cf. [DuS4, Lemma 4.1] in the 2-dimensional case) gives an analytical description of the bubbling. Lemma 4.1. Suppose that ui ? u weakly in W 1,n (B, Rm ) and ui |∂B → u|∂B uniformly in L∞ (∂B, Rm ). Then, given ε > 0, there exist R > 0, a measurable set G, G ⊂ B, and maps  ui ∈ W 1,n (B, Rm ), such that after passage to a subsequence: (i)  ui = u on B \ G with ᏸn (G) < ε; (ii)  ui |∂B = u|∂B ; (iii)  ui (x) = ui (x) if |ui (x)| ≥ R or |ui (x) − u(x)| ≥ 1; (iv) limi→∞  ui − ui L∞ (B,Rm ) = 0; (v)  ui ? u weakly in W 1,n (B, Rm ) as i → ∞; (vi) lim supi→∞ [DG ( ui ) + DG (u)] ≤ ε + lim inf i→∞ [D(ui ) − D(u)]; (vii) if the ui assume values in a closed subset A of Rm which admits neighbourhood retractions that have Lipschitz constant arbitrarily close to 1 on neighbourhoods of compact subsets, then the x˜n can also be chosen to have values in A. Proof. Using Rellich’s theorem and Egoroff’s theorem in turn, we can find R > 3, 1/2 ≥ δn ↓ 0, and G ⊂ B measurable with ᏸn (G) < ε and DG (u) < ε ! (ε ! is determined later), such that after passage to a subsequence, we have  u|∂B L∞ ≤ (1/3)R, supB\G |u| ≤ (1/3)R, supB\G |ui − u| ≤ δi , and  ui |∂B − u|∂B L∞ ≤ δi . We choose

472

DUZAAR AND GROTOWSKI

η ∈ C 1 (R) with η = 1 on (−∞, (1/3)R], η = 0 on [(2/3)R, ∞), and 0 ≤ −η! ≤ 4/R on R, and we define ϑi by ϑi (t) = 1 for t ≤ δi , ϑi (t) = ((1/t) − 1) /((1/δi ) − 1) for δi ≤ t ≤ 1 and ϑi (t) = 0 for t ≥ 1. We further define


  ui = ui + η ◦ |u| ϑi ◦ |ui − u| u − ui . (4.1) Note that ϑi ◦ |ui − u| and η ◦ |u| both take the value 1 on ∂B. Parts (i) and (ii) then follow directly, due to our choices of G, η, and ϑi . We note that if |ui (x)| ≥ R, then |ui (x) − u(x)| ≥ R/3 > 1 or |u(x)| ≥ (2/3)R. For |ui (x) − u(x)| ≥ R/3 > 1, the definition of ϑi ensures ϑi (|ui (x) − u(x)|) = 0. If |u(x)| ≥ (2/3)R, we have η(|u(x)|) = 0. These combine to show (iii). Since 0 ≤ η ≤ 1 and supt≥0 ϑi (t)t ≤ δi → 0 as i → ∞, we have also established (iv). In order to show (vi) we differentiate (4.1) to obtain




D ui = Dui + η ◦ |u| D ϑi ◦ |ui − u| u − ui   (4.2)
 u 

! · Du ϑi ◦ |ui − u| u − ui + η ◦ |u| |u| (with the interpretation u/|u| · Du = 0 for u = 0). Using the identity tϑi! (t) + ϑi (t) = −δi /(1 − δi ) for t > δi , we have

 D ui = 1 − η ◦ |u| Dui + η ◦ |u| Du   (4.3)
  u
· Du u − ui if |u − ui | ≤ δi , + η! ◦ |u| |u| and

  

 ⊥
 δi P Dui D ui = 1 − η ◦ |u| ϑi ◦ |ui − u| P Dui + 1 + η ◦ |u| 1 − δi


 δi + η ◦ |u| ϑi ◦ |ui − u| P ⊥ Du − η ◦ |u| P Du 1 − δi  

 u 
 + η! ◦ |u| · Du ϑi ◦ |ui − u| u − ui if |u − ui | > δi , |u|

(4.4)

where P denotes the field of rank-1 orthogonal projections  


P : Rm $ ξ −→ |ui − u|−2 ui − u · ξ ui − u (with P ⊥ = id −P ). For almost all x ∈ G with |ui (x)−u(x)| ≤ δi we therefore have, via (4.3), |D ui | ≤ |Dui | + |Du| +

4 δi |Du|, R

EXISTENCE AND REGULARITY FOR H -SYSTEMS

473

and, via (4.4) for |ui (x) − u(x)| > δi , we have  1/2 1 δi 4 2 |D ui | ≤ |P ⊥ Dui |2 + |P Du | + |P ⊥ Du| + |P Du| + δi |Du|; i 2 1 − δi R (1 − δi ) that is, we have (almost everywhere on G) |D ui | ≤

1 |Dui | + 2|Du|. 1 − δi

After applying Young’s inequality, for λ > 0 we have  
 4n 1 DG (u). + λ D (u ) + ui ≤ DG  G i (1 − δi )n λ

(4.5)

Letting i → ∞ and noting δi → 0, this becomes

 

4n ui ) + DG (u) ≤ (1 + λ) lim sup DG (ui ) − DG (u) + 2 + DG (u) lim sup DG ( λ i→∞ i→∞  

4n ≤ (1 + λ) lim sup D(ui ) − D(u) + 2 + DG (u). λ i→∞ (4.6)





In the last inequality, we use the fact that lim supi→∞ DB\G (ui ) ≥ DB\G (u) (note ui ? u in W 1,n (B, Rm )). We now fix λ > 0, such that λ supi D(ui ) ≤ (1/2)ε, and then we fix ε! such that DG (u) < ε ! and ((2 + (4n /λ))) ε! < (1/2)ε. Part (vi) then follows from (4.6) after passing to a subsequence such that we can replace lim sup by lim inf in (4.6). From (vi) we have supi DG ( ui ) < ∞. Furthermore (cf. part (i)),  ui = u on B \ G, that is, supi D( ui ) < ∞. Combining this with the weak convergence of ui to u and with part (vi), this shows (v). To see (vii), we apply the above construction with (1/2)ε in place of ε. Then  ui (x) = ui (x) ∈ A if |ui (x)| ≥ R. Further, by (iv) we have  ui − ui L∞ (B,Rm ) = δi → 0 as i → ∞, so  ui (x) lies either in A or in a uniform δi -tubular neighbourhood of {a ∈ A : |a| ≤ R}, which we denote by Uδi . Given this, we can find a Lipschitz neighbourhood retraction π : V → A such that Uδi ⊂ V and Lip(π|Uδi ) is arbitrarily close to 1 for i sufficiently large. Then (i)–(vi) also follow if we replace  u by π ◦ ui . We can interpret lim inf i→∞ [D(ui )−D(u)] as the n–Dirichlet integral of the bubble that separates under the passage to the weak limit ui ? u. In order to establish lower semicontinuity for the energy functional EH (u) = D(u) + nVH (u, u0 ) with respect to weak convergence in ᏿(u0 , A), we need to control the H –volume jump lim supi→∞ n |VH (ui , u0 ) − VH (u, u0 )|.This is accomplished by passing from ui to  ui and by using a suitable isoperimetric condition, which is defined below. We first recall the standard definition of an (unrestricted) isoperimetric condition (cf. [St1, (3.7)] and [DuS3, Definition 3.1]).

474

DUZAAR AND GROTOWSKI

Definition 4.2. Consider 0 < s ≤ ∞, 0 ≤ c < ∞, and A ⊂ Rn+1 . (1) An (unrestricted) isoperimetric condition of type c, s is valid for H and A if     n|Q, H "| = n  iQ H " ≤ c M(∂Q) (4.7) A

for all integer-multiplicity rectifiable (n + 1)-currents Q with spt Q ⊂ A and with M(∂Q) ≤ s. Here iQ is the multiplicity function of Q. (2) Suppose that every spherical n-current T with support in A and M(T ) ≤ s is uniquely homologically trivial in A, that is, there exists an integer-multiplicity rectifiable (n + 1)-current with spt Q ⊂ A, M(Q) < ∞, and ∂Q = T . We say that H satisfies a spherical isoperimetric condition of type c, s on A if we have      n|Q, H "| = n  iQ H " ≤ c M(T ) (4.8) A

for all T , Q as above. Remark 4.3. (1) If A = Rn+1 (or, more generally, A is homologically n-aspherical), then an unrestricted isoperimetric inequality of type c, s implies a spherical isoperimetric condition of type c, s. (2) If H satisfies a spherical isoperimetric condition of type c, s on A, we can conclude from Lemma 3.3(ii) and Definition 3.4 that the H -volume VH (u, v) is defined for all u, v ∈ W 1,n (B, A) with u − v ∈ W01,n (B, Rn+1 ). Further, we conclude that we have the estimate
 n|VH (u, v)| ≤ c M Ju − Jv . (4.9) In the following theorem we apply this isoperimetric condition to obtain existence results. Theorem 4.4. Let A be a closed subset of Rn+1 which admits neighbourhood retractions that have Lipschitz constant arbitrarily close to 1 on neighbourhoods of compact subsets; let H : A → R be a bounded, continuous function that satisfies a spherical isoperimetric condition of type c, s; and let u0 ∈ W 1,n (B, A) be a fixed reference surface for which the inequality (1 + σ )D(u0 ) ≤ s holds for some 1 < σ ≤ ∞. Further, let ᏿(u0 , A; σ ) denote the class of all surfaces  u ∈ ᏿(u0 , A) with D( u ) ≤ σ D(u0 ). Then we have the following. (i) If σ < ∞ and c ≤ 1 or if σ = ∞ and c < 1, then the variational problem


 EH  (4.10) u =D  u + nVH  u, u0 −→ min in ᏿(u0 , A; σ ) has a solution. (ii) If c≤

σ −1 σ +1

(respectively, c < 1, if σ = ∞),

(4.11)

EXISTENCE AND REGULARITY FOR H -SYSTEMS

475

then the variational problem (4.10) has a solution v with D(v) < σ D(u0 ). If the inequality (4.11) is strict or if u0 is itself not a solution to (4.10), then D(u) < σ D(u0 ) for every solution u to (4.10). (iii) If A is the closure of a C 2 -domain in Rn+1 and |H | ≤ ᏷∂A

pointwise on ∂A,

(4.12)

then every minimum u of (4.10) with D(u) < σ D(u0 ) is a weak solution of the Dirichlet problem Ᏸ(H, u0 ) in A. If, in addition, |H (a)| < ᏷∂A (a) in a given point a ∈ ∂A and u0 |∂B omits some neighbourhood of a, then there exists a neighbourhood V of a in Rn+1 such that u(B) ∩ V = ∅. Proof. (i) From (3.3), for  u ∈ ᏿(u0 , A; σ ) we have 



u + D u0 ≤ (σ + 1)D u0 ≤ s, M Ju˜ − Ju0 ≤ D 

(4.13)

u, u0 ) is defined for all  u ∈ ᏿(u0 , A; σ ). Using (4.9) and (4.13), we have so that VH ( 


 

 u ≥D  u − nVH  u, u0  ≥ (1 − c)D  u − cD u0 ; (4.14) EH  that is, EH is bounded from below on ᏿(u0 , A; σ ). We now choose a minimizing sequence (ui )i∈N for (4.10), and we note that (4.14) implies that supi D(ui ) < ∞ if σ = ∞ and c < 1. For finite σ this follows directly from the definition of ᏿(u0 , A; σ ). After passing to a subsequence, we can assume that ui converges to a map u ∈ ᏿(u0 , A; σ ) weakly in W 1,n and pointwise almost everywhere. For given ε > 0 we apply Lemma 4.1 and obtain, after passage to a subsequence, surfaces  ui ∈ ᏿(u0 , A) with limi→∞  ui − ui L∞ (B,Rn+1 ) = 0. From Lemma 3.5(i), we thus have that VH ( ui , u0 ) and VH ( ui , ui ) are well defined, and furthermore we have


 ui , u 0 − V H ui , u 0 = V H  ui , ui −→ 0 as i −→ ∞. (4.15) VH  (The proof of Lemma 3.5(i) shows that we do not need to assume that A admits uniform Lipschitz neighbourhood retractions, since in the current situation, from Lemma 4.1(iii) we have  ui (x) = ui (x) for |ui (x)| ≥ R.) Choosing ε < (1/2)D(u), via (3.3) and Lemma 4.1(vi) we obtain
 


ui + DG (u) ≤ 2ε + D ui − D(u) < σ D u0 ≤ s (4.16) M Ju˜ i − Ju ≤ DG  for i sufficiently large (for G ⊂ B given by Lemma 4.1). Thus we conclude, from the spherical isoperimetric condition (note c ≤ 1), Remark 4.3, and (4.16), the inequality 


 ui , u  ≤ cM Ju˜ i − Ju ≤ 2ε + D ui − D(u) (4.17) nVH  for i sufficiently large. Next we wish to show 


 ui , u 0 = V H  ui , u + VH u, u0 . VH 

(4.18)

476

DUZAAR AND GROTOWSKI

To see this, note that (4.17) guarantees the existence of VH ( ui , u), (4.15) ensures that VH ( ui , u0 ) is well defined, and the existence of VH (u, u0 ) is guaranteed by the fact that u ∈ ᏿(u0 , A; σ ). Therefore, we have the existence of rectifiable (n + 1)-currents Iu˜ i ,u , Iu˜ i ,u0 , and Iu,u0 with support in A, all of which are uniquely determined by their boundaries Ju˜ i − Ju , Ju˜ i − Ju0 , and Ju − Ju0 . Thus we have Iu˜ i ,u0 = Iu˜ i ,u + Iu,u0 , since the currents on both sides have the same boundary. This shows (4.18). Using (4.15), (4.18), and (4.17), we have, for i sufficiently large,


 EH ui = D ui + nVH ui , u0


 = EH (u) − D(u) + D ui + nVH ui , u0 − nVH u, u0
 

 ui , u − nVH  ui , u i = EH (u) + D ui − D(u) + nVH 
 ui , u i ≥ EH (u) − 2ε − nVH  ≥ EH (u) − 3ε. This shows that u minimizes the H -energy in the class ᏿(u0 , A; σ ). To see (ii), we note that EH (u) ≤ EH (u0 ) for solutions of (4.10). Hence we have
 D(u) = EH (u) − nVH u, u0

 ≤ EH u0 − nVH u, u0

 = D u0 − nVH u, u0



 ≤ D u0 + c D(u) + D u0


 ≤ 1 + c(1 + σ ) D u0
 ≤ σ D u0 , where we have used, in turn, inequality (3.3), the fact that VH (u0 , u0 ) = 0, the isoperimetric condition, and inequality (4.11). The strict inequality D(u) < σ D(u0 ) occurs in the following situations: when σ = ∞; when c < (σ +1)/(σ −1) if σ < ∞; or in the case where u0 is not a solution of (4.10), that is, E(u) < E(u0 ). On the other hand, if u0 solves (4.10), then D(u0 ) < σ D(u0 ) since σ > 1. Part (iii) follows from Lemma 3.5(ii) in light of the results of Section 2. Remark 4.5. (1) In the case A = Rn+1 , it is not, in fact, necessary to assume that the integer-multiplicity rectifiable (n+1)-currents Iu,u ˜ 0 occurring in the proof of Theorem 4.4 have support in A. As long as we have that H is bounded and ᏸn+1 measurable on some closed set A˜ ⊃ A, we can weaken Definition 4.2(ii) by allowing ˜ (That is, we only need to require that T is uniquely homologically trivial spt Q ⊂ A. ˜ in A.) (2) A natural choice of reference surface u0 is a minimizer of the n–Dirichlet integral relative to given boundary data, that is, D(u0 ) ≤ D( u ) for all  u ∈ ᏿(u0 , A).

EXISTENCE AND REGULARITY FOR H -SYSTEMS

477

The existence of such a minimizer is guaranteed, for example, if we consider Dirichlet boundary data γ ∈ C 0 (∂B, A), which admits an extension in W 1,n (B, A). The  above proof then goes through if we use ᏿(γ , A) =  u ∈ W 1,n (B, A) :  u|∂B = γ = ∅ in place of ᏿(u0 , A), and if we use ᏿(γ , A; σ ) = { u ∈ ᏿(γ , A) : D( u ) ≤ σ D(u0 )}, where u0 minimizes the n–Dirichlet integral in ᏿(γ , A), in place of ᏿(u0 , A; σ ). 5. Geometric conditions sufficient for existence. In this section we combine the results of [DuS3] concerning isoperimetric inequalities with Theorem 4.4 to obtain conditions on the Dirichlet boundary data ϕ ∈ C 0 (∂B, A) and on the prescribed mean curvature H which are sufficient to ensure the existence of a (weak) solution of the Dirichlet problem Ᏸ(H, ϕ). The first result is a Wente-type theorem. We consider Dirichlet boundary data ϕ ∈ C 0 (∂B, A) that admits a W 1,n (B, A)-extension, and we denote by u0 ∈ W 1,n (B, A) the D-minimizing map with u0 |∂B = ϕ and set dϕ = D(u0 ). Theorem 5.1. Let A be the closure of a C 2 -domain in Rn+1 such that the minimum of the principal curvatures ᏷∂A (viewed with regard to the inward-pointing normal) is positive at every point a ∈ ∂A. Further, consider Dirichlet boundary data ϕ ∈ C 0 (∂B, A) as above, and consider H : A → R, bounded and continuous, satisfying  αn+1 sup |H | ≤ n (5.1) 2dϕ A and |H (a)| ≤ ᏷∂A (a)

for a ∈ ∂A.

(5.2)

Then there exists a weak solution u ∈ W 1,n (B, A) to the Dirichlet problem Ᏸ(H, ϕ); that is,
 Dxi |Du|n−2 Dxi u = H ◦ u · ux1 × · · · × uxn in B, u|∂B = ϕ

on ∂B.

Proof. We extend H via H ≡ 0 on Rn+1 \ A to a bounded, measurable function. For a closed rectifiable n-current T with support in A and mass not greater than s, the results of Section 3 show that there exists a unique rectifiable (n + 1)-current Q satisfying ∂Q = T . The isoperimetric inequality (1.6) then implies n|Q, H "| ≤ n sup |H | · M(Q) ≤ n γn+1 sup |H |s 1/n M(T ). A

A

(5.3)

That is, H satisfies an isoperimetric condition of type nγn+1 supA |H |s 1/n , s on Rn+1 . Thus the conditions of Theorem 4.4(i) (keeping in mind Remark 4.5(i)) are therefore satisfied with σ = (s/dϕ ) − 1 if s > 2dϕ and nγn+1 supA |H |s 1/n ≤ 1. If we further require

478

DUZAAR AND GROTOWSKI

nγn+1 sup |H |s 1/n ≤ A

σ − 1 s − 2dϕ = , σ +1 s

then we can apply (ii) of Theorem 4.4. Noting that the maximum of the function s & → (s − 2dϕ )/(s 1+(1/n) ) on (2dϕ , ∞) occurs for s = 2(n + 1)dϕ , we obtain the sufficient condition  2(n + 1)dϕ − 2dϕ 1 αn+1 n sup |H | ≤ = .

1+(1/n) nγ 2dϕ n+1 2(n + 1)dϕ A The remaining conclusions follow from Theorem 4.4(iii). We can exploit the fact that the functions iu,u0 and iQ introduced in Section 3 are actually in BV (Rn+1 , Z), and hence in L1+(1/n) (Rn+1 , Z), to give a different set of sufficient conditions. Compare with [St1, Theorem 6.1] and [St2, Theorem 3.3]. Theorem 5.2. Let A and ϕ be as in Theorem 5.1. Further, let H : A → R be a bounded, continuous function satisfying  A

|H |

n+1



1 dx < 1 + n

n+1

αn+1

(5.4)

and |H (a)| ≤ ᏷∂A (a)

for a ∈ ∂A.

Then there exists a weak solution u ∈ W 1,n (B, A) to the Dirichlet problem Ᏸ(H, ϕ). Proof. As in the proof of Theorem 5.1, we extend H via H ≡ 0 on Rn+1 \ A to a bounded, measurable function on Rn+1 . For a closed rectifiable n-current T with support in A and its associated (n + 1)-current Q satisfying ∂Q = T and multiplicity function iQ , we use Hölder’s inequality and [Fe, 4.5.9 (31)] in order to obtain     iQ H " n|Q, H "| = n  Rn+1 n/(n+1)  1/(n+1)  (n+1)/n n+1 |iQ | dx |H | dx ≤n Rn+1

=

n −1/(n+1) α n + 1 n+1

Rn+1

 A

|H |n+1 dx

1/(n+1)

M(T ). −1/(n+1)

n/(n+1) That is, H satisfies an isoperimetric condition of type c, ∞ for c = αn+1 
 n+1 dx . Hence the conditions of Theorem 4.4 (with s = σ = ∞) are therefore |H | A satisfied if c < 1; this is precisely (5.4). The following corollary is immediate.

EXISTENCE AND REGULARITY FOR H -SYSTEMS

479

Corollary 5.3. Let A, ᏷∂A , and ϕ be as above, and let H be a bounded, continuous function on A for which (5.2) and    1 n+1 αn+1 (5.5) sup |H | < 1 + n ᏸn+1 (A) A hold. Then there exists a weak solution u ∈ W 1,n (B, A) to the Dirichlet problem Ᏸ(H, ϕ). In the case A = B R (0) ⊂ Rn+1 , conditions (5.5) and (5.2) simplify to sup |H | <

BR (0)

n+1 1 , n R

|H (a)| ≤

1 R

for a ∈ ∂BR (0).

That is, Corollary 5.3 contains the results of [DuF3, Satz 2.1] as a special case (cf. [MY, Theorem 4]) in the case of constant H . Theorem 5.4. Let A and ϕ be as in Theorem 5.1, and let H : A → R be bounded, be continuous, and satisfy  n+1 1/(n+1) t n+1 sup ᏸ {a ∈ A : |H (a)| ≥ t} =: c < 1 (5.6) t>0 αn+1 in addition to (5.2). Then there exists a weak solution u ∈ W 1,n (B, A) to the Dirichlet problem Ᏸ(H, ϕ). Proof. We extend H as before. Following the arguments of the proof of [St2, Proposition 5.1] and noting (5.5), we obtain an isoperimetric condition of type c, ∞ with c < 1. That is, for every rectifiable n-current T with ∂T = 0 and spt T ⊂ A and for the unique rectifiable n + 1-current Q satisfying ∂Q = T , we have n|Q, H "| ≤ c · M(T ).  = Rn+1 ) are Thus the conditions of Theorem 4.4 (with s = σ = ∞, c < 1, and A satisfied. 6. Regularity of solutions. In this section we discuss the regularity of solutions to (4.10). We call a domain G ⊂ Rn+1 locally convex up to Lipschitz transformations if G = int(G) and if for every point a0 ∈ ∂G, we can find a neighbourhood U of a0 and a bi-Lipschitz mapping f from the component of a0 in U ∩G to some closed convex set. The domain G is called uniformly locally convex up to Lipschitz transformations if there is a constant E independent of a0 , 0 < E ≤ 1, such that U and f can be chosen to satisfy U ⊃ BE (a0 ),

Lip(f ) ≤ E−1 ,

Lip(f −1 ) ≤ E−1 .

(Compare with [St1, Remark 3.9] and the comments thereafter.)

(6.1)

480

DUZAAR AND GROTOWSKI

Theorem 6.1. Let A, H , and u0 satisfy the conditions of Theorem 4.4 with associated parameters σ, s, and c. Further, let A be the closure of a domain that is uniformly locally convex up to Lipschitz transformations. Then every solution u of (4.10) is Hölder continuous inside B; further, u ∈ C 0 (B, Rn+1 ) if u|∂B ∈ C 0 (∂B, Rn+1 ). Proof. Our goal is to prove that the inequality DBρ (x0 ) (u) ≤ DBr (x0 ) (u)

 ρ nα r

(6.2)

holds for all x0 ∈ B and 0 < ρ ≤ r < min{r0 , 1 − |x0 |}. We can then apply Morrey’s Dirichlet growth theorem [Mor, 3.5.2] to conclude the local Hölder continuity of u with exponent α. To show (6.2) we begin by fixing x0 ∈ B, and we set u(r, ω) = u(x0 +rω) = ur (ω) for ω ∈ S n−1 and 0 ≤ r ≤ 1 − |x0 |. The function   n/2  r  ∂u 2 1 1   + |dω u|2 ρ n−1 dω dρ (6.3) ((r) := DBr (x0 ) (u) = √ ρ2 nn 0 S n−1  ∂ρ  is absolutely continuous on [0, 1 − |x0 |], and for almost all r in this interval we have  1 |dω u(r, ·)|n dω ≤ r(! (r). (6.4) G(r) := √ nn S n−1 From now on, we only consider r such that (6.4) holds. Sobolev’s embedding theorem then ensures  (6.5) osc u(r, ·) = sup |u(r, ω) − u(r, ω! )| ≤ c(n) n G(r). S n−1

ω,ω! ∈S n−1

Our aim is to obtain an estimate for G(r). Denoting by 0 < E ≤ 1 the constant from (6.1), we consider the cases G(r) ≥ (E/2c(n))n and G(r) < (E/2c(n))n separately. In the former case, using ((r) ≤ D(u), we have     2c(n) n 2c(n) n ((r) ≤ G(r)D(u) ≤ σ D(u0 )r(! (r). (6.6) E E In the latter case, from (6.5) we have the inequality oscS n−1 ur < E/2. That is, we can find a1 with a1 ∈ ur (S n−1 ) = u(∂Br (x0 )) ⊂ BE/2 (a1 ) ∩ A. If BE/2 (a1 ) is not contained in A, then we can find a0 ∈ ∂A with {ta0 + (1 − t)a1 : 0 ≤ t ≤ 1} ⊂ BE/2 (a1 ) ∩ BE (a0 ). With f as in (6.1) we define h ∈ W 1,n (Br (x0 ), Rn+1 ) to be the DBr (x0 ) -minimizing map with boundary values f ◦ u|∂Br (x0 ) , and further we define w = f −1 ◦ h ∈ W 1,n (Br (x0 ), Rn+1 ). These are well defined, since u(∂Br (x0 )) is contained in the component of a0 in A ∩ BE (a0 ) and, hence, h(∂Br (x0 )) = f ◦ u(∂Br (x0 )) in the convex set Im(f ), so that h(B r (x0 )) ⊂ Im(f ). For w we have 

 u|Br (x0 ) − w ∈ W01,n Br (x0 ), Rn+1 , (6.7) w ∈ W 1,n Br (x0 ), A ,

EXISTENCE AND REGULARITY FOR H -SYSTEMS

481

and DBr (x0 ) (w) ≤ E−n DBr (x0 ) (h) ≤ E−n DBr (x0 ) (f ◦ u) ≤ E−2n DBr (x0 ) (u).

(6.8)

(Here we extend f to a map of all of A with the same Lipschitz constant; see Kirszbraun’s theorem [Fe, 2.10.43].) If BE/2 (a1 ) ⊂ A, we simply define w := h to be the DBr (x0 ) -minimizing map with boundary data u|∂Br (x0 ) ; in this case, too, we have (6.7) and (6.8). The next step is to show the existence of r0 > 0 such that the inequality DBr (x0 ) (u) ≤ M0 DBr (x0 ) (w)

(6.9)

holds for all Br (x0 ) ⊂ B with r ≤ r0 for a constant M0 independent of r and x0 . We now define  u on B \ Br (x0 ),  u= (6.10) w on Br (x0 ), and note that  u ∈ W 1,n (B, A) and  u − u0 ∈ W01,n (B, Rn+1 ). If D( u ) > σ D(u0 ), then we have from (6.10), since D(u) ≤ σ D(u0 ),
 u = DBr (x0 ) (w), DBr (x0 ) (u) < DBr (x0 )  u ) ≤ σ D(u0 ), we can and hence we have (6.9) with M0 = 1. On the other hand, if D( take  u as a comparison surface for problem (4.11), which leads to EH (u) ≤ EH ( u ), or equivalently, from (6.10),


 DBr (x0 ) (u) ≤ DBr (x0 ) (w) + n VH  u, u0 − VH u, u0 . (6.11) We now consider the spherical n-current Ju˜ − Ju . From (6.10) and (6.8) we have

  M Ju˜ − Ju ≤ DBr (x0 ) (w) + DBr (x0 ) (u) ≤ E−2n + 1 DBr (x0 ) (u). (6.12) Since DBr (x0 ) (u) becomes arbitrarily small as ᏸn (Br (x0 )) converges to zero, we can find positive r1 depending only on s such that M(Ju˜ − Ju ) ≤ s for r ≤ r1 . (Note that E depends only on A and not on the parameters s, σ , and c.) This guarantees the existence of an integer-multiplicity rectifiable (n+1)-current Iu,u ˜ with support in A and boundary Ju˜ − Ju . Denoting by Iu,u , I the integer-multiplicity rectifiable ˜ 0 u,u0 (n+1)-currents with support in A with boundary Ju˜ −Ju0 (respectively, Ju −Ju0 ), we have Iu,u u, u) = VH ( u, u0 ) − VH (u, u0 ), and hence ˜ = Iu,u ˜ 0 − Iu,u0 . This shows VH ( from (6.11) we have
 DBr (x0 ) (u) ≤ DBr (x0 ) (w) + nVH  u, u (6.13) if 0 < r ≤ r1 .

482

DUZAAR AND GROTOWSKI

Since H satisfies a spherical isoperimetric condition of type c, s, we can use (4.9) and (6.12) to estimate n|VH ( u, u)| by 
 

nVH  (6.14) u, u  ≤ c M Ju˜ − Ju ≤ c DBr (x0 ) (w) + DBr (x0 ) (u) . From (6.14) and (6.13), we have, if c < 1, DBr (x0 ) (u) ≤

1+c DB (x ) (w); 1−c r 0

this is precisely (6.9), with M0 = (1 + c)/(1 − c). In the case c = 1, we use the isoperimetric inequality (1.6) and (6.12) to bound |VH ( u, u)| from above by 
  1+(1/n)

VH  u, u  ≤ H L∞ M Iu,u ≤ γn+1 H L∞ M Ju˜ − Ju ˜ 1+(1/n)
1/n
≤ γn+1 H L∞ E−2n + 1 DBr (x0 ) (u). DBr (x0 ) (u) Thus, given ε > 0, we can determine r0 , 0 < r0 ≤ r1 , such that n|VH ( u, u)| ≤ ε DBr (x0 ) (u). From (6.13) we thus have DBr (x0 ) (u) ≤

1 DB (x ) (w); 1−ε r 0

(6.15)

that is, (6.9) is also valid in this case (in fact, for M0 arbitrarily close to 1, since we can choose r0 as small as  we please). We next define p := − S n−1 f ◦ u(r, ω) dω and  r  for ω ∈ S n−1 , 0 ≤ ρ < , p 2    v(ρ, ω) :=  2ρ 2ρ r  p+ − 1 f ◦ u(r, ω) for ω ∈ S n−1 , ≤ ρ ≤ r.  2− r r 2 Using Poincaré’s inequality, we have  n    r  ∂v  2n − 1 n−1   ρ |f ◦ u(r, ω) − p|n dω  (ρ, ω) dω dρ = n r/2 S n−1 ∂ρ S n−1    dω (f ◦ u)(r, ω)n dω ≤ c(n) S n−1    −n dω u(r, ω)n dω. ≤ c(n)E S n−1

For the tangential component we obtain  r       1 dv(ρ, ω)n dω dρ ≤ E−n log 2 dω u(r, ω)n dω. r/2 ρ S n−1 S n−1

EXISTENCE AND REGULARITY FOR H -SYSTEMS

483

Combining this with (6.4), we have  −n DBr (x0 ) (v) ≤ c(n)E |du(r, ω)|n dω ≤ c(n)E−n r(! (r). S n−1

The DBr (x0 ) (·)-minimality of h yields DBr (x0 ) (h) ≤ DBr (x0 ) (v) ≤ c(n)E−n r(! (r). Combining this with (6.9) and the definition of w (recall that w is either h or f −1 ◦h, depending on whether BE/2 (a) ⊂ A), we have ((r) = DBr (x0 ) (u) ≤ M0 DBr (x0 ) (w) ≤ M0 E−n DBr (x0 ) (h) ≤ M0 c(n)E−2n r(! (r).

(6.16)

  Setting M1 := max (2c(n)/E)n , c(n)M0 E−2n , we have from (6.6) and (6.16) ((r) ≤ M1 r(! (r)

for almost all 0 < r ≤ min{r0 , 1 − |x0 |},

and hence, with α := (nM1 )−1 ,  ρ nα ((ρ) ≤ ((r) for 0 < ρ ≤ r ≤ min{r0 , 1 − |x0 |}. r This yields (6.2) and, hence, by the comments above, completes the proof of interior regularity. If u|∂B ∈ C 0 (∂B, Rn+1 ), we can generalize [HiK, Lemma 3] directly to the current setting. This yields u ∈ C 0 (B, Rn+1 ), as desired. Higher interior regularity for solutions of the Dirichlet problem Ᏸ(H, ϕ) (e.g., C 1,β for Lipschitz continuous H ) follows from the arguments of [HL, Section 3]. References [C] [DHKW]

[DT] [Du1] [Du2] [DuF1]

R. Courant, Dirichlet’s Principle, Conformal Mapping, and Minimal Surfaces, Interscience, New York, 1950. U. Dierkes, S. Hildebrandt, A. Küster, and O. Wohlrab, Minimal Surfaces, Vol. 1: Boundary Value Problems, Grundlehren Math. Wiss. 295; Vol. 2: Boundary Regularity, Grundlehren Math. Wiss. 296, Springer-Verlag, Berlin, 1992. W. Ding and G. Tian, Energy identity for a class of approximate harmonic maps from surfaces, Comm. Anal. Geom. 3 (1995), 543–554. F. Duzaar, Variational inequalities and harmonic mappings, J. Reine Angew. Math. 374 (1987), 39–60. , On the existence of surfaces with prescribed mean curvature and boundary in higher dimensions, Ann. Inst. H. Poincaré Anal. Non Linéaire 10 (1993), 191–214. F. Duzaar and M. Fuchs, Existenz und Regularität von Hyperflächen mit vorgeschriebener mittlerer Krümmung, Analysis 10 (1990), 193–230.

484 [DuF2] [DuF3]

[DuF4] [DuF5] [DuS1] [DuS2] [DuS3] [DuS4] [EG] [Fe] [GS1] [GS2] [HL] [He] [Hi1] [Hi2] [HiK]

[Mor] [MY] [Q] [SU] [Si] [St1] [St2] [W]

DUZAAR AND GROTOWSKI , On the existence of integral currents with prescribed mean curvature vector, Manuscripta Math. 67 (1990), 41–67. , Einige Bemerkungen über die Existenz orientierter Mannigfaltigkeiten mit vorgeschriebener mittlerer Krümmungsform, Z. Anal. Anwendungen 10 (1991), 525–534. , Einige Bemerkungen über die Regularität von stationären Punkten gewisser geometrischer Variationsintegrale, Math. Nachr. 152 (1991), 39–47. , A general existence theorem for integral currents with prescribed mean curvature form, Boll. Un. Mat. Ital. B (7) 6 (1992), 901–912. F. Duzaar and K. Steffen, Boundary regularity for minimizing currents with prescribed mean curvature, Calc. Var. Partial Differential Equations 1 (1993), 355–406. , λ minimizing currents, Manuscripta Math. 80 (1993), 403–447. , Existence of hypersurfaces with prescribed mean curvature in Riemannian manifolds, Indiana Univ. Math. J. 45 (1996), 1045–1093. , Parametric surfaces of least H -energy in a Riemannian manifold, Math. Ann. 314 (1999), 197–244. L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, Stud. Adv. Math., CRC Press, Boca Raton, Fla., 1992. H. Federer, Geometric Measure Theory, Grundlehren Math. Wiss. 153, Springer-Verlag, Berlin, 1969. R. Gulliver and J. Spruck, The Plateau problem for surfaces of prescribed mean curvature in a cylinder, Invent. Math. 13 (1971), 169–178. , Existence theorems for parametric surfaces of prescribed mean curvature, Indiana Univ. Math. J. 22 (1972), 445–472. R. Hardt and F. Lin, Mappings minimizing the Lp norm of the gradient, Comm. Pure Appl. Math. 40 (1987), 555–588. E. Heinz, Über die Existenz einer Fläche konstanter mittlerer Krümmung bei vorgegebener Berandung, Math. Ann. 127 (1954), 258–287. S. Hildebrandt, Einige Bemerkungen über Flächen beschränkter mittlerer Krümmung, Math. Z. 115 (1970), 169–178. , On the Plateau problem for surfaces of constant mean curvature, Comm. Pure Appl. Math. 23 (1970), 97–114. S. Hildebrandt and H. Kaul, Two-dimensional variational problems with obstructions, and Plateau’s problem for H -surfaces in a Riemannian manifold, Comm. Pure Appl. Math. 25 (1972), 187–223. C. B. Morrey, Multiple Integrals in the Calculus of Variations, Grundlehren Math. Wiss. 130, Springer-Verlag, New York, 1966. L. Mou and P. Yang, Multiple solutions and regularity of H -systems, Indiana Univ. Math. J. 45 (1996), 1193–1222. J. Qing, On singularities of the heat flow for harmonic maps from surfaces into spheres, Comm. Anal. Geom. 3 (1995), 297–315. J. Sacks and K. Uhlenbeck, The existence of minimal immersions of 2-spheres, Ann. of Math. (2) 113 (1981), 1–24. L. Simon, Lectures on Geometric Measure Theory, Proc. Centre Math. Appl. Austral. Nat. Univ. 3, Austral. Nat. Univ. Press, Canberra, 1983. K. Steffen, Isoperimetric inequalities and the problem of Plateau, Math. Ann. 222 (1976), 97–144. , On the existence of surfaces with prescribed mean curvature and boundary, Math. Z. 146 (1976), 113–135. H. Wente, An existence theorem for surfaces of constant mean curvature, J. Math. Anal.

EXISTENCE AND REGULARITY FOR H -SYSTEMS

[Wr]

485

Appl. 26 (1969), 318–344. H. Werner, Das problem von Douglas für Fläschen konstanter mittlerer Krümmung, Math. Ann. 133 (1957), 303–319.

Mathematisches Institut der Humboldt-Universität zu Berlin, Unter den Linden 6, D10099 Berlin, Germany

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

VANISHING CYCLES AND MONODROMY OF COMPLEX POLYNOMIALS WALTER D. NEUMANN and PAUL NORBURY

1. Introduction. In this paper we describe the trivial summand for monodromy around a fibre of a polynomial map Cn → C, generalising and clarifying work of Artal Bartolo, Cassou-Noguès, and Dimca [2], who proved similar results under strong restrictions on the homology of the general fibre and singularities of the other fibres. They also showed that a polynomial map f : C2 → C has trivial global monodromy if and only if it is “rational of simple type” in the terminology of Miyanishi and Sugie. We refine this result and correct the Miyanishi-Sugie classification of such polynomials, pointing out that there are also nonisotrivial examples. Let f : Cn → C be a nonconstant polynomial map. The polynomial describes a family of complex affine hypersurfaces f −1 (c), c ∈ C. It is well known that the family is locally trivial, so the hypersurfaces have constant topology, except at finitely many irregular fibres f −1 (ci ), i = 1, . . . , m whose topology may differ from the generic or regular fibre of f . Definition 1.1. If f −1 (c) is a fibre of f : Cn → C, choose sufficiently small that all fibres f −1 (c ) with c ∈ D 2 (c) − {c} are regular and let N(c) := f −1 (D 2 (c)). Let F = f −1 (c ) be a regular fibre in N(c). Then
 Vq (c) := Ker Hq (F ; Z) −→ Hq (N(c); Z)
 V q (c) := Cok H q (N(c); Z) −→ H q (F ; Z) are the groups of vanishing q-cycles and vanishing q-cocycles for f −1 (c). They have the same rank, which we call the number of vanishing q-cycles for f −1 (c). Choose a regular value c0 for f and paths γi from c0 to ci for i = 1, . . . , m, which are disjoint except at c0 . We can use these paths to refer homology or cohomology of a regular fibre near one of the irregular fibres f −1 (ci ) to the homology or cohomology of the reference regular fibre F =
f −1 (c0 ).  The fundamental group  = π1 C − {c1 , . . . , cm } acts on the homology H∗ (F ; Z) and cohomology H ∗ (F ; Z). If this action is trivial, we say that f has trivial global monodromy group. This action has the following generators. Let hq (ci ) : Hq (F ) → Hq (F ) and hq (ci ) : H q (F ) → H q (F ) be the monodromy Received 30 November 1998. Revision received 4 May 1999. 1991 Mathematics Subject Classification. Primary 14B05, 32S30, 32S50; Secondary 14H50, 57M25. Authors’ research supported by the Australian Research Council. 487

488

NEUMANN AND NORBURY

about the fibre f −1 (ci ) (obtained by translating the fibre F along the path γi until close to the fibre f −1 (ci ), then in a small loop around that
fibre and  back along q γi ). We are interested in the fixed group H q (F )h (ci ) = Ker 1 − hq (ci ) of this local monodromy. Theorem 1.2. For q > 0,1 the maps H q (F ; Z) → V q (ci ) induce an isomorphism H (F ; Z) ∼ = q

m 

V q (ci ).

i=1

Under this isomorphism we have  
V q (ci ), Ker 1 − hq (cj ) = K q (cj ) ⊕ i =j

for some K q (cj ) ⊂ V q (cj ). Hence the subgroup of cohomology fixed under global monodromy is H q (F ; Z) =

m 

K q (ci ).

i=1

Finally, there is a natural short exact sequence

 0 −→ Cok 1 − hq−1 (c) −→ H q+1 N(c), ∂N(c) −→ K q (c) −→ 0. The above exact sequence lets one compute dim(K q (c)) inductively in terms of numbers of vanishing cycles and betti numbers of H ∗ (N(c), ∂N(c)). The following theorem localizes the computation of H ∗ (N(c), ∂N(c)) into the singular fibre. Theorem 1.3. Let H∗ (f −1 (c), ∞) denote H∗ (f −1 (c), U ), where U is a regular neighbourhood of infinity (e.g., U = {z ∈ f −1 (c) : ||z|| > R} for large R). Then we have a natural isomorphism H q+1 (N(c), ∂N(c)) ∼ = H2n−q−1 (f −1 (c), ∞). Under the assumptions that F has homology only in dimension (n−1) and that all singularities of fibres of f are isolated, Artal-Bartolo,
Cassou-Nogués, and Dimca [2] proved the dimension formulae for Ker 1 − hn−1 (c) and H n−1 (F ; Z) that follow from the above theorems. Polynomials f (x1 , . . . , xn ) = x1 g(x2 , . . . , xn ) are examples of polynomials with trivial global monodromy that do not satisfy their assumptions for n > 2. The first displayed formula of Theorem 1.2 was proved (in homology) by Broughton [3]; see also [4] and [18]. The homology version of the above results is the following theorem. 1 All

results also hold for q = 0 if H q and Hq are read as reduced (co)homology throughout.

VANISHING CYCLES AND MONODROMY

489

Theorem 1.4. For q > 0, the inclusions Vq (ci ) → Hq (F ; Z) induce an isomorphism Hq (F ; Z) ∼ =

m 

Vq (ci ).

i=1

Moreover, Im(1 − hq (c)) ⊂ Vq (c), and there is a natural exact sequence
 

0 −→Im 1 − hq (c) −→Vq (c) −→Hq+1 N(c), ∂N(c) −→Ker 1 − hq−1 (c) −→ 0 and an isomorphism Hq+1 (N (c), ∂N(c)) ∼ = H 2n−q−1 (f −1 (c), ∞). The groups H2n−2 (f −1 (c), ∞) and H 2n−2 (f −1 (c), ∞) are freely generated by the fundamental classes of the irreducible components of f −1 (c), so for q = 1 the above results give us the following corollary. Corollary 1.5. If rc is the number of irreducible components of f −1 (c), then and V1 (c)/ Im(1 − h1 (c)) ∼ = = Zrc −1 .

K 1 (c) ∼ Zrc −1

In particular, the 1-dimensional monodromy with Q-coefficients about a fibre f −1 (c) is trivial if and only if the number of irreducible components of this fibre exceeds by 1 the number of its vanishing 1-cycles. For an irreducible fibre, this says this monodromy is trivial if and only if the fibre has no vanishing 1-cycles. By Theorem 1.6, this generalises Michel and Weber’s positive answer in [11] to Dimca’s question of whether the local monodromy around a reduced and irreducible irregular fibre of a polynomial f : C2 → C must be nontrivial (the same answer is implicit in Theorem 1 of [2]). The conditions are needed here: one can find f : C2 → C having an irregular fibre with trivial local monodromy, this fibre having any number of components. Such examples exist with nonreduced fibres or fibres of arbitrarily high genus. Recall that a polynomial f : Cn → C is primitive if its regular fibres are irreducible; equivalently, it is not of the form g ◦ h with g : C → C and h : Cn → C polynomial maps and deg g > 1. Theorem 1.6. A fibre of a primitive polynomial f : C2 → C has no vanishing cycles if and only if it is regular. By [18], this result holds in any dimension for a fibre with “isolated W -singularities at infinity.” We prove all of the above theorems in Section 2. When the fibre f −1 (c) is reduced with isolated singularities, there is a quick alternative proof of Corollary 1.5 for homology. Namely, let F0 be the nonsingular core of f −1 (c), obtained by intersecting f −1 (c) with a very large ball and then removing small regular neighbourhoods of its singularities. Then F0 can be isotoped into a nearby regular fibre F , and it is not hard to see the following (e.g., [15]).

490

NEUMANN AND NORBURY

Proposition 1.7. Under the above assumption, Hq (F, F0 ) is isomorphic to Vq (c) by an isomorphism that fits in the commutative diagram Hq (F, F0 ) O

∼ =

/ Vq (c) ⊆

Hq (F )

1−hq

 / Hq (F ).

Since F0 has rc topological components, Corollary 1.5 for homology follows in this case using q = 1 and the long exact homology sequence for the pair (F, F0 ). The following consequence of the monodromy results was proved by Artal Bartolo et al. in [2]. The improvement of the second sentence is from Dimca [7]. Theorem 1.8. A primitive polynomial f : C2 → C has trivial global monodromy group if and only if f is rational of simple type, in the sense of Miyanishi and Sugie [12]. The same conclusion already follows if only the monodromy h1 (∞) at infinity of f is trivial (h(∞) is the monodromy around a very large circle in C, so h1 (∞) is the product of the h1 (ci )). A polynomial f : C2 → C is rational if its generic fibre is rational (i.e., genus zero). Simple type means that if we take a nonsingular compactification Y = C2 ∪ E of C2 such that f extends to a holomorphic map f : Y → CP 1 , then f is of degree 1 on each horizontal irreducible component of the compactification divisor E (E is a union of smooth rational curves E1 , . . . , En with normal crossings and a component Ei is called horizontal if f | Ei is nonconstant). We discuss Theorem 1.8 and refinements of it in Section 3. In Section 4 we describe corrections to Miyanishi and Sugie’s classification of rational polynomials of simple type. Details of this will appear elsewhere. Acknowledgements. This research was supported by the Australian Research Council. We thank the referees for some useful comments. 2. Proofs of the main theorems Proof of Theorem 1.2. The direct sum statement has been proved in [3] (see also [18]), but a proof is quick, so we include it for completeness and notation. For each irregular value ci , we construct a neighbourhood Ni = f −1 (D 2 (ci )) of the corresponding irregular fibre as in Definition 1.1, with chosen small enough that the disks D 2 (ci ) are disjoint. Let c0 be a regular value outside all these disks  and choose disjoint paths γi joining c0 to each disk D 2 (ci ). Let P = m i=1 γi and m D = i=1 D 2 (ci ), so K = P ∪ D is the union of these paths and disks. Then Cn deformation retracts onto f −1 (K). The Mayer-Vietoris sequence for f −1 (K) = f −1 (P ) ∪ f −1 (D) gives

491

VANISHING CYCLES AND MONODROMY q

0 −→ H (F ) ⊕

(1)

m 

q

H (Ni ) −→

i=1

m 

H q (F ) −→ 0

(q > 0).

i=1

m q Since the ith summand of the sum 1 H (Ni ) maps trivially to all but the ith summand of m H (F ), this shows the following proposition. q 1 Proposition 2.1. H q (Ni ) → H q (F ) is injective with cokernel V q (ci ) (by Definition 1.1). factoring source and target of the middle isomorphism of (1) by the subgroup Thus, H q (Ni ) gives the desired isomorphism ∼ =

q

H (F ) −−→

(2)

m 

V q (ci ).

i=1

The long exact sequence for the pair (Ni , F ) shows that we have a commutative diagram with exact rows: 0

/ H q (Ni )

/ H q (F )

/ H q+1 (N , F ) i

/ H q (F )

 / V q (ci )

/0

∼ =

0

/ H q (Ni )

/ 0.

We now claim that we can identify the long exact sequence of the triple (Ni , ∂Ni , F ) as follows: H q (∂Ni , F ) 

/ H q+1 (N , ∂N ) i i

/ H q+1 (N , F ) i

/ H q+1 (N , ∂N ) i i

 / V q (ci )

∼ =

H q−1 (F )

/ H q+1 (∂N , F ) i

∼ =

∼ =

 / H q (F ).

The first and fourth vertical isomorphisms are seen by thickening F within ∂N and then using excision and the Künneth formula: H q (∂Ni , F ) ∼ = H q (F × I, F × ∂I ) ∼ = H q−1 (F ). We have already shown the third vertical isomorphism. Thus the above diagram is proved. Now consider the composition H q (F ) → V q (ci ) → H q (F ) where the second map is the map of the above diagram. Tracing the definitions, we see it is the composition: H q (F ) → H q+1 (∂Ni , F ) → H q (F ), where the first map is the boundary map for the pair. We claim that this composition is 1 − hq (ci ). Indeed, ∂Ni is isomorphic to the mapping torus F ×h(ci ) S 1 , so there is a map F × I → Ni that identifies the ends

492

NEUMANN AND NORBURY

of F ×I by h(ci ). The map H q+1 (∂Ni , F ) → H q (F ) is induced by the map of chain groups Cq (F ) → Cq+1 (Ni ) that takes a q-chain A in F to the (q + 1)-chain A × I in F × I mapped to Ni . The boundary map H q (F ) → H q+1 (Ni , F ) is induced by the map that lifts a (q + 1) chain in (Ni , F ) to the corresponding chain in F × I and then takes its boundary. Composing these maps of chains clearly gives the chain map 1−h(ci )( . The induced composition in q-cohomology is thus 1−hq (ci ), as claimed. Since the composition H q (F )  → V q (ci ) → H q (F ) is 1 − hq (ci
), and H q (F  )→ q V (ci ) is surjective with kernel j =i V q (cj ), it follows that Ker 1 − hq (ci ) con  tains j =i V q (cj ). It hence has the form K q (ci ) ⊕ j =i V q (cj ) in terms of the isomorphism of (2). Thus the second statement of Theorem 1.2 follows. The final exact sequence of the theorem then follows by replacing the first term of the bottom sequence of the above diagram by its image and the last arrow by its kernel. Proof of Theorem 1.3. Let Ni0 be f −1 (D 2 (ci )) ∩ D 2n where D 2n is first chosen of large enough radius that f −1 (ci ) is transverse to the boundary of it and all larger disks (for the existence of such a radius, even if f −1 (ci ) has nonisolated singularities, see, e.g., Proposition 2.3.1 of [1]), and is then rechosen small enough that ∂D 2n is transverse to f −1 (ci ) for all ci ∈ D 2 (ci ). Put ∂0 Ni0 := ∂Ni ∩ Ni0 , Fi0 := f −1 (ci ) ∩ D 2n and Ci := f −1 (ci ) − int(Fi0 ). Then the inclusion of Ni − Ci in Ni is a homotopy equivalence and the inclusion of ∂Ni into ∂Ni ∪ (Ni − Ni0 − Ci ) is a homotopy equivalence, so we have H q+1 (Ni , ∂Ni ) ∼ = H q+1 (Ni − Ci , ∂Ni ∪ (Ni − 0 Ni − Ci )). Excision then shows that this is isomorphic to H q+1 (Ni0 , ∂Ni0 − ∂Fi0 ), and this equals H q+1 (Ni0 , ∂0 Ni0 ) by homotopy equivalence. Putting ∂1 Ni0 := ∂Ni0 − int(∂0 Ni0 ), Poincaré-Lefschetz duality gives H q+1 (Ni0 , ∂0 Ni0 ) ∼ = H2n−q−1 (Ni0 , 0 0 0 0 ∂1 Ni ). But the pair (Ni , ∂1 Ni ) is homotopy equivalent to (Fi , ∂Fi0 ). By excision, H∗ (Fi0 , ∂Fi0 ) = H∗ (f −1 (ci ), ∞). The proof of the homology versions of these results is essentially the same, so we omit it. Proof of Theorem 1.6. Since f is primitive, any nonreduced fibre of f has more than one component and thus has vanishing cycles by Corollary 1.5. For a reduced fibre, on the other hand, the desired result is implicit in several places in the literature. For instance, it follows immediately from the result that for primitive f : C2 → C, the Euler characteristic of an irregular fibre always exceeds that of the regular fibre (Proposition 1 of Suzuki2 [20]), together with the following easy facts: (1) The number of vanishing cycles for f −1 (ci ) is χ(Ni ) − χ(F ). (2) For an irregular fibre in any dimension, χ(f −1 (ci )) = χ(Ni ) (see, e.g., [2, comments preceding Théorème 3]). It also follows from the more general result of Siersma and Tibár [18] that a 2 The

case of reduced fibre, which is all we need here, is already in [19]. Reference [9] is often cited for this, but only seems to prove the case of a nonsingular fibre.

VANISHING CYCLES AND MONODROMY

493

polynomial fibre in any dimension with “isolated W -singularities at infinity” is regular if and only if it has no vanishing cycles, since a reduced fibre of f : C2 → C satisfies this condition. 3. Discussion of Theorem 1.8. In this section we assume that f is primitive; this is clearly implied by global trivial monodromy. We give a mild improvement (implicit in [7]) of Theorem 1.8. Proposition 3.1. (1) The global monodromy on the closed fibre F is trivial if and only if f has rational generic fibres. (2) If we consider the subgroup B ⊂ H1 (F ) generated by small loops around the punctures of F , then the global monodromy restricted to B is trivial if and only if f is degree 1 on all horizontal curves. (3) The above two statements also hold with “global monodromy” replaced by “monodromy at infinity.” Proof. Let Y = C2 ∪ E be as described just after Theorem 1.8. E is a union of smooth rational curves E1 , . . . , En with normal crossings. Since H1 (F ) and H1 (F ) are torsion free, triviality of monodromy for homology or cohomology are equivalent. (1) Deligne’s invariant cycle theorem [6] gives an epimorphism H 1 (Y ) → H 1 (F ) . But H 1 (Y ) = 0, so if the global monodromy is trivial, then H 1 (F ) = 0. The converse is trivial. (2) Note that if f is degree > 1 on some horizontal curve E, then the homology classes represented by the punctures where F meets E get permuted nontrivially as we circle a branch point of f | E. (3) This follows as in [7] by the same proofs as above if we replace Y by a −1 neighbourhood Y0 of the fibre f (∞) and apply the invariant cycle theorem of [5] ([7] cites [8, Theorem 7.13]), which says that H 1 (Y0 ) → H 1 (F )h(∞) is surjective. −1 H 1 (Y0 ) = 0 since Y0 retracts onto f (∞), which is a simply connected union of some of the rational curves Ei . We can refine the proof of the second part of the proposition to obtain a stronger −1 result. Let pi1 , . . . , piki be the points where f (ci ) meets horizontal curves, and for each j = 1, . . . , ki , let δij be the degree of f on a small neighbourhood of the point pij in its horizontal curve. Thus the generic fibre F near f −1 (ci ) has δij punctures near pij that are cyclically permuted by the monodromy around ci . It follows that the restriction of 1 − h1 (ci ) to the subgroup B of the above proposition has image of  dimension kji=1 (δij − 1). Denote


  eci : = dim Im 1 − h1 (ci ) − dim Im 1 − h1 (ci ) | B ki   

δij − 1 . = dim Im 1 − h1 (ci ) − j =1

494

NEUMANN AND NORBURY


 This measures the “extra” part of Im 1−h1 (ci ) that does not arise from the homology at infinity. It is clear that if eci = 0, then the local monodromy h1 (ci ) : H1 (F ) → H1 (F ) of −1 the closed fibre around f (ci ) is trivial. The converse is not true for arbitrary maps of a surface, but the following theorem implies that it is for our local monodromy map. Theorem 3.2. With V 1 (ci ) := Ker(H1 (F ) → H1 (N i )), we have
 Im 1 − h1 (ci ) ⊂ V 1 (ci ), and both these groups have rank eci . Moreover, m 

eci ≥ 2 genus(F ).

i=1

Proof. The inclusion Im(1−h1 (ci )) ⊂ V 1 (ci ) is clear, while the fact that they have the same dimension is shown in [2, Section III, part 2(c)]. We sketch the argument. The dimension equality in question is, in fact, generally true for a neighbourhood −1 N = f (D) of a singular fibre of a fibration f of a projective surface over a curve. −1 Here D is a disk about a point c, and F c = f (c) is the only singular fibre over this disk. The ingredients are: • N deformation retracts onto F c ; • H1 (F ) → H1 (F c ) = H1 (N) is surjective mod torsion (actually strictly surjective in our case, since f is primitive); thus dim V1 (c) = dim Ker(H1 (F ) → H1 (N)) = dim H1 (F ) − dim H1 (F c ); • dim(Im(1 − h1 (c)) also equals dim H1 (F ) − dim H1 (F c ) (this is a consequence of the fact that the nullity of the intersection form of N is 1). We leave the details of each of these ingredients to the reader (or see [2] and [10]). As in [2], following Kaliman [10], we denote the above number:

−1
  kci := dim Im 1 − h1 (ci ) = dim V 1 (ci ) = dim H1 F − dim H1 f (ci ) . We must show kci = eci . We have a short exact sequence (3)


 0 −→ B −→ H1 (F ) −→ H1 F −→ 0,

and taking the image of 1 − h(ci ) applied to this sequence gives a sequence 

 0 −→ Z j (δij −1) −→ Im 1 − h1 (ci ) −→ Im 1 − h1 (ci ) −→ 0. (4) This sequence is exact except possibly at its middle term  (this holds for a homomorphic image of any short exact sequence). The cokernel of Z j (δij −1) → Im(1−h1 (ci ))

495

VANISHING CYCLES AND MONODROMY

has, by definition, dimension eci . Since the sequence induces a surjection of this cokernel to Im(1 − h1 (ci )), we see that e ci ≥ k c i .

(5)

That this is, in fact, equality follows from [2, Corollaire 5(ii)], but we give a proof since we need one of its ingredients. Corollary 1.5 implies ki

  
δij − 1 + eci + rci − 1. dim V1 (ci ) = dim Im 1 − h1 (ci ) + rci − 1 = j =1

 Summing this over i and applying the consequence dim V1 (ci ) = dim H1 (F ) = 1 − χ (F ) of Theorem 1.4 on the left and the Riemann-Hurwitz formula on the right gives 1 − χ (F ) =

m
   dE − 1 + eci + rci − 1 ,




i=1

where the first sum on the right is over all horizontal curves E and dE is the degree of f on E. Since dE is the number of punctures of F , this simplifies to (6)

2 genus(F ) = 1 − δ +

m  (eci + rci − 1), i=1

where δ is the number of horizontal curves. But Kaliman proves this equation in [10] with eci replaced by kci , so the inequalities (5) must be equalities. The final inequality of the theorem follows from (6) and the formula δ −1 ≥

m  (rci − 1), i=1

of Kaliman [10]. A consequence of the above proof is the exactness of the kernel sequence (and hence also the image sequence (4)) of 1−h(ci ) applied to the short exact sequence (3). 1−h

Indeed, if we replace each group A in (3) by the chain complex 0 → A −−→ A → 0, then the resulting short exact sequence of chain complexes has long exact homology sequence 0 → Ker(1 − h1 | B) → Ker(1 − h1 ) → Ker(1 − h1 ) → Cok(1 − h1 | B) → Cok(1 − h1 ) → Cok(1 − h1 ) → 0. The equality in (5) implies that the middle map of this sequence has rank zero, and hence is the zero map since Cok(1 − h1 | B) is free abelian.

496

NEUMANN AND NORBURY

4. Classification of rational polynomials of simple type. The classification in [12] mistakenly assumes isotriviality (all regular fibres of f are conformally isomorphic to each other) at one stage in the proof (page 346, lines 10–11). There are in fact also many nonisotrivial 2-variable rational polynomials of simple type, the simplest being f (x, y) = x(1+xy)(1+axy)+xy of degree 5, whose regular fibres f −1 (c) are 4-punctured CP 1 ’s such that the cross-ratio of the punctures varies linearly with c. In this section we list the nonisotrivial rational polynomials of simple type. We list their regular splice diagrams (see [13], [14]), since this gives a useful description of the topology. For each case there are several possible topologies for the irregular fibres, depending on additional parameters. We have a proof that these examples complete the classification but it is tedious and not yet written down in full detail, so the result should be considered tentative. Let p, q, P , Q be positive integers with P q − pQ = 1, and let r and a1 , . . . , ar  be positive integers. Let A = ri=1 ai , B = AQ + P − Q, C = Aq + p − q, and let bi = qQai + 1 for i = 1, . . . , r. Then the following is the regular splice diagram of a rational polynomial of simple type:

❜

Q ❜−q 1 ❄

B ❜−C 1

1 r1

❄

a1 −b1✟✟ ❜ 1 1 ✟✟ −Q ❜✟✟ . ❄ .. 1 .. . q ❍❍ ❍❍ −P −br❍❍ ❜ar ✛ ❜ 1 1 p ❄ ❜

❜

❜

There is one further degree-8 example that does not fall in the above family. The splice diagram is ❜

2 ❜ −5 1 ❄

1 ❜ −1 2 ❄

1 r1

−3 ❜1 ✲ 2 ❄

In all of these examples, the curve obtained by filling the puncture corresponding to the second arrowhead from the left has constant conformal type as we vary the regular fibre f −1 (c), and that puncture varies linearly with c ∈ C. References [1] [2]

S. Akbulut and H. King, Topology of Real Algebraic Sets, Math. Sci. Res. Inst. Publ. 25, Springer-Verlag, New York, 1992. E. Artal-Bartolo, P. Cassou-Noguès, and A. Dimca, “Sur la topologie des polynômes complexes” in Singularities (Oberwolfach, 1996), Progr. Math. 162, Birkhäuser, Basel, 1998, 317–343.

VANISHING CYCLES AND MONODROMY [3]

[4] [5] [6] [7] [8]

[9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]

497

S. A. Broughton, “On the topology of polynomial hypersurfaces” in Singularities, Part 1 (Arcata, Calif., 1981), Proc. Sympos. Pure Math. 40, Amer. Math. Soc., Providence, 1983, 167–178. , Milnor numbers and the topology of polynomial hypersurfaces, Invent. Math. 92 (1988), 217–241. C. H. Clemens, Degeneration of Kähler manifolds, Duke Math. J. 44 (1977), 215–290. P. Deligne, Théorie de Hodge, II, Inst. Hautes Études Sci. Publ. Math. 40 (1971), 5–57. A. Dimca, Monodromy at infinity for polynomials in two variables, J. Algebraic Geom. 7 (1998), 771–779. F. Guillén, V. Navarro Aznar, P. Pascual Gainza, and F. Puerta, Hyperrésolutions cubiques et descente cohomologique, Lecture Notes in Math. 1335, Springer-Verlag, Berlin, 1988. ˜ Hà Huy Vui and Lê Dung Tráng, Sur la topologie des polynômes complexes, Acta Math. Vietnam. 9 (1984), 21–32. S. Kaliman, Two remarks on polynomials in two variables, Pacific J. Math. 154 (1992), 285– 295. F. Michel and C. Weber, On the monodromies of a polynomial map from C 2 to C, preprint. M. Miyanishi and T. Sugie, Generically rational polynomials, Osaka J. Math. 17 (1980), 339–362. W. D. Neumann, Complex algebraic plane curves via their links at infinity, Invent. Math. 98 (1989), 445–489. , Irregular links at infinity of complex affine plane curves, to appear in Quart. J. Math. Oxford Ser. (2). W. D. Neumann and P. Norbury, Unfolding polynomial maps at infinity, preprint, 1999. , Rational polynomials of simple type, in preparation. H. Saito, Fonctions entières qui se réduisent à certains polynômes, II, Osaka J. Math. 14 (1977), 649–674. ˘ Singularities at infinity and their vanishing cycles, Duke Math. J. D. Siersma and M. Tibar, 80 (1995), 771–783. M. Suzuki, Propriétés topologiques des polynômes de deux variables complexes, et automorphismes algébriques de l’espace C2 , J. Math. Soc. Japan 26 (1974), 241–257. , Sur les opérations holomorphes du groupe additif complexe sur l’espace de deux variables complexes, Ann. Sci. École Norm. Sup. (4) 10 (1977), 517–546.

Department of Mathematics and Statistics, The University of Melbourne, Victoria 3010, Australia; [email protected]; [email protected]

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

DOUBLE HILBERT TRANSFORMS ALONG POLYNOMIAL SURFACES IN R3 ANTHONY CARBERY, STEPHEN WAINGER, and JAMES WRIGHT 1. Introduction. In this paper we consider the Lp -boundedness of operators defined formally as

  ds dt , f x − s, y − t, z − P (s, t) Hf (x, y, z) = st |s|≤1 |t|≤1 where P (s, t) is a polynomial in s and t with P (0, 0) = 0, and ∇P (0, 0) = 0. We call H the (local) double Hilbert transform along the surface (s, t, P (s, t)). The operator may be precisely defined for a Schwartz function f by integrating where  ≤ |s| ≤ 1 and η ≤ |t| ≤ 1, and then taking the limit as , η → 0. The corresponding 1-parameter problem has been extensively studied (see [RS1], [RS2], and [S], for example). The type of operator that we are concerned with in this paper has been previously studied in [NW], [RS3], and [V]. In those works, operators that are in some ways more general than ours are considered, but only under an appropriate dilation invariance, which in our setting would force P to be a monomial. If P (s, t) = s m t n , then according to [RS3] (see Section 5 below for the precise statement), for any p, 1 < p < ∞, H is bounded in Lp if and only if at least one of m and n is even. Our present result is stated in terms of the Newton diagram of P , which we describe below. Recently Phong and Stein have shown how the Newton diagram also plays a decisive role in describing the mapping properties of certain degenerate Fourier integral operators (see [PS]). Main theorem. For any p, 1 < p < ∞, Hf Lp ≤ Ap f Lp if and only if for each (m, n) that is a corner point of the Newton diagram corresponding to P , at least one of m and n is even. Received 12 November 1998. 1991 Mathematics Subject Classification. Primary 42B20. Carbery partially supported by Engineering and Physical Sciences Research Council grant numbers GR/78574 and GR/M03085. Wainger partially supported by National Science Foundation grant number DMS-9731647. Wright partially supported by an Australian Research Council grant and Engineering and Physical Sciences Research Council grant number GR/M03085. Research at Mathematical Sciences Research Institute was partially supported by National Science Foundation grant number DMS-9701755. 499

500

CARBERY, WAINGER, AND WRIGHT

In higher dimensions where we consider the analogous k parameter Hilbert transform along a surface given as the graph of a polynomial P (x), x ∈ Rk , one might conjecture that Lp -boundedness holds precisely when there is at least k − 1 even powers in each monomial corresponding to a corner point of the Newton diagram associated to the polynomial P . This would be consistent with the general results in [RS3]. However, we show that if P (s, t, u) = s 4 t 4 u2 + s 2 t 2 u4 + s 3 t 3 u3 , the analogous triple Hilbert transform in R4 is unbounded, and thus the strict 4-dimensional analogue of our theorem is false. We hope to return to this matter in a future paper. The idea of our argument is to divide the region of (s, t) integration into parts in which P (s, t) may be replaced by one of the monomials corresponding to a corner point of the Newton diagram, making an error that is bounded in Lp . Then we may apply the Ricci-Stein result to obtain the Lp -estimate for each piece. The organization of our paper is as follows: In Section 2 we discuss the Newton diagram of P and give a basic splitting of our operator. In Section 3 we introduce our approximating operator and give some preliminary estimates. Section 4 contains the proof of the Lp -boundedness of the difference between H and the approximating operator. In Section 5 we show that the Lp -boundedness of the approximating operator under our hypothesis follows from the work of Ricci and Stein, and thus we complete the proof of the Lp -boundedness under the hypothesis of the theorem. In Section 6 we discuss the necessity of the hypothesis of the theorem, and in Section 7 we consider the example in four dimensions mentioned above. Acknowledgements. We would like to thank Professors M. Christ, F. Ricci, and E. M. Stein for discussing this material with us. 2. The Newton diagram and the splitting of the region of integration. We write  P (s, t) = am,n s m t n , (m,n)∈

where (m, n) ∈  if and only if am,n  = 0. For each (m, n) in , we let   Qm,n = (x, y) ∈ R2 | x ≥ m and y ≥ n .  Set Q = (m,n)∈ Qm,n . Then the Newton diagram  of P is the smallest (closed) convex set containing Q.  is an unbounded polygon with a finite number of corners. We denote the set of corners by D. Then D ⊂ . The sufficiency part of the main theorem is that if for every (m, n) in D, at least one of m and n is even, then H is bounded in Lp , 1 < p < ∞. We begin by choosing an odd C ∞ -function φ(s), defined on the real line, nonnegative for s ≥ 0, and supported in 1/2 ≤ |s| ≤ 2 such that ∞ 

1 2p φ(2p s) = . s p=−∞

DOUBLE HILBERT TRANSFORMS ALONG SURFACES

We then define Hp,q f (x, y, z) = 2p+q





501

  f x − s, y − t, z − P (s, t) φ(2p s)φ(2q t) ds dt.

It suffices to consider the operator H0 =



Hp,q .

p≥0 q≥0

The set D consists of a finite number of points G1 , . . . , Gr , with Gj = (mj , nj ). We may choose the order of the points so that mj +1 is strictly greater than mj . Then nj +1 is strictly less than nj . For 1 ≤ j ≤ r − 1, we set λj = (nj − nj +1 )/(mj +1 − mj ), so that λj is the absolute value of the slope of the line joining Gj to Gj +1 . The convexity of  implies that the λj are decreasing. Next we write H0 = rj =1 M(j ) for certain operators M(j ). This corresponds to the splitting of the range of integration mentioned in the introduction. First suppose r ≥ 3. We then set  M(1) = Hp,q . p/q≥λ1 p≥0, q≥0

For 2 ≤ j ≤ r − 1, let



M(j ) =

Hp,q .

λj −1 >p/q≥λj p≥0, q≥0

Finally, set M(r) =



Hp,q .

p/q<λr−1 p≥0, q≥0

If r = 2, then there is only one λ, and we put  Hp,q M(1) = p/q≥λ p≥0, q≥0

and M(2) =



Hp,q .

p/q<λ p≥0, q≥0

If r = 1, set M(1) = H0 . Finally, for each j , let Z(j ) denote the set of pairs (p, q) occurring in the sum defining M(j ). The points (p, q) in Z(j ) lie inside or on the boundary of an infinite triangle. If, for example, 2 ≤ j ≤ r − 1, this triangle is formed from the intersection of two half spaces determined by the vectors vj = Gj +1 − Gj and wj = Gj −1 − Gj .

502

CARBERY, WAINGER, AND WRIGHT

3. The approximating operators and some preliminary estimates. For (p, q) in Z(j ), we set
  p+q f x − s, y − t, z − amj , nj s mj t nj φ(2p s)φ(2q t) ds dt Sp,q f (x, y, z) = 2 and



Q(j ) =

Sp,q .

(p,q)∈Z(j )

Let E(j ) = M(j ) − Q(j ). Our main task in the proof for the sufficiency part of our theorem is to demonstrate the following proposition. Proposition 1. We have the estimate E(j )f Lp ≤ A(p, j )f Lp ,

1 < p < ∞.

We let µp,q denote the multiplier corresponding to Hp,q − Sp,q for (p, q) in Z(j ). So (1) µp,q (ξ, η, γ )

     = exp iξ 2−p s+iη2−q t exp iγ P 2−p s, 2−q t  

− exp iγ 2−pmj 2−qnj amj ,nj s mj t nj φ(s)φ(t) ds dt. We need some estimates for µp,q with (p, q) in Z(j ). Lemma 2. For any (u, v) in , and (p, q) in Z(j ), (2)

2−up 2−vq ≤ 2−mj p 2−nj q .

(This lemma asserts that each monomial occurring in P (s, t) is dominated by s mj t nj for (s, t) in the region of integration corresponding to M(j ).) Proof. For each (u, v) in , one of the following three possibilites occurs: (a) u ≥ mj and v ≥ nj , (b) u < mj , v ≥ nj , and v − nj ≥ λj −1 , mj − u or (c) u > mj , v ≤ nj , and nj − v ≤ λj . u − mj In case (a), the estimate (2) clearly holds. In case (b), we are concerned with the ratio

DOUBLE HILBERT TRANSFORMS ALONG SURFACES

503

2−pu 2−qv = 2p(mj −u) 2−q(v−nj ) 2−pmj 2−qnj ≤ 2p(mj −u) 2−qλj −1 (mj −u) ≤ 1 since p/q ≤ λj −1 . Case (c) is treated similarly. Lemma 3. Assume (p, q) is in Z(j ). Then |µp,q (ξ, η, γ )| ≤ A|γ |2−pmj 2−qnj

(3) and (4)

|µp,q (ξ, η, γ )| ≤

A (|ξ |2−p + |η|2−q + |γ |2−pmj 2−qnj )σ

for some σ > 0. Proof. The estimate (3) follows from (1), the mean value theorem, and Lemma 2. Also, to Lemma 2, the polynomial Q(s, t) =  2pmj 2qnj P (2−p s, 2−q t) =  according j k k bj,k s t is uniformly in any C class and satisfies |bj,k | ≥ |amj ,nj |. Since the norms |bj,k | and sup1≤|α|≤d inf |s|,|t|≤2 |∂ α Q(s, t)| are equivalent on the finitedimensional vector space of nonconstant polynomials of degree at most d, we see that some derivative of Q is uniformily bounded below, and so the estimate (4) follows from [S, p. 342, Proposition 5]. As mentioned above, each Z(j ) consists of all (p, q) lying inside or on the boundary of an infinite triangle. We want to further split Ej according to the distance (p, q) lies from the boundary of this triangle. To this end, for 2 ≤ j ≤ r, choose νj with λj < νj < λj −1 , ν1 > λ1 and νr < λr . Let 

p Zj+ = (p, q) ∈ Z(j ) ≥ νj , q 

p − Zj = (p, q) ∈ Z(j ) < νj . q Correspondingly, set Ej+ = and

Ej− =





Hp,q − Sp,q



(p,q)∈Zj+





 Hp,q − Sp,q .

(p,q)∈Zj−

To prove Proposition 1, it suffices to show the next proposition. Proposition 4. We have the estimate  −  E f  p ≤ A(p, j )f Lp , j L

1 < p < ∞,

504 and

CARBERY, WAINGER, AND WRIGHT

  Ej+ f Lp ≤ A(p, j )f Lp ,

1 < p < ∞.

We give the estimate for Ej− . The estimate for Ej+ is similar. For N ≥ 0 and 2 ≤ j ≤ r − 1, we define       ZjN = (p, q) ∈ Zj− | p mj +1 − mj − q nj − nj +1 = N . Z1N and ZrN are defined similarly. ZjN represents the points (p, q) in Zj− that are at a distance N from the upper edge of the triangle defined by Z(j ). The upper edge of the triangle defined by Z(j ) is the line in the positive quadrant that is orthogonal to Gj +1 − Gj . We let    EjN = Hp,q − Sp,q . (p,q)∈ZjN

Proposition 4 is a consequence of the following estimate, whose proof is postponed until the next section. Proposition 5. For 1 < p < ∞ and 1 ≤ j ≤ r,  N  E f  p ≤ A(j, p)2−σ (j,p)N f Lp j L for some σ (j, p) > 0. For (p, q) ∈ ZjN , we have an improvement over the estimate (3), which is used in the proof of Proposition 5. Lemma 6. For (p, q) in ZjN , (5)

µp,q (ξ, η, γ ) ≤ A|γ |2−σ N 2−pmj 2−qnj ,

for some σ > 0. We give the proofs for 2 ≤ j ≤ r − 1. The proofs for j = 1 and j = r are similar. Proof. From (1), we see that it suffices to show that for (u, v) in  with (u, v)  = (mj , nj ), (6)

2−pu 2−qv ≤ 2−σ N 2−pmj 2−qnj

for some σ > 0, which may depend on u, v, mj and nj . We divide the proof into a number of cases. Suppose first that u > mj . Then, since  is convex and contains (u, v), nj − v nj − nj +1 ≤ . u − mj mj +1 − mj

DOUBLE HILBERT TRANSFORMS ALONG SURFACES

505

So it follows that

 (u−mj )/(mj +1 −mj ) 2−pu 2−qv ≤ 2−N . −pm −qn j j 2 2 Next suppose u < mj . Then v is greater than nj . We wish to show that          q v − nj − p mj − u > σ p mj +1 − mj − q nj − nj +1 . Since  is convex, (v − nj ) ≥ λj −1 (mj − u). Also, p < qνj . Therefore, it suffices to show that      qλj −1 − p mj − u > σ qνj mj +1 − mj . Since we are allowing σ to depend on λj −1 , mj , u, νj , and mj +1 (but not on p and q), and since p/q < νj < λj −1 , this last inequality is now clear. Finally, we have the case u = mj . Then v > nj , and we wish to show        q v − nj > σ p mj +1 − mj − q nj − nj +1 . Since p < νj q, it is again clear that we can find σ . Thus the proof of Lemma 6 is complete. 4. The proof of Proposition 5. We prove Proposition 5 with a Littlewood-Paley N, j decomposition, which depends on N. Thus we construct functions ψ7 (ξ, η, γ ) so  N, j that 7 ψ7 (ξ, η, γ ) = 1 for (ξ, η, γ )  = 0 and so that for each ξ, η, γ , only a uniN, j formly bounded number of the ψ7  = 0. For convenience we drop the superscripts N and j . We then define U7 by  U 7 f (ξ, η, γ ) = ψ7 (ξ, η, γ )f(ξ ). Next we write

EjN f =







 Hp,q − Sp,q Uq+7 f,

7 (p,q)∈Z N j

and we put



V7 =



 Hp,q − Sp,q Uq+7 .

(p,q)∈ZjN

We prove three estimates for V7 : (7)

V7 f L2 ≤ A2−σ N f L2 ,

(8)

V7 f L2 ≤ A2−σ |7| f L2 ,

(9)

V7 f Lp ≤ A(|7| + N)f Lp ,

for any p, 1 < p ≤ 2. Given (7), (8), and (9), Proposition 5 follows from interpolation and duality in a standard manner.

506

CARBERY, WAINGER, AND WRIGHT

We would like to motivate the construction of the functions ψ7 . We begin by noting that there is a natural one-parameter family of dilations associated to EjN . In fact, if we envision dilations of the form   δ7 (ξ, η, γ ) = 2−σ1 7 ξ, 2−σ2 7 η, 2−σ3 7 γ , it would be natural to choose σ1 , σ2 , and σ3 so that the support of the model kernel  (p,q)∈Z N Sp,q is preserved. This kernel is supported in a region around the curve j

s ∼ 2−p , t ∼ 2−q , u ∼ 2−r , where p(mj +1 − mj ) − q(nj − nj +1 ) = 0 and r = pmj − qnj . So if we fix σ2 = 1, then we should have σ1 = and σ3 =

nj − nj +1 mj +1 − mj

(nj − nj +1 )mj nj mj +1 − nj +1 mj + nj = . mj +1 − mj mj +1 − mj

We then choose a C0∞ -function ψ supported in an annulus with center at the origin so that    ψ δ7 (ξ, η, γ ) = 1 7

away from the origin. We now wish to define BN (ξ, η, γ ) so that if δq BN (ξ, η, γ ) ∼ 1, then

2−p |ξ | + 2−q |η| + 2−pmj 2−qnj |γ | ∼ 1.

This can be achieved by defining   BN (ξ, η, γ ) = 2−(N/(mj +1 −mj )) ξ, η, 2−(mj N/(mj +1 −mj )) γ . Finally, define

  ψ7 (ξ, η, γ ) = ψ δ7+q BN (ξ, η, γ ) .

Now (3) and (4) imply (8), while (4) and (5) imply (7). This completes the proof for the L2 -boundedness of H . We now turn to the proof of (9). Note that V7 = V7,N is a convolution operator with kernel K7 = K7,N . To prove (9) it suffices to prove that K7,N satisfies a Hörmander condition with linear growth in 7, N . Namely, (10)
ρ(x,y,z)>Cρ(u,v,w)

K7,N (x + u, y + v, z + w) − K7,N (x, y, z) dx dy dz ≤ A(N + |7|)

DOUBLE HILBERT TRANSFORMS ALONG SURFACES

507

for a large C. Here, ρ(x, y, z) = |x|1/σ1 + |y|1/σ2 + |z|1/σ3 . Then ρ(δj (x, y, z)) = 2−j ρ(x, y, z). To prove (10), we split V7 into two parts: 

Hp,q

(p,q)∈ZjN

and



Sp,q ,

(p,q)∈ZjN

and we prove (10) for  the kernel of each part separately.That is, we prove (10) where K7,N is the kernel of (p,q)∈Z N Hp,q . The proof for (p,q)∈Z N Sp,q is similar and j j  is actually contained in the proof for (p,q)∈Z N Hp,q by changing the polynomial P . j Then

   p+q 2 Lq+7 x − s, y − t, z − P (s, t) φ(2p s, 2q t) ds dt, K7,N (x, y, z) = (p,q)∈ZjN

where   −1 (x, y, z) , L7 (x, y, z) = D(N, 7)L δ7−1 BN  = ψ, L and

 −1  . D(N, 7) = det δ7−1 BN

Thus the integral on the left-hand side of (10) is at most  I (p, q), (p,q)∈ZjN

where (after changing variables in s and t) I (p, q) = D(N, 7 + q)

×

   −1 −1 L δ7+q BN (x+u−s, y+v−t)z+w−P 2−p s, 2−q t ρ(x,y,z)≥Cρ(u,v,w)    −1 −1  − L δ7+q BN x − s, y − t, z − P 2−p s, 2−q t φ(s)φ(t) ds dt dx dy dz.

508

CARBERY, WAINGER, AND WRIGHT

We shall obtain three estimates on I (p, q): (11) (12)

I (p, q) ≤ A,  α I (p, q) ≤ A 2q+7 ρ(u, v, w) 1 2α2 N

for some α1 , α2 > 0, and (13)

I (p, q) ≤ A

1 2q+7 ρ(u, v, w)

if 2qβ ρ(u, v, w) ≥ 1 for an appropriate β > 0. We use estimate (11) when 2qβ ρ(u, v, w) ≤ 1 and 2q+7 ρ(u, v, w)α1 2α2 N ≥ 1, that is, for at most A(N + |7|) values of q. Otherwise, we use (12) or (13). Thus (11), (12), and (13) imply (10), proving Proposition 5. We turn to the proof of the estimates (11), (12), and (13). In the discussion of these estimates, we let x = (x, y, z) and u = (u, v, w). If we make the change of variables −1 −1 x  = δ7+q BN x,

the estimate (11) becomes clear. If we make the same change of variables and use the mean value theorem, we see that  −1 −1   −1  I (p, q) ≤ δ7+q BN u ≤ A2α2 N δ7+q u  α   −1  α1  ≤ A2α2 N ρ δ7+q u ≤ A2α2 N 27+q ρ u 1 for appropriate α1 and α2 , proving (12). To see (13), we estimate the contributions from the difference in the integrand −1 −1 separately. In each term, we make a change of variables x  = δ7+q BN x. Then the more complicated of the two terms becomes

φ(s)φ(t) ds dt ρ(δ7+q BN x)≥Cρ(u)

    L x + δ −1 B −1 u − δ −1 B −1 2−p s, 2−q t, P 2−p s, 2−q t dx 7+q N 7+q N

= φ(s)φ(t) ρ(BN x)>C27+q ρ(u)

      L x + δ −1 B −1 u − δ −1 B −1 2−p s, 2−q t, P 2−p s, 2−q t dx. 7+q N 7+q N Finally, we make a change of variables   −1 −1   −1 −1  −p x  = x + δ7+q BN u − δ7+q BN 2 s, 2−q t, P 2−p s, 2−q t , and we have to check that the region of integration in x  is contained in the region ρ(x  ) > C1 27+q ρ(u), for some constant C1 . Since each coordinate of BN x  is less than

DOUBLE HILBERT TRANSFORMS ALONG SURFACES

509

the corresponding coordinate of x  , it is enough to see that if ρ(BN x) > C27+q ρ(u), then ρ(BN x  ) > C1 27+q ρ(u). So it is enough to see that      ρ 2−p s, 2−q t, P 2−p s, 2−q t < ρ u . But ρ(2−p s, 2−q t, P (2−p s, 2−q t)) < A2−βq , so the proof of (13), and hence Proposition 5 and Proposition 1, is complete. 5. Completion of the proof of the main theorem. The sufficiency now follows directly from a special case of a general result of Ricci and Stein, which we now describe. Suppose for each pair of integer points I = (p, q) ∈ Z2 , we have an associated (I ) probability measure µ(I ) supported in the unit cube of R3 . Define the dilated µI of µ(I ) by


(I )

R3

f dµI =


R3

  f 2−p x, 2−q y, 2−pm 2−qn z dµ(I ) (x, y, z),

where m and n are fixed integers. The following proposition is a special case of [RS3, Theorem 5.1]. Proposition. Suppose  (I ) (ξ )| ≤ C(1 + |ξ |)− for some C,  > 0; (i) |µ  (ii) µ(I ) (λej ) = 0 for all λ ∈ R and 1 ≤ j ≤ 3, where {ej } form the canonical basis for R3 .  (I ) (ξ , 0, ξ ) ≡ 0 or In addition, if either m or n is zero, we further require µ 1 3  (I ) µ (0, ξ2 , ξ3 ) ≡ 0, respectively.  (I ) Then convolution with the kernel K = I ∈Z2 µI is bounded on Lp (R3 ), 1 < p < ∞. If we define µ(I ) f =





  ds dt f s, t, s mj t nj φ(s)φ(t) st

if I = (p, q) is in Z(j ) and otherwise µ(I ) = 0, then the boundedness of Q(j ) =



Sp,q

(p,q)∈Z(j )

follows directly from the above proposition. 6. The proof of the necessity of the main theorem. We suppose (k, 7) is a corner  point of the Newton diagram of P , P (s, t) = (m,n)∈ am,n s m t n , with both k and 7 odd. We wish to show that H is not bounded in L2 (R3 ). Without loss of generality,

510

CARBERY, WAINGER, AND WRIGHT

we may assume ak,7 = 1. Now Hf = µ ∗ f , where µ is a tempered distribution. If φ is a test function, then
 1  φ s, t, P (s, t) ds dt. µ(φ) = lim ,η→0 ≤|t|≤1 st η≤|s|≤1 We consider a family µδ of dilates of µ such that µδ → ν as δ → 0, where
 ds dt  , φ s, t, s k t 7 ν(φ) = lim ≤|t| ,η→0 st η≤|s| where the convergence is as distributions. If µ ∗ f were bounded in L2 , then µδ ∗ f would be uniformly bounded in L2 , and so f → ν ∗ f would be bounded in L2 . However, f → ν ∗ f is not bounded on L2 (see [RS3]). We commence with the details of µδ → ν. Since (k, 7) is a corner point of the Newton diagram of P , there are positive numbers a and b so that ak + b7 < am + bn for every (m, n) in  \ {(k, 7)}. We then define µδ (φ) = µ(φδ ), where   φδ (s, t, u) = δ −a δ −b δ −(ak+b7) φ δ −a s, δ −b t, δ −(ak+b7) u . So after a change of variables, we see

µδ (φ) = |s|≤1/δ a

where

  ds dt , φ s, t, Pδ (s, t) st |t|≤1/δ b

Pδ (s, t) = s k t 7 +



δ (m,n) am,n s m t n ,

(m,n)∈ (m,n)=(k,7)

with m,n = am + bn − ak − b7 > 0 for (m, n)  = (k, 7). Fix α < a and β < b to be small positive numbers to be determined later. Then µδ (φ) − ν(φ)

     ds dt φ s, t, Pδ (s, t) − φ s, t, s k t 7 = st |s|≤1/δ α |t|≤1/δ β

  ds dt   ds dt + φ s, t, Pδ (s, t) + 1/δα ≤|s|≤1/δa φ s, t, Pδ (s, t) |s|≤1/δ α st st |t|≤1/δ b 1/δ β ≤|t|≤1/δ b

   ds dt  ds ds + φ s, t, s k t 7 + |s|≤1/δα φ s, t, s k t 7 α st st |s|≥1/δ β |t|≥1/δ

= A(δ) + B(δ) + C(δ) + D(δ) + E(δ). Clearly D(δ) and E(δ) → 0 as δ → 0. We write A(δ) = A1 (δ) + A2 (δ) + A3 (δ) + A4 (δ), where the integrands in A1 , A2 , A3 , and A4 are the same as those in A, but the

511

DOUBLE HILBERT TRANSFORMS ALONG SURFACES

regions of integration in A1 , A2 , A3 , and A4 depend upon a large number σ , which in turn depends on α and β. In A1 the range of integration is δ σ ≤ |s| ≤ (1/δ)α , δ σ ≤ |t| ≤ (1/δ)β . In A2 we have |s| ≤ δ σ , δ σ ≤ |t| ≤ (1/δ)β . In A3 the range is δ σ ≤ |s| ≤ (1/δ)α , |t| ≤ δ σ . Finally, in A4 we have |s| ≤ δ σ , |t| ≤ δ σ . It is clear by the mean value theorem that A1 is dominated by CφC 1 δ η−d(α+β) log(1 + (1/δ)) where d is the degree of P and η = min m,n . This shows that A1 (δ) → 0 as δ → 0 if α and β are chosen small enough. To estimate A2 , A3 , and A4 , we do not try to use the difference, but rather the fact that we are integrating only over a small set. Consider, for example, the A4 case

  ds dt   ds dt φ s, t, P = lim (s, t) . φ s, t, Pδ (s, t) δ σ |s|≤δ σ ≤|s|≤δ →0 st st |t|≤δ σ ≤|t|≤δ σ We write Pδ (s, t) = Pδ1 (s, t) + Pδ2 (s) + Pδ3 (t) with |Pδ1 (s, t)| ≤ C(|s| |t|). Clearly we may replace Pδ by Pδ2 + Pδ3 . Then     φ s, t, Pδ2 (s) + Pδ3 (t) − φ s, 0, Pδ3 (s)
1  ∂   φ s, r1 t, Pδ2 (s) + r1 Pδ3 (t) dr1 = 0 ∂r1
1
1  ∂ ∂   = φ r2 s, r1 t, r2 Pδ2 (s) + r1 Pδ3 (t) dr1 dr2 ∂r ∂r 1 2 0 0
1  ∂   + φ 0, r1 t, r1 Pδ33 (t) dr1 . 0 ∂r1 Therefore,
lim

→0

≤|s|≤δ σ ≤|t|≤δ σ

  ds dt φ s, t, Pδ (s, t) st

1
1 dr1 dr2 = 0

0


|s|≤δ σ

|t|≤δ σ

E(s, t, r1 , r2 , δ)

ds dt , st

where |E(s, t, r1 , r2 , δ)| ≤ CφC 2 |st|. Thus the contribution to A4 from φ(s, t, Pδ (s, t)) is ᏻ(δ 2σ ). The contribution from φ(s, t, s k t 7 ) is simpler. The terms A2 and A3 are dealt with in a similar manner. To deal with B(δ), we replace φ(s, t, Pδ (s, t)) by
1      ∂   φ s, t, Pδ (s, t) − φ s, 0, Pδ (s, 0) = φ s, rt, Pδ (s, rt) dr. ∂r 0 Since |(∂/∂r)φ(s, rt, Pδ (s, rt))| ≤ C(|t| + |t|d )/((1+|s|)N ) for any N, |B(δ)| ≤ Cδ η for some positive η. A similar argument can be used to treat C(δ), and the argument is complete.

512

CARBERY, WAINGER, AND WRIGHT

7. The four-dimensional example. Here we wish to show that
   ds dt du Hf (x, y, z, w) = |s|≤1 f x −s, y −t, z−u, w− s 2 t 2 u4 +s 4 t 4 u2 +s 3 t 3 u3 |t|≤1 stu |u|≤1

defines an unbounded operator on L2 (R4 ). By considering the multiplier corresponding to H evaluated at points where the variables dual to (x, y, z) are zero, it suffices to show that
1
1
1  ds dt du  2 4 4 4 2 2 M(γ ) = eiγ [s t u +s t u ] sin γ s 3 t 3 u3 stu 0 0 0 is an unbounded function of γ . Let γ  1. After some changes of variables, we find


γ 1/6

M(γ ) =

dt t

0




t 0

dv v




γ 1/6 /v

eiv

6 [s 2 +s 4 ]

0

  ds . sin s 3 v 6 s

We wish to see that the contribution from 0 ≤ t ≤ 1 is bounded. By using the fact that | sin s 3 v 6 | ≤ s 3 v 6 , we see that the contribution to the s integral  from 0 ≤ s ≤ (v  /v 2 ) is at most v  , while an argument using van der Corput’s lemma  shows that the contribution from s > (v  /v 2 ) is dominated by v  . This shows that the contribution from 0 ≤ t ≤ 1 is bounded. So
γ 1/6
t
1/6     ds dt dv γ /v M(γ ) = + ᏻ(1). exp iv 6 s 2 + s 4 sin s 3 v 6 t 0 v 0 s 1 We now wish to show that we can change the interval of integration in s to [0, ∞]. Again using van der Corput’s lemma, we see that if we integrate for s > (γ 1/6 /v), the contribution would be ᏻ(v −2 γ −2/3 ). Thus for v > (1/γ )1/3 γ  , we can replace the range of integration by [0, ∞]. For v < (1/γ )1/3 γ  , we divide the interval s > (γ 1/6 )/v into two parts, s < γ δ /(v (3/2)+δ ) and s > γ δ /(v (3/2)+δ ). The contribution from s > γ δ /(v (3/2)+δ ) is at most 1 γ 4δ




γ 1/6

1

dt t




γ −(1/3)+

v 4δ dv = ᏻ(1).

0

The contribution from s < γ δ /(v (3/2)+δ ) is at most


γ 1/6 1

dt t




γ −(1/3)+

γ 3δ v (9/2)+3δ

0

v 6 dv = ᏻ(1)

if δ is small. Thus we see


γ 1/6

M(γ ) = 1

dt t




t 0

dv v




∞ 0

    ds + ᏻ(1). exp iv 6 s 2 + s 4 sin v 6 s 3 s

DOUBLE HILBERT TRANSFORMS ALONG SURFACES

513

Next we want to replace the v integral [0, t] by [0, ∞]. By using the fourth derivative test in van der Corput’s lemma, we see that we need to consider only small s. Then using the second derivative criterion, we see that we need only worry about s < 1/(v 5/2 ). Then we complete the reduction using the fact that | sin(v 6 s 3 )| ≤ v 6 s 3 . Thus 1 M(γ ) = A log γ + ᏻ(1), 6 where
∞
∞     ds dv A= . exp iv 6 s 2 + s 4 sin v 6 s 3 v s 0 0 Interchanging the order of integration and replacing v 6 by v, we see
∞
    dv ds ∞  1 . exp iv s 2 + s 4 + s 3 − exp iv s 2 + s 4 − s 3 A= 12i 0 s 0 v So



     dv 1 ∞ ds ∞  Im A = − cos v s 2 + s 4 + s 3 − cos v s 2 + s 4 − s 3 12 0 s 0 v  2 
∞ 4 3 s +s +s ds 1 ln 2 =− 12 0 s s + s4 − s3  = 0. References

[NW]

[PS] [RS1] [RS2] [RS3] [S]

[V]

L2 -boundedness

A. Nagel and S. Wainger, of Hilbert transforms along surfaces and convolution operators homogeneous with respect to a multiple parameter group, Amer. J. Math. 99 (1977), 761–785. D. H. Phong and E. M. Stein, The Newton polyhedron and oscillatory integral operators, Acta Math. 179 (1997), 105–152. F. Ricci and E. M. Stein, Harmonic analysis on nilpotent groups and singular integrals, I: Oscillatory integrals, J. Funct. Anal. 73 (1987), 179–194. , Harmonic analysis on nilpotent groups and singular integrals, II: Singular kernels supported on submanifolds, J. Funct. Anal. 78 (1988), 56–84. , Multiparameter singular integrals and maximal functions, Ann. Inst. Fourier (Grenoble) 42 (1992), 637–670. E. M. Stein, Harmonic Analysis: Real-Variable Methods, Orthogonality, and Oscillatory Integrals, Princeton Math. Ser. 43, Monogr. Harmon. Anal. 3, Princeton Univ. Press, Princeton, 1993. J. Vance, Lp -boundedness of the multiple Hilbert transform along a surface, Pacific J. Math. 108 (1983), 221–241.

Carbery: Department of Mathematics and Statistics, University of Edinburgh, Edinburgh EH9 3JZ, Scotland; [email protected] Wainger: Department of Mathematics, University of Wisconsin, Madison, Wisconsin 53706, USA; [email protected] Wright: School of Mathematics, University of New South Wales, Sydney, New South Wales 2052, Australia; [email protected]

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

ON THE SPECTRUM OF CERTAIN DISCRETE SCHRÖDINGER OPERATORS WITH QUASIPERIODIC POTENTIAL FLORIN P. BOCA Throughout this paper, we denote
2d =
2 (Zd ), d ≥ 1, with canonical orthonormal basis {u(n)}n , and L2d = L2 (Td ). The torus Td is freely identified with Rd /Zd = [0, 1)d . We set x = dist(x, Zd ), x ∈ Rd . The vector in Zd , whose kth component is zero if k  = j and one if k = j , is denoted by ej , 1 ≤ j ≤ d. Set also e = e1 + · · · + ed and 0 = (0, . . . , 0) ∈ Zd . In the sequel, we often refer to the following two functions: hz (θ ) = z −

d


2 cos 2πθj ,

j =1

G(z) = (2π )−d

θ = (θ1 , . . . , θd ) ∈ Rd , z ∈ C,

 Td

log |hz (θ)| dθ,

z ∈ C \ [−2d, 2d].

For each λ > 0, denote cλ (z) = G(z) − log λ and consider Pλ = {z ∈ C; cλ (z) = 0}

and

Cλ = {z ∈ C; cλ (z) ≥ 0} ∩ [−2d, 2d],

which are compact subsets of C. Notice (see [6]) that there exist constants b1 = a1 = 1 and bd = eG(2d) > ad = eG(0) > 0, d ≥ 2 (NB: The function G considered in this paper differs from the G defined in [6] by a translation of z by 2d), such that Sλ = Pλ ∪ Cλ looks like Sλ = Cλ =

−2d

Sλ = −2d

  *  Cλ

and 2d _ V) h p Pλ

Pλ = ∅

6

* 

if λ ≤ ad ,

2d

if ad < λ ≤ bd ,

 Cλ

Received 25 January 1999. 1991 Mathematics Subject Classification. Primary 47N50. Author’s research supported by an Engineering and Physical Sciences Research Council Advanced Fellowship and by a grant of the Romanian Academy. 515

516

FLORIN P. BOCA

Sλ =

−2d ·

· 2d

and

Cλ = ∅

if λ > bd .

For d = 1, Sλ coincides with the interval [−2, 2] if λ ≤ 1; Sλ coincides with the ellipse centered at the origin, with semiaxis λ + λ−1 along the x-axis and λ − λ−1 along the y-axis if λ > 1. Throughout this paper, α = (α1 , . . . , αd ) denotes a fixed d-tuple of real numbers and λ a fixed complex number. Consider the unitary operators U1 , . . . , Ud , V1 , . . . , Vd that act on
2d by Uj u(n) = u(n − ej ),

Vj u(n) = e2π inj αj u(n),

n = (n1 , . . . , nd ) ∈ Zd .

They satisfy the commutation relations Uj Uk = Uk Uj ,

V j Vk = V k Vj ,

Uj Vk = e2π iδj, k αj Vk Uj

(1)

for all 1 ≤ j , k ≤ d, where δj,k denotes the Kronecker symbol. Consider also the bounded operator Hα (λ) on
2d defined by Hα (λ) =

d
 j =1

 Uj + Uj∗ + λV1 · · · Vd ,

or equivalently, Hα (λ) u(n) =

d
     u n + ej + u n − ej + λe2π in,α u(n),

n ∈ Zd .

(2)

j =1

We recall that a d-tuple α = (α1 , . . . , αd ) ∈ Rd is called typically diophantine if there exist constants c, σ > 0 (depending on α) such that   −σ/2   
d
  d   nj αj  n2j  ≥c   j =1 j =1 for all (n1 , . . . , nd ) ∈ Zd \{0}. Although this set has full Lebesgue measure in Rd , it is strictly smaller than the set of irrational d-tuples. For example, Liouville numbers are not typically diophantine; recall that α ∈ R is called Liouville if there exist sequences of integers {pn }n , {qn }n , qn ≥ 1, gcd(pn , qn ) = 1, and a constant c > 0 such that, for all n,





α − pn ≤ c .

qn

nqn

DISCRETE SCHRÖDINGER WITH QUASIPERIODIC POTENTIAL

517

A d-tuple α ∈ Rd is called irrational if 1, α1 , . . . , αd are linearly independent over Q. The spectrum of Hα (λ) has been investigated by P. Sarnak ([6], see also the expository paper [7]), who proved the following theorem. Theorem (see [6]). Assume that α ∈ Rd is typically diophantine and λ ∈ C, |λ|  = ad . Then the following hold. (i) Sp(Hα (λ)) = S|λ| . (ii) Hα (λ) has no eigenvalues in C|λ| . (iii) Hα (λ) has eigenvalues that are dense on the lemniscate P|λ| if |λ| > ad . The proof of (ii) only uses the fact that α is irrational. Notice also that (iii) is not true for any irrational α ∈ Rd . For instance, if d = 1 and α ∈ R is a Liouville number, then Gordon’s argument (see [2]; see also [7, Thm. 7.3]) applies since V (n) = λe2π inα is a Gordon potential, showing that for any λ ∈ C, the operator Hα (λ) has no eigenvalues on
2 . The aim of this paper is to extend the first part of Sarnak’s result to any irrational α ∈ Rd . More precisely, we prove the following statement. Theorem 1. Assume that α ∈ Rd is irrational and λ ∈ C. Then   Sp Hα (λ) = S|λ| . To prove Theorem 1, we regard Hα (λ) as an element of the C ∗ -tensor product algebra Aα = Aα1 ⊗ · · · ⊗ Aαd and compute the spectrum of ρ(Hα (λ)), where ρ is an automorphism of Aα described subsequently. The inclusion Sp(Hα (λ)) ⊂ Sλ is proved by using a version of Weyl’s uniform distribution theorem (see [10]) contained in Lemma 2 below. For the converse inclusion, we compute the trace of the resolvent of Hα (λ) in Aα and use some properties of the Brown measure (see [1]). The next lemma is a version of Weyl’s uniform distribution theorem and is equivalent to the unique ergodicity of the rotation by (e2π iα1 , . . . , e2π iαd ) on Td (see, e.g., [4, Thm. 9.2]). Lemma 2. Let f ∈ C(Td ) and let α ∈ Rd be an irrational d-tuple. Then n

1
f (t + kα) = (2π)−d n→∞ n lim

k=1

 Td

f (x) dx

uniformly in t ∈ Rd . We recall some basic properties of rotation algebras. For each α ∈ R, the universal Aα is generated by two unitary operators u and v subject to the commutation relation uv = e2π iα vu. The algebra Aα is endowed with the canonical ergodic action π : T2 → Aut(Aα ) defined by πz1 ,z2 (u) = z1 u, πz1 ,z2 (v) = z2 v. It is also

C ∗ -algebra

518

FLORIN P. BOCA

endowed with the faithful tracial state τ : Aα → C, defined by  τ (a) = αg (a) dg, a ∈ Aα , T2

 which acts on finite sums of monomials by τ ( m,n am,n um v n ) = a0,0 , am,n ∈ C. We are particularly interested in the order-4 ∗-automorphism ρ0 of Aα that acts  on its generators by ρ0 (u) = v −1 , ρ0 (v) = u. This corresponds to γg0 , where g0 = 01 −1 0 , in −π ibdα ud v b , (Z) → Aut(A ), γ (u) = e the Brenken-Watatani representation γ : SL 2 θ g   γg (v) = e−π iacα uc v a for g = ac db ∈ SL2 (Z). For irrational α, the C ∗ -algebra Aα is simple (hence all its representations are faithful) and τ is the unique tracial state on Aα . Consider now an irrational α = (α1 , . . . , αd ) ∈ Rd and the universal C ∗ -algebras Aαj generated by unitaries uj and vj that satisfy uj vj = e2π iαj vj uj , 1 ≤ j ≤ d. By (1), we get faithful representations πj : Aαj → Ꮾ(
2d ) defined by πj (uj ) = Uj , πj (vj ) = Vj ; therefore a representation π : Aα = Aα1 ⊗max · · · ⊗max Aαd → Ꮾ(
2d ), acting as π(a1 ⊗ · · · ⊗ ad ) = π1 (a1 ) · · · πd (ad ), aj ∈ Aαj , where ⊗max denotes the maximal C ∗ -tensor product (see [9]). Since Aαj are nuclear C ∗ -algebras for all 1 ≤ j ≤ d, Aα coincides with the spatial C ∗ -tensor product Aα1 ⊗ · · · ⊗ Aαd . Moreover, each Aαj is simple; hence Aα is a simple C ∗ -algebra (see [9]) and, as a result, the representation π is faithful. We also get automorphisms ρ and πz,z , z = (z1 , . . . , zd ), z = (z1 , . . . , zd ) ∈ Td , of Aα defined by ρ(a1 ⊗ · · · ⊗ ad ) = ρ0 (a1 ) ⊗ · · · ⊗ ρ0 (ad ), respectively, by πz,z (a1 ⊗ · · · ⊗ ad ) = πz1 ,z1 (a1 ) ⊗ · · · ⊗ πzd ,zd (ad ), aj ∈ Aαj . For λ ∈ C and θ = (θ1 , . . . , θd ), t = (t1 , . . . , td ) ∈ Rd , we set hα (λ) =

d
j =1

α (λ, θ, t) = H

d
j =1

  I ⊗ · · · ⊗ uj + u∗j ⊗ · · · ⊗ I + λv1 ⊗ · · · ⊗ vd ∈ Aα ,



   e2π iθj Vj + e−2π iθj Vj∗ + λe2π i(t1 +···+td ) U1 · · · Ud ∈ Ꮾ
2d ,

  α (λ, θ ) = H α λ, θ, 0 , H

  α λ, 0 . α (λ) = H H

α (λ, θ ) acts on
2 by The operator H d α (λ, θ ) u(n) = λu(n − e) + H

d


  2 cos 2π θj + nj αj u(n),

n ∈ Zd .

(3)

j =1

Since ∗-automorphisms preserve spectrum, we get for all θ, t ∈ Rd ,           Sp Hα (λ) = Sp π hα (λ) = Sp hα (λ) = Sp πθ,t hα (λ)      α (λ, θ, t) = Sp πρπθ,t hα (λ) = Sp H     α (λ) . α (λ, θ) = Sp H = Sp H

(4)

DISCRETE SCHRÖDINGER WITH QUASIPERIODIC POTENTIAL

519

We notice for further use that τd : Aα → C defined by τd (a1 ⊗ · · · ⊗ ad ) = τ (a1 ) · · · τ (ad ), aj ∈ Aαj , 1 ≤ j ≤ d extends to a faithful tracial state on Aα and that τd πz,z = τd ρ = τd . Since α is irrational, τd is the unique tracial state on Aα . The C ∗ -algebras Aα and π(Aα ) are isomorphic since π is faithful, and hence π(Aα ) has also a unique tracial state τ0 and τ0 π = τd ; thus τ0 (U1m1 V1n1 · · · Udmd Vdnd ) = δm1 ,0 δn1 ,0 · · · δmd ,0 δnd ,0 , mj , nj ∈ Z. For each N ∈ N∗ , set FN = {(n1 , . . . , nd ) ∈ Zd ; |n1 |, . . . , |nd | ≤ N} and denote by PN the orthogonal projection from
2d onto
2 (FN ). Consider the states φN on Ꮾ(
2d ) defined by φN (x) =

Tr(PN xPN ) = (2N + 1)−d Tr(PN )




 xu(n), u(n) ,

|n1 |,...,|nd |≤N

  x ∈ Ꮾ
2d .

A direct computation provides 

 φN U1k1 V1l1 · · · Udkd Vdld = (2N + 1)−d δk1 ,0 · · · δkd ,0

d


e2π ilj rαj

j =1 |r|≤N

for all k1 , l1 , . . . , kd , ld ∈ Z, and thus d      lim φN U1k1 V1l1 · · · Udkd Vdld = δkj ,0 δlj ,0 = τ0 U1k1 V1l1 · · · Udkd Vdld

N→∞

j =1

and therefore limn→∞ φN (a) = τ0 (a) for all a in the subspace S spanned in π(Aα ) by the monomials U1k1 V1l1 · · · Udkd Vdld , kj , lj ∈ Z. Since S is norm dense in π(Aα ), it follows that {φN (a)}N is Cauchy and therefore


 τ0 (a) = lim φN (a) = lim (2N + 1)−d au(n), u(n) (5) N→∞

N →∞

|n1 |,...,|nd |≤N

for all a ∈ π(Aα ). Notice that equality (4) shows also Sp(hα (λ)) = Sp(hα (|λ|)). We assume from now on that λ ∈ [0, ∞). Furthermore, (4) shows that Sp(hα (λ)) ⊂ Sλ is equivalent to the existence of some α (λ, θ )) ⊂ Sλ . The last inclusion is a consequence of the next θ ∈ Rd such that Sp(H two propositions. α (λ, θ)− zI Proposition 3. Assume z ∈ C\Sλ . Then, for all θ ∈ Rd , the operator H is onto. / [−2d, 2d] or c < 0. Proof. Denote c = cλ (z). We have either c > 0 and z ∈   2 Let η = n ηn u(n) ∈
d . We prove the existence of ξ = n ξn u(n) ∈
2d such that α (λ, θ )ξ − zξ = η or, according to (3), H   d ηn z − j =1 2 cos 2π θj + nj αj ξn , n ∈ Z d . (6) + ξn+e = λ λ

520

FLORIN P. BOCA

Assume first c > 0 and z ∈ / [−2d, 2d]. Take
αn,m ηm , ξn = −

(7)

m

where αn,m =

    0

r k=0



z−

λr

d

j =1 2 cos 2π



   θ j + nj + k αj

if m − n = re, r ∈ N, otherwise.

The meaning of (7) is that ξ = −Xη, where X = (αn,m )n,m∈Zd . It is plain to check  that if X is a bounded operator on
2d , then ξ = n ξn u(n) ∈
2d and (ξn )n satisfy (6).  To prove that X is bounded, write X = r≥0 Xr , where Xr is obtained from X by replacing all entries αn,m for which m − n  = re, r ∈ N, by zero. Each Xr is bounded since Xr  = sup |αn,n+re | = sup r n

≤

n



λr



k=0

z−

λr

d

j =1 2 cos 2π



   θ j + nj + k α j

r .

dist z, [−2d, 2d]

Moreover, by Lemma 2, the sequence {r −1 log |αn,n+re |}r≥1 converges uniformly in n to −c < 0 when r → ∞. As a result, there exists r0 > 0 such that for all r ≥ r0 , one has Xr  = supn∈Zd |αn,n+re | ≤ e−cr/2 , showing that X is bounded. Finally, assume c < 0 and take
ξn = βn,m ηm , m

where

  λ−1      d r−1  
    −r βn,m = λ 2 cos 2π θj + nj − k αj z−   j =1  k=1   0 

if n − m = e, if n − m = re, r ≥ 2, otherwise.

If Y = (βn,m )n,m∈Zd is bounded on
2d , then ξ = n ξn u(n) ∈
2d . It is easy to check  that (ξn )n also satisfies (6). To prove that Y is bounded, write Y = r≥1 Yr , where Yr is obtained from Y by replacing all entries βn,m with n − m  = re, r ∈ N∗ , by zero. Each Yr is bounded since  r−1 |z| + 2d . Yr  = sup |βn,n−re | ≤ λr n

DISCRETE SCHRÖDINGER WITH QUASIPERIODIC POTENTIAL

521

If z ∈ / [−2d, 2d], Lemma 2 shows that {r −1 log |βn,n−re |}r≥1 converges uniformly in n to c < 0 when r → ∞. So, there exists r0 > 0 such that for all r ≥ r0 , one has Yr  ≤ ecr/2 for r ≥ r0 and Y is bounded. If z ∈ [−2d, 2d], choose a point z  = z such that Re z = Re z and c = G(z ) − log λ < 0. Since |z − t| ≤ |z − t|, t ∈ R, it follows that



r−1 d


    r − 1 1
1 log |βn,n−re | ≤ log λ log

z − 2 cos 2π θj + nj − k αj

− r r r k=1

j =1

for all n ∈ Zd and r ≥ 1. But, again by Lemma 2, the right-hand side tends uniformly in n to c < 0 when r → ∞; thus lim supr→∞ r −1 log |βn,n−re | < 0 and Y is bounded in this case too. Proposition 4. Assume z ∈ C \ Sλ . Then the following hold. α (λ) − zI is one-to-one. (i) If z ∈ / [−2d, 2d] and cλ (z)  = 0, the operator H α (λ, θ)− (ii) If z ∈ [−2d, 2d] and cλ (z)  = 0, there exists θ = θ(z) ∈ Rd such that H zI is one-to-one. Proof. For z ∈ / [−2d, 2d], Weyl’s uniform distribution theorem yields




r d

 

1
log

z − 2 cos 2π tj + sαj

r→∞ r lim

s=0

j =1



0 d


 

1


= lim log z − 2 cos 2π tj + sαj

= G(z) (8) r→−∞ |r| s=r j =1

for all t = (t1 , . . . , td ) ∈ Rd . For z ∈ [−2d, 2d], we use the Birkhoff ergodic theorem and the fact that a countable intersection of full Lebesgue measure sets has full Lebesgue measure to get the following for all k ∈ Zd and almost all θ ∈ Rd :



r d


 

1
log

z − 2 cos 2π θj + kj αj + sαj

r→∞ r lim

s=0

j =1




0 d



 1
= lim log

z − 2 cos 2π θj + kj αj + sαj

= G(z). (9) r→−∞ |r| s=r j =1

θ = 0 in case (i). In case (ii), take θ such that (9) holds and such that Put d d 2 j =1 cos 2π(θj + kj αj + sαj )  = z for all s ∈ Z and k ∈ Z .  α (λ, θ) − zI , then (3) provides If ξ = n ξn u(n) ∈
2d is an eigenvalue of H    z − dj =1 2 cos 2π θj + nj αj ξn+e = ξn , n ∈ Z d . (10) λ

522

FLORIN P. BOCA

Assume ξ  = 0; for instance, ξk  = 0 for some k ∈ Zd . By (10), it follows that ξk+re = βr ξk for all r ∈ Z, where   d r−1 
      −r   2 cos 2π θj + kj + s αj z− λ s=0 j =1 βr =  λr           −1 d s=r z − j =1 2 cos 2π θj + kj + s αj

if r ≥ 1, if r ≤ −1.

By (8), (9), and the choice of θ, one gets lim

r→∞

1 1 log |βr | = cλ (z) = − lim log |βr |. r→−∞ r |r|

When cλ (z) < 0, this implies limr→−∞ |βr | = ∞, and thus limr→−∞ |ξk+re | = ∞, which contradicts ξ ∈
2d ; while for cλ (z) > 0, we get limr→∞ |βr | = ∞ = limr→∞ |ξk+re |, which is again a contradiction. Corollary 5. For any irrational α ∈ Rd and any λ ∈ C, one has   Sp hα (λ) ⊂ Sλ . Proof. Let z ∈ C\Sλ . By Propositions 3 and 4, there exists θ = θ(z) ∈ Rd such that α (λ, θ )). Since the spectrum of H α (λ, θ) is independent of θ and coincides z∈ / Sp(H with the spectrum of hα (λ), we get the desired inclusion. The first step in the proof of the opposite inclusion consists in computing the trace of the resolvent of hα (λ). Proposition 6. Assume that α = (α1 , . . . , αd ) ∈ Rd is irrational and that λ > 0. Then, for all z ∈ C \ Sλ , one has   if cλ (z) < 0, 0   −1  = τd z − hα (λ) dθ  / [−2d, 2d]. if cλ (z) > 0 and z ∈ (2π )−d Td hz (θ) Proof. If z ∈ C \ Sλ , then c = cλ (z)  = 0 and z − hα (λ) is invertible. Notice that since τd = τd ρ = τ0 πρ, we have     −1  −1    α (λ) −1 . = τd ρ (z − hα λ) = τ0 z − H τd z − hα (λ) α (λ))−1 ) on C \ Sλ allows us to assume without The analyticity of z → τ0 ((z − H loss of generality that z ∈ / [−2d, 2d] in the case c < 0.

DISCRETE SCHRÖDINGER WITH QUASIPERIODIC POTENTIAL

523

α (λ))−1 with respect to the Let {αk,r }k,r∈Zd be the matrix coefficients of (z − H 2 orthonormal basis {u(n)}n of
d ; that is,
 α (λ) −1 u(k) = z−H αk,r u(r).



(11)

r

Set βr = z −

d

j =1 2 cos 2π rj αj ,

r ∈ Zd . Then (11) and

  α (λ) u(r) = βr u(r) − λu(r − e) z−H provide u(k) =




 
  αk,r βr u(r) − λu(r − e) = αk,r βr − λαk,r+e u(r),

r

r

which gives αk,r+e =

βr αk,r − δk,r , λ

k, r ∈ Zd ,

(12)

and therefore, for all k ∈ Zd and n ≥ 1, αk,k−ne =

λn αk,k . βk−e · · · βk−ne

(13)

Consider first the case c < 0 (hence implicitly λ > ad ); that is, z belongs to the “interior” of Pλ . Then (13) and Weyl’s theorem provide 1 log |αk,k−ne | = log λ − G(z) = −c > 0. n→∞ n lim

As a result, if we assume αk,k  = 0 for some k, then we get limn→∞ |αk,k−ne | = ∞, α (λ))−1 . We conclude that αk,k = 0, k ∈ Zd , contradicting the boundedness of (z − H which we combine with (5) to get 
  α (λ) −1 = lim (2N + 1)−d τ0 z − H αk,k = 0. N →∞

|n1 |,...,|nd |≤N

When c > 0, we use (13) to obtain the following for all k ∈ Zd and all n ≥ 1: αk,k+ne = λ−(n−1) βk+e · · · βk+ne αk,k+e .

(14)

Assume αk,k+e  = 0. Then (14) and Weyl’s theorem provide limn→∞ |αk,k+ne | = α (λ))−1 . Therefore αk,k+e = 0, ∞, which contradicts again the boundedness of (z− H k ∈ Zd , and by (12), αk,k = βk−1

524

FLORIN P. BOCA

for all k ∈ Zd , which together with (5) provides τ0



−1 

α (λ) z−H




= lim (2N + 1)−d N→∞

αk,k

|n1 |,...,|nd |≤N




= lim (2N + 1)−d N→∞

|n1 |,...,|nd |≤N

z−

d

(15)

1

j =1 2 cos 2πnj αj

.

Using the fact that the linear span of functions of form g(t1 , . . . , td ) = f1 (t1 ) · · · fd (td ), tj ∈ T, fj ∈ C(T), is uniformly dense in C(Td ) together with Weyl’s theorem, one gets 
−d −d f (n1 α1 , . . . , nd αd ) = (2π) f (t) dt lim (2N + 1) N→∞

Td

|n1 |,...,|nd |≤N

for all f ∈ C(Td ). Hence by (15), τ0

    α (λ) −1 = (2π)−d z−H = (2π)−d

d

Td

z−

Td

dt . hz (t)



dt

j =1 2 cos 2πtj

Corollary 7. If α ∈ Rd is irrational and λ > 0, then   Pλ ⊂ Sp hα (λ) . Proof. Assume that I ⊂ Pλ is an open arc such that I ∩ Sp(hα (λ)) = ∅. Since the function z → τd ((z − hα (λ))−1 ) is analytic on C \ Sp(hα (λ)), Proposition 6 implies that it is identically zero on C \ Sp(hα (λ)). In particular,  dθ =0 T hz (θ) for all z with |z| > 2d + λ, which is impossible. Determinants and Lidskii’s theorem play an important role in the spectral analysis of non-selfadjoint operators (see [5, XII.17]). It is therefore expected that the results of L. G. Brown on Lidskii’s theorem for type II von Neumann algebras (see [1]) could provide useful devices for such problems. Let M be a W ∗ -algebra endowed with a faithful, normal, semifinite trace τ , and let T ∈ M ∩L1 (M, τ ). According to [1], there exists a unique nonnegative measure µT on Sp(T ) \ {0} such that    τ log |1 − zT | = log |1 − zw| dµT (w) C

DISCRETE SCHRÖDINGER WITH QUASIPERIODIC POTENTIAL

525

for all z ∈ C (equality between subharmonic functions). By [1, Thm. 3.10], if f is holomorphic in a neighborhood of Sp(T ) ∪ {0} and f (0) = 0, then    τ f (T ) = f (ζ ) dµT (ζ ). (16) C

Moreover, if τ (1) = 1, then M ⊂ L1 (M, τ ) and µT (Sp(T ) \ {0}) = 1. If we also have 0 ∈ / Sp(T ), then µT (Sp(T )) = 1. We seek to apply this to the operator T = hα (λ) regarded as an element of the type II1 factor M = πτd (Aα ) , which contains the C ∗ -algebra Aα . Before completing the proof of the inclusion Cλ ⊂ Sp(hα (λ)) in Theorem 1, we make some remarks on certain probability measures on R. Consider the functions f, fd ∈ L1 (R) defined by f (t) =

√

1

π 4−t2

χ(−2,2) (t),

t ∈ R,

fd = f ∗ · · · ∗ f ,   

d ≥ 1,

d times

and the probability measures µ and µd on R with densities f and, respectively, fd , d ≥ 1. Set also  uj = I ⊗ · · · ⊗ uj ⊗ · · · ⊗ I,

d
  hd = hα (0) = u∗j ,  uj +

d ≥ 1.

j =1

The Brown measure of the selfadjoint operator hd ∈ Aα coincides with the probability measure  µd supported on R and defined by  µd (p) = τd (p(hd )), p ∈ C[X]. But τd is faithful, and hence supp( µd ) = Sp(hd ). On the other hand,  uj is the multiplication operator Mzj on L2d , and hence hd coincides with the multiplication operator by d 2 µd ) = Sp(hd ) = [−2d, 2d]. j =1 2 cos 2π xj on Ld and, as a result, supp( For d = 1, the equality τ ((u1 + u∗1 + λv1 )n ) = τ ((u1 + u∗1 )n ) yields 

   n  1 x n d µ1 (x) = τ hn1 = τ u1 + u∗1 = π −2 2



2π

(2 cos t)n dt =

0



xn dx √ −2 π 4 − x 2 2

for all n ∈ N. Therefore the Stone-Weierstrass theorem provides  µ1 = µ. For d ≥ 2, z1 + z¯ 1 , . . . , zd + z¯ d are independent random variables; hence  µd = µ d and, in particular, supp(µd ) = [−2d, 2d]. We need one more standard thing about the Cauchy transform of µd . The Cauchy transform  1 g(t) dt Ᏻ(z) = π R z−t of g ∈ L1 (R) is an analytic function on C \ supp(g), and one has Ᏻ(x ± iy) = v(x + iy) ∓ iu(x + iy)

526

FLORIN P. BOCA

for all x, y ∈ R, where  (x − t)g(t) 1 dt, v(x + iy) = π R (x − t)2 + y 2

1 u(x + iy) = π



yg(t) dt. 2 2 R (x − t) + y

It is well known (see, e.g., [8, p. 185]) that lim ε ! 0 u(x + iε) = g(x) and limε!0 v(x + iε) = 0 for almost all x ∈ R. As a result, one gets the Sohotski-Plemelj formula   2ig(x) = lim Ᏻ(x − iε) − Ᏻ(x + iε) , x a.e., ε!0

which shows that if I ⊂ R is an open interval and g(x)  = 0 for almost all x ∈ I , then Ᏻ cannot extend analytically from H+ = {z ∈ C ; Im z > 0} to H− = {z ∈ C ; Im z < 0} through I . Since fd (x) > 0 for almost all x ∈ (−2d, 2d), the Cauchy transform Ᏻµd of fd is analytic on C \ [−2d, 2d] and cannot extend analytically from H+ to H− through an arc from (−2d, 2d). The next proposition completes the proof of Theorem 1. Proposition 8. Assume that α ∈ Rd is irrational and λ ∈ [0, ∞). Then   Cλ ⊂ Sp hα (λ) . Moreover, if λ ≤ ad , the Brown measure µα,λ of hα (λ) coincides with µd . Proof. Assume first λ ≤ ad . In this case, Sp(hα (λ)) ⊂ [−2d, 2d]. Using (16) and the definition of τd and τ , we see that for all n ∈ N,  2d  2d      n n n n x dµα,λ (x) = τd hα (λ) = τd hα (0) = τd hd = x n dµd (x), (17) −2d

−2d

which we combine with the Stone-Weierstrass theorem to get µα,λ = µd ; hence     Cλ = [−2d, 2d] = supp(µα,λ ) ⊂ Sp hα (λ) = Sp Hα (λ) . To complete the proof, we compare the Cauchy transforms of µd and µα,λ . They coincide at infinity for all λ since for all z with |z| > 2d + λ, we can use (16) and τd (hα (λ)n ) = τd (hn0 ) to get  
  dµd (t)
−n−1 2d n = z t dµd (t) = z−n−1 τd hn0 π Ᏻµd (z) = −2d R z−t n≥0

=


n≥0

z

−n−1



τd hα (λ)

n≥0

n



 =

dµα,λ (ζ ) = π Ᏻµα,λ (z). |ζ |≤2d+λ z − ζ

The functions Ᏻµd and Ᏻµα,λ are analytic on {z ∈ C; cλ (z) > 0} \ Cλ and coincide on {z ∈ C; |z| > 2d + λ}; hence Ᏻµα,λ cannot extend analytically from H+ to H− through an (open) arc from Cλ . Since Ᏻµα,λ is analytic on C\Sp(hα (λ)), we conclude that Cλ ⊂ Sp(hα (λ)).

DISCRETE SCHRÖDINGER WITH QUASIPERIODIC POTENTIAL

527

For d = 1, the computation of the Brown measure of hα (λ) can be done explicitly for λ > 1 too. In this case, Jensen’s formula provides √



z + z2 − 4

, z ∈ C.

G(z) = log

2 The spectrum of hα (λ) is the ellipse Pλ = {a cos t + ib sin t ; t ∈ R}, a = λ + λ−1 , b = λ − λ−1 , which does not contain zero. Thus by (16),   −1  dµα,λ (ζ ) , z ∈ C \ Pλ . τ z − hα (λ) = z−ζ Pλ This is further equal by Proposition 6 to zero if cλ (z) < 0 and to   1 1 dθ dζ 1 =√ = 2 2π T z − 2 cos 2πθ 2πi |ζ |=1 ζ − zζ + 1 z2 − 4 if cλ (z) > 0. On the other hand, a direct computation gives   2π 0 if cλ (z) < 0, dt 1 1 = √ if cλ (z) > 0; 2π 0 z − a cos t − ib sin t z2 − 4 therefore,  2π   −1  1 1 dt f (ζ ) = dζ = τ z − hα (λ) 2π 0 z − a cos t − ib sin t 2πi Pλ z − ζ and dµα,λ (ζ ) =

1 f (ζ ) dζ 2πi

on Pλ ,

where

1 , t ∈ R. b cos t + ia sin t The inclusion Sλ ⊂ Sp(Hα (λ)) can also be proved using Sarnak’s theorem and results from [3] on the continuity of the field of C ∗ -algebras (Aα )α∈[0,1) . f (a cos t + ib sin t) =

Acknowledgments. I am grateful to R. Gologan for stimulating discussions. References [1]

[2] [3]

L. G. Brown, “Lidskii’s theorem in the type II case” in Geometric Methods in Operator Algebras (Kyoto, 1983), ed. H. Araki and E. G. Effros, Pitman Res. Notes Math. Ser. 123, Longman Sci. Tech., Harlow, 1986, 1–35. A. Ya. Gordon, On the point spectrum of the one-dimensional Schrödinger operator (in Russian), Uspekhi Mat. Nauk 31 (1976), no. 4, 257–258. U. Haagerup and M. Rørdam, Perturbations of the rotation C ∗ -algebras and of the Heisenberg commutation relation, Duke Math. J. 77 (1995), 627–656.

528 [4] [5] [6] [7] [8] [9] [10]

FLORIN P. BOCA R. Mañé, Ergodic Theory and Differentiable Dynamics, Ergeb. Math. Grenzgeb. (3) 8, Springer-Verlag, Berlin, 1987. M. Reed and B. Simon, Methods of Modern Mathematical Physics, IV: Analysis of Operators, Academic Press, New York, 1978. P. Sarnak, Spectral behavior of quasiperiodic potentials, Comm. Math. Phys. 84 (1982), 377– 401. B. Simon, Almost periodic Schrödinger operators: A review, Adv. in Appl. Math. 3 (1982), 463–490. E. M. Stein and G. Weiss, Introduction to Fourier Analysis on Euclidean Spaces, Princeton Math. Ser. 32, Princeton Univ. Press, Princeton, 1971. M. Takesaki, Theory of Operator Algebras, I, Springer-Verlag, New York, 1979. H. Weyl, Über die Gleichverteilung von Zahlen mod. Eins, Math. Ann. 77 (1916), 313–352.

School of Mathematics, Cardiff University, P.O. Box 926, Senghennydd Road, Cardiff CF2 4YH, United Kingdom; [email protected] On leave from: Institute of Mathematics of the Romanian Academy, P.O. Box 1-764, Bucharest 70700, Romania; [email protected]

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

ON THE STRUCTURE OF THE SELBERG CLASS, III: SARNAK’S RIGIDITY CONJECTURE J. KACZOROWSKI and A. PERELLI

1. Introduction. In this paper we deal with a problem, raised by Peter Sarnak, about rigidity in the Selberg class ᏿. Roughly speaking, the problem is as follows: Is every continuous one-parameter family Ᏺ = {F (s; ξ )}ξ ∈R of functions in ᏿ a shifted family, that is, k
  Fj s + ihj (ξ ) F (s; ξ ) = j =1

with Fj ∈ ᏿ and hj (ξ ) continuous? We need some preliminaries in order to give a precise formulation of the problem and our results. We refer to the survey paper [4] for the basic notation, definitions, and results about ᏿. Given a function F ∈ ᏿, we denote by an (F ) its nth coefficient and write  r   1 θF = Im 2 , µj − 2 j =1

where the µj appear in the -factors of a functional equation of F (s). We recall that the shift θF is an invariant of F (s) (see [4, Section 8]). Moreover, for every entire F ∈ ᏿ and every θ ∈ R, the shifted function Fθ (s) = F (s +iθ) belongs to ᏿. We also recall the Selberg orthonormality conjecture, asserting that if F, G ∈ ᏿ are primitive functions, then   ap (F )ap (G) 1 if F = G, = δF,G log log x+O(1) as x −→ ∞, where δF,G = p 0 if F  = G. p≤x We further recall that under Selberg orthonormality conjecture, ᏿ has unique factorization into primitive functions, the only primitive function with a pole at s = 1 is the Riemann zeta function ζ (s), and Fθ (s) is a primitive function if θ ∈ R and if F ∈ ᏿ are primitive and entire (see [4, Section 4]). We say a primitive function F ∈ ᏿ is normal if θF = 0. Assuming Selberg orthonormality conjecture, we normalize any primitive function F ∈ ᏿ in the following way: Received 26 May 1999. 1991 Mathematics Subject Classification. Primary 11M41; Secondary 11M99. Authors’ research partially supported by the State Committee for Scientific Research of Poland (KBN) grant number 2 PO3A 024 17. 529

530

KACZOROWSKI AND PERELLI

if F (s) has a pole at s = 1, then F˜ (s) = F (s) = ζ (s), already primitive and normal; if F (s) is entire, then F˜ (s) = F−θF /dF (s), primitive and normal. Here dF denotes the degree of F (s). We say a function F ∈ ᏿ is normal if it is a product of normal primitive functions. In view of the unique factorization, every F ∈ ᏿ can be normalized in a unique way as follows: if F (s) =

k


Fi (s), Fi (s) primitive, then F˜ (s) =

i=1

It is easy to see that

k


F˜i (s).

i=1

˜ F (s) ∼ G(s) ⇐⇒ F˜ (s) = G(s)

is an equivalence relation in ᏿, and we choose the set of normal functions F ∈ ᏿ as representatives of ᏿/ ∼. Another problem raised by Sarnak deals with the countability of ᏿ modulo shifts. We may state the problem by asking if (1.1)

᏿/ ∼ is countable.

Observe that in order to state (1.1), we essentially need to assume the Selberg orthonormality conjecture. Observe also that the simpler equivalence relation in ᏿, F (s) ≈ G(s) ⇐⇒ G(s) = Fθ (s) for some θ ∈ R, does not give rise to a countable quotient ᏿/ ≈. In fact, the functions ζ (s)L(s +iθ, χ), χ primitive Dirichlet character, are nonequivalent under ≈ for every θ ∈ R. However, if we restrict ≈ to the set ᏼ of all primitive functions in ᏿, then we expect that (1.2)

ᏼ/ ≈ is countable.

Assertions (1.1) and (1.2) are two forms of the countability conjecture for ᏿. Observe that (1.2) does not require the assumption of the Selberg orthonormality conjecture, and it is easy to see that the two forms of the countability conjecture are equivalent under the Selberg orthonormality conjecture. Hence, under the Selberg orthonormality conjecture, we refer indifferently to (1.1) or (1.2) as the countability conjecture. A one-parameter family Ᏺ = {F (s; ξ )}ξ ∈R of functions in ᏿ is called a continuous family if for every s0 ∈ C\{1}, the mapping ξ  −→ F (s0 ; ξ ) is continuous. A continuous family Ᏺ is called a shifted family if there exist primitive functions Fj ∈ ᏿ and continuous functions hj : R → R, j = 1, . . . , k, such that (1.3)

F (s; ξ ) =

k
j =1

  Fj s + ihj (ξ )

531

ON THE STRUCTURE OF THE SELBERG CLASS, III

for all (s, ξ ) ∈ C × R. Finally, we state the rigidity conjecture for ᏿: every continuous family is a shifted family. Sarnak remarked that the countability and rigidity conjectures are closely related, and in fact our first result shows that the rigidity conjecture is a consequence of the countability conjecture, under the Selberg orthonormality conjecture. Theorem 1. Assume the Selberg orthonormality conjecture and the countability conjecture. Then every continuous family is a shifted family. We remark that the Selberg orthonormality conjecture is used in a rather mild way in the proof of Theorem 1. In fact, it is clear from the proof that Theorem 1 still holds if we replace the Selberg orthonormality conjecture by its corollary, asserting that ᏿ has unique factorization and that the shifts of entire primitive functions are still primitive. In such a case, the countability conjecture is assumed in the form (1.2). Theorem 1 has been independently obtained by U. Vorhauer and E. Wirsing (see [7]). We remark that, in our approach, Theorem 1 is a consequence of Baire’s theorem in general topology, while in Vorhauer-Wirsing’s approach, it follows from a theorem of Sierpi´nski on a connectedness property of the real line. Now we turn to our main result. Consider a one-parameter family Ᏺ = {F (s; ξ )}ξ ∈ᐁ of Dirichlet series in the complex variable s, where ᐁ ⊃ R is a domain in C. We denote by an (ξ ) the nth coefficient of such Dirichlet series. The family Ᏺ is called an analytic family if (i) the mapping (s, ξ )  → F (s; ξ ) is holomorphic on (C \ {1}) × ᐁ; (ii) for every n ∈ N, the mapping ξ  → an (ξ ) is holomorphic on a domain ᐁn ⊃ R; (iii) for every ξ ∈ R, the function F (s; ξ ) belongs to ᏿. Observe that, in particular, the restriction to R of an analytic family is a continuous family. In the case of an analytic family Ᏺ, we expect that the functions hj (ξ ) in (1.3) can be chosen to be holomorphic on a suitable domain ᐂ ⊃ R. In this case, Ᏺ is called an analytic shifted family. We remark that the functions hj (ξ ) coming from Theorem 1 are, even in the case of an analytic family, not necessarily holomorphic. This can be seen considering the analytic family  Ᏺ = F (s + i sin ξ )F (s + i cos ξ ) ξ ∈C , F ∈ ᏿ entire and primitive, (1.4) where the functions hj (ξ ) coming from the proof of Theorem 1 are in fact h1 (ξ ) = max(sin ξ, cos ξ )

and

h2 (ξ ) = min(sin ξ, cos ξ ),

ξ ∈ R.

However, in accordance with our expectation, we have the following theorem. Theorem 2. Assume the Selberg orthonormality conjecture and the countability conjecture. Then every analytic family is an analytic shifted family.

532

KACZOROWSKI AND PERELLI

We remark that, by Corollary 4.1, we have hj (ξ ) ∈ R for ξ ∈ R. We also remark that the above domains ᐁ and ᐂ are not necessarily coincident. In fact, consider the family  



(1.5) F (s; ξ ) = F s + i ξ 2 + 1 F s − i ξ 2 + 1 where F ∈ ᏿ is entire. In this case, we can choose ᐁ = C, while the functions h1 (ξ ) and h2 (ξ ) have branch points at ξ = ±i. However, Theorem 2 excludes that such a possibility happens for ξ ∈ R. Examples (1.4) and (1.5) also give some indication of the difficulties one must face in order to prove Theorem 2. In fact, the proof of Theorem 2 is based on tools from complex analysis in several variables, and part of our analysis deals with problems coming from the branches of the involved functions. We refer to Section 2 for an outline of the proof of Theorem 2. We remark that the Selberg orthonormality conjecture is used in Theorem 2 more substantially than in Theorem 1 (see Lemma 5.1). We have been informed by Vorhauer and Wirsing that they now have a proof of Theorem 2 based on the theory of Riemann surfaces instead of complex analysis in several variables. We observe that Theorems 1 and 2 are in fact corollaries of the following more general result, which can be proved exactly in the same way. Let Ꮿ0 be a countable set of primitive functions of ᏿ and let Ꮿ denote the multiplicative semigroup generated by the functions in Ꮿ0 and all their shifts. We have the following theorem. Theorem 3. Suppose that the Selberg orthonormality conjecture holds in Ꮿ. Then every continuous (resp., analytic) family of functions in Ꮿ is a shifted (resp., analytic shifted) family. Theorem 3 enables us to obtain unconditional versions of Theorems 1 and 2 for certain subclasses of ᏿. For example, we may choose Ꮿ0 to be the set of all Dirichlet L-functions formed with primitive characters. More generally, Theorem 3 shows, essentially, that Theorems 1 and 2 hold unconditionally in the class of automorphic L-functions. In fact, the properties of the Rankin-Selberg convolution of automorphic L-functions can be used instead of Selberg orthonormality conjecture when dealing with such a class; we refer to Bump [1] for the required properties of automorphic L-functions. Let Ᏺ = {F (s; ξ )}ξ ∈R be a continuous family. In agreement with the rigidity conjecture, we expect that there exists a choice of the data of the functions F (s; ξ ) that is continuous in ξ . Therefore, the main invariants associated with F (s; ξ ), and in particular the degree d(ξ ), are expected to be continuous in ξ . This is, of course, the case under the hypotheses of Theorem 1. We shall deal with this and similar problems in a future paper. Given two continuous families Ᏺi = {Fi (s; ξ )}ξ ∈R , i = 1, 2, we define  Ᏺ1 Ᏺ2 = F1 (s; ξ )F2 (s; ξ ) ξ ∈R .

ON THE STRUCTURE OF THE SELBERG CLASS, III

533

Clearly, Ᏺ1 Ᏺ2 is a continuous family as well. In analogy with the definition of primitivity in ᏿, we say that a continuous family Ᏺ is primitive if Ᏺ = Ᏺ1 Ᏺ2 implies that Ᏺ1 or Ᏺ2 is the trivial family {1}ξ ∈R . The hypotheses of Theorem 1 imply the existence of a factorization into primitive families for every continuous family. However, (1.4) shows that such a factorization is not necessarily unique. In the case of analytic families, from Theorem 2, we have the following corollary. Corollary. Assume the Selberg orthonormality conjecture and the countability conjecture. Then every analytic family has unique factorization into primitive analytic families. We finally remark that, with the aim of deducing results like those of Theorems 1 and 2, variants of the definition of continuous and analytic family are certainly possible. In fact, the well-known formula for the nth coefficient of a Dirichlet series allows us to show that there is some freedom in the choice of such definitions. We only note, without discussing details, that such variants can be expressed, for example, in terms of uniform bounds for the functions F (s; ξ ), the coefficients an (ξ ), or for the abscissa of absolute convergence σ (ξ ). Acknowledgments. The authors wish to thank the KBN and the Dipartimento di Metodi e Modelli Matematici of the University of Genova for their generous support. 2. Outline of the proof of Theorem 2. Our starting point is the expression for F (s; ξ ) provided by Theorem 1, that is, the basic identity (1.3) for (s, ξ ) ∈ C × R, from which we get an identity of type (2.1)

ap (ξ ) =

k 

  νj ap Fj p −ihj (ξ )

for every prime p and ξ ∈ R,

j =1

where the νj are certain positive integers. We want to invert (2.1) and hence, for a given prime k-tuple p = (p1 , . . . , pk ), we consider the holomorphic function  : Ck → Ck given by   k k     −ixj   −ixj (x) =  ν j ap 1 F j p 1 , . . . , νj apk Fj pk  , x = (x1 , . . . , xk ). j =1

j =1

The Selberg orthonormality conjecture implies that the Jacobian determinant det J (x) of the function (x) does not vanish for some x = (h1 (ξ ), . . . , hk (ξ )) and some prime k-tuple p (see Lemma 5.1 and (5.5), (5.7)). Hence the inverse function theorem provides the holomorphic continuation of the functions hj (ξ ) to a certain domain Ᏸ (see Section 5). If Ᏸ ⊃ R, then Theorem 2 is proved; otherwise we may assume that ξ ∗ = sup ξ < ∞, ξ ∈Ᏸ∩R

534

KACZOROWSKI AND PERELLI

and our aim is to obtain the holomorphic continuation of the functions hj (ξ ) at ξ ∗ . Observe that the functions hj (ξ ), obtained by the above argument, do not necessarily coincide on Ᏸ ∩ R with the functions hj (ξ ) coming from Theorem 1 (see also (1.4)). However, it is easy to see that the limit of hj (ξ ) as ξ → ξ ∗− exists for every j = 1, . . . , k. We denote such a limit by xj∗ , define hj (ξ ∗ ) = xj∗ , and write x ∗ = (x1∗ , . . . , xk∗ ). It is therefore tempting to try to use the Selberg orthonormality conjecture again in order to show that there exists a prime k-tuple p such that det J (x ∗ )  = 0, thus obtaining the required holomorphic continuation of the functions hj (ξ ). However, the following example shows that in general this is not the case. Consider, in fact, the family given in (1.4). Then in this case, (2.1) becomes   ap (ξ ) = ap (F ) p −i sin ξ + p −i cos ξ , and hence it is easy to check that if ξ ∗ = π/4, then det J (x ∗ ) = 0

for every p = (p1 , p2 ).

The main part of the proof of Theorem 2 is therefore devoted to the analysis at x ∗ of the analytic set  J = x ∈ Ck : det J (x) = 0 . To this aim, we consider the holomorphic functions A : # → Ck and H : Ᏸ → Ck given by     and H (ξ ) = h1 (ξ ), . . . , hk (ξ ) , A(ξ ) = ap1 (ξ ), . . . , apk (ξ ) where # is a small disc at ξ ∗ , and write y ∗ = A(ξ ∗ ). Clearly    H (ξ ) = A(ξ ) for ξ ∈ # ∩ Ᏸ, H (ξ ∗ ) = x ∗ , and

(x ∗ ) = y ∗ .

By successive restrictions around x ∗ , y ∗ , and ξ ∗ , we construct two branched coverings |X∗ : X ∗ −→ Y ∗

and

A|#∗ : #∗ −→ Y ∗ ,

where x ∗ , y ∗ , and ξ ∗ are inner points of certain 1-dimensional irreducible analytic sets X ∗ , Y ∗ , and #∗ , respectively. Moreover, such coverings have the property that X ∗ ∩ J ⊂ {x ∗ } and Ᏸ ∩ #∗ is a nonempty open set. This construction depends on tools from complex analysis in several variables and on the unique factorization in the class ᏿∗ generated by all complex shifts of functions in ᏿ (see Sections 4 and 6). Observe that since we are interested in a local analysis at the three points ξ ∗ , x ∗ , and y ∗ , we are in fact dealing with germs of analytic sets, holomorphic functions, and coverings.

535

ON THE STRUCTURE OF THE SELBERG CLASS, III

Once the above situation is established, we get the analytic continuation of H (ξ ) as a multivalued function on #∗ \ {ξ ∗ } by means of      H γ (t) = )t A γ (t) , t ∈ [0, 1], where γ is any path in #∗ avoiding ξ ∗ , and )t (y) is the local inverse of the function (x) at the point A(γ (t)). Moreover, the unique factorization in ᏿∗ implies that ξ ∗ is an algebraic branch point of H (ξ ) (see Lemma 6.10). Finally, by the theory of uniformization, we prove that H (ξ ) has holomorphic continuation to #∗ as a onevalued function, and Theorem 2 follows (see Section 7). 3. Proof of Theorem 1. We prove Theorem 1 under the assumptions that ᏿ has unique factorization and the shifts of entire primitive functions are still primitive. Moreover, we assume the countability conjecture in the form of (1.2). ˜ the set of all finite We fix a set ᏼ0 of representatives of ᏼ/ ≈ and denote by ᏿ products of functions in ᏼ0 . Let Ᏺ = {F (s; ξ )}ξ ∈R be a continuous family, and for ˜, every function F ∈ ᏿ r
Fj (s), F (s) = j =1

let (3.1)

 Ꮽ(F ) = ξ ∈ R : F (s; ξ ) =

r








Fj s + iθj for some real θ1 , . . . , θr .

j =1

Clearly R=



Ꮽ(F ),

˜ F ∈᏿

˜ is countable. Hence, given any interval I , by and from our hypotheses we have that ᏿ ˜ and a nonempty interval I0 ⊂ I such Baire’s theorem there exist a function F0 ∈ ᏿ that   (3.2) Ꮽ F0 ⊃ I0 . If F0 (s) = 1 identically, then F (s; ξ ) = 1 identically for every ξ ∈ Ꮽ(F0 ), and hence by continuity we have F (s; ξ ) = 1 identically for every ξ ∈ Ꮽ(F0 ). Otherwise, writing k
Fj (s) with Fj ∈ ᏼ0 , F0 (s) = j =1

from (3.1) we deduce the existence of functions gj : Ꮽ(F0 ) → R such that F (s; ξ ) =

k
j =1

  Fj s + igj (ξ ) ,

  ξ ∈ Ꮽ F0 .

536

KACZOROWSKI AND PERELLI

Consider the partition P = J1 ∪ · · · ∪ Jr of {1, . . . , k}, with r minimal, such that Fi = Fj for i, j ∈ Jν and ν = 1, . . . , r. We rearrange the functions gj (ξ ) to form new functions   hj : Ꮽ F0 −→ R such that for each ν = 1, . . . , r, (hj (ξ ))j ∈Jν is a permutation of (gj (ξ ))j ∈Jν for every ξ ∈ Ꮽ(F0 ), and, moreover, hi (ξ ) ≤ hj (ξ ) for i < j , i, j ∈ Jν . Clearly the functions hj (ξ ) satisfy F (s; ξ ) =

(3.3)

k


  Fj s + ihj (ξ ) ,

  ξ ∈ Ꮽ F0 .

j =1

Lemma 3.1. Assume that ᏿ has unique factorization and that the shifts of entire primitive functions are primitive. Then Ꮽ(F0 ) is closed and the functions hj (ξ ), j = 1, . . . , k, are continuous. Proof. Let ξ0 ∈ Ꮽ(F0 ) and ξn → ξ0 with ξn ∈ Ꮽ(F0 ). We start by observing that hj (ξn ) is bounded for every j = 1, . . . , k. In fact, by the functional equation of the functions Fj (s) and the properties of the  function, we find a σ0 < 0 and a function K(t), satisfying K(t) ≥ c > 0 for all t ∈ R and K(t) → ∞ as |t| → ∞, such that Fj (σ0 +it) ≥ K(t) for j = 1, . . . , k. Since, by continuity, F (σ0 ; ξn ) is bounded, from (3.3) we therefore see that all hj (ξn ) are also bounded. Suppose that ξn and ξn are two subsequences of ξn such that     lim hj ξn = αj and lim hj ξn = βj , j = 1, . . . , k. n→∞

n→∞

Then, for any fixed s  = 1, by continuity we get k
j =1





Fj s + iαj = lim

n→∞

k
j =1

     Fj s + ihj ξn = lim F s; ξn n→∞

    = F s; ξ0 = lim F s; ξn n→∞

= lim

n→∞

k
j =1

k   
  Fj s + ihj ξn = Fj s + iβj . j =1

Hence by unique factorization there exists a permutation σ of the set {1, . . . , k} such that (3.4)

Fj (s) = Fσ (j ) (s)

and

αj = βσ (j ) .

From (3.4) we see that the partition P defined above is such that (αj )j ∈Jν is a permutation of (βj )j ∈Jν for ν = 1, . . . , r. But for i, j ∈ Jν , we have hi (ξn ) ≤ hj (ξn ) if i < j ; hence αi ≤ αj and βi ≤ βj if i < j . Therefore (3.5)

αj = βj ,

j = 1, . . . , k,

ON THE STRUCTURE OF THE SELBERG CLASS, III

537

and hence   lim hj ξn

(3.6)

n→∞

exists for every j = 1, . . . , k. Moreover, the same argument shows that such a limit does not depend on the sequence ξn . Now suppose that ξ0 ∈ Ꮽ(F0 ) \ Ꮽ(F0 ). Then we define     hj ξ0 = lim hj ξn , n→∞

and by continuity, for s  = 1 we have 





(3.7) F s; ξ0 = lim F s; ξn = lim n→∞

k




n→∞

k  
   Fj s + ihj ξn = Fj s + ihj ξ0 ,



j =1

j =1

which is a contradiction since (3.7) implies that ξ0 ∈ Ꮽ(F0 ). Therefore, Ꮽ(F0 ) is closed. Finally, let ξ0 ∈ Ꮽ(F0 ) and ξn → ξ0 with ξn ∈ Ꮽ(F0 ). Then, in view of (3.6), writing   lj = lim hj ξn , j = 1, . . . , k n→∞

and arguing as in (3.7), for s  = 1 we get k


       Fj s + ihj ξ0 = F s; ξ0 = lim F s; ξn n→∞

j =1

= lim

n→∞

=

k


k


   Fj s + ihj ξn

j =1

  Fj s + ilj .

j =1

Hence the argument leading to (3.5) shows in this case that lj = hj (ξ0 ) for j = 1, . . . , k; therefore, the functions hj (ξ ) are continuous and Lemma 3.1 is proved. From Lemma 3.1, (3.2), and (3.3), we immediately get the following proposition. Proposition 3.1. Assume that ᏿ has unique factorization and that the shifts of entire primitive functions are primitive. Then the countability conjecture in the form (1.2) implies that every nonempty interval I ⊂ R has a nonempty subinterval I0 such that (1.3) holds with continuous functions hj (ξ ) for ξ ∈ I0 . Let X denote the family of all intervals (α, β) ⊂ R satisfying the following:

538

KACZOROWSKI AND PERELLI

(i) there exist F1 , . . . , Fk ∈ ᏼ0 such that (1.3) holds with continuous functions hj (ξ ) for ξ ∈ (α, β); (ii) (α, β) is maximal with property (i). Clearly all intervals in X are disjoint. Let    X = R\ I= Ꮽ(F ) ∩ X . I ∈X

˜ F ∈᏿

Suppose X  = ∅. Then since X is a closed subset of R, it is a complete metric space with the metric induced by R. We apply Baire’s theorem to X in the same way as before, thus getting an interval ˜ such that (a, b) and a function F0 ∈ ᏿     ∅  = (a, b) ∩ X ⊂ Ꮽ F0 ∩ X = Ꮽ F0 ∩ X, since Ꮽ(F0 ) is closed by Lemma 3.1. Therefore, the same argument of Lemma 3.1 gives (3.8)

F (s, ξ ) =

k


  Fj s + ihj (ξ ) ,

ξ ∈ (a, b) ∩ X,

j =1

with Fj ∈ ᏼ0 and hj (ξ ) continuous on (a, b) ∩ X. By Proposition 3.1, there exists I0 ∈ X such that (a, b) ∩ I0  = ∅, and for ξ ∈ I0 we have (3.9)

F (s, ξ ) =

K


  Gj s + igj (ξ )

with Gj ∈ ᏼ0 and gj (ξ ) continuous.

j =1

Since (a, b) contains points from X, we can assume without loss of generality that (a, b) ∩ I0 = (α, β) with a < β < b, and from the maximality property of I0 , we have β ∈ X. Hence by (3.8) and (3.9), we have k K
 
  Fj s + ihj (β) = Gj s + igj (β) . j =1

j =1

By unique factorization we have k = K, and we can assume without loss of generality that Fj = Gj and hj (β) = gj (β) for j = 1, . . . , k. This means that formula (3.8) holds everywhere in (a, b), and hence (a, b) ⊂ I0 , which is a contradiction. Hence we have X = ∅, that is,  R= I. I ∈X

By compactness, every finite closed interval [a, b] ⊂ R has a finite cover I1 ∪· · ·∪IN by intervals Ij ∈ X. But such intervals are open and disjoint; hence N = 1 and Theorem 1 follows.

ON THE STRUCTURE OF THE SELBERG CLASS, III

539

4. The class ᏿∗ . We define ᏿∗ to be the multiplicative semigroup generated by all complex shifts of functions F ∈ ᏿. Hence every F ∈ ᏿∗ can be written as (4.1)

F (s) =

k


  Fj s + z j ,

Fj ∈ ᏿ and zj ∈ C.

j =1

᏿∗

in the same way as it is defined in ᏿. In view of the We define primitivity in factorization in ᏿, we may assume that all functions Fj (s) in (4.1) are primitive in ᏿. We have the following lemma. Lemma 4.1. Assume the Selberg orthonormality conjecture. Then a function in ᏿∗ is primitive in ᏿∗ if and only if it is a complex shift of a primitive function in ᏿. Proof. We start by observing that if F (s) is primitive in ᏿∗ , then k = 1 in (4.1), and hence F (s) is a complex shift of a primitive function in ᏿. Conversely, let F ∈ ᏿ be primitive in ᏿, z ∈ C and write F (s + z) as a product of primitive functions in ᏿∗ . In view of the above observation, shifting by −z we get (4.2)

F (s) =

k


  Fj s + z j ,

Fj ∈ ᏿ primitive in ᏿ and zj ∈ C.

j =1

We may clearly assume that zj = xj + iyj with x1 = · · · = xh < xh+1 ≤ · · · ≤ xk , and comparing pth coefficients in (4.2), we get (4.3)

  k  ap F j ap (F ) = . p zj j =1

Suppose that x1 < 0. Multiplying both sides of (4.3) by ap (F1 )pz1 −1 and summing for p ≤ x, we obtain       k   ap (F )ap F1  ap F j ap F 1 (4.4) = . 1+zj −z1 p 1−z1 p p≤x p≤x j =1 Since x1 < 0, in view of the Ramanujan conjecture axiom for the Selberg class and of the Selberg orthonormality conjecture, the left-hand side of (4.4) is bounded as x → ∞, and the right-hand side is " log log x, which is a contradiction. Now suppose that x1 > 0. Multiplying both sides of (4.3) by ap (F )p−1 and summing for p ≤ x, a similar argument leads again to a contradiction. Hence x1 = 0.

540

KACZOROWSKI AND PERELLI

Taking the square of the modulus of both sides of (4.3), dividing by p, and summing for p ≤ x we get (4.5)

 |ap (F )|2 p≤x

p

=

    k  k   ap Fj ap Fm j =1 m=1 p≤x

p 1+zj +zm

.

Arguing as before, the left-hand side of (4.5) is asymptotically equal to log log x and the right-hand side is greater than or equal to h log log x + O(1). Hence h = 1, and (4.4) becomes (4.6)

      ap (F )ap F1 p −iy1  ap F1 2 = + O(1) = log log x + O(1). p p p≤x p≤x

From (4.6) and the Selberg orthonormality conjecture, we have   F (s) = F1 s + iy1 ,

(4.7)

and hence k = 1 and Lemma 4.1 follows. The argument leading to (4.7) also proves the following lemma. Lemma 4.2. Assume the Selberg orthonormality conjecture and let F, G ∈ ᏿ be primitive. If F (s) = G(s + z), z ∈ C, then z = iθ with θ ∈ R. Now we turn to the main result of this section, which is the analogue of a wellknown result for ᏿. Proposition 4.1. Assume the Selberg orthonormality conjecture. Then ᏿∗ has unique factorization. Proof. Suppose that there exists a function in ᏿∗ with two distinct factorizations. Then by Lemma 4.1, there exist Fj , Gm ∈ ᏿ primitive in ᏿, and complex numbers zj = xj + iyj , wj = uj + ivj such that (4.8)

k


h  
  Fj s + z j = Gm s + w j

j =1

m=1

with Fj (s + zj )  = Gm (s + wm ) for every j, m. Moreover, in view of the argument in the proof of Lemma 4.1, we may assume that 0 = x 1 ≤ x 2 ≤ · · · ≤ xh

and

0 = u 1 ≤ u 2 ≤ · · · ≤ uk .

ON THE STRUCTURE OF THE SELBERG CLASS, III

541

Comparing coefficients in (4.8), multiplying by ap (F1 )pz1 −1 , and summing for p ≤ x we get         k  h    ap Fj ap F1 ap Gm ap F1 = . p 1+wm −z1 p 1+zj −z1 j =1 p≤x m=1 p≤x Arguing as in Lemma 4.1, we obtain that the left-hand side is greater than c log log x for some constant c > 0; hence there exists a 1 ≤ m0 ≤ k such that          ap Gm0 ap F1  ap Gm 0 ap F 1  . = log log x ∼ 1+um0 +i vm0 −y1 p 1+wm0 −z1 p≤x p≤x p Therefore, um0 = 0, and hence by the Selberg orthonormality conjecture we have     F1 s + z 1 = G m0 s + w m0 , which is a contradiction. Corollary 4.1. Assume the Selberg orthonormality conjecture and suppose that F (s) =

k


  Fj s + z j

j =1

with F, Fj ∈ ᏿ and zj ∈ C, j = 1, . . . , k. Then zj = iθj with θj ∈ R. Proof. We may assume that the functions Fj (s) are primitive in ᏿. By the factorization of F (s) into primitive functions in ᏿, we get (4.9)

k


h
  Fj s + zj = F (s) = Gm (s),

j =1

Gm ∈ ᏿ ,

m=1

say. By Lemma 4.1, (4.9) provides two factorizations of F (s) into primitive functions in ᏿∗ , and hence Corollary 4.1 follows from Proposition 4.1 and Lemma 4.2. From Corollary 4.1, we see that if a shift of a function in ᏿ is still in ᏿, then such a shift must be purely imaginary, of course under the Selberg orthonormality conjecture. 5. Holomorphic continuation to a domain. For a given integer N ≥ 1, let G1 , . . . , GN ∈ ᏿ be distinct primitive functions, let l1 , . . . , lN be positive integers, and let p = (p1 , . . . , pN ) be an N-tuple of distinct primes. Writing  N  D(G, p) = det lj apm Gj m,j =1 , we have the following lemma.

542

KACZOROWSKI AND PERELLI

Lemma 5.1. Assume the Selberg orthonormality conjecture. Then there exists a prime N-tuple p with D(G, p)  = 0. Proof. By contradiction, let r be the minimal value of the N’s such that D(G, p) = 0 for every prime N-tuple p. Then r ≥ 2 by a result of M. R. Murty and V. K. Murty on the strong multiplicity one for ᏿ (see [4, Proposition 4.1]). Denoting by
j =1

m=1

Multiplying by ap1 (G1 )/p1 and summing for p1 ≤ x, from the Selberg orthonormality conjecture we get       r r
  ap1 Gσ (1) ap1 G1
  0 =  lj  sgn σ apm Gσ (m) p1 p ≤x j =1



=

r


j =2

 =

r




lj   lj 

j =2

 =

r


 lj 

m=2

1

r  
  sgn σ δGσ (1) ,G1 log log x + O(1) apm Gσ (m)

 σ ∈
m=2



 sgn σ

σ ∈
r










r
j =2

 lj 











apm Gσ (m)

log log x + O(1)

m=2

sgn σ

σ ∈
j =2

Hence

σ ∈
r




apm Gσ (m)

log log x + O(1).

m=2



 sgn σ

σ ∈
r




apm Gσ (m)





= 0.

m=2

˜ p) ˜ = (G2 , . . . , Gr ) and p˜ = (p2 , . . . , pr ), which contra˜ = 0 with G Therefore, D(G, dicts the minimality of r. By Theorem 1 we may fix an interval I0  = ∅ such that for ξ ∈ I0 , (5.1)

F (s; ξ ) =

k
j =1

ν  Fj s + ihj (ξ ) j

ON THE STRUCTURE OF THE SELBERG CLASS, III

543

with νj ≥ 1 integers, hj (ξ ) continuous functions, and Fj (s) primitive and normal, j = 1, . . . , k. It is easy to see that we may assume, after possibly changing k and νj , shrinking the interval I0 , and renumbering the functions Fj (s) and hj (ξ ), that for every ξ ∈ I0 , (5.2)

hi (ξ )  = hj (ξ ) if Fi = Fj .

In fact, consider j = 1, 2 in (5.1); if F1 = F2 , then either (i) h1 (ξ ) = h2 (ξ ) for every ξ ∈ I0 or (ii) there exists ξ0 ∈ I0 with h1 (ξ0 )  = h2 (ξ0 ). In the first case, we simply write F1 (s + ih1 (ξ ))F2 (s + ih2 (ξ )) = F1 (s + ih1 (ξ ))2 , while in the second there exists a subinterval ∅  = I0 ⊂ I0 on which h1 (ξ )  = h2 (ξ ). Now apply the same argument to the new expression for F (s; ξ ), replacing I0 by I0 , and so on. The iteration stops after a finite number of steps, producing an expression of type (5.1) satisfying (5.2). Comparing pth coefficients in (5.1), for every ξ ∈ I0 we have (5.3)

ap (ξ ) =

k 

  νj ap Fj p −ihj (ξ ) .

j =1

Since the functions Fj (s) are primitive and normal, in view of (5.2) we may apply Lemma 5.1 for every fixed ξ ∈ I0 , with N = k, Gj (s) = Fj (s +ihj (ξ )), and lj = νj . Writing

  −ih (ξ ) k J (ξ, p) = det νj apm Fj pm j , m,j =1

we have J (ξ, p) = D(G, p), and hence from Lemma 5.1 we have that for every ξ ∈ I0 , there exists a prime k-tuple p(ξ ) such that   J ξ, p(ξ )  = 0. Now fix a point ξ0 ∈ I0 and write   (5.4) p = p ξ0 = (p1 , . . . , pk ). Moreover, with the choice in (5.4), write J (ξ ) = J (ξ, p). Since J (ξ0 )  = 0 and J (ξ ) is a continuous function, after possibly shrinking I0 we may assume that for every ξ ∈ I0 , (5.5)

J (ξ )  = 0.

Let the primes p1 , . . . , pk be as in (5.4) and, with abuse of notation, write     and am Fj = apm Fj , m = 1, . . . , k. am (ξ ) = apm (ξ )

544

KACZOROWSKI AND PERELLI

Hence from (5.3) we have that for ξ ∈ I0 , (5.6)

am (ξ ) =

k 

  −ih (ξ ) ν j am F j p m j ,

m = 1, . . . , k.

j =1

Now consider the holomorphic function  : Ck → Ck given by x = (x1 , . . . , xk )  −→ y = (y1 , . . . , yk )   k k     −ixj   −ixj νj a1 Fj p1 , . . . , νj ak Fj pk  . = j =1

j =1

The determinant det J (x) of the Jacobian matrix J of the function (x) is   ∂(y1 , . . . , yk ) det J (x) = det ∂(x1 , . . . , xk )  k 


  −ix k k log pm det νj am Fj pm j , = (−i) m,j =1

m=1

and hence, choosing x = x(ξ ) = (h1 (ξ ), . . . , hk (ξ )) with ξ ∈ I0 , we have  k 
(5.7) log pm J (ξ ). det J (x) = (−i)k m=1

Therefore, by (5.5) we get (5.8)

det J (x)  = 0

for x = x(ξ ) and ξ ∈ I0 . Moreover, for x = x(ξ ) and ξ ∈ I0 , from (5.6) we have   (x) = a1 (ξ ), . . . , ak (ξ ) . By (5.8) and the inverse function theorem, (x) is locally biholomorphic at any such point x = x(ξ ); that is, for every ξ ∈ I0 there exists a ball B(x) in Ck at (x), x = x(ξ ), on which the local inverse )(y) of the function (x) is defined and holomorphic. Now fix ξ0 ∈ I0 and a ball Bξ0 in C at ξ0 on which all functions am (ξ ), m = 1, . . . , k, are holomorphic and satisfy   a1 (ξ ), . . . , ak (ξ ) ∈ B(x(ξ0 )) for every ξ ∈ Bξ0 . Clearly, for ξ ∈ I0 ∩ Bξ0 we have     (5.9) h1 (ξ ), . . . , hk (ξ ) = ) a1 (ξ ), . . . , ak (ξ ) , and we define h1 (ξ ), . . . , hk (ξ ) by (5.9) for ξ ∈ Bξ0 \(I0 ∩ Bξ0 ). Therefore, such functions hj (ξ ) are holomorphic on Bξ0 and satisfy (5.6). Moreover, by analytic

ON THE STRUCTURE OF THE SELBERG CLASS, III

545

continuation, (5.1) is satisfied for every ξ ∈ Bξ0 . The above argument provides the holomorphic continuation of h1 (ξ ), . . . , hk (ξ ) as one-valued functions satisfying (5.1), and hence (5.6) as well, on a domain Ᏸ ⊃ I0 . Moreover, all functions am (ξ ), m = 1, . . . , k, are holomorphic on Ᏸ, Ᏸ ∩ R is an interval, and since hj (ξ ) ∈ R for ξ ∈ I0 , j = 1, . . . , k, then by the reflection principle we have hj (ξ ) ∈ R for ξ ∈ Ᏸ ∩ R, j = 1, . . . , k. We remark that the functions hj (ξ ) obtained by the above argument do not necessarily coincide for ξ ∈ Ᏸ ∩ R with the functions hj (ξ ) coming from Theorem 1 (see also (1.4)). If such domain Ᏸ already contains R, then Theorem 2 is proved. Otherwise, without loss of generality, we may assume that ξ ∗ = sup ξ < ∞. ξ ∈Ᏸ∩R

In the next sections we prove that the functions hj (ξ ), j = 1, . . . , k, have holomorphic continuation to some ball Bξ ∗ in C at ξ ∗ as one-valued functions satisfying (5.1), and Theorem 2 immediately follows. 6. Analytic sets and coverings. In this section we introduce certain analytic sets and coverings. We refer to Grauert-Fritzsche [2], Grauert-Remmert [3], Kaup-Kaup [5], and Remmert [6] for the basic definitions and results we need. We only recall the main general results we use in our concrete situation. We wish to thank Ettore Carletti and Giacomo Monti Bragadin for useful discussions about such results. Lemma 6.1 (Proper mapping theorem). Let Ꮽ be a connected analytic set and let ϕ : Ꮽ → Ck be a locally proper holomorphic mapping. Then ϕ(Ꮽ) is locally analytic in Ck and irreducible. Proof. See [6, Hilfsatz p. 355 and Satz 23]. Lemma 6.2. Let Ꮽ be an irreducible analytic set in a domain U ⊂ Ck with dimC Ꮽ ≥ 1 and let f (z) be a holomorphic function on U, not identically vanishing on Ꮽ. Then   dimC Ꮽ ∩ {z ∈ U : f (z) = 0} = dimC Ꮽ − 1. Proof. See [2, Theorem 6.18]. Lemma 6.3 (Open mapping theorem). Let Ꮽ and Ꮾ be irreducible d-dimensional analytic sets. Then every finite holomorphic mapping ϕ : Ꮽ → Ꮾ is open. Proof. See [3, p. 107]. Lemma 6.4. Let Ꮽ and Ꮾ be analytic sets. Then every surjective, open, and finite holomorphic mapping ϕ : Ꮽ → Ꮾ is a branched covering of Ꮾ.

546

KACZOROWSKI AND PERELLI

Proof. See [3, p. 137]. Coming back to the concrete situation of Section 5, let δ > 0 be such that all functions am (ξ ), m = 1, . . . , k, are holomorphic on the closure # of the disc # = #(δ) in C with center at ξ ∗ and radius δ. Moreover, with abuse of notation, we denote by Ᏸ = Ᏸ(δ) the connected component containing the interval (ξ ∗ − δ, ξ ∗ ) of the intersection of the domain Ᏸ of Section 5 with #(δ). We consider the holomorphic functions A : # −→ Ck

and

H : Ᏸ −→ Ck

  A(ξ ) = a1 (ξ ), . . . , ak (ξ )

and

  H (ξ ) = h1 (ξ ), . . . , hk (ξ ) .

defined by

Clearly Ᏸ and # are domains, Ᏸ ⊂ #, A(ξ ) is holomorphic on #, and for ξ ∈ Ᏸ we have   (6.1)  H (ξ ) = A(ξ ). Moreover, we write

y ∗ = A(ξ ∗ ).

We first observe that we may assume that A(ξ ) is nonconstant on #. In fact, if the am (ξ ) are all constant on #, then they are identically constant, and hence, by (5.9), the functions hj (ξ ) are constant on Ᏸ. Therefore, from (5.1) we have that F (s; ξ ) is constant on Ᏸ; hence it is a constant function of ξ and Theorem 2 follows in this case. We may therefore assume that a1 (ξ ) is nonconstant on #, and hence a1 (ξ ) is not identically vanishing on #. Moreover, since # is compact, the function a1 (ξ ) has a finite number of zeros in #, and the set A−1 ({y ∗ }) is finite. Hence we may choose δ > 0 in such a way that (6.2)

{ξ ∈ # : a1 (ξ ) = 0} ⊂ {ξ ∗ }

and

A−1 ({y ∗ }) = {ξ ∗ }.

We have the following lemma. Lemma 6.5. There exists a δ0 > 0 such that, writing #0 = #(δ0 ), the holomorphic mapping   A|#0 : #0 −→ A #0 is a q-fold branched covering with q ≥ 1 and branch point at most at ξ ∗ . Proof. This is a well-known result in view of the above properties of the function A(ξ ) and of the domain # (see, for example, [5, pp. 130–134]). Clearly q − 1 is the minimum order of zero of the functions aj (ξ ) at ξ ∗ , j = 1, . . . , k.

ON THE STRUCTURE OF THE SELBERG CLASS, III

We write

  Y0 = A # 0

547

 

Ᏸ0 = Ᏸ δ0 .

and

Therefore, in particular, ξ ∗ and y ∗ are inner points of #0 and Y0 , respectively, Ᏸ0 ⊂ #0 is a nonempty domain, #0 \ {ξ ∗ } is arcwise connected, and A(#0 \ #0 ) ∩ A(#0 ) = ∅. We have the following lemma. Lemma 6.6. Y0 is an irreducible analytic set in a domain V ⊂ Ck and dimC Y0 = 1. Proof. We first show that Y0 is locally analytic in Ck and irreducible. If aj (ξ ∗ )  = 0 for some j = 1, . . . , k, then the holomorphic mapping (6.3)

A|#0 : #0 −→ Ck

has Jacobian matrix with rank = 1 at every point of #0 . Hence Y0 is clearly locally analytic in Ck and, moreover, Y0 is irreducible by the lemma quoted in the proof of Lemma 6.1. Otherwise, the argument is more involved, and we use Lemma 6.5 to prove that the holomorphic mapping (6.3) is locally proper. In fact, let y 0 ∈ Y0 , y 0  = y ∗ , write (6.4)

A−1 ({y 0 }) = {ξ1 , . . . , ξq },

and let Bξj be a sufficiently small disc in C at ξj , B ξj ⊂ #0 \ {ξ ∗ }, j = 1, . . . , q. Suppose that every compact neighborhood Ky 0 of y 0 in Y0 is such that A

−1





Ky 0  ⊂

q 

Bξ j .

j =1

q Then, by a simple compactness argument, there exist a sequence ξn ∈ #0 \( j =1 Bξj ) q and a point ξ0 ∈ #0 \ ( j =1 Bξj ) such that ξn → ξ0 and A(ξn ) → y 0 . Hence     A ξ0 = lim A ξn = y 0 , n→∞

which is a contradiction in view of (6.4) and of the properties of the covering A|#0 : #0 → Y0 . Hence there exists a compact neighborhood Ky 0 of y 0 in Y0 such that q    A−1 Ky 0 ⊂ Bξ j . j =1

Since Ky 0 is closed, A−1 (Ky 0 ) is a closed bounded set inside an open set and therefore it is compact. The same argument can be applied to y ∗ as well, even in a simplified way, since ∗ y has a unique counterimage in #0 . Therefore, the holomorphic mapping (6.3) is

548

KACZOROWSKI AND PERELLI

locally proper, and hence Lemma 6.1 implies that Y0 is locally analytic in Ck and irreducible. Since Y0 is locally analytic in Ck , for every y 0 ∈ Y0 there exist a ball By 0 in Ck at y 0 and holomorphic functions fj (y), j = 1, . . . , l, on By 0 such that Y0 ∩ By 0 = {y ∈ By 0 : fj (y) = 0, j = 1, . . . , l}. Therefore, writing



V =

By ,

y∈Y0

we see that V is a domain in Ck since Y0 is connected. Hence Y0 is an analytic set in V . Finally, from (6.2), we have that the Jacobian matrix JA (ξ0 ) = (a1 (ξ0 ), . . . , ak (ξ0 )) of the function A(ξ ) has rank JA (ξ0 ) = 1 at every point ξ0  = ξ ∗ . Hence dimC Y0 = 1, and Lemma 6.6 follows. In order to define H (ξ ∗ ), we observe that although the functions hj (ξ ) obtained in Section 5 do not necessarily coincide for ξ ∈ Ᏸ ∩ R with the functions hj (ξ ) coming from Theorem 1, we still have that lim H (ξ )

(6.5)

ξ →ξ ∗− ξ ∈R

exists. In fact, denoting here by gj (ξ ) the functions hj (ξ ) coming from Theorem 1, by unique factorization the values hj (ξ ) are a permutation of the values gj (ξ ) for every ξ ∈ Ᏸ ∩R. Moreover, the gj (ξ )’s are continuous on a neighborhood of ξ ∗ , and the hj (ξ )’s are continuous on a left neighborhood of ξ ∗ . Therefore, the limit (6.5) exists. In view of (6.5), we define H (ξ ∗ ) = lim H (ξ ) = x ∗ ,

(6.6)

ξ →ξ ∗− ξ ∈R

and hence (6.1) extends to ξ ∗ as well, that is, (x ∗ ) = y ∗ . In fact, by (6.1) and (6.6), we have  

  (x ∗ ) =   lim H (ξ ) = lim  H (ξ ) = lim A(ξ ) = A(ξ ∗ ) = y ∗ . ξ →ξ ∗− ξ ∈R

ξ →ξ ∗− ξ ∈R

ξ →ξ ∗− ξ ∈R

Now consider the analytic set in Ck  J = x ∈ Ck : det J (x) = 0 .

ON THE STRUCTURE OF THE SELBERG CLASS, III

549

Since J (ξ ) is holomorphic and not identically vanishing on Ᏸ0 , we have that  (6.7) D0 = ξ ∈ Ᏸ0 : J (ξ ) = 0 is a discrete subset of Ᏸ0 . Therefore, in particular, dimC J = k − 1 in view of (5.7) and (6.7). Moreover, we may assume that Ᏸ0 \ D0 is still connected. In view of (6.1) and since H (Ᏸ0 ) is connected, we denote by U the connected component of −1 (V ) containing H (Ᏸ0 ), where V is the domain in Lemma 6.6, and consider the set   X = −1 Y0 ∩ U. Observe that U is a domain, X ⊃ H (Ᏸ0 ), and X is an analytic set in U . In fact, if f1 (y), . . . , fl (y) define locally Y0 at y 0 , then   gj (x) = fj (x) , j = 1, . . . , l, define locally X at a point x 0 ∈ −1 ({y 0 }). Moreover, the Jacobian matrix Jg (x) of the functions gj (x), j = 1, . . . , l, is   (6.8) Jg (x) = Jf (x) J (x), where Jf (y) is the Jacobian matrix of the functions fj (y), j = 1, . . . , l. In view of (5.7), (6.7), (6.8), and Lemma 6.6, we have that (6.9)

rank Jg (x) = k − 1

on H (Ᏸ0 \ D0 ), and hence H (Ᏸ0 \ D0 ) is a connected subset of the set of regular points of X. Therefore, there exists an irreducible component of X containing H (Ᏸ0 ). We denote by X0 such an irreducible component and observe that x ∗ ∈ X0 since, by construction, x ∗ lies in the same connected component of H (Ᏸ0 ). Moreover, both X0 and X0 \ {x ∗ } are arcwise connected, and x ∗ is an inner point of X0 . Therefore, the above argument and (6.9) prove the following lemma. Lemma 6.7. Assume the Selberg orthonormality conjecture. Then X0 is an irreducible analytic set in U with dimC X0 = 1. Moreover, X0 ⊃ H (Ᏸ0 ), X0 \ {x ∗ } is arcwise connected, and x ∗ is an inner point of X0 . Consider further the analytic subset of X0   D1 = X0 ∩ J ∪ −1 ({y ∗ }) . Lemma 6.8. Assume the Selberg orthonormality conjecture. Then D1 is a discrete subset of X0 . Proof. By (5.7), we have that det J (x) is not identically vanishing on X0 . Hence from Lemmas 6.2 and 6.7, we get (6.10)

dimC (X0 ∩ J ) = dimC X0 − 1 = 0.

550

KACZOROWSKI AND PERELLI

By Lemma 6.7, dimC D1 is either 0 or 1. If dimC D1 = 1, then by (6.10), we have   dimC X0 ∩ −1 ({y ∗ }) = 1, and hence (x) = y ∗ for every x ∈ X0 since X0 is irreducible. Therefore, by (6.1), the function A(ξ ) is constant on Ᏸ0 , which is a contradiction. Hence dimC D1 = 0, and D1 is therefore a discrete subset of X0 . In view of Lemma 6.8, there exists a ball Bx ∗ in Ck at x ∗ such that    Bx ∗ ∩ X0 ∩ J ∪ −1 ({y ∗ }) = {x ∗ }, that is, X0 ∩ Bx ∗ is singular at most at x ∗ and contains no counterimages of y ∗ other than x ∗ itself. We write X ∗ = X 0 ∩ Bx ∗ and observe that by Lemma 6.7, the ball Bx ∗ can be chosen in such a way that the following lemma holds. Lemma 6.9. Assume the Selberg orthonormality conjecture. Then X ∗ is an irreducible analytic set in U ∗ = U ∩ Bx ∗ with dimC X ∗ = 1. Moreover, X∗ ∩ J ⊂ {x ∗ }, X ∗ ∩ H (Ᏸ0 )  = ∅, X ∗ \ {x ∗ } is arcwise connected, and x ∗ is an inner point of X ∗ . An important property of the holomorphic mapping |X∗ : X ∗ → Y0 is given by the following. Lemma 6.10. Assume the Selberg orthonormality conjecture. Then each fiber of the holomorphic mapping |X∗ : X ∗ −→ Y0 has at most q(k!) elements. Proof. Fix a point ξ0 ∈ Ᏸ0 with H (ξ0 ) ∈ X ∗ and write     x 0 = H ξ0 and y0 =  x0 . Let x 1 ∈ X∗ , x 1  = x ∗ . Since X ∗ \ {x ∗ } is arcwise connected, there exists a path  : [0, 1] → X ∗ from x 0 to x 1 avoiding x ∗ . Hence g : [0, 1] → (X ∗ ) given by   (6.11) g(t) =  (t) is a path in (X ∗ ) from y 0 to y 1 = (x 1 ), avoiding y ∗ . Since y ∗ ∈ / g([0, 1]), by Lemma 6.5 the path g[0,1] can be lifted to #0 in a unique way to a path γ : [0, 1] → #0 from ξ0 to a certain point ξ1 ∈ A−1 ({y 1 }) and avoiding ξ ∗ . In particular, we have   (6.12) A γ (t) = g(t). Since the Jacobian determinant of the function A(ξ ) has rank equal to 1 at every point of γ ([0, 1]) and (x) is locally biholomorphic at any point of ([0, 1]), by (6.11)

ON THE STRUCTURE OF THE SELBERG CLASS, III

551

and (6.12) we obtain the holomorphic continuation of H (ξ ) along the path γ[0,1] by means of      H γ (t) = )t A γ (t) , t ∈ [0, 1], where the function )t (y) is the local inverse of the function (x) at the point A(γ (t)). Clearly, we have   H ξ1 = x 1 . (6.13) Moreover, since for ξ ∈ Ᏸ0 we have (6.14)

F (s; ξ ) =

k


ν  Fj s + ihj (ξ ) j ,

j =1

and Ᏸ0 ∩ A−1 ((X ∗ )) contains an open set, by analytic continuation (6.14) holds on γ ([0, 1]), where the functions hj (ξ ), j = 1, . . . , k, are defined on γ ([0, 1]) by        h1 γ (t) , . . . , hk γ (t) = H γ (t) , t ∈ [0, 1]. Now let y ∈ (X ∗ ) and let x ∈ −1 ({y}). Since the fiber of |X∗ at y ∗ consists of the single element x ∗ , we may assume that y  = y ∗ . Moreover, A−1 ({y}) consists exactly of q elements. Write x = (α1 , . . . , αk ) and consider a path [0,1] from x 0 to x, avoiding x ∗ . By the above construction, we get a point ξ ∈ A−1 ({y}) satisfying, by (6.13) and (6.14), (6.15)

F (s; ξ ) =

k


 ν Fj s + iαj j .

j =1

By Proposition 4.1, for every fixed ξ ∈ A−1 ({y}), we have that if (6.15) holds with (α1 , . . . , αk ) replaced by (β1 , . . . , βk ), then (β1 , . . . , βk ) is a permutation of (α1 , . . . , αk ). Hence there are at most k! points x ∈ −1 ({y}) for which the above construction leads to the same point ξ ∈ A−1 ({y}). Therefore, there are at most q(k!) points in −1 ({y}), and Lemma 6.10 is proved. The last lemma of this section is the following. Lemma 6.11. Assume the Selberg orthonormality conjecture. Then the holomorphic mapping |X∗ : X ∗ −→ (X∗ ) is a q -fold branched covering with 1 ≤ q ≤ q(k!) and branch point at most at x ∗ . Proof. In view of Lemmas 6.6, 6.9, and 6.10, from Lemma 6.3 we have that the holomorphic mapping |X∗ : X ∗ → Y0 is open. Therefore, Lemma 6.11 follows from Lemmas 6.4 and 6.10.

552

KACZOROWSKI AND PERELLI

Write Y ∗ = (X ∗ ),

#∗ = A−1 (Y ∗ ),

and

Ᏸ∗ = Ᏸ0 ∩ #∗ .

Observe that #∗ is an open neighborhood of ξ ∗ and we may assume that #∗ \ {ξ ∗ } is arcwise connected. Moreover, Y ∗ is an irreducible analytic set containing y ∗ as an inner point, and Ᏸ∗ is open and contains an interval of the form (ξ ∗ − ε, ξ ∗ ), with some ε > 0. We summarize the basic results obtained in this section by the following proposition. Proposition 6.1. Assume the Selberg orthonormality conjecture. Then there exist an open neighborhood #∗ of ξ ∗ and 1-dimensional irreducible analytic sets X ∗ and Y ∗ such that the holomorphic mappings A|#∗ : #∗ −→ Y ∗

and

|X∗ : X ∗ −→ Y ∗

are a q-fold and a q -fold branched covering with q, q ≥ 1 and branch point at most at ξ ∗ and x ∗ , respectively. Moreover, x ∗ and y ∗ are inner points of X ∗ and Y ∗ , respectively, X ∗ ∩ J ⊂ {x ∗ }, #∗ \ {ξ ∗ } is arcwise connected, and Ᏸ∗ is an open set containing an interval of type (ξ ∗ − ε, ξ ∗ ). 7. Proof of Theorem 2 and of Corollary. We first provide the holomorphic continuation of H (ξ ) to #∗ \ {ξ ∗ } along paths. Fix a point ξ0 ∈ (ξ ∗ − ε, ξ ∗ ) and write     and x 0 = H ξ0 . y 0 = A ξ0 Given a path γ : [0, 1] → #∗ starting at ξ0 and avoiding ξ ∗ , we write ξ1 = γ (1) and consider the path g : [0, 1] → Y ∗ defined by   g(t) = A γ (t) , t ∈ [0, 1]. Observe that g[0,1] is a path from y 0 to y 1 = A(ξ1 ) and avoids y ∗ . Hence we can lift in a unique way g[0,1] to a path  : [0, 1] → X∗ from x 0 to a certain point x 1 ∈ −1 ({y 1 }), avoiding x ∗ and satisfying    (t) = g(t), t ∈ [0, 1]. Moreover, for every t ∈ [0, 1], there exists a ball Bg(t) in Ck at g(t) such that the local inverse )t (y) of the function (x) is defined and holomorphic on Bg(t) . Observe that the above construction is justified by Proposition 6.1. As a consequence, we obtain the holomorphic continuation of H (ξ ) along any path γ[0,1] by      H γ (t) = )t A γ (t) , t ∈ [0, 1], and hence the holomorphic continuation of H (ξ ) as a multi-valued function on #∗ \ {ξ ∗ }. Moreover, by analytic continuation the functions hj (ξ ) defined by   h1 (ξ ), . . . , hk (ξ ) = H (ξ )

ON THE STRUCTURE OF THE SELBERG CLASS, III

553

satisfy (5.1) for ξ ∈ #∗ \ {ξ ∗ }. Therefore, ξ ∗ is an isolated singularity of H (ξ ) and, by construction, H (ξ ) is bounded for ξ ∈ #∗ \ {ξ ∗ }. Hence we can exclude the possibility that ξ ∗ is a pole or an essential singularity of H (ξ ). From Lemma 6.10, we have that if ξ ∗ is a branch point, then it is an algebraic branch point. Hence, by the theory of uniformization, for j = 1, . . . , k we have

 1/r hj (ξ ) = fj ξ − ξ ∗ , 0 < |ξ − ξ ∗ | < ρ, with a positive integer r, a real number ρ > 0, and a one-valued holomorphic function fj (z) in the punctured disc 0 < |z| < ρ. Therefore, writing fj (z) =

∞ 

aj,n zn ,

0 < |z| < ρ,

n=0

we have hj (ξ ) =

∞ 

 n/r aj,n ξ − ξ ∗ ,

0 < |ξ − ξ ∗ | < ρ.

n=0

Since (5.1) holds for ξ ∈ #∗ \ {ξ ∗ }, by Corollary 4.1 we have   hj (ξ ) ∈ R for ξ ∈ #∗ \ {ξ ∗ } ∩ R, and hence aj,n ∈ R ξ∗ − ρ

for n ≥ 0 and j = 1, . . . , k.

ξ ∗,

<ξ < consider the symmetric point ξ = ξ ∗ + (ξ − ξ ∗ )eπ i ∈ R. Given Hence, by the same principle as before, we have aj,n eπ in/r ∈ R as well. Therefore,

for n ≥ 0 and j = 1, . . . , k

aj,n  = 0 $⇒ eπin/r ∈ R $⇒ r | n,

and hence hj (ξ ) is a one-valued function for j = 1, . . . , k. It follows that ξ ∗ is a removable singularity, and hence the functions hj (ξ ) are in fact one-valued, are holomorphic on #∗ , and, by analytic continuation, satisfy (5.1) for every ξ ∈ #∗ . Theorem 2 is therefore proved. Now we turn to the proof of the Corollary. The existence of a factorization is immediate from Theorem 2. In order to prove that the factorization is unique we assume, by contradiction, the existence of an identity of type Ᏺ1 · · · Ᏺk = Ᏻ1 · · · Ᏻh ,

Ᏺj , Ᏻi primitive analytic families and Ᏺj  = Ᏻi .

554

KACZOROWSKI AND PERELLI

Therefore, F1 (s; ξ ) · · · Fk (s; ξ ) = G1 (s; ξ ) · · · Gh (s; ξ ),

ξ ∈ R,

and by Theorem 2, the involved functions are primitive since the families are primitive. Hence by the unique factorization in ᏿, we have that h = k and the functions on the left-hand side are a permutation of those on the right-hand side for every ξ ∈ R. As a consequence, there exist j0 , i0 and a set Ꮽ ⊂ R with an accumulation point such that Fj0 (s; ξ ) = Gi0 (s; ξ ),

ξ ∈ Ꮽ.

But two analytic families coincide if their members coincide for every ξ in a set having an accumulation point; hence Ᏺj0 = Ᏻi0 , which is a contradiction. References [1] [2] [3] [4] [5] [6] [7]

D. Bump, Automorphic Forms and Representations, Cambridge Stud. Adv. Math. 55, Cambridge Univ. Press, Cambridge, 1997. H. Grauert and K. Fritzsche, Several Complex Variables, Grad. Texts in Math. 38, SpringerVerlag, New York, 1976. H. Grauert and R. Remmert, Coherent Analytic Sheaves, Grundlehren Math. Wiss. 265, Springer-Verlag, Berlin, 1984. J. Kaczorowski and A. Perelli, “The Selberg class: A survey” in Number Theory in Progress (Zakopane-Ko´scielisko, 1997), Vol. 2, de Gruyter, Berlin, 1999, 953–992. L. Kaup and B. Kaup, Holomorphic Functions of Several Variables: An Introduction to the Fundamental Theory, de Gruyter Stud. Math. 3, de Gruyter, Berlin, 1983. R. Remmert, Holomorphe und meromorphe Abbildungen komplexer Räume, Math. Ann. 133 (1957), 328–370. U. Vorhauer and E. Wirsing, On Sarnak’s rigidity conjecture, preprint, 1999.

Kaczorowski: Faculty of Mathematics and Computer Science, Adam Mickiewicz ´ Poland; [email protected] University, 60-769 Poznan, Perelli: Dipartimento di Matematica, Università di Genova, Via Dodecaneso 35, 16146 Genova, Italy; [email protected]

Vol. 101, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY IN LOEWNER SPACES ZOLTÁN M. BALOGH and PEKKA KOSKELA APPENDIX BY JUSSI VÄISÄLÄ To Professor Hans Martin Reimann 1. Introduction. Let X and Y be metric spaces and f : X → Y a homeomorphism. Then the distortion of f at a point x ∈ X is H (x) := lim sup r→0

where

L(x, r) , l(x, r)

(1)


 L(x, r) := sup |f (x) − f (y)| : |x − y| ≤ r ,
 l(x, r) := inf |f (x) − f (y)| : |x − y| ≥ r ,

and by |x − y| we denote the distance between x and y in a metric space. We say that f is quasiconformal if there is a constant H so that H (x) ≤ H for every x ∈ X. This infinitesimal condition is easy to state but not easy to use. For instance, it is not clear from the definition if the inverse mapping is quasiconformal as well. It was recently shown by Heinonen and Koskela [7] that, for a large class of metric spaces, quasiconformal mappings satisfy a stronger, global condition: H (x, r) :=

L(x, r) ≤ H < ∞ l(x, r)

for all x ∈ X and all r > 0. This holds under the conditions that X = Y is an AhlforsDavid Q-regular Loewner space with Q > 1 and that f maps bounded sets to bounded sets. Let us call maps that satisfy the above global inequality quasisymmetric. This condition is weaker than the usual definition of quasisymmetry but results in the same class of mappings for a large class of metric spaces including the ones considered in Theorems 1.1, 1.2, and 1.3; see also [7]. Here the (Ahlfors-David) Q-regularity of the metric space X means that X is equipped with a Borel measure µ and there is a constant Cµ ≥ 1 such that   Cµ−1 r Q ≤ µ B(r) ≤ Cµ r Q , Received 13 November 1998. Revision received 27 April 1999. 1991 Mathematics Subject Classification. Primary 30C65. Balogh’s research partially supported by the Swiss National Science Foundation. Koskela’s research partially supported by Academy of Finland grant number 34082 and the Swiss National Science Foundation. 555

556

BALOGH AND KOSKELA

for all balls B(r) ⊂ X of radius r < diam X. As the Loewner property requires more terminology, we do not give the definition here; see Section 2. It is worth pointing out here that the Euclidean spaces Rn and the Heisenberg groups are Q-regular Loewner spaces. Furthermore, Bourdon and Pajot [4] and, independently, Laakso [11] have recently constructed examples of Q-regular Loewner spaces with Q > 1 not necessarily an integer. In this paper we address the question of removable singularities for quasiconformal mappings. This is a central question in the theory, and it dates back to the pioneering work of Ahlfors and Beurling [1] and is still under active research. More precisely, we are interested in conditions on a compact subset E of a Q-regular Loewner space that would guarantee that each quasiconformal mapping of X \ E that maps X \ E into X and maps bounded sets to bounded sets extends to a quasisymmetric homeomorphism of X onto X. When X is the Euclidean space Rn , n ≥ 2, this question is relatively well understood: E is removable in the above sense if the (n − 1)-dimensional Hausdorff measure of E is zero. On the other hand, there exist nonremovable Cantor sets of Hausdorff dimension n − 1. For the existence of such nonremovable sets, see the papers [2], [3] of Bishop. The positive result is due to Väisälä [17] and is based on a characterization of (sense-preserving) quasiconformal mappings by means of the differential inequality |Df (x)|n ≤ KJf (x). Indeed, any homeomorphism f is quasiconformal whose coordinate functions have locally n-integrable weak partial derivatives and whose differential satisfies a.e. the above differential inequality for a fixed finite constant K. The removability then follows from a removability theorem for Sobolev functions (i.e., locally n-integrable functions with locally n-integrable weak partial derivatives) and a result of Reshetnyak that allows one to show that f admits a homeomorphic extension. For the question on homeomorphic extension to a set E, see also the paper [12] by Martio and Näkki. There is no analytic characterization known for quasiconformality in the setting of a Q-regular Loewner space and, even if one could obtain such a characterization, the result of Reshetnyak is not available in this generality. Thus there is no real hope for strong removability results as in the Euclidean case. This is demonstrated by the following example. Example. Given 0 <  < 1, pick a closed totally disconnected set F ⊂ [0, 1]  (F ∩ B(x, r)) ≥ Cr  for each x ∈ F and all of Hausdorff dimension  so that H∞  0 < r < 1, where H∞ (A) is the -dimensional Hausdorff content of A. Consider R ¯ 1) ⊂ Rn as the first coordinate axis in Rn , n ≥ 2, and glue the closed unit ball B(0, n n ¯ 1) ∪F R is the disjoint to a copy of R along F to obtain a space X. Then X = B(0, ¯ 1) and Rn with points in the two copies of F ⊂ B(0, ¯ 1) and F ⊂ Rn union of B(0, identified. The Euclidean metrics of the two spaces have a natural extension to a ¯ 1) ∪F Rn lying in the two different metric on X: the distance between x, y ∈ B(0, parts of the union is given by

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

557

  |x − y| := inf |x − z| + |z − y| . z∈F

The space X equipped with this metric and the Euclidean volume is an n-regular Loewner space (cf. [7, Theorem 6.15]). However, E = F ∪ F , where F = F + x0 , |x0 | = 3, fails to be removable for quasiconformal mappings of X \ E into X. For ¯ 1) \ E and example, define f : X \ E → X \ (E − x0 ) to be id|B(0,1)\E on B(0, ¯ n (id −x0 )|Rn \E on R \E. Then f is a quasiconformal homeomorphism of X \E onto X \ (E − x0 ) that does not even have a continuous extension to E. Thus there can be no removability theorem in terms of the Hausdorff measure only. Based on this example there is little doubt that an additional topological assumption is needed. We formulate such a condition as the separation axiom S in Definition 4.1 below. Under this topological condition we prove the following abstract removability theorem. Theorem 1.1. Let X be a proper, unbounded, Q-regular Loewner space with Q > 1 that satisfies the separation axiom S. Then there exists t > 1 such that for each compact, spherically t-porous set E ⊂ X, any quasiconformal mapping f : X \ E → f (X \ E) = Y ⊂ X for an open Y that maps bounded sets to bounded sets extends to a quasisymmetric homeomorphism f˜ of X onto X. Here the requirement that f map bounded sets to bounded sets means that f (A) is bounded if and only if A is bounded, and the properness of X requires that each closed ball in X be compact. The separation axiom S is defined in Section 4. For example, Rn , the Heisenberg group, or more generally any complete manifold, satisfies the separation axiom while the example discussed above does not. The spherical t-porosity means the following: for each x ∈ E there is a sequence rj tending to zero with E ∩ B(x, trj ) \ B(x, t −1 rj ) = ∅ for all j. The above theorem is surprisingly sharp. First of all, the condition that f (X \ E) be open is essential as we require the extension to be quasisymmetric. To see this, simply consider X = {z ∈ C : Re z ≥ 0} with E = ∅ and f (z) = log(z + 1). Then f is not quasisymmetric. We also give an example of a space X and a mapping f that satisfy all assumptions above but openness of the image, and f does not even extend homeomorphically over a set E consisting of two points. Moreover the example discussed before Theorem 1.1 suggests that some kind of a separation condition is necessary. In that example, the set E can be selected so as to be t-porous for some t. As the constant t in Theorem 1.1 is allowed to depend on the given data, this example does not show the necessity of a separation condition, but a slightly more complicated example given in Section 4 qualifies for this conclusion. One wonders if, under the assumptions of Theorem 1.1, one could also remove some sets that are not totally disconnected. In the Euclidean case this can be done when n > 2. This hope turns out to be futile as we also exhibit, for given n, a situation as in Theorem 1.1 so that a compact set E of finite 1-dimensional measure is not removable.

558

BALOGH AND KOSKELA

Theorem 1.1 in Euclidean space is true for any t > 1 (see [6, Theorem 4.2]). We do not have an example showing that we need large t in the statement of Theorem 1.1 rather than any t > 1. If we assume a priori that f is a global homeomorphism, we obtain a more general result. Theorem 1.2. Let X be a Q-regular, unbounded Loewner space with Q > 1. Then there exists t > 1 such that, for each compact, spherically t-porous set E ⊂ X, any homeomorphism f : X → X that maps bounded sets to bounded sets and is quasiconformal in X \ E is quasisymmetric in X. Above, we did not assume that X be proper; under the properness assumption it is immediate that f maps bounded sets to bounded sets. We do not know if also sufficiently small continua could be removed in the setting of Theorem 1.2. For a precise result in the Euclidean setting, see the recent work [8] of Jones and Smirnov. Theorem 1.2 is obtained as a consequence of the following result. Theorem 1.3. Let X be a Q-regular, unbounded Loewner space with Q > 1. Let f : X → X be a homeomorphism that maps bounded sets to bounded sets and so that for some t > 1, some constant H , and all x ∈ X, L(x, s) lim inf sup ≤ H. r→0 r/t≤s≤tr l(x, s) Then f is quasisymmetric in X. The proof of Theorem 1.3 is based on a modification of the arguments of Heinonen and Koskela [6] that were used to show that quasiconformal self mappings of a Carnot group of dimension at least 2 are quasisymmetric. The paper is organized as follows. Section 2 contains some necessary definitions. We prove Theorem 1.3 in Section 3, which also contains a slight extension of this result. The proofs of Theorems 1.1, 1.2, and the above mentioned examples are given in Section 4. The appendix written by Jussi Väisälä deals with the separation condition. 2. Preliminaries. A connected Q-regular metric space (X, d, µ) has the Loewner property if there is a function " : (0, ∞) → (0, ∞) such that modQ (E, F ) ≥ "(t) whenever E and F are two disjoint nondegenerate continua in X and dist(E, F ) . min{diam E, diam F } Here modQ (E, F ) stands for the Q-modulus of the curve family connecting E to F :  modQ (E, F ) = inf ρ Q dµ, t ≥ #(E, F ) :=

X

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

559

where the infimum is taken over all admissible Borel densities ρ : X → [0, ∞], that is, for which  γ

ρ ds ≥ 1

on any rectifiable curve γ connecting E and F in X. If U ⊆ X is a domain in X such that E, F ⊂ U , we use the notation modQ (E, F, U ) for the relative modulus, that is, we consider the (smaller) family of curves that connect E and F and are contained in U . This concept of a Loewner space was introduced in [7] where it was shown that, under a mild local connectivity property, a proper Q-regular space is a Loewner space if and only if the space supports a suitable Poincaré inequality. It then follows that there are a plethora of Loewner spaces. A metric space X is called C-linearly locally connected, or C-LLC (C ≥ 1), if for x ∈ X and r > 0, any two points in B(x, r) can be joined by a rectifiable curve in B(x, Cr) and any two points in X \ B(x, r) can be joined by a rectifiable curve in X \B(x, r/C). It was shown in [7] that each Q-regular Loewner space is C-LLC with a constant that only depends on ", Q, and the Q-regularity constant. Let now X and Y be metric spaces and f : X → Y a homeomorphism. For t ≥ 1 we introduce the distortion H (x, t) of f at the point x ∈ X by L(x, s) , r/t≤s≤tr l(x, s)

H (x, t) := lim inf sup r→0

where

(2)


 L(x, s) := sup |f (x) − f (y)| : |x − y| ≤ s ,
 l(x, s) := inf |f (x) − f (y)| : |x − y| ≥ s ,

and by |x − y| we denote the distance between x and y in a metric space. Remarks. (a) Because lim inf r→0

L(x, r) L(x, r) ≤ H (x, t) ≤ lim sup , l(x, r) r→0 l(x, r)

(3)

we see that the uniform boundedness of H (x, t) in x is a weaker condition than quasiconformality that amounts to uniform boundedness of the right side of (3). Notice that H (x, t) is an increasing function of t and that H (x, 1) = lim inf r→0 L(x, r)/ l(x, r). (b) Usually, in the definition of quasiconformality, one considers the extremal dilatations on spheres, that is,
 L1 (x, s) := sup |f (x) − f (y)| : |x − y| = s ,
 l1 (x, s) := inf |f (x) − f (y)| : |x − y| = s , and the corresponding limit quotients.

560

BALOGH AND KOSKELA

On the other hand, if Y is C-LLC one can easily check that     1 B f (x), l1 (x, s) ⊂ f B(x, s) ⊂ B f (x), CL1 (x, s) . C Thus

1 l1 (x, s), C and so, in the setting of the metric spaces considered in this paper, there is no loss of generality in working with L(x, s) and l(x, s), which are better suited for our purposes. L(x, s) ≤ CL1 (x, s),

l(x, s) ≥

3. Proof of Theorem 1.3. In [7] it was proven that if X and Y are Q-regular metric spaces such that X is Loewner and Y is C-LLC for some C ≥ 1, then any quasiconformal homeomorphism that maps bounded sets to bounded sets is quasisymmetric. On the other hand, it was shown in [6] that if X = Rn , then the much weaker condition that H (x, 1) is uniformly bounded in x implies quasisymmetry. We prove here that already the Loewner condition allows one to relax the quasiconformality to uniform boundedness of H (x, t) in x for any fixed t > 1. Our argument is a modification of the proofs in [6]. The novelty here is the use of the covering lemma (Lemma 3.2). For the convenience of the reader, we give a rather detailed proof of the crucial part (Lemma 3.3) of the argument even though this results in repeating the reasoning in [6]. Theorem 3.1. Let X, Y be Q-regular metric spaces, where X is a Loewner space and Y is C-LLC for some C ≥ 1, and let f : X → Y be a homeomorphism that maps bounded sets to bounded sets. If for some t > 1 and some constant H , H (x, t) ≤ H for all x ∈ X, then f is quasisymmetric. Theorem 1.3 is a direct consequence of the above theorem as each Loewner space is LLC by [7]. Lemma 3.2. Let X be a Q-regular metric space, and let 1 < t < 2. There is a number N = N(t, Cµ ) such that for any collection of balls Ꮾ = {B(x, rx ) : x ∈ A ⊆ X} with supx∈A rx < ∞ there exist subcollections Ꮾ1 , . . . , ᏮN of Ꮾ with the following properties: (1) each Ꮾk contains countably many disjoint balls;   (2) the set A of the centers of balls in Ꮾ is covered by N k=1 B∈Ꮾk tB, where tB is a ball with the same center as B and t times the radius of B. Proof. The proof is a modification of the proof of the Besicovitch covering theorem. The first step consists of the following claim. Claim. There exists N = N(t, Cµ ) such that if a finite collection Ꮿ of balls has the properties: (a) all balls in Ꮿ intersect the one with the smallest radius,

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

561

(b) for any two balls B, B ∈ Ꮿ the ball tB does not contain the center of B , then )Ꮿ ≤ N. Proof. Let B0 be the ball with the smallest radius in Ꮿ and that intersects all the others. Using (b) we obtain that

for every ball B ∈ Ꮿ. It then follows that

r B 0 ≤ r B ≤ r B0

1 , t −1





B∈Ꮿ

2 B ⊆ 1+ t −1

(4)

B0 ,

and by the Q-regularity we obtain



 t +1 Q Q B ≤ Cµ rB0 . µ t −1

(5)

B∈Ꮿ

Let us notice that the center of B  = B is at distance of at least rB (t − 1) from B by property (b). By (4) this means that the center of B is of distance at least rB0 (t − 1) away from B. We obtain this way a number of )Ꮿ disjoint small balls of radius rB0 (t − 1) contained in ∪B∈Ꮿ B. Denoting by S the union of these, we see that µ(S) ≥ )Ꮿ

1 Q (t − 1)Q rB0 . Cµ

(6)

From (5) and (6) it follows that

)Ꮿ ≤ (Cµ )2

t +1 (t − 1)2

Q

=: N,

proving the claim. Our argument now proceeds as in the proof of the Besicovitch covering theorem, and we only briefly sketch it following [15, p. 15]. Imagine the subcollections Ꮾ1 , . . . , ᏮN as rows of disjoint balls that we create with the following procedure starting with a largest ball and proceeding in order of the size. Place “a largest ball” B1 in the first row. Throw away all balls whose centers are covered by tB1 . Take a next largest ball B2 . If B2 is disjoint from B1 place it in the first row. If not, then place it in the second row. Throw away all balls whose centers are covered by tB2 . At the nth step, place Bn in the earliest row that keeps balls in each row disjoint. Throw away all balls whose centers are covered by tBn . Proceed by transfinite induction. The whole list certainly covers A since we only throw away balls whose centers are already covered. Each row consists of disjoint balls by construction. We show that in the process we create at most N rows, where

562

BALOGH AND KOSKELA

N comes from the claim: Assume that some ball B0 is put in the (N +1)th row. Then there are balls B1 , . . . , BN already in the first N rows with radii larger than that of B0 and all of them intersect B0 . Consider the collection Ꮿ := {B0 , . . . , BN }. Then Ꮿ has properties (a) and (b) from the claim and thus )Ꮿ ≤ N , a contradiction. In the above argument there might not be a largest ball that we need in our successive selections, and then we choose a nearly largest. To deal with this technical difficulty we can use a version of the claim with condition (a) replaced by the condition that all balls in Ꮿ intersect one with at most twice the minimal radius. We invite the reader to fill in the details (cf. also [13, pp. 31–33]). Remark. The Besicovitch covering theorem is just a stronger form of the above lemma that holds for the value of t = 1 in Rn . This is the reason why the uniform boundedness of H (x, 1) implies quasisymmetry in the n-regular Loewner space Rn . As it was observed in [10, p. 17] , this covering property fails already in the Heisenberg group. Proof of Theorem 3.1. By the Loewner property of X (cf. [7, Theorem 4.9]), the quasisymmetry of f : X → Y follows from the next result. Lemma 3.3. Let us suppose that X, Y, f , and t satisfy the assumptions of Theorem 3.1. Then for any two disjoint continua E, F ⊆ X such that y ∈ f (E) ⊂ B(y, l) and f (F ) ⊂ Y \ B(y, L), for some y ∈ Y and L > 2l, we have 

L 1−Q , modQ (E, F ) ≤ C log l where the constant C ≥ 1 depends only on the data associated to X, Y, f , and t. Proof. There is no loss of generality to assume that 1 < t < 2. The proof consists of a construction of an admissible density ρ : X → [0, ∞] for the pair (E, F ) so that

  L 1−Q Q ρ dµ ≤ C log . (7) l X This is achieved by a special collection of balls Ꮾ and the formula  

1 L −1 diam f B χ2B (x). log ρ(x) := C1 l dist(f B, y) diam B

(8)

B∈Ꮾ

We now list sufficient conditions on Ꮾ to guarantee that (7) holds and that ρ is an admissible density. (P1) The collection Ꮾ consists of balls B(x, r) that are good in the sense that L(x, r)/ l(x, r) < 2H . The radii of such good balls we call good scales. (P2) Ꮾ covers the set f −1 (B(y, L) \ B(y, l)) ⊂ X, f B ∩ B(y, L) \ B(y, l)  = ∅ and for any B ∈ Ꮾ, diam f B < 2−j0 −1 L where j0 is the smallest integer such that 2−j0 L < l.

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

563

(P3) Ꮾ splits into N + 1 subcollections (N from Lemma 3.2) Ꮾ1 , . . . , ᏮN +1 such that 1/t Ꮾk is a pairwise disjoint collection of balls for k = 1, . . . , N + 1. (P4) For each B ∈ Ꮾ there is a subset VB ⊂ B such that µ(f B) ≤ D1 µ(f VB ) for some uniform constant D1 ; moreover we require that if Bi , Bj , ∈ Ꮾk , Bi  = Bj , then VBi ∩ VBj = ∅. Note that for an arbitrary x ∈ X there are plenty of small good balls around it by the condition H (x, t) ≤ H . By the continuity of f : X → Y , property (P2) is also easy to get. Concerning (P3) and (P4), we see that one can choose VB = 1/tB for B in the first N subcollections Ꮾk . For B ∈ ᏮN +1 the construction of VB is more complicated. We postpone the construction of Ꮾ to the end of the proof and we next explain why the density defined by (8) works if Ꮾ satisfies (P1), . . . , (P4). Let us first show the inequality in (7). In order to estimate the integral X ρ Q dµ, it is useful to introduce the function ᐂ : Ꮾ → R+ by

 L −1 diam f B ᐂ(B) = log . l dist(f B, y) By [6, Proposition 2.9], there is a uniform constant C2 > 0 such that   ρ Q dµ ≤ C2 ᐂ(B)Q . X

B∈Ꮾ

We can show that 



Q

ᐂ(B) ≤ C3

B∈Ꮾ

L log l

1−Q (9)

for some uniform constant C3 > 0. We adopt the notation M  M if M ≤ CM for some uniform constant C = C(X, Y, f, t). To estimate the left side of (9), we make a partition of the balls into classes Ꮾj , j = 0, . . . , j0 , where Ꮾj consists of the balls in Ꮾ for which f B either lies in the annular region B(y, 2−j L) \ B(y, 2−j −1 L) or touches ∂B(y, 2−j L). Then 

 L −Q   (diam f B)Q ᐂ(B)Q ≤ log  Q , l B j B∈Ꮾj dist(f B, y) and because for good balls (diam f B)Q  µ(f B),

  L −Q   −j −Q  Q ᐂ(B)  log µ(f B). 2 L l j B

j

B∈Ꮾ

564

BALOGH AND KOSKELA

For the second sum on the right we write 

µ(f B) =



µ(f B),

k=1 B∈Ꮾj ∩Ꮾk

B∈Ꮾj

where by property (P4) of Ꮾ,   µ(f B)  B∈Ꮾj ∩Ꮾk

N +1 

    Q µ(f VB )  µ B y, 2−j L  2−j L ,

B∈Ꮾj ∩Ꮾk

and consequently



 Q µ(f B)  2−j L .

B∈Ꮾj

This shows that

 L −Q ᐂ(B)  log (j0 + 1), l and since, by the definition, j0  log L/ l, we obtain (9).  It is not hard to check that there is a uniform constant C1 such that γ ρ ds ≥ 1 for any curve γ connecting E and F . This follows from the fact that f γ then has to pass through the annular region B(y, L) \ B(y, l) that is covered by {f B : B ∈ Ꮾ}. Here we do not need the special properties of the covering Ꮾ. We leave the details to the reader with the hint to use the partition of Ꮾ into the classes Ꮾj as above. Now we give the construction of the collection Ꮾ with the required properties. As already mentioned, we do the construction of Ꮾ in such a way that (P1) and (P2) clearly hold. The difficulty is to achieve (P3) and (P4). For x ∈ f −1 (B(y, L) \ B(y, l)) let us choose the radii rx so that we have the conditions diam f B(x, rx ) < 2−j0 −1 L and 

Q



L(x, r) < 2H l(x, r)

for

1 rx ≤ r ≤ rx . t

This is possible as H (x, t) < H , for all points x ∈ X, and f is continuous. Let us define D := (20H 2 )Q Cµ2 and divide the set U = f −1 (B(y, L) \ B(y, l)) into two subsets Ᏸ and Ᏻ, where Ᏸ consists of points x ∈ U for which the doubling property   µ f B(x, rx )   ≤D (10) µ f B(x, rx /t) holds, while Ᏻ contains the points x ∈ U for which huge gaps occur in the image side, that is,   µ f B(x, rx )   > D. (11) µ f B(x, rx /t)

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

565

Applying Lemma 3.2 to the collection {B(x, rx /t); x ∈ Ᏸ}, we obtain the subcollections Ꮾ1 , . . . , ᏮN of the collection {B(x, rx ) : x ∈ Ᏸ} that cover Ᏸ and have properties (P1), (P2), (P3), and also (P4) with VB := 1/tB. We next construct the collection ᏮN +1 that covers Ᏻ and has the required properties. Let us first note that, by the definitions of L(x, s) and l(x, s),     B f (x), l(x, s) ⊆ f B(x, s) ⊆ B f (x), L(x, s) , for x ∈ X and s > 0. As both rx and rx /t are good distortion scales, the Q-regularity yields  r Q  r Q x x L x, ≤ (2H )Q l x, t t  

rx  t Q   (2H ) Cµ µ f B(x, rx ) ≤ D (2H )Q Cµ2 ≤ L(x, rx )Q D (2H )2Q Cµ2 l(x, rx )Q . < D ≤ (2H )Q Cµ µ f B x,

Consequently, by the choice of D, we obtain  r  1 x L x, < l(x, rx ), t 5 which gives that

  r    x B f (x), 5L x, ⊂ f B(x, rx ) , t

for each x ∈ Ᏻ. We use the classical (1/5)-covering theorem (cf. [13, p. 24]) in Y to subtract from the collection {B(f (x), 5L(x, rx /t)) : x ∈ Ᏻ} a countable subcollection     r  x : i = 1, 2, . . . ᏮᏳ = B f (xi ), 5L xi , i t  such that f (Ᏻ) ⊆ B∈ᏮᏳ B and, for i  = j ,    r   rxj  x B f (xi ), L xi , i ∩ B f (xj ), L xj , = ∅. t t

(12)

For each xi ∈ Ᏻ as above, denote by ri the least radius r > 0 such that f (B(xi , r)) contains the ball B(f (xi ), 5L(xi , rxi /t)). Then clearly rxi /t ≤ ri ≤ rxi and l(xi , ri ) = 5L(xi , rxi /t).

566 Putting now

BALOGH AND KOSKELA






ᏮN+1 := B(xi , ri ) : i = 1, 2, . . . ,

we obtain a collection that covers Ᏻ, and also 1/t ᏮN +1 is a pairwise disjoint collection by (12). Finally define VBi := f −1 B(f (xi ), L(xi , rxi /t)). It is clear that VBi ⊆ B(xi , ri ) and VBi ∩ VBj = ∅ for i  = j by (12). It remains to show the estimate in (P4). As ri is also a good distortion scale since rxi /t ≤ ri ≤ rxi , the Q-regularity gives      µ f B(xi , ri ) ≤ µ B f (xi ), L(xi , ri ) ≤ Cµ L(xi , ri )Q ≤ Cµ (2H )Q l(xi , ri )Q  r Q x = Cµ (10H )Q L xi , i t    ≤ Cµ2 (10H )Q µ f VBi . In conclusion, property (P4) holds with D1 := D = (20H 2 )Q Cµ2 . This finishes the proof. 4. Proofs of Theorems 1.1 and 1.2. We begin this section with some definitions. We say that a set C ⊂ X separates two points x, y ∈ X if each curve connecting x and y in X necessarily intersects C. The following topological condition is essential. Definition 4.1. A metric space X satisfies the separation axiom S if, for each compact subset F of X, there is a constant λ > 0 so that the following holds. If a finite union A ⊂ F of pairwise disjoint continua of diameters less than λ separates points x, y ∈ X, then a component of A separates x, y in X. Examples. The separation axiom S clearly follows under the assumption that if a compact set C separates a pair x, y of points in X, then necessarily a component of C separates these points. One can prove this stronger separation property for Rn , n ≥ 2, by looking at appropriate Mayer-Vietoris sequences in the zero cohomology and using the fact that H 1 (Rn ) = 0 for n ≥ 2 (cf. [18, p. 61]). Thus the separation axiom S holds for the Euclidean space Rn . For the Heisenberg groups and for the Carnot groups the same is true as they are homeomorphic to some Rn via the exponential map. A torus embedded in R3 is a simple example of a proper Loewner space that does not satisfy the above stronger separation condition but does satisfy axiom S. In fact, given any metric connected n-manifold X, n ≥ 2, and a compact subset F of X, there exists a constant λ = λ(X, F ) > 0 so that if A ⊂ F is a compact set whose components have diameters at most λ and if A separates points x, y in X, then a component of A separates x, y in X. The reason behind this is that the injectivity radius on F is bounded away from zero and thus the question can, with some work,

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

567

be reduced to the Euclidean situation. A proof by Jussi Väisälä of the separation condition for connected metric manifolds is given in the appendix. It is easy to check that the example described before the statement of Theorem 1.1 does not satisfy the separation axiom S suggesting the necessity of this condition. Definition 4.2. A compact set E ⊂ X is spherically t-porous, t > 1, if for each x ∈ E there exists a sequence (rxn ), rxn  = 0, rxn → 0, of radii such that   A x, rxn , t ∩ E = ∅, where A(x, rxn , t) = B(x, trxn ) \ B(x, rxn /t). Since we do not require any uniformity of the sequence of radii, the above spherical porosity condition allows big sets in terms of the Hausdorff dimension as indicated by the following proposition. Proposition 4.3. There exists a set E ⊂ Rn of Hausdorff dimension n which is spherically t-porous for all t > 1. Proof. For simplicity we only give the proof in the case n = 2. Let E1 be the unit n(k) square in R2 , and, for k ≥ 2, let Ek = ∪i Eki be a collection of disjoint, equally spaced squares of the same size such that Ek ⊂ Ek−1 , constructed as follows. Choose i a square Ek−1 ⊂ Ek−1 of side length dk−1 and subdivide it into smaller squares of

i . We side length dk = dk−1 /2k . We obtain a number of 22k subsquares of Ek−1 2 remove some of these so that we are left with a number of 22(k −k) equally spaced small squares. Notice that the distance of two neighboring squares in Ek is δk ≈ 2k dk . Furthermore, observe that   i area Ek ∩ Ek−1 1  i  = 2k for all i = 1, . . . , n(k − 1) and k ≥ 2. #k := 2 area Ek−1 2

2

To calculate the Hausdorff dimension of E := ∩k≥1 Ek , we apply the formula (cf. [14, Proposition 2.2]) k+1 | log #i | lim sup i=1 ≥ 2 − dim E. (13) | log dk | k→∞ k+1 2  2 It is clear that dk = 2− i=1 i and so | log dk | ≈ k 3 while k+1 i=1 | log #i | ≈ k ; thus the left side of (13) vanishes. This shows that dim E = 2. From δk ≈ 2k dk it follows that E is t-porous for each t > 1.

Let X and Y be unbounded metric spaces, and let f : X → Y be a quasiconformal homeomorphism onto its image f (X) ⊆ Y . We say that f is of class Ᏺ if it maps bounded sets to bounded sets and its image f (X) is open in Y . Under this notation, Theorem 1.1 reads as follows.

568

BALOGH AND KOSKELA

Theorem 4.4. Let X be a proper, unbounded, Q-regular Loewner space with Q > 1 that satisfies the separation axiom S. Then there exists t > 1 such that for any compact, spherically t-porous set E ⊂ X, each quasiconformal mapping f : X \ E → X that is of class Ᏺ extends to a quasisymmetric homeomorphism f˜ : X → X. Proof. Here is an outline of the proof. In Step I, we show that f has a continuous extension to a map f˜ : X → f˜(X); Step II shows that f˜ : X → f˜(X) is injective; and in Step III, we show that f˜(X) = X and that f˜ is quasisymmetric. Step I: Continuous extension. The idea is as follows: For x ∈ E choose a sequence of good scales rxn → 0, that is, A(x, rxn , t) ∩ E = ∅. Choose a sequence of points xn ∈ ∂B(x, rxn ) ⊂ A(x, rxn , t). Then (f (xn )) is a bounded sequence (f maps bounded sets to bounded sets) and since X is proper, it has a subsequence (also denoted by (f (xn ))) converging to a point y = y(x). Let us define  f (x), x ∈ X \ E, ˜ f (x) = y(x), x ∈ E. The proof of the continuity of f˜ is divided in two further steps. Step I.A: Annular continuity. We show that there exists a number 1 < t1 ≤ t such that     f xn −→ y(x) for any sequence of points xn ∈ A x, rxn , t1 . Step I.B: Continuity. We show that     f xn −→ y(x) for any sequence of points xn ∈ B x, rxn \ E, xn −→ x. Before starting the proof, let us note that our task is to exhibit a value t > 1 such that the proofs of Step I.A and Step I.B work. We choose a priori a large value t > 1 and we comment on its magnitude at the end. We use the notation M ≈ M if M  M and M  M. The proof of Step I.A is based on the following lemma. Lemma 4.5. There exists a number 1 < t1 ≤ t such that whenever x ∈ E, y ∈ B(x, rx /t) \ E, and z1 , z2 ∈ A(x, rx , t1 ) we have |f (y) − f (z1 )| ≈ |f (y) − f (z2 )| for any good scale rx > 0. Proof. Let us define t1 := t/C1 > 1, where C1 is a large (uniform) constant to be determined later. Choose y ∈ B(x, rx /t) \ E and let z1 , z2 ∈ A(x, rx , t1 ). Let us assume that 1 |f (y) − f (z2 )|, |f (y) − f (z1 )| < 4C 2

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

569

where C > 1 is the LLC constant of X. Let us choose a point z ∈ X far away from y so that f (z) is far from f (y). By the C-LLC property of X, we can join the points f (y) and f (z1 ) by a curve γ1 ⊂ B(f (y), C|f (y) − f (z1 )|). Similarly join f (z) and f (z2 ) by another curve γ2 ⊂ X \ B(f (y), |f (y) − f (z2 )|/C). Let us write F1 = f −1 (γ1 ∩ f A(x, rx , 2t1 )) and F2 = f −1 (γ2 ∩ f A(x, rx , 2t1 )). Then F1 , F2 have subcurves E1 , E2 that join z1 and z2 to ∂A(x, rx , 2t1 ). If the (uniform) constant C1 is large enough we can apply [7, Lemma 3.17] to conclude that   modQ E1 , E2 , A(x, rx , t) ≥ δ > 0, for some uniform δ > 0. On the other hand, by the arguments in the proof of Lemma 3.3 we obtain the estimate 

  |f (y) − f (z2 )| 1−Q modQ E1 , E2 , A(x, rx , t)  log 2 , C |f (y) − f (z1 )| which implies that |f (y)−f (z2 )|  |f (y)−f (z1 )|. This ends the proof of the lemma. Let us consider a sequence (xn ) such that xn ∈ A(x, rxn , t1 ). By Lemma 4.5 we have           f x − f xn+1   f xn − f xn+1  −→ 0 as n −→ ∞. n Since f (xn ) → y(x) we conclude Step I.A. For the proof of Step I.B, we need the following preparatory lemmas. Lemma 4.6. Let F ⊂ X \ E be a compact set and x, y ∈ X \ E such that F separates x and y and also separates x and E. Then f (F ) separates f (x) and f (y). Proof. Let us suppose by contradiction that there exists a curve γ : [0, 1] → X such that γ (0) = f (x), γ (1) = f (y), and γ ([0, 1]) ∩ f (F ) = ∅. It is clear that γ is not contained in f (X \ E); for otherwise f −1 γ would join x and y in (X \ E) \ F . Let us define
 t0 := sup t ∈ [0, 1] : γ (s) ∈ f (X \ E) for any 0 ≤ s ≤ t . Then 0 < t0 < 1. Let tn  t0 and yn = γ (tn ). Then yn ∈ f (X \ E), and we write xn := f −1 yn . Let us recall that by our assumption on f we have that A ⊂ X is bounded if and only if f (A) is bounded. Thus our sequence (xn ) is bounded. Since X is proper we find a subsequence (still denoted (xn )) such that xn → x0 ∈ X. If x0 ∈ X \ E, then yn = f (xn ) → f (x0 ) and so f (x0 ) = γ (t0 ) ∈ f (X \ E). On the other hand, f (X \ E) is open in X. This gives a contradiction to the maximality of t0 . If x0 ∈ E, then f −1 (γ |[0,t0 ) ) must intersect F because F separates x and E. This is impossible as γ (t) ∈ / f (F ).

570

BALOGH AND KOSKELA

Lemma 4.7. Any two points x, y ∈ X \ E can be joined by a curve γ˜ ⊂ X \ E. Proof. Consider a curve γ that joins x and y in X and let E := E ∩ γ . Then E is compact and, by the (1/5)-covering theorem (cf. [13, p. 24]), we can cover it with a finite collection of good balls {B(xi , ri ) : i = 1, . . . , K} with centers xi ∈ E and such that {B(xi , ri /5) : i = 1, . . . , K} are disjoint. On the other hand, (B(xi , tri ) \ B(xi , ri /t))∩E = ∅ and we see that in fact the collection {B(xi , ri /5) : i = 1, . . . , K} covers E . One can also choose the ri > 0 so small that {x, y} ∩ (∪i B(xi , ri /5)) = ∅. Then γ must enter and leave each B(xi , ri /5). Let us denote by yi the point where γ enters B(xi , ri /5) for the first time, and let zi denote the point where γ leaves B(xi , ri /5) the last time. By [7, Remark 3.19], there exists a constant C2 (depending only on X) such that yi and zi can be connected by an arc γi ⊂ B(xi , C2 ri /5) \ B(xi , ri /(5C2 )). If t > 5C2 , then we obtain that B(xi , C2 ri /5) \ B(xi , ri /(5C2 )) ∩ E = ∅ and we can correct γ by replacing its arc between yi and zi by γi for i = 1, . . . , K. The new curve γ˜ obtained above connects x and y without meeting E. The claim of the lemma follows. We can now turn to the proof of Step I.B. Pick a point x ∈ E and suppose that there exists a sequence of points xn ∈ B(x, rxn /t1 ) \ E such that |f (xn ) − f˜(x)| > 

for all n ∈ N.

Let us choose a point x 0 ∈ X that is far away from E so that f (x 0 ) is far away from f˜(E). We construct a set Fn ⊂ X \ E with the following properties: (i) Fn separates xn and E, and it is a union of kn pairwise disjoint continua Fni ; (ii) Fn1 separates xn and x 0 whereas Fni does not for i > 1; (iii) dn := maxi diam f (Fni ) → 0 as n → ∞; moreover f (Fn1 ) → f˜(x) as n → ∞; (iv) ∪n Fn is bounded. Let us assume that we have already constructed the sets (Fn )n with the required properties. By properties (i) and (ii), together with Lemma 4.6, we conclude that f (Fn ) separates f (xn ) and f (x 0 ). As X is proper, ∪n Fn is compact and thus the separation axiom S shows that f (Fnin ) already separates f (xn ) and f (x 0 ) for all large n for some index in ∈ {1, . . . , kn }. On the other hand, by the property (ii), f (Fn1 ) must be the continuum separating f (xn ) and f (x 0 ) in X. Since f (Fn1 ) → f˜(x), there exists an n0 ∈ N such that f (Fn1 ) ⊂ B(f˜(x), /C) for n ≥ n0 , where C is the LLC constant of X. Both f (xn ) and f (x 0 ) are in X \B(f˜(x), ) and so they can be joined by a curve in X \B(f˜(x), /C). This is a contradiction as f (Fn1 ) separates f (xn ) and f (x 0 ). Thus Step I.B is complete, once we have constructed the set Fn with the above properties. Let us first construct Fn1 . Recall that by Step I.A, f (A(x, rxn , t1 )) → f˜(x) as n → ∞. If A(x, rxn , t1 ) happens to be connected, put Fn1 := A(x, rxn , t1 ). Otherwise start with the annulus A(x, rxn , 2) and notice that, by [7, Remark 3.19], any pair of points in A(x, rxn , 2) can be joined by a curve in A(x, rxn , t1 ) provided that t1 > C2 ,

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

571

where C2 is some large uniform constant. Then we define A1n to be the union of such curves connecting the points in A(x, rxn , 2) to a fixed point in A(x, rxn , 2) and 1 lying in A(x, rxn , t1 ). Then A1n is connected and bounded. Set Fn1 = An . Because A(x, rxn , 2) ⊂ Fn1 , our set Fn1 separates xn and x 0 . Finally, it is clear that Fn1 is compact and connected, and that f (Fn1 ) → f˜(x) as n → ∞. Let us now construct the continua Fn2 , . . . , Fnkn . By Lemma 4.7 there exists a curve γn connecting xn and x 0 in X \ E. Then δn := dist(γn ∪ Fn1 , E) > 0. Following the line of the proof of Lemma 4.7, we cover E by a finite collection of good balls {B(xni , rxni ) : i = 2, . . . , kn } that are pairwise disjoint and satisfy rxni < δn /t. For each xni we consider the annulus A(xni , rxni , t1 ) and define the continua Fni ⊂ A(xni , rxni , t1 ) in a similar way as in the definition of Fn1 above. We may clearly assume that n Fni satisdiam f (A(xni , rxni , t1 )) ≤ 1/n. We leave to the reader to check that Fn = ∪ki=1 fies all the listed properties. This finishes the proof of the continuity of f˜ : X → f˜(X). Remark. Tracing back for the magnitude of the constant t > 1, we observe that the above proof works provided t > 10C1 C2 , where C1 and C2 are the (uniform) constants from Lemma 3.17 and Remark 3.19, respectively, in [7]. Step II: Injectivity of f˜. Let us assume by contradiction that there exist two points x, y ∈ X such that f˜(x) = f˜(y) = z. We can find a sequence of radii rn  0 such that the preimage Cn := f˜−1 (B(z, rn )) contains a curve γn ⊂ X \E with diam γn >  for  = |x − y|/2 and any n ∈ N. Let us suppose that Cn and γn with the above property have already been constructed, and consider C := ∩n Cn . Then f˜(C) = {z}. We claim that there are at least two points x1 , y1 ∈ (X \ E) ∩ C, which is a contradiction as f : X \ E → X is injective. We first prove the existence of a point x1 ∈ (X \ E) ∩ C. If such a point x1 does not exist, then C ⊆ E. We can cover E with a finite number of pairwise disjoint balls with radii much smaller than . The union of these balls forms an open neighborhood of C and thus, for n large enough, Cn must be contained in the union of these balls. This is impossible as γn ⊂ Cn . This argument gives the existence of another point y1 ∈ (X \ E) ∩ C as well. Let us now turn to the construction of Cn and γn . Without loss of generality assume that x, y ∈ E. Since f˜ is uniformly continuous in a large ball containing E, for each r > 0 there exists δ > 0 such that for the points x , x in the ball with |x − x | < δ we have that |f˜(x ) − f˜(x )| < r/M, where M ≥ 2 is a constant to be determined shortly. Let us choose rn  0 and let δn  0 be as above. Choose xn , yn ∈ X \ E so that |xn − x| < δn , |yn − y| < δn . Then |f (xn ) − z| < rn /M and |f (yn ) − z| < rn /M. By Lemma 4.7 we can connect xn and yn by a curve γn ⊂ X \ E. Consider a covering of E by a finite number of pairwise disjoint balls with radii smaller than δn and such that the union of these balls does not intersect γn . Construct

572

BALOGH AND KOSKELA

the associated annular continua Fni as before and define Fn := ∪i f (Fni ) ∪ An , where An is the annular continuum constructed using ∂B(z, rn ). There is a constant C0 determined by the LLC constant of X so that An ⊂ X \ B(z, rn /C0 ) and so that the diameter of An is no more than C0 rn . As we may choose the radii of the balls as small as we wish, we may assume that the diameters of the Fni do not exceed δn , and thus the above continuity estimate guarantees that the diameters of f (Fni ) do not exceed rn /M. Choose now M so large that the An component of Fn is contained in X \ B(z, rn /(2C0 )); notice that the continua f (Fni ) are pairwise disjoint and thus each f (Fni ) contained in the An component necessarily intersects An . In any case the diameter of this component is no more than (C0 + (2/M))rn . We further require, as we may, that C/M ≤ 1/2C0 . We claim that for large enough n, Fn does not separate f (xn ) and f (yn ) in X. Assume that Fn separates f (xn ) and f (yn ). If n is large enough, the separation axiom S shows that already a component of Fn separates f (xn ) and f (yn ). (Notice that ∪n Fn is compact as X is proper.) Since f (xn ), f (yn ) ∈ B(z, rn /M), the C-LLC property of X shows that An cannot separate. On the other hand, f (Fni ) cannot separate either, since f (γn ) connects f (xn ) and f (yn ) without meeting f (Fni ). Consequently, we can join f (xn ) and f (yn ) by a curve 2n in B(z, rn ) such that 2n ∩ (∪i f (Fni )) = ∅. Then, reasoning as in the proof of Lemma 4.6, we see that 2n ⊂ f (X \E). Hence γn := f −1 (2n ) ⊂ Cn is the desired curve in X \ E that joins xn and yn . This finishes the proof of Step II. Let us observe that, by the properness of X and because f maps bounded sets to bounded sets (i.e., A bounded if and only if f (A) bounded), it follows from the injectivity that f is a homeomorphism onto its image. Step III: Quasisymmetry of f˜. Let us show first that f˜(X) = X. We claim that ˜ f (X) = f (X \ E). Clearly f˜(X) ⊆ f (X \ E). To see the opposite inclusion, let y ∈ f (X \ E). Choose a sequence of points yj ∈ f (X \ E) such that yj → y. Then the sequence of points xj = f −1 (yj ) has a subsequence (denoted the same way) converging to a point x ∈ X. Thus f˜(x) = lim f (xj ) = lim yj = y, and we conclude that y ∈ f˜(X). Furthermore, since f (X \ E) is open in X, we infer that ∂ f˜(X) ⊂ f˜(E). Let us suppose that y1 ∈ X \ f˜(X), and fix y2 ∈ f (X \ E). Any curve that joins y1 and y2 in X must meet ∂ f˜(X) ⊂ f˜(E). Consequently f˜(E) separates y1 and y2 . Choose again a sequence (indexed by the positive integers n = 1, 2, . . . ) of covers of E by unions of pairwise disjoint small balls {B(xni , rni ) : xni ∈ E}. We associate to each B(xni , rni ) an annular continuum Fni as before. One checks easily, reasoning as in the proof of Lemma 4.6, that the finite union ∪i f (Fni ) of pairwise disjoint continua separates y1 and y2 in X. We may clearly assume that maxi diam f (Fni ) → 0 and that maxi diam Fni → 0. By the separation axiom S, we conclude that one of the sets, f (Fni ), should already separate y1 and y2 when n is sufficiently large. This is a contradiction as X is LLC and d(f (Fni ), {y1 , y2 }) ≥ δ > 0 with some δ > 0

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

573

independent of i, n (provided n is large). In order to prove quasisymmetry of f˜ : X → X, we apply Theorem 3.1. We need to check that H (x, t) for f˜ is uniformly bounded in x for some t > 1. It is enough to consider points x ∈ E. One can simply repeat the consideration in the proof of Lemma 4.5. This argument proves Theorem 1.2 as well. We leave the details to the reader. Remarks. (a) It is clear from the proof that there is a version of Theorem 4.4 for f : X \ E → f (X \ E) ⊂ Y with Y a metric space not necessarily equal to X. One assumes that X is proper, Q-regular with Q > 1, and Loewner. The target space Y is assumed to be proper, Q-regular with the same Q, and LLC, and in addition, it is required to satisfy the separation axiom S. (b) The unboundedness of the space X is not essential for these results. The bounded case can be treated similarly with some extra work. (c) It seems that we do not use the full strength of the separation axiom. We do not, however, know of any elegant topological assumption weaker than the separation axiom that would suffice for the proof (cf. Example 2). We now present examples that establish the necessity of the topological assumptions in our result. Our first example shows that one needs to assume f (X \ E) to be open for the statement of Theorem 4.4. 2 Example 1. Let F = ∪∞ 1 [2j, 2j + 1] × {0} ⊂ R , and define X to be the space obtained by gluing two copies of {(x, y) : x ≥ 1} along F. By [7, Theorem 6.15], X is a 2-regular Loewner space. Clearly X is also proper and satisfies the separation axiom S. Let E = {p1 , p2 }, where both p1 and p2 are the point (1, 0) but in the two different parts of the space X. Consider the map f : X \ E → X defined by f (x, y) = (x + 2, y) for x ∈ X \ E. Then f satisfies all the conditions of the theorem except for the openness of the image. The continuous extension f˜ of f is not injective as f˜(p1 ) = f˜(p2 ) = (3, 0) ∈ F .

The next example shows the necessity of the separation axiom. Example 2. The construction is again done using gluing. This time, let F = 2 ∪∞ 1 Fj × {0} ⊂ R , where Fj is a copy of the usual (1/3)-Cantor set on the interval [2j, 2j + 1], and define X to be the space obtained by gluing two copies of R2 along F. Again, by [7, Theorem 6.15], X is a 2-regular Loewner space. Clearly X is also proper. Given t > 1 pick a regular t-porous Cantor set Et ⊂ [0, 1] constructed as the usual (1/3)-Cantor set but by removing at each stage of the construction a middle interval of length s times the length of the interval in question, where 1/3 < s < 1 is determined by the value of t. Then it is easy to construct a (“piecewise linear”) quasisymmetric mapping ht : R → R so that ht (Et ) = F1 and ht (Fj ) = Fj +1 for j = 1, 2, . . . . For constructions of this type, see [5]. Let gt be the Beurling-Ahlfors extension of ht to R2 . Set E to be the union of the two copies of Et , one in each copy of R2 , and define ft : X \ E → X \ F1 by restricting gt to the complement of

574

BALOGH AND KOSKELA

Et in each copy of R2 . Then ft is a quasiconformal mapping of X \ E onto the open subset X \F1 of X that maps bounded sets to bounded sets and that has no continuous injective extension to E. Our last example shows that one can only remove totally disconnected sets under the assumptions of Theorem 1.1. Example 3. Let n ≥ 2, and consider F0 = [0, 1] ⊂ Rn as embedded in Rn . Let n F = ∪∞ 1 (F0 + 2j e), where e ∈ R is any unit vector, and define, as before, X to be the space obtained by gluing two copies of Rn along F. Then X is a proper, n-regular Loewner space and satisfies the separation axiom S. We define E as the union of the two copies of F0 located in the two different parts of X. Then the quasiconformal mapping f : X \ E → X \ (F0 + 2e) given by f (x) = x + 2e has no continuous injective extension to E.

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

575

APPENDIX JUSSI VÄISÄLÄ

We show that connected metric manifolds have the property S (see Definition 4.1). We first recall a result in Euclidean topology. Lemma A.1. Let C ⊂ Rn be a continuum, let H be a component of Rn \ C, and let W be a connected open neighborhood of C. Then W ∩ H is connected. Proof. Set V = W ∪(Rn \H ). The set Rn \H is connected by [16, Section V.14.4]; see also [16, Section V.5]. Hence V is connected. Moreover, V and H are open with V ∪ H = Rn . Since W ∩ H = V ∩ H , the lemma follows from [16, Cor. 2, p. 112]. Theorem A.2. Let X be a metric connected n-manifold, n ≥ 2. Then X satisfies the separation axiom S. Proof. Let F ⊂ X be compact. For each p ∈ F choose neighborhoods U (p) ⊂ V (p) of p such that diam V (p) < diam X/2 and such that the pair (V (p), U (p)) is homeomorphic to (Rn , B n ). Choose a finite set P ⊂ F such that F is covered by {U (p) : p ∈ F }. Choose λ > 0 such that (1) λ is a Lebesgue number for this covering and (2) λ < dist(U (p), X \ V (p)) for all p ∈ P . We show that λ has the property of Definition 4.1. Suppose that A ⊂ F is a finite disjoint union of continua of diameters less than λ, and that A separates points x, y in X. We show that x and y are separated by a component of A. Case 1. There is p ∈ P such that U (p) contains x and y. Since A∩V (p) separates x, y in V (p), it follows from [16, Section V.14.3] that a component E of A ∩ V (p) separates x, y in V (p). The E-component C of A meets U (p), since otherwise x and y are joined by the connected set U (p) in V (p) \ E. Since diam C < λ < dist(U (p), X \ V (p)), we have C ⊂ V (p), and hence C = E. Assume that C does not separate x, y in X. Then there is an arc γ ⊂ X \C from x to y. Choose a compact set B with C ⊂ B ⊂ V (p) such that (V (p), B) is homeomorphic to (Rn , B¯ n ). Since γ  ⊂ V (p), there is a subarc α of γ joining x and V (p) \ B in V (p). Similarly, let β ⊂ γ join y and V (p) \ B in V (p). Now the connected set α ∪ β ∪ (V (p) \ B) joins x and y in V (p) \ C. This contradiction proves Case 1. General case. Let C be a component of A. By a hole of C we mean a component of X\C. Since diam C < λ, there is p ∈ P with C ⊂ U (p). Since X\U (p) is connected, it lies in a hole H of C. Since diam U (p) < diam X/2, we have diam H > diam X/2. It follows that each component C of A has a unique hole, called the outer hole of C, characterized by the property diam H > diam X/2 and also by the property that it contains X \ U (p) whenever p ∈ P and C ⊂ U (p).

576

BALOGH AND KOSKELA

Assume that no component of A separates x, y. Then, for each component C of A, the points x and y lie in the same hole of C. If this is not the outer hole, then x, y ∈ U (p) for some p ∈ P , and we are in Case 1. Hence we may assume that x and y lie in the outer hole of each component of A. Every arc α ⊂ X from x to y meets A. Let k(α) be the number of the components of A meeting α. Choose α such that k(α) is minimal. Let x1 be the first point of α in A, let C be the x1 -component of A, and let y1 be the last point of α in C. Then the subarcs α[x, x1 ) and α(y1 , y] lie in the outer hole H of C. Choose p ∈ P with C ⊂ U (p). Now U (p) \ (A \ C) is open and contains C. Let W be its C-component. Since U (p) is homeomorphic to Rn , we can argue, as in Case 1, to conclude that H ∩ U (p) is connected, hence a component of U (p) \ C. Applying Lemma A.1 with Rn , H replaced by U (p), H ∩ U (p), we see that W ∩ H is connected. Choose x ∈ α[x, x1 ) and y ∈ α(y1 , y] such that the arcs α[x , x1 ] and α[y1 , y] lie in W . Join x and y by an arc β in W ∩ H . Now α[x, x ] ∪ β ∪ α[y , y] contains an arc α from x to y with k(α ) < k(α). This contradicts the minimality of k(α) and proves the theorem. Remark. The following stronger result is true: Let X be a metric connected nmanifold, n ≥ 2, and let F ⊂ X be compact. Then there is λ > 0 such that if A ⊂ F is a compact set whose components have diameters less than λ and if A separates points x, y in X, then a component of A separates x, y in X. The proof is more complicated than that of Theorem A.2. Acknowledgements. This research was begun when the second author was visiting at the University of Berne. We wish to thank Hans Martin Reimann for his interest in and support of this work. We also thank the referee for valuable comments. References [1]

L. Ahlfors and A. Beurling, Conformal invariants and function-theoretic null-sets, Acta Math. 83 (1950), 101–129. [2] C. Bishop, Some homeomorphisms of the sphere conformal off a curve, Ann. Acad. Sci. Fenn. Ser. A I Math. 19 (1994), 323–338. [3] , Non-removable sets for quasiconformal and bilipschitz mappings in Rn , preprint. [4] M. Bourdon and H. Pajot, Poincaré inequalities and quasiconformal structure on the boundary of some hyperbolic buildings, Proc. Amer. Math. Soc. 127 (1999), 2315–2324. [5] F. W. Gehring and J. Väisälä, Hausdorff dimension and quasiconformal mappings, J. London Math. Soc. (2) 6 (1973), 504–512. [6] J. Heinonen and P. Koskela, Definitions of quasiconformality, Invent. Math. 120 (1995), 61– 79. [7] , Quasiconformal maps in metric spaces with controlled geometry, Acta Math. 181 (1998), 1–61. [8] P. W. Jones and S. Smirnov, On removable sets for quasiconformal mappings and Sobolev functions, preprint. [9] R. Kaufman and J.-M. Wu, On removable sets for quasiconformal mappings, Ark. Mat. 34 (1996), 141–158. [10] A. Korányi and H. M. Reimann, Foundations for the theory of quasiconformal mappings on

QUASICONFORMALITY, QUASISYMMETRY, AND REMOVABILITY

[11] [12] [13] [14] [15] [16] [17] [18] [19]

577

the Heisenberg group, Adv. Math. 111 (1995) 1–87. T. J. Laakso, Ahlfors Q-regular spaces with arbitrary Q admitting a weak Poincaré inequality, to appear in Geom. Funct. Anal. O. Martio and R. Näkki, Continuation of quasiconformal mappings (in Russian), Sibirsk. Mat. Zh. 28 (1987), 162-170. P. Mattila, Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability, Cambridge Stud. Adv. Math. 44, Cambridge Univ. Press, Cambridge, 1995. C. McMullen, Area and Hausdorff dimension of Julia sets of entire functions, Trans. Amer. Math. Soc. 300 (1987), 329–342. F. Morgan, Geometric Measure Theory: A Beginner’s Guide, Academic Press, Boston, 1988. M. H. A. Newman, Elements of the Topology of Plane Sets of Points, 2d ed., Cambridge Univ. Press, New York, 1961. J. Väisälä, Removable sets for quasiconformal mappings, J. Math. Mech. 19 (1969/1970), 49–51. R. L. Wilder, Topology of Manifolds, Amer. Math. Soc. Colloq. Publ. 32, Amer. Math. Soc., New York, 1949. J.-M. Wu, Removability of sets for quasiconformal mappings and Sobolev spaces, Complex Variables Theory Appl. 37 (1998), 491–506.

Balogh: Mathematisches Institut, Universität Bern, Sidlerstrasse 5, CH-3012 Bern, Switzerland; [email protected] Koskela: Department of Mathematics, University of Jyväskylä, P.O. Box 35, FIN-40351 Jyväskylä, Finland; [email protected].fi Väisälä: Department of Mathematics, University of Helsinki, P.O. Box 4, FIN-00014 Helsinki, Finland; jussi.vaisala@helsinki.fi