Coxeter groups, Lorentzian lattices, and K3 surfaces. 25 June, 27 July 1998

Richard E. Borcherds, ^{∗}

D.P.M.M.S., 16 Mill Lane, Cambridge, CB2 1SB, England.

e-mail: reb@dpmms.cam.ac.uk

www home page www.dpmms.cam.ac.uk/˜ reb

Contents.

1. Introduction.

2. Notation and statement of main theorem.

3. Proof of main theorem.

4. The structure of Γ_{Ω}.

5. Examples.

1. Introduction.

The main result of this paper describes the normalizer N_{W_{Π}}(W_{J}) of a finite parabolic subgroup W_{J} of a (possibly infinite) Coxeter group W_{Π}. More generally we describe N_{W_{Π}.Γ_{Π}}(W_{J}) where Γ_{Π} is a group of diagram automorphisms of the Coxeter diagram Π of W_{Π}. By taking Π to be Conway’s Coxeter diagram of the reflection group of II_{1,25} we compute the automorphism groups of some Lorentzian lattices and K3 surfaces.

In the case when W_{Π} is a finite Coxeter group (and Γ_{Π} = 1) the normalizer of W_{J} has been described by Howlett [H]. His result states that N_{W_{Π}}(W_{J}) is a split extension W_{J}.W′_{J}, where W′_{J} = W_{Ω}.Γ_{Ω} is in turn a split extension with W_{Ω} a Coxeter group and Γ_{Ω} a more mysterious group acting on Ω. Howlett showed by case by case analysis that if Π is connected (and W_{Π} is finite) then Γ_{Ω} is an elementary abelian 2-group and is a subgroup of Aut(J). When W_{Π} is infinite the normalizer N_{W_{Π}}(W_{J}) has a similar structure, except that the group Γ_{Ω} can be more complicated. Although there is still a canonical map from Γ_{Ω} to Aut(J), the kernel can be non trivial, though it has finite cohomological dimension. The kernel is trivial in the case of finite W_{Π} considered by Howlett because any finite group of finite cohomological dimension must be trivial. For example, the case when J = A_{1}, Γ_{Π} = 1 has been done by Brink [Br], who showed that Γ_{Ω} is a free group (and therefore has cohomological dimension at most 1). We will extend Brink’s result to Coxeter diagrams of arbitrary finite reflection groups. More precisely we construct a category Q_{4} using Π and J and prove that the classifying space of this category is a classifying space of the group Γ_{Ω}. The main point about this category Q_{4} is that it is often finite and can often be written down explicitly, in which case we can easily read off a presentation of Γ_{Π}. For example, if J = A_{1} we show that the classifying space of this category Q_{4} is 1-dimensional, so its fundamental group is free and we recover Brink’s result. After writing this paper I discovered that Brink and Howlett had previously announced a related description of the normalizer of a parabolic subgroup of a Coxeter group; see [B-H], and see example 2.8 for the relation between their result and theorem 2.7.

For later applications we need some generalizations as follows. First of all, instead of calculating the normalizer of W_{J} in a Coxeter group W_{Π}, we calculate the normalizer in an

∗ Supported by a Royal Society professorship and an NSF grant. extension W_{Π}.Γ_{Π}, where Γ_{Π} is a group of diagram automorphisms. Secondly, we sometimes want to compute not the full normalizer, but a subgroup with image contained in some subgroup Γ_{J} of Aut(J). Thirdly, we sometimes want to vary the choice of the Coxeter group W_{Ω}, which we do by varying a certain normal subgroup R of Γ_{J}. For example, in calculating the automorphism groups of K3 surfaces we take W_{Ω} to be generated by reflections of norm −2 vectors rather than by all reflections, so we take R = 1.

Section 2 contains a statement of the main result (theorem 2.7) describing a classifying category for the group Γ_{Ω}, and section 3 contains the proof of this result. Section 4 contains some more information about the structure of Γ_{Ω}. In section 5 we give some applications of theorem 2.7, and in particular show how to describe the automorphism groups of some Lorentzian lattices by embedding them in II_{1,25} and using the description of Aut(II_{1,25}) in [C]. The idea of studying Lorentzian lattices by embedding them as orthogonal complements of root lattices in II_{1,25} comes from Conway and Sloane ([C-S]).

Work of I. Piatetski-Shapiro and I. R. Shafarevich [P-S] shows that there is a map from the automorphism group of a K3 surface to the group of automorphisms of its Picard lattice modulo the group generated by reflections of norm −2 vectors which has finite kernel and co-finite image, so in practice if we want to describe the automorphisms of a K3 surface the main step is to calculate the automorphism group of its Picard lattice. Kondo showed in [K] that the automorphism groups of some K3 surfaces could be studied by embedding their Picard lattice as the orthogonal complement of a root lattice in II_{1,25}. We use Kondo’s idea to describe the automorphism groups of some K3 surfaces in terms of combinatorics of the Leech lattice. In particular we reprove some results of Vinberg [V] on the “most algebraic” K3 surfaces and extend them to the “next most algebraic” K3 surface. Kondo showed in [K] that the automorphism group of the Kummer surface of a generic genus 2 Jacobian was generated by the classically known automorphisms together with some new automorphisms found by Keum [Ke], and we show how to use Kondo’s results to describe the structure of this group. Kondo and Keum [K-K] have recently proved similar results for some Kummer surfaces associated to the products of two elliptic curves.

Kondo recently found another mysterious connection between automorphism groups of K3 surfaces and Niemeier lattices [K98], and used this to give a short proof of Mukai’s classification [Mu] of the finite groups that act on K3 surfaces.

I would like to thank I. Cherednik, I. Grojnowski, R. B. Howlett, J. M. E. Hyland, S. Kondo, U. Ray, and G. Segal for their help.

2. Notation and statement of main theorem.

This section states the main result (theorem 2.7) describing normalizers of parabolic subgroups of Coxeter groups.

We recall some basic definitions about Coxeter systems. For more about them see [Hi] or [Bo]. A pair (W, S) is called a Coxeter system if W is a group with a subset S such that W has the presentation

⟨s : s ∈ S|(ss′)^{m_{ss′}} = 1 when m_{ss′} < ∞⟩

where m_{ss′} ∈ {1, 2, 3, …, ∞} is the order of ss′, and m_{ss′} = 1 if and only if s = s′. A diagram automorphism of S is an automorphism of the set S that extends to an automorphism of the group W, and Aut(S) means the group of diagram automorphisms of S. We say that (W, S) is irreducible if S is not a union of two disjoint commuting subsets. The number of elements of S is called its rank. The Coxeter system is called spherical if W has finite order. The irreducible spherical Coxeter diagrams are A_{n} (n ≥ 1), B_{n} = C_{n} (n ≥ 2), D_{n} (n ≥ 4), E_{6}, E_{7}, E_{8}, F_{4}, G_{2}^{(n)} = I_{2}(n) (n ≥ 5), H_{3}, and H_{4}. It is also sometimes useful to define the Coxeter diagrams B_{1} = C_{1} = A_{1}, D_{3} = A_{3}, D_{2} = A_{1}^{2}, E_{5} = D_{5}, E_{4} = A_{4}, E_{3} = A_{2}A_{1}, G_{2}^{(4)} = B_{2} = C_{2}, G_{2}^{(3)} = A_{2}, G_{2}^{(2)} = A_{1}^{2}.

If (W_{Π}, Π) is a Coxeter system then we write V_{Π} for the (possibly infinite dimensional) real vector space with a basis of elements e_{s} for s ∈ Π, and put a symmetric bilinear form on V_{Π} by defining

(e_{s}, e_{s′}) = 2 cos(π/m_{ss′}).

Note that we normalize the roots e_{s} so that they have norm (e_{s}, e_{s}) = −2 rather than 1; this is done to be consistent with the usual conventions in algebraic geometry.

The Coxeter group W_{Π} acts on V_{Π} with the element s ∈ Π ⊆ W_{Π} acting as the reflection v ↦ v + (v, e_{s})e_{s} in the hyperplane e_{s}^{⊥}. Any subgroup Γ_{Π} of Aut(Π) acts on V_{Π} by permutations of the elements e_{s}, so we get an action of W.Γ_{Π} on V_{Π}, and hence on the dual space V_{Π}^{∗}. We write Δ^{+} for the set of positive roots of W_{Π}. We define the fundamental chamber C_{Π} ⊆ V_{Π}^{∗} of W_{Π} by

C_{Π} = {x ∈ V_{Π}^{∗}|x(r) ≥ 0 for all r ∈ Π and x(r) > 0 for almost all r ∈ Δ^{+}}.

(Recall that “almost all” means “all but a finite number of”.) A theorem due independently to Tits and Vinberg states that no two distinct points of C_{Π} are conjugate under W_{Π}, and the subgroup of W_{Π} fixing all points of some subset A of W_{Π} is generated by the reflections in the faces of C_{Π} containing A. In particular W_{Π} acts simply transitively on the conjugates of C_{Π}. The union W_{Π}(C_{Π}) of all conjugates of C_{Π} under W_{Π} is given by

W_{Π}(C_{Π}) = {x ∈ V_{Π}^{∗}|x(r) > 0 for almost all r ∈ Δ^{+}}.

In particular W_{Π}(C_{Π}) is convex and closed under multiplication by positive real numbers. If x ∈ W_{Π}(C_{Π}) then the set of roots vanishing on x is a finite root system.

Note that W_{Π}(C_{Π}) is usually slightly smaller than the Tits cone, which is defined in the same way except that we omit the condition that x(e_{s}) > 0 for all but a finite number of s in the definition of the fundamental domain. The Tits cone can be thought of as obtained from W_{Π}(C_{Π}) by “adding some boundary components”. The reason for using W_{Π}(C_{Π}) rather than the Tits cone is that the cone W_{Π}(C_{Π}) has the property that the subgroup of W_{Π} fixing any vector of it is finite.

We fix a spherical subset J of Π. In particular we get a spherical Coxeter system (W_{J}, J). Suppose K is an isometry of Coxeter diagrams from J into Π. We write W_{K} for the finite reflection group generated by K(J). There is a natural homomorphism p : N_{W_{Π}.Γ_{Π}}(W_{K}) ↦ Aut(J). We let N_{W_{Π}.Γ_{Π}}(W_{K}; Γ_{J}) be the subgroup of elements whose image is in a subgroup Γ_{J} of Aut(J). We are interested in describing the group N_{W_{Π}.Γ_{Π}}(W_{J}; Γ_{J}). Most of the time we take Γ_{J} = Aut(J) in which case N_{W_{Π}.Γ_{Π}}(W_{J}; Γ_{J}) = N_{W_{Π}.Γ_{Π}}(W_{J}), but it is occasionally useful to use other values of Γ_{J}; see example 5.7 and theorem 4.1.

We define W′_{K} to be the subgroup of N_{W_{Π}.Γ_{Π}}(W_{K}; Γ_{J}) mapping K(J) to itself.

Lemma 2.1. N_{W_{Π}.Γ_{Π}}(W_{K}; Γ_{J}) = W_{K}.W′_{K}.

Proof. This follows immediately from the fact that W_{K} acts simply transitively on the Weyl chambers of W_{K}, and W′_{K} is the subgroup of N_{W_{Π}.Γ_{Π}}(W_{K}; Γ_{J}) fixing a Weyl chamber of W_{K}. This proves lemma 2.1.

The group W′_{K} acts on the subspace V_{Π}^{∗W_{K}} of V_{Π}^{∗} of all vectors fixed by W_{K}. We now construct a reflection group W_{Ω_{K}} acting on V_{Π}^{∗W_{K}}. We choose a normal subgroup R of Γ_{J}. (The subgroup R is used to control the reflection group W_{Ω} defined below. Often we want W_{Ω} to be as large as possible and we take R = Γ_{J}, but sometimes we want to take a smaller W_{Ω}; see examples 5.3, 5.4 and 5.5.) We define the group W_{Ω_{K}} to be the subgroup of W_{Π} ∩ W′_{K} generated by elements w ∈ W_{Π} ∩ W′_{K} such that w acts on V_{Π}^{∗W_{K}} as a reflection and acts on J as an element of R. We define Ω_{K} to be the Coxeter diagram of W_{Ω_{K}} and C_{Ω_{K}} to be its fundamental chamber. If K is the identity map from J to J ⊆ Π then we write W_{Ω}, C_{Ω}, and Ω instead of W_{Ω_{K}}, C_{Ω_{K}}, and Ω_{K}. Note that W_{Ω_{K}} is obviously contained in the inverse image of R in W_{Π} ∩W′_{K}, but can be much smaller; see for example the discussion of D_{4} in example 5.7.

We define the group Γ_{Ω_{K}} to be the subgroup of W′_{K} of elements w with w(C_{Ω_{K}}) = C_{Ω_{K}}.

Lemma 2.2. The group W′_{K} is a semidirect product W′_{K} = W_{Ω_{K}} .Γ_{Ω_{K}}.

Proof. We first show that the group W_{Ω_{K}} acts faithfully on V_{Π}^{∗W_{K}}. More generally we will show that if w ∈ W_{Π} ∩W′_{K} acts trivially on V_{Π}^{∗W_{K}}, then w = 1. To see this we observe that w fixes the point x ∈ C_{Π} ∩ V_{Π}^{∗W_{K}} such that x(e_{s}) = 0 if s ∈ K(J) and x(e_{s}) = 1 if s ∉ K(J). Therefore w is in the subgroup W_{K} of W_{Π} generated by the simple reflections of W_{Π} fixing x. On the other hand w maps K(J) into itself as w ∈ W′_{K}. This implies that w = 1 because 1 is the only element of W_{K} mapping K(J) into itself. This proves that the group W_{Ω_{K}} acts faithfully on V_{Π}^{∗W_{K}}.

Lemma 2.2 now follows from the fact that W_{Ω_{K}} acts simply transitively on the conjugates of C_{Ω_{K}} under W′_{K}, and Γ_{Ω_{K}} is the stabilizer of C_{Ω_{K}}. This proves lemma 2.2.

Warning: the group Γ_{Ω_{K}} need not act faithfully on V_{Π}^{∗W_{K}} (though it does act faithfully on V_{Π}^{∗W_{K}} × J).

We define a classifying category of a group Γ to be a category whose geometric realization is a classifying space for Γ. (Recall from [Q] that the geometric realization of a category is a space with a 0-cell for each object and an n-cell for each sequence of n composable morphisms if n > 0.) For example, the category with one object whose morphisms are the elements of Γ (with composition given by group multiplication) is a classifying category for Γ.

We have more or less reduced the problem of describing N_{W_{Π}.Γ_{Π}}(W_{J}; Γ_{J}) to that of describing Γ_{Ω_{K}}. (The Coxeter diagram Ω of W_{Ω} can be described once we know Γ_{Ω}.) The main theorem of this paper describes Γ_{Ω_{K}} by giving an explicit classifying category for it. To define this category we need some more definitions.

Suppose that S is the Coxeter diagram of a finite reflection group G of a finite dimensional vector space with no vectors fixed by G. Fix a Weyl chamber C of G, so that the walls of C correspond to the points of S. Then there is a unique element σ_{S} of G taking C to −C called the opposition involution. The involution −σ_{S} acts on the the Coxeter diagram S, and its action on S does not depend on the choice of C. This action can be described as follows. The points of the Coxeter diagram correspond to the simple roots of C. This set of roots is the same as the set of simple roots of σ_{S}(C) = −C multiplied by −1. Hence −σ_{S} acts on this set of simple roots, in other words on the Coxeter diagram S. The involution −σ_{S} of S can be described explicitly as follows. On diagrams of type A_{1}, B_{n} = C_{n}, D_{2n}, E_{7}, E_{8}, F_{4}, G_{2}^{(2n)}, H_{3}, H_{4} for any n ≥ 1 the involution −σ_{S} is the trivial automorphism of S, while for diagrams of types A_{n+1}, D_{2n+1}, E_{6}, G_{2}^{(2n+1)} for n ≥ 1 the involution −σ_{S} is the unique nontrivial automorphism of the Coxeter diagram S. Finally if the diagram S is a union of connected components then −σ_{S} acts on each connected component as described above.

Suppose that J and S are Coxeter diagrams. Suppose that K and K′ are two isometries from J into S. We define K and K′ to be adjacent if there is a point s of S not in K(J) such that K(J) ∪ s is spherical and σ_{K(J)∪s}σ_{K(J)} takes K to K′. If K is adjacent to K′ then K′ is adjacent to K. We define two isometries K, K′ of J into S to be associate if there is a sequence of isometries K = K_{1}, K_{2}, …, K_{n} = K′ from J into S such that K_{i} and K_{i+1} are adjacent for 1 ≤ i < n. We write \overline{K} for the equivalence class of all isometries from J into S that are associate to K.

Remark. The term associate is closely related to the same term in algebraic group theory, where two parabolic subgroups are called associate if their Levi factors are conjugate. There is a small difference because a standard parabolic subgroup corresponds to a subdiagram of the Dynkin diagram, while we work with labeled subdiagrams of a Coxeter diagram.

Suppose that S is a spherical Coxeter diagram. Recall that R is a normal subgroup of Γ_{J} ⊆ Aut(J). We define an isometry K : J ↦ S to be R-reflective if K is adjacent to r(K) for some r ∈ R. We define the equivalence class \overline{K} to be R-reflective if at least one of its elements is R-reflective. (The reason for the name “R-reflective” appears in lemmas 3.8 and 3.9.)

Example 2.3. Suppose J is A_{1} and S is A_{3}, with the isometries from J into S labeled as K_{1}, K_{2}, K_{3} in the obvious way. Then the isometry K_{1} is adjacent to K_{2} as σ_{K_{1}(J)∪K_{2}(J)}σ_{K_{1}} takes K_{1} to K_{2}. Similarly K_{2} is adjacent to K_{3}, and K_{1} is associate to K_{3} but not adjacent to K_{3}. The isometries K_{1} and K_{3} are adjacent to themselves and K_{2} is not adjacent to itself, so the isometry K_{2} is not R-reflective but the isometries K_{1} and K_{3} are. So K_{2} is not R-reflective but the equivalence class \overline{K_{2}} is.

Example 2.4. Suppose K is D_{5} and S is D_{6}. Then there are exactly two isometries K_{1}, K_{2} : J ↦ S of S, which are adjacent to each other but not to themselves. These two isometries are exchanged by the nontrivial automorphism of D_{5}. Hence if R contains just 1 ∈ Aut(D_{5}) then K_{1}, K_{2}, and the equivalence class \overline{K_{1}} = \{K_{1}, K_{2}\} are not R-reflective, but if R is the whole of Aut(D_{5}) = Z/2Z then all of them are R-reflective.

Example 2.5 Suppose K is A_{3} and S is D_{5}. Then there are 8 isometries K : J ↦ S. These form two equivalence classes under the relation of being associate, one of size 2 and one of size 6. This shows that two isometries from a connected diagram J into S need not be associate to conjugates of each other under Aut(J).

Example 2.6 Suppose J is A_{2}. If S is A_{n} for n ≥ 2 then there are two equivalence classes of isometries K : J ↦ S, which are exchanged by the nontrivial automorphism of A_{2}. However if S is D_{n} (n ≥ 4), E_{6}, E_{7}, or E_{8} then there is only one equivalence class, as the A_{2} can be reversed by doing a “three point turn” around the point of valence 3 in S.

We define a poset P_{3}^{+} as follows. The objects of P_{3}^{+} are pairs (S, \overline{K}) consisting of a spherical subdiagram S of Π and an equivalence class \overline{K} of isometries from J into S. We define the partial order on P_{3}^{+} by putting (S, \overline{K}) ≤ (S′, \overline{K′}) if S ⊆ S′ and \overline{K} ⊆ \overline{K′}. We define P_{3} to be the sub-poset of P_{3}^{+} of elements (S, \overline{K}) such that \overline{K} is not R-reflective.

Note that the condition that \overline{K} is not R-reflective is quite restrictive and implies that S is usually not much larger than J and in any case has at most twice the rank of J. In particular S cannot contain any root orthogonal to K(J) as this implies that K is R-reflective for any R.

Suppose that P is a poset acted on by a group G. We define the homotopy quotient of P by G to be the following category Q. The objects of Q are the elements of P. The morphisms from p_{1} ∈ P to p_{2} ∈ P correspond to the group elements g ∈ G such that g(p_{1}) ≤ p_{2}, and composition of morphisms is given by multiplication of group elements. If G is trivial this is the usual category associated to the poset P, and if P has just one point this is the usual category with one object associated to the group G. If we take a full set of representatives of the orbits of G on P, then the full sub category of Q with these objects is a skeleton of the category Q.

The poset P_{3} is acted on by the group Γ_{J} × Γ_{Π} ⊆ Aut(J) × Aut(Π). We define Q_{3} to be the homotopy quotient of P_{3} by Γ_{J} ×Γ_{Π}, and we construct the category Q_{4} as a skeleton of Q_{3} as above. In other words the objects of Q_{4} are a complete set of representatives for the orbits of Γ_{J} × Γ_{Π} on the elements of P_{3} and the morphisms from (S, \overline{K}) to (S′, \overline{K′}) correspond to group elements γ ∈ Γ_{J} × Γ_{Π} such that γ((S, \overline{K})) ≤ (S′, \overline{K′}).

The main result of this paper is the following description of the classifying space of Γ_{Ω}.

Theorem 2.7. Suppose we are given the following objects. (W_{Π}, Π) A Coxeter system.

Γ_{Π} A subgroup of Aut(Π). (W_{J}, J) A spherical Coxeter system with J ⊆ Π.

Γ_{J} A subgroup of Aut(J). R A normal subgroup of Γ_{J}.

Define W_{Ω}, Γ_{Ω}, and Q_{4} as above, so that the group N_{W_{Π}.Γ_{Π}}(W_{J}; Γ_{J}) has the structure

W_{J}.W_{Ω}.Γ_{Ω}

where W_{Ω} is a Coxeter group. Then the component of Q_{4} containing the object (J, \overline{id_{J}}) is a classifying category for the group Γ_{Ω}.

Theorem 2.7 gives a presentation of the group Γ_{Ω} because Γ_{Ω} is the fundamental group of the category Q_{4} with respect to the basepoint (J, \overline{id_{J}}), and it is easy to write down a presentation of the fundamental group of any connected category Q as follows. Choose a spanning tree T for the underlying 1-complex of Q (which has a point for each object of Q and a 1-cell for each morphism). Then the fundamental group Γ_{Ω} of Q has a presentation as follows. The group Γ_{Ω} has a generator \overline{g} for each morphism g of Q. The relations are \overline{gh} = \overline{g}\overline{h} whenever gh is defined, and g = 1 for g in the spanning tree.

Example 2.8. Suppose we put R = Γ_{J} = Aut(J) and Γ_{Π} = 1. Then we see from theorem 2.7 that N_{W_{Π}}(W_{J}) = W_{J}.W_{Ω}.Γ_{Ω}, where Γ_{Ω} is the fundamental group of the component of Q_{4} corresponding to J. So in particular theorem 2.7 describes normalizers of finite parabolic subgroups of Coxeter groups. More generally, Brink and Howlett [B-H] have described a presentation for normalizers of possibly infinite parabolic subgroups of Coxeter groups. It is not trivial to see that the presentation given by [B-H] is equivalent to the one given by theorem 2.7 (though of course this follows from the fact that they are both presentations of the same group). Howlett pointed out to me that their result only requires considering subdiagrams S of Π whose rank is at most 2 + rank(J) to get the relations of Γ_{Ω}, and of rank at most 1 + rank(J) to get the generators of Γ_{Ω}. It seems possible that a similar simplification could be made to theorem 2.7 if all that is required is a presentation rather than a classifying space. Perhaps the natural map from π_{i}(Q_{4}^{j}) to π_{i}(Q_{4}) is an isomorphism for i < j and an epimorphism for i = j, where Q_{4}^{j} is the full sub-category of Q_{4} whose objects are the elements (S, \overline{K}) such that rank(S) ≤ rank(J)+j. If so, the map from π_{1}(Q_{4}^{2}) to π_{1}(Q_{4}) would be an isomorphism, so this would give a closer connection to the presentation of Brink and Howlett. Their result also suggests that theorem 2.7 could be generalized by allowing W_{J} to be infinite and modifying the definition of Q_{4} to allow subdiagrams S such that W_{J} has finite index in W_{S}.

3. Proof of main theorem.

This section gives the proof of the theorem 2.7. The idea of the proof is to construct categories and functors according to the following diagram.

Q_{1} ⟵ Q_{2} ⟶ Q_{3} ⟵ Q_{4}

It is easy to show that a component of Q_{1} is a classifying category for Γ_{Ω}. We also show that the functors between the categories are all homotopy equivalences, so a component of Q_{4} is a classifying category for Γ_{Ω}, which is what we wanted to prove.

We define an isometry from J into the roots of W_{Π} to be primitive if it is conjugate under W_{Π} to an isometry from J into Π. An example of a non-primitive isometry is an isometry from A_{1}^{4} into the roots of D_{4}.

We define a category Q_{1} as follows. We define the poset P_{1} to be the poset of pairs (C, K) where K is a primitive isometry from J into the (possibly non-simple) roots of W_{Π}, and C is a Weyl chamber of the reflection group W_{Ω_{K}} of V_{Π}^{∗W_{K}}. The partial order on P_{1} is the trivial one with (C_{1}, K_{1}) ≤ (C_{2}, K_{2}) if and only if (C_{1}, K_{1}) = (C_{2}, K_{2}). The objects of P_{1} are acted on in the natural way by the group W_{Π}.Γ_{Π} via its action on V_{Π}, and by the group Γ_{J} via its action on J. We define the category Q_{1} to be the homotopy quotient of P_{1} by the group Γ_{J} × W_{Π}.Γ_{Π}.

Lemma 3.1. The component of Q_{1} containing the object id_{J} : J ↦ J ⊆ Π is a classifying category for the group Γ_{Ω}.

Proof. This follows immediately from the fact that Q_{1} is a groupoid such that the automorphism group of the object K is the group Γ_{Ω_{K}}. This proves lemma 3.1.

Let C_{Π} be the Weyl chamber of W_{Π} defined in section 2. By a face of C_{Π} we mean a nonempty intersection of C_{Π} with some of the hyperplanes bounding C_{Π}. The faces of C_{Π} of codimension n correspond to the spherical subdiagrams of Π of rank n. We define a Π-cell to be a conjugate of a face of C_{Π} under W_{Π}. The cone X is the union of all Π-cells, and the intersection of two Π-cells is either empty or another Π-cell.

We define a category Q_{2} and posets P_{2}, P_{2}^{+} as follows. The objects of the poset P_{2}^{+} are the pairs (D, K) where K is a primitive isometry from J into the roots of W_{Π}, and D is a Πcell contained in V_{Π}^{∗W_{K}}. We define the partial order on P_{2}^{+} by saying (D_{1}, K_{1}) ≤ (D_{2}, K_{2}) if D_{1} ⊆ D_{2} and K_{1} = K_{2}. We define P_{2} to be the sub-poset of P_{2}^{+} of elements (D, K) such that D is not contained in a reflection hyperplane of W_{Ω_{K}}. The category Q_{2} is defined to be the homotopy quotient of P_{2} by Γ_{J} × W_{Π}.Γ_{Π}.

Lemma 3.2. Suppose G is a group, P_{1} and P_{2} are G-posets, and f is a morphism of Gposets from P_{2} to P_{1}. Also suppose that for any Y ∈ P_{1} the poset f^{−1}(Y) is contractible (in other words the corresponding simplicial complex is contractible). Then the functor induced by f between the homotopy quotient categories Q_{2}, Q_{1} of the posets P_{2} and P_{1} by the group G is a homotopy equivalence.

Proof. If f is a functor from a category Q_{2} to a category Q_{1} and Y is an object of Q_{1} then we write f^{−1}(Y) for the fiber of f over Y, in other words the sub category of Q_{2} whose morphisms are those mapped to the identity of Y by f. We write Y \f for the category consisting of pairs (X, v) with v : Y ↦ f(X), where a morphism from (X, v) to (X′, v′) is a morphism w : X ↦ X′ such that f(w)v = v′. Then a result due to Quillen (the corollary to theorem A on page 9 of [Q]) states that f is a homotopy equivalence provided that for all Y in Q_{1} the poset f^{−1}(Y) is contractible and the functor from f^{−1}(Y) to Y \f taking X to (X, id_{Y}) has a right adjoint. (Here id_{Y} is the identity morphism of Y.)

We will use Quillen’s result to show that f is a homotopy equivalence. For any object Y of P_{1} the category f^{−1}(Y) is just the category of the poset f^{−1}(Y), which is contractible by assumption. So it only remains to check the condition about the existence of a right adjoint from Y \f to f^{−1}(Y). The category Y \f has as objects pairs (X, v) with v ∈ G, v(Y) ≤ f(X) and there is a morphism from (X, v) to (X′, v′) if and only if v^{−1}(X) = v^{′−1}(X′), in which case the morphism is unique. We define a functor g from Y \f to f^{−1}(Y) on objects by g((X, v)) = v^{−1}(X). It is easy to check that this extends in a unique way to morphisms. It is a right adjoint to f because Y ≤ g((X, v)) if and only if there is a morphism (necessarily unique) from f(Y) to (X, v), both conditions being equivalent to v(Y) ≤ X. This shows that the conditions of Quillen’s result are satisfied, so f is a homotopy equivalence. This proves lemma 3.2.

Lemma 3.3. The functor f is a homotopy equivalence from Q_{2} to Q_{1}.

Proof. By lemma 3.2 it is sufficient to check that for each Y ∈ P_{1}, the sub poset f^{−1}(Y) of P_{2} is contractible. The poset f^{−1}(Y) is the poset of a cell decomposition of a convex cone in a real vector space. As any convex set is contractible, the poset f^{−1}(Y) is also contractible. This proves lemma 3.3.

Lemma 3.4. Suppose (W, S) is a spherical Coxeter system acting on the vector space V_{S} with Weyl chamber C. Suppose K is an isometry from J into S. Let V_{S}^{∗W_{K}} be the subspace of V^{∗} fixed by W_{K}, where W_{K} is the reflection group whose simple roots are the points K(J). The walls of C ∩ V_{S}^{∗W_{K}} correspond to the points in S not in the image of K; let s be one of these points and let s^{⊥} ∩ V_{S}^{∗W_{K}} be the wall in V_{S}^{∗W_{K}} corresponding to s. Choose w ∈ W so that w(C) is the (unique) Weyl chamber of W such that w(C) ∩ V_{S}^{∗W_{K}} is the cell in V_{S}^{∗W_{K}} on the other side of s^{⊥} ∩ V_{S}^{∗W_{K}} to C ∩ V_{S}^{∗W_{K}} and such that C and w(C) are both in the same Weyl chamber of W_{K}. Then

w = σ_{K(J)∪s}σ_{K(J)}.

Proof. We can reduce to the case when S = K(J)∪s, so that V_{S}^{∗W_{K}} is one dimensional and s^{⊥} ∩ V_{S}^{∗W_{K}} is just the point 0. Then σ_{S}(σ_{K(J)}(C)) = −σ_{K(J)}(C) which contains −C ∩V_{S}^{∗W_{K}}, so σ_{S}(σ_{K(J)}(C))∩V_{S}^{∗W_{K}} is a cell on the other side of s^{⊥} ∩V_{S}^{∗W_{K}} to C ∩V_{S}^{∗W_{K}}. Moreover σ_{K(J)}(C) is in the opposite Weyl chamber of K(J) to C, and σ_{S}(σ_{K(J)}(C)) is in the opposite Weyl chamber to σ_{K(J)}(C), so σ_{S}(σ_{K(J)}(C)) is in the same Weyl chamber of K(J) as C. This shows that the element w of the lemma is σ_{S}σ_{K(J)}. This proves lemma 3.4.

A result similar to the following lemma (using subsets of S rather than isometries K : J ↦ S) is given in [H, lemma 5] when S is finite and in [D] for arbitrary S.

Lemma 3.5. Suppose (W, S) is a Coxeter system, J is a spherical Coxeter diagram, and K and K′ are two isometries from J into S. Then K and K′ are conjugate under W if and only if they are associate.

Proof. First suppose that K and K′ are adjacent. Then σ_{K∪K′}(σ_{K}(K)) = K′, so K and K′ are conjugate under W. Next suppose K and K′ are associate. Then by definition we can find a sequence K = K_{1}, K_{2}, …, K_{n} = K′ such that K_{i} and K_{i+1} are adjacent for all i. Hence K = K_{1} and K′ = K_{n} are also conjugate under W. So associate isometries from J into S are conjugate under W.

Conversely, suppose that K and K′ are conjugate by an element w ∈ W. Consider the subspace V_{S}^{∗W_{K}} of V_{S}^{∗}, and the codimension 0 cells in it of the form V_{S}^{∗W_{K}} ∩ C for some Weyl chamber C of W. For any two such cells, for example D = V_{S}^{∗W_{K}} ∩ C and D′ = V_{S}^{∗W_{K}} ∩ w(C), we can find a sequence D = D_{1}, D_{2}, …, D_{n} = D′ such that D_{i} and D_{i+1} are adjacent by a face of codimension 1 in V_{S}^{∗W_{K}}. For each i let C_{i} be the (unique) Weyl chamber whose intersection with V_{S}^{∗W_{K}} is D_{i} and that is contained in the Weyl chamber of W_{K}. We identify each C_{i} with C_{Π} using w_{i}. The set K(J) is a subset of the simple roots of C_{i}, so K_{i} = w_{i}^{−1}(K) maps J to the simple roots of C_{Π}. The isometries K_{i} and K_{i+1} are adjacent for all i, because the element w_{i+1}w_{i}^{−1} mapping C_{i} to C_{i+1} is equal to σ_{K(J)∪s_{i}}σ_{K(J)}, where s_{i} is the simple root of C_{i} orthogonal to D_{i} ∩ D_{i+1} but not to D_{i}. Therefore K = K_{1} and K′ = K_{n} are adjacent. This proves lemma 3.5.

Suppose that P is a W -poset for a group W with the property that if p ≤ w(p) for w ∈ W, p ∈ P then p = w(p). We define the quotient W \P of P by W to be the poset whose elements are the orbits W p of W acting on P, where we put W p ≤ W q if w(p) ≤ q for some w ∈ W. This should not be confused with the homotopy quotient of P by W.

Lemma 3.6. The Γ_{J} × Γ_{Π} posets P_{3}^{+} and W_{Π}\P_{2}^{+} are isomorphic.

Proof. We will construct an isomorphism f of posets from W_{Π}\P_{2}^{+} to P_{3}^{+}. Suppose (D, K) is an element of P_{2}^{+} representing an element of W_{Π}\P_{2}^{+}. We can find an element w of W_{Π} such that w(D) ⊆ C_{Π} and w(K(J)) ⊆ Π. We define f((D, K)) to be (S, \overline{w(K)}), where S is the set of simple roots of C_{Π} orthogonal to w(D). We check that this is well defined even though w is not unique. To prove this we can assume that D ⊆ C_{Π} and K(J) ⊆ Π. Then the different possibilities for w are elements of the group generated by the reflections fixing D and the Weyl chamber of K(J). But these elements take K to an associated isometry K : J ↦ S, so the equivalence class \overline{K} is well defined by lemma 3.5. These elements also take S to S, so S is well defined. This proves that (S, \overline{K}) is uniquely defined.

The isomorphism f of posets from W_{Π}\P_{2}^{+} to P_{3}^{+} obviously preserves the Γ_{J} × Γ_{Π} action on both posets. This proves lemma 3.6.

Lemma 3.7. Suppose (W, S) is a spherical Coxeter system, K is an isometry K : J ↦ S, and s is a point of S not in K(J). Then there is an element of W mapping V_{S}^{∗W_{K}} to itself and acting on V_{S}^{∗W_{K}} as reflection in s^{⊥} ∩ V_{S}^{∗W_{K}} if and only if −σ_{K(J)∪s} maps K(J) to itself. If such an element of W exists, then there is a unique such element w mapping K to itself, given by w = σ_{K(J)∪s}σ_{K(J)}.

Proof. If an element of w maps V_{S}^{∗W_{K}} to itself then there is a unique element of W with the same action on V_{S}^{∗W_{K}} and mapping K(J) to itself because W_{K} acts simply transitively on its Weyl chambers, so we may assume that w maps K(J) to itself. If in addition w acts on V_{S}^{∗W_{K}} as reflection in s^{⊥} ∩ V_{S}^{∗W_{K}} then by lemma 3.4 w must be σ_{K(J)∪s}σ_{K(J)}.

Conversely if −σ_{K(J)∪s} maps K(J) to itself then σ_{K(J)∪s}σ_{K(J)} maps K(J) to itself and acts on V_{S}^{∗W_{K}} as reflection in s^{⊥} ∩ V_{S}^{∗W_{K}}. This proves lemma 3.7.

Lemma 3.8. Suppose K is an isometry from J into a spherical subdiagram S of Π. Then K : J ↦ S is R-reflective if and only if the Π-cell S^{⊥} ∩ C_{Π} is contained in a reflection hyperplane of W_{Ω_{K}} of the form s^{⊥} ∩ V_{Π}^{∗W_{K}} for s ∈ Π.

Proof. First suppose that K : J ↦ S is R-reflective. Then there is a point s ∈ S not in K(J) such that w = σ_{K(J)∪s}σ_{K(J)} acts on K(J) as an element of R. So w is a reflection of W_{Ω} corresponding to the hyperplane s^{⊥} ∩V_{Π}^{∗W_{K}}, and this hyperplane contains S^{⊥} ∩C_{Π}.

Conversely, suppose that S^{⊥} ∩ C_{Π} is contained in a reflection hyperplane of w ∈ W_{Ω_{K}} of the form s^{⊥} ∩ V_{Π}^{∗W_{K}} for s ∈ Π. Then we must have s ∈ S because S^{⊥} ∩ C_{Π} ⊆ s^{⊥}. The element w must be equal to σ_{K(J)∪s}σ_{K(J)}, and this element acts on K as an element of R because w ∈ W_{Ω}. Therefore K : J ↦ S is R-reflective. This proves lemma 3.8.

Lemma 3.9. Suppose K is an isometry from J into a spherical subdiagram S of Π. Then the equivalence class \overline{K} is R-reflective if and only if the Π-cell S^{⊥} ∩ C_{Π} of V_{Π}^{∗W_{K}} corresponding to S is contained in a reflection hyperplane of W_{Ω_{K}}.

Proof. Suppose the cell of V_{Π}^{∗W_{K}} corresponding to S is contained in a reflection hyperplane of W_{Ω_{K}}. Choose a Weyl chamber C_{Π} for W_{Π} such that this reflection hyperplane is a wall of C_{Π} ∩ V_{Π}^{∗W_{K}}. Then by lemma 3.8 the corresponding isometry K′ : J ↦ S is R-reflective, and by lemma 3.5 is associate to K. So \overline{K} is R-reflective.

Conversely suppose that \overline{K} is R-reflective. Then w(K) is R-reflective and has image in S for some w ∈ W_{Ω}. By lemma 3.8 this implies that w(S^{⊥} ∩ C_{Π}) is contained in a reflection hyperplane of W_{Ω}, so the same is true of S^{⊥} ∩ C_{Π}. This proves lemma 3.9.

Lemma 3.10. Suppose P is a G-poset for some group G. Suppose that W is a normal subgroup of G such that if p ≤ q and w(p) ≤ q for some w ∈ W, p, q ∈ P, then w = 1. Then the homotopy quotient Q_{2} of P by G is equivalent to the homotopy quotient Q_{3} of W \P by G/W.

Proof. Recall from [M, theorem 1, page 91] that if f is any functor from a category Q_{2} to a category Q_{3} then f is an equivalence if the following two conditions are satisfied:

1 Any object of Q_{3} is isomorphic to some object in the image of f.

2 For any two objects p, q of Q_{2}, f induces an isomorphism from Mor(p, q) to Mor(f(p), f(q)). We will apply this to show that our categories Q_{2} and Q_{3} are equivalent. We define

f on objects by f(p) = W p, and define f on morphisms using the obvious homomorphism from G to G/W. Condition 1 above is satisfied because every element of P_{3} is the image of an element of P_{2}, so every object of Q_{3} is the image of an object of Q_{2}. Suppose p and q are objects of P_{2}. The set of morphisms of Q_{2} from p to q can be identified with the set of elements g of G such that g(p) ≤ q, and Mor_{Q_{3}}(f(p), f(q)) can also be identified with the set of elements g of G such that g(p) ≤ q, so condition 2 above is satisfied. This shows that f is an equivalence and proves lemma 3.10.

Lemma 3.11. If p ≤ q and w(p) ≤ q for some w ∈ W_{Π}, p, q ∈ P_{2}, then w = 1.

Proof. Suppose that p = (D, K). Then q = (D_{1}, K) for some D_{1} containing D as p ≤ q. But then w(D) ⊆ D_{1}, so w(D) = D as no two distinct subsets D, w(D) of D_{1} are conjugate under W_{Π}, as D_{1} is contained in a fundamental domain of W_{Π}. Hence we can assume that w fixes D as well as K.

The subgroup of W_{Π} fixing D is a finite reflection group W_{D} generated by the reflections of W_{Π} fixing D because D ⊆ C_{Π}. The subgroup of W_{Π} fixing K is generated by the reflections fixing all elements of K(J). Any such reflection is in W_{Ω_{K}} because R contains 1. However, the condition that p = (D, K) ∈ P_{2} implies that there are no reflections of W_{Ω_{K}} fixing D. Hence the subgroup of W_{D} fixing K is trivial. So any element w ∈ W_{Π} such that w(p) ≤ q is trivial. This proves lemma 3.11.

Lemma 3.12. The categories Q_{2} and Q_{3} are equivalent.

Proof. Lemmas 3.10 and 3.11 show that there is an equivalence of categories from Q_{2} to the homotopy quotient of W_{Π}\P_{2} by Γ_{J} ×Γ_{Π}. Lemma 3.6 shows that the Γ_{J} ×Γ_{Π} posets P_{3}^{+} and W_{Π}\P_{2}^{+} are isomorphic. Lemma 3.9 shows that the subset P_{2} of P_{2}^{+} corresponds under this isomorphism to the subset W_{Π}\P_{2} of W_{Π}\P_{2}^{+}, so the Γ_{J} × Γ_{Π} posets P_{3} and W_{Π}\P_{2} are isomorphic. Therefore the category Q_{2} is equivalent to the homotopy quotient of P_{3} by Γ_{J} × Γ_{Π}, which is just Q_{3}. This proves lemma 3.12.

Lemma 3.13. The natural injection from Q_{4} to Q_{3} is an equivalence of categories.

Proof. This follows from the fact that Q_{4} is a skeleton of Q_{3}, so the natural injection is an equivalence of categories. This proves lemma 3.13.

We can now prove theorem 2.7. By lemmas 3.3, 3.12, and 3.13, the categories Q_{4} and Q_{1} are homotopy equivalent. So by lemma 3.1, the component of Q_{4} containing (J, \overline{id_{J}}) is a classifying category for Γ_{Ω}. This proves theorem 2.7.

4. The structure of Γ_{Ω}.

Theorem 4.1. The kernel of the natural map from Γ_{Ω} to Γ_{J} ×Γ_{Π} has finite cohomological dimension.

Proof. The classifying category of the kernel is just a component of the category of the poset P_{3}. A case by case check on possible Coxeter diagrams shows that the lengths of chains in P_{3} are bounded (by rank(J) + 1 for example), so the corresponding simplicial complex has finite dimension at most rank(J). Therefore the kernel has cohomological dimension at most rank(J). This proves theorem 4.1.

Corollary 4.2. If Γ_{Π} has finite virtual cohomological dimension, then so does Γ_{Ω}.

Proof. This follows immediately from theorem 4.1 and the fact that Γ_{J} is finite and standard properties of the virtual cohomological dimension.

Howlett showed that if W_{Π} is finite then the group Γ_{Ω} is a subgroup of Aut(J) × Aut(Π). We can deduce this from theorem 4.1 as follows. If W_{Π} is finite then so is the kernel of the map from Γ_{Ω} to Aut(J) × Aut(Π). On the other hand this kernel has finite cohomological dimension by theorem 4.1. But any finite group of finite cohomological dimension must be trivial, so the natural map from Γ_{Ω} to Aut(J) × Aut(Π) is injective. If W_{Π} is infinite then this kernel is usually infinite, as can be seen from most of the examples below. The fact that this kernel no longer vanishes is the main reason why normalizers of parabolic subgroups of Coxeter groups are more complicated to describe when the Coxeter group is infinite.

5. Examples.

Example 5.1. Suppose that J is A_{1} and the group Γ_{Π} is trivial. In this case Brink [Br] gave an elegant description of the group Γ_{Ω} as follows. Form the graph obtained from the Coxeter graph Π by keeping only the edges of odd order. Then for any point J = A_{1} of this new graph, the centralizer of the corresponding reflection (which is the group Γ_{Ω} corresponding to J) is the fundamental group of this graph with basepoint the chosen point, and in particular Γ_{Ω} is a free group.

We now check that this is equivalent to the description given by theorem 2.7. The only subdiagrams S with a non R-reflective class \overline{K} are the points of the Coxeter graph or the edges of odd order together with their endpoints. So the category Q_{4} has an object for each point or odd order edge of the Coxeter graph. The only non-identity morphisms correspond to inclusions of points in edges. The classifying space of this category is just the first barycentric subdivision of Brink’s graph. In particular the fundamental group of this category with some object as basepoint is canonically isomorphic to the fundamental group of Brink’s graph with the corresponding point as basepoint. This verifies that the description of Γ_{Ω} in [Br] is equivalent to the description in theorem 2.7.

The following lemma can often be used to find the fundamental group of a category with at most 2 objects.

Lemma 5.2. Suppose A and B are subgroups of a group. Let Q be the category with 2 objects p and q such that Mor(p, p) = A, Mor(p, q) = BA, Mor(q, q) = B, Mor(q, p) = ∅, with composition defined in the obvious way. Then π_{1}(Q) = A ∗_{A∩B} B.

Proof. This can be proved by writing down a set of generators and relations for the fundamental group, and checking that they are equivalent to a set of generators and relations for A ∗_{A∩B} B. We will leave the details to the reader.

For most of the examples below we will take W_{Π}.Γ_{Π} to be the group of automorphisms of the even 26 dimensional Lorentzian lattice II_{1,25} not exchanging the two cones of norm 0 vectors. According to Conway [C], the Coxeter group W_{Π} has a simple reflection r_{λ} ∈ Π for each vector λ of the Leech lattice Λ = Π, and the order of r_{λ}r_{μ} is 1, 2, 3, or ∞ according to whether (λ − μ)^{2} is 0, 4, 6, or greater than 6. The group Γ_{Π} is the automorphism group Λ.Aut(Λ) = Λ.(Z/2Z).Co_{1} of the affine Leech lattice, where Λ is the subgroup of translations and Co_{1} is Conway’s largest sporadic simple group.

If J is a spherical subdiagram of Λ then there is a homomorphism from N_{W_{Π}.Γ_{Π}}(W_{J})/W_{J} to the automorphism group of the lattice J^{⊥}. This has finite kernel and co-finite image, so theorem 2.7 can usually be used to describe the automorphism group of the lattice J^{⊥}. Most of the remaining examples in this section use this idea.

Example 5.3. Suppose L is the even Lorentzian lattice of dimension 20 and determinant 3. Let W^{(2)}(L) be the subgroup of Aut(L) generated by reflections of norm −2 vectors of L. Vinberg showed in [V] that Aut(L)^{+}/W^{(2)}(L) was the automorphism group of a certain K3 surface modulo a cyclic subgroup, and also showed that this group was an extension of a group of order 72 by a free product of 12 groups of order 2. We will show how to recover Vinberg’s description of Aut(L)^{+}/W^{(2)}(L) from theorem 2.7.

We take J to be an E_{6} ⊂ Λ so that L = J^{⊥}, and take R = 1 ⊂ Γ_{J} = Aut(E_{6}) = Z/2Z. The category Q_{4} contains exactly two objects, corresponding to an E_{6} and an E_{7} in Λ. For each subdiagram X of Λ we write G(X) for the automorphisms of Π = Λ mapping X into itself. The morphisms from the E_{6} object to itself form a group G(E_{6}) of order 72. The morphisms from the E_{7} object to itself form a group Z/2Z × G(E_{7}) of order 2 × 6 = 12 (where the Z/2Z comes from the group Γ_{J}). The morphisms from E_{6} to E_{7} can be identified with Z/2Z × G(E_{6}). By lemma 5.2 the group Γ_{Ω} is isomorphic to G(E_{6})∗_{G(E_{7})}(G(E_{7})×Z/2Z). If A, B, and C are any groups with B ⊆ A then A∗_{B} (B×C) is a semidirect product of a normal subgroup isomorphic to the free product of |A|/|B| copies of C and with the quotient by this normal subgroup isomorphic to A. Hence Γ_{Ω} has a normal subgroup isomorphic to the free product of 12 copies of Z/2Z, and the quotient is the group G(E_{6}) of order 72. This is equivalent to the description of this group given by Vinberg in [V]. In fact we can describe the various parts of Vinberg’s description in terms of the Leech lattice as follows: the group of order 2 is the group of automorphisms of E_{6}, the group of order 72 is the subgroup of Aut(Λ) mapping an E_{6} into itself, and the number 12 is the number of E_{7}’s of Λ containing an E_{6}.

The group Aut(L)^{+} also contains reflections in norm −6 vectors. The quotient by the full reflection group is finite of order 72, isomorphic to G(E_{6}). In this case the category Q_{4} contains just one point. Note that the reflections of norm −6 vectors induce the nontrivial automorphism of E_{6}. The 12 elements of order 2 in the paragraph above are in fact reflections of norm −6 vectors. See [V] or [B] for more details of this case.

If L is the even Lorentzian lattice of determinant 4 and dimension 20, which is again the Picard lattice of a K3 surface, then Vinberg gave a similar description of the automorphism group as an extension ((Z/2Z) ∗ (Z/2Z) ∗ (Z/2Z) ∗ (Z/2Z) ∗ (Z/2Z)).S_{5} of the symmetric group S_{5} by the free product of 5 group of order 2. This group can also be calculated using theorem 2.7. The corresponding category Q_{4} has 2 objects, corresponding to a D_{6} or D_{7} in Λ, the group S_{5} is the subgroup of Aut(Λ) mapping the D_{6} to itself, the group Z/2Z is the group of automorphisms of the D_{6} diagram, and the number 5 of copies of Z/2Z is the number of D_{7}’s containing a D_{6}.

Example 5.4 As a more complicated example we will describe the automorphism group of the Picard lattice L of the “next most algebraic K3 surface”, in other words L is the 20 dimensional even Lorentzian lattice of determinant 7. We take Π = Λ, Γ_{Π} = Λ.Aut(Λ), J = A_{6}, R = 1, Γ_{J} = Aut(A_{6}) = Z/2Z. Then Γ_{Ω} is the subgroup of elements of Aut(L) fixing a Weyl chamber of the reflection group generated by the reflections of norm −2 vectors. By theorem 2.7 the group Γ_{Ω} is the fundamental group of the category Q_{4}. The category Q_{4} has exactly 5 objects, corresponding to the 5 orbits of Coxeter diagrams A_{6}, A_{7}, D_{7}, E_{7}, and D_{8} with a non R-reflective isometry from A_{6} into them. (The group Λ.Aut(Λ) acts transitively on the embeddings of any of these Coxeter diagrams into Λ.) Note that for D_{8} and E_{7} there is only one equivalence class of isometries from A_{6} = J into it, while for A_{6}, A_{7}, and D_{7} there are two classes, which are exchanged by Aut(J) = Z/2Z. The category Q_{4} looks like this.

A_{7}(48)
(672) ↗  ↘ (192)
E_{7}(12) ⟵ A_{6}(336) ⟶ D_{8}(16)
(672) (672) ↘ (2016) ↗ (48)
D_{7}(24)

Here the numbers are the numbers of morphisms between pairs of objects in Q_{4}.

Example 5.5. Kondo in [K] studied the automorphism group of a generic Jacobian Kummer surface by embedding its Picard lattice L as the orthogonal complement of a certain J = A_{3}A_{1}^{6} in II_{1,25}, and used this to describe a generating set for the automorphism group. (Note that the Leech lattice contains more than 1 orbit of subdiagrams of the form A_{3}A_{1}^{6}; the one used by Kondo has largest possible stabilizer in Aut(Λ).) By results of Nikulin [N] the automorphism group of the K3 surface is the subgroup of Aut(L)^{+}/W^{(2)}(L) of elements acting on L′/L as ±1. This is just the group Γ_{Ω} of theorem 2.7 where we take Π = Λ, Γ_{Π} = Λ.Aut(Λ), J to be Kondo’s A_{3}A_{1}^{6}, R = 1, and Γ_{J} = the subgroup of order 2 of Aut(J) generated by the nontrivial automorphism of the A_{3}. The category Q_{4} is not connected and the component containing J seems quite complicated. Some partial calculations I have done suggest that the automorphism group of the generic Jacobian Kummer surface might be

(W.(Z/2Z)^{5}) ∗ (Z/2Z) ∗ (Z/2Z) ∗ (Z/2Z) ∗ (Z/2Z) ∗ (Z/2Z) ∗ (Z/2Z)

where W is a Coxeter group of rank 32+60 generated by the 16 projections, 16 correlations, sixty Cremona transformations and the (Z/2Z)^{5} is generated by sixteen translations and a switch [K], but I have not proved this rigorously.

It is much easier to work out the group Γ_{Ω} for R and Γ_{J} replaced by Aut(J) = Z/2Z × S_{6}. In this case the group Γ_{Ω} has a normal subgroup of index |S_{6}| isomorphic to the quotient of the automorphism group of a generic Jacobian Kummer surface by the Coxeter group generated by reflections in norm −4 vectors. Theorem 2.7 describes the Γ_{Ω} as the fundamental group of a component of the finite category Q_{4}. This component has just two elements, corresponding to the Coxeter diagrams A_{3}A_{1}^{6} and A_{5}A_{1}^{5}. Using lemma 5.2 and the results in [K] we see that

Γ_{Ω} = ((Z/2Z)^{5}.S_{6}) ∗_{S_{5}} (S_{5} × Z/2Z)

where (Z/2Z)^{5}.S_{6} is the subgroup of Γ_{Π} fixing A_{3}A_{1}^{6} ([K, lemma 4.5]), and S_{5} × Z/2Z is the subgroup of Γ_{Π} fixing A_{5}A_{1}^{5}. In particular Γ_{Ω} has a normal subgroup which is the free product of 192 groups of order 2, and the quotient by this normal subgroup is (Z/2Z)^{5}.S_{6}. If we change R to 1 but keep Γ_{J} = Aut(J) then Γ_{Ω} becomes the group Aut(L)^{+}/W^{(2)}(L) which appears to be

(W.(Z/2Z)^{5}.S_{6}) ∗_{S_{5}} (S_{5} × (Z/2Z))

though I have not proved this rigorously.

For more examples of automorphism groups of Kummer surfaces, corresponding to the cases J = D_{4}^{2}, D_{4}A_{3}, D_{4}A_{2}, or A_{3}^{2}, see [K-K].

Example 5.6 Suppose that J in theorem 2.7 contains no components of types A_{n} (n ≥ 1) or D_{5}, and assume that R = Γ_{J} = Aut(J). Then the map from Γ_{Ω} to Γ_{J} × Γ_{Π} is injective. This follows because a case by case check over all irreducible spherical Coxeter diagrams shows that any isometry from J into a strictly larger spherical Coxeter diagram is R-reflective.

Example 5.7 We show how to explain Vinberg’s result [V, V-K] that the reflection group of I_{1,n} has finite index if and only if n ≤ 19. Following Conway and Sloane [C-S] we write the even sublattice L of I_{1,n} as D_{25−n}^{⊥} in II_{1,25} for n ≤ 23, where D_{3} = A_{3} and D_{2} = A_{1}^{2}. In theorem 2.7 we take Π = Λ, Γ_{Π} = Λ.Aut(Λ), J = D_{25−n}, R = Γ_{J} a subgroup of order 2 of Aut(J) (which is equal to Aut(J) for n ≠ 21). Then the quotient of Aut(L)^{+} by its reflection subgroup is the group Γ_{Ω}. For n ≤ 19 the group Γ_{Ω} is finite by example 5.6. For n = 20 this argument breaks down because J is the “exceptional” case D_{5} of example 5.6. For 20 ≤ n ≤ 23 we can still describe the group Γ_{Ω} explicitly using theorem 2.7; see [B, theorem 6.6] for details. When n = 21 this gives a natural example with Γ_{J} ≠ Aut(J).

If we take J to be D_{4} and take R = Γ_{J} to be the symmetric group S_{3} = Aut(D_{4}) instead of a group of order 2 then W_{Ω}.Γ_{Ω} is the automorphism group of the even sublattice of I_{1,21}, and Γ_{Ω} is a finite group. See [B, p. 149] for details.

Example 5.8. The groups Γ_{Ω} have many of the properties of arithmetic groups; for example, they often have finite classifying categories. (It follows easily from [S] that arithmetic groups have this property.) It is natural to ask when they are arithmetic. There seems to be no obvious general way of deciding this. The following argument can often be used to show that Γ_{Ω} is not arithmetic. First of all a theorem due to Margulis [Ma, page 3] implies that if a group is an arithmetic subgroup of a group of rank at least 2, then all normal subgroups are either in the center or of finite index. Secondly, a theorem of Borel and Serre [B-S, 11.4.4] says that if a group Γ is arithmetic in a Lie group G then d = r + vcd(Γ) where d is the dimension of the symmetric space of G and r is the rank of G and vcd(Γ) is the virtual cohomological dimension of Γ. Let us use these results to prove that the group Γ_{Ω} of example 5.3 (the automorphism group of a K3 surface) is not arithmetic in any Lie group G. The group G must have rank 1 by the theorem of Margulis, as Γ_{Ω} has non abelian free subgroups of finite index and so cannot be an arithmetic subgroup of a group of rank at least 2. Its virtual cohomological dimension is one, so by the theorem of Borel and Serre [B-S, 11.4.4] the symmetric space of G must have dimension 1+1 = 2. But any finite subgroup of Γ_{Ω} must fix a point of this symmetric space, and therefore acts faithfully on the 2 dimensional tangent space of this point. But Γ_{Ω} has finite subgroups that are too large to have 2 dimensional faithful representations. Hence Γ_{Ω} is not arithmetic. Note that Γ_{Ω} has subgroups of finite index that are free and therefore arithmetic.

References.

[B] R. E. Borcherds, Automorphism groups of Lorentzian lattices, J.Alg. Vol 111 (1987)

No. 1, pp. 133-153. [B-S] A. Borel, J.-P. Serre, Corners and arithmetic groups. Comment. Math. Helv. 48

(1973), 436–491. [B-H] B. Brink, R. B. Howlett, Normalizers of Parabolic Subgroups in Coxeter Groups, 1998

preprint, available from http://www.maths.usyd.edu.au:8000/u/˜ bobh/

[Bo] N. Bourbaki, Groupes et algèbres de Lie. Chapitres IV-VI. Actualités Scientifiques et Industrielles, No. 1337 Hermann, Paris 1968.

[Br] B. Brink, On centralizers of reflections in Coxeter groups. Bull. London Math. Soc. 28 (1996), no. 5, 465–470.

[C] J. H. Conway, The automorphism group of the 26-dimensional even unimodular Lorentzian lattice. J. Algebra 80 (1983), no. 1, 159–163. Reprinted as chapter 27 of “Sphere packings, lattices and groups.” Grundlehren der Mathematischen Wissenschaften 290. Springer-Verlag, New York, 1993. ISBN: 0-387-97912-3

[C-S] J. H. Conway, N. J. A. Sloane, Leech roots and Vinberg groups. Proc. Roy. Soc. London Ser. A 384 (1982), no. 1787, 233–258. Reprinted as chapter 28 of “Sphere packings, lattices and groups.” Grundlehren der Mathematischen Wissenschaften 290. Springer-Verlag, New York, 1993. ISBN: 0-387-97912-3

[D] V. V. Deodhar, On the root system of a Coxeter group, Comm. Algebra 10 (1982) 611–630.

[Hi] H. Hiller, Geometry of Coxeter groups. Research Notes in Mathematics, 54. Pitman (Advanced Publishing Program), Boston, Mass.-London, 1982. ISBN: 0-273-08517-4

[H] R. B. Howlett, Normalizers of parabolic subgroups of reflection groups. J. London Math. Soc. (2) 21 (1980), no. 1, 62–80.

[Ke] J. H. Keum, Automorphisms of Jacobian Kummer surfaces. Compositio Math. 107 (1997), no. 3, 269–288.

[K-K] J. H. Keum, S. Kondo, The automorphism groups of Kummer surfaces associated with

the product of two elliptic curves, 1998 preprint.

[K] S. Kondo, The automorphism group of a generic Jacobian Kummer surface. J. Alge-

braic Geometry 7 (1998) 589–609.

[K98] S. Kondo, Niemeier lattices, Mathieu groups and finite groups of symplectic automorphisms of K3 surfaces. Duke Math. Journal Vol 92 (1998) no. 3, 593-598.

[M] S. MacLane, Categories for the working mathematician. Graduate Texts in Mathematics, Vol. 5. Springer-Verlag, New York-Berlin, 1971.

[Ma] G. A. Margulis, “Discrete subgroups of semisimple Lie groups”. Ergebnisse der Mathematik und ihrer Grenzgebiete (3), 17. Springer-Verlag, Berlin, 1991. ISBN: 3-540-12179-X.

[Mu] S. Mukai, Finite groups of automorphisms of K3 surfaces and the Mathieu group. Invent. Math. 94 (1988), no. 1, 183–221.

[N] V. V. Nikulin, An analogue of the Torelli theorem for Kummer surfaces of Jacobians. (Russian) Izv. Akad. Nauk SSSR Ser. Mat. 38 (1974), 22–41. Integral symmetric bilinear forms and its applications, Math. USSR Izv., 14 (1980), 103-167.

[P-S] I. R. Shafarevich, I. I. Piatetski-Shapiro, Torelli’s theorem for algebraic surfaces of type K3. (Russian) Izv. Akad. Nauk SSSR Ser. Mat. 35 1971 530–572. English translation in pages 516-557 of Collected mathematical papers by Igor R. Shafarevich. Springer-Verlag, Berlin-New York, 1989. ISBN: 3-540-13618-5

[Q] D. Quillen, Higher algebraic K-theory. I. Algebraic K-theory, I: Higher K-theories (Proc. Conf., Battelle Memorial Inst., Seattle, Wash., 1972), pp. 85–147. Lecture Notes in Math., Vol. 341, Springer, Berlin 1973.

[S] J.-P. Serre, Cohomologie des groupes discrets. Prospects in mathematics (Proc. Symp., Princeton Univ., Princeton, N.J., 1970), pp. 77–169. Ann. of Math. Studies, No. 70, Princeton Univ. Press, Princeton, N.J., 1971.

[V] È. B. Vinberg, The two most algebraic K3 surfaces. Math. Ann. 265 (1983), no. 1,

1–21. [V-K] È. B. Vinberg, I. M. Kaplinskaja, The groups O_{18,1}(Z) and O_{19,1}(Z). (Russian) Dokl.

Akad. Nauk SSSR 238 (1978), no. 6, 1273–1275.