REPRESENTATIONS OF THE ALTERNATING GROUP WHICH ARE IRREDUCIBLE OVER SUBGROUPS

ALEXANDER S. KLESHCHEV AND JAGAT SHETH

Abstract. Let F be an algebraically closed field of characteristic p ≥ 0 and A_{n} be the alternating group on n letters. The main goal of this paper is to describe the pairs (G, E), where E is an irreducible F A_{n}-module and G < A_{n} is a proper subgroup of A_{n} such that the restriction E↓_{G} is irreducible. We will be able to give a list of all such pairs, provided p > 3. The case p = 0 was treated by Saxl. The problem is important for the classification of maximal subgroups in finite classical groups.

Introduction

Let F be an algebraically closed field of characteristic p ≥ 0 and A_{n} be the alternating group on n letters. The main goal of this paper is to describe the pairs (G, E), where E is an irreducible F A_{n}-module and G < A_{n} is a proper subgroup of A_{n} such that the restriction E↓_{G} is irreducible. We will be able to give a list of all such pairs, provided p > 3. The case p = 0 was treated by Saxl [27]. We note that (in characteristic p) the problem is important for the classification of maximal subgroups in finite classical groups.

A similar problem for the symmetric group has been solved in [3] (also for p > 3). This yields a solution of our problem for all irreducible F A_{n}-modules E which lift to the symmetric group Σ_{n}. However, not all irreducible F A_{n}-modules lift, and so we need some further investigation to complete the problem.

To describe the main result we first explain how the irreducible F A_{n}-modules can be parametrised. From now on we will assume that p > 2 as the case p = 2 is quite exceptional for A_{n} and also because it is excluded in our main result anyway. We start from the symmetric group, referring the reader to [13] for the standard facts on its representation theory. In particular, to every p-regular partition λ of n one can associate the irreducible F Σ_{n}-module D^{λ}. Denote by sgn = sgn_{Σ_{n}} the 1-dimensional sign representation of Σ_{n}. Then D^{λ} ⊗ sgn is irreducible so there should exist a p-regular partition λ^{M} with

D^{λ} ⊗ sgn ≅ D^{λ^{M}}.

The bijection λ ↦ λ^{M} on the set of p-regular partitions is called the Mullineux bijection. This bijection can be described explicitly using a combinatorial algorithm suggested by Mullineux, see [26, 23, 12, 1].

It follows easily from the Clifford theory (see e.g. [11]) that the restriction D^{λ}↓_{A_{n}} is irreducible if and only if D^{λ} ⊗sgn ≅ D̸^{λ}. If this is the case, denote this irreducible restriction by E^{λ}. Of course, E^{λ} ≅ E^{λ^{M}}. On the other hand, if D^{λ} ⊗ sgn ≅ D^{λ} then the restriction D^{λ}↓_{A_{n}} splits as a direct sum E_{+}^{λ} ⊕ E_{−}^{λ} of two irreducible F A_{n}-modules. We have λ = λ^{M}, in which case we say that λ is Mullineux-fixed. Finally,

{E^{λ} | λ ≠ λ^{M}} ∪ {E_{+}^{λ}, E_{−}^{λ} | λ = λ^{M}}

is a complete set of irreducible F A_{n}-modules, and distinct modules L and M from this set are isomorphic if and only if L ≅ E^{λ}, M ≅ E^{λ^{M}} for some p-regular partition λ with λ ≠ λ^{M}.

1991 Mathematics Subject Classification: 20C20, 20C30, 20B35, 20B20.

The first author gratefully acknowledges the support of the NSF (grant DMS-9900134).

We introduce some elementary combinatorial notions used in the statement of the main result. Let λ be a p-regular partition of n. Represent it in the form

λ = (l_{1}^{a_{1}}, l_{2}^{a_{2}}, …, l_{k}^{a_{k}}) for l_{1} > l_{2} > ⋯ > l_{k} > 0, a_{i} > 0.

Then λ is called a Jantzen-Seitz partition (or JS-partition for short) if

l_{i} − l_{i+1} + a_{i} + a_{i+1} ≡ 0 (mod p) for all i with 1 ≤ i ≤ k − 1.

We will not distinguish between a partition λ = (λ_{1} ≥ λ_{2} ≥ …) and its Young diagram

λ = {(i, j) ∈ ℤ_{>0} × ℤ_{>0} | j ≤ λ_{i}}.

Elements (i, j) ∈ ℤ_{>0} × ℤ_{>0} are called nodes. The residue of a node A = (i, j), written res A, is defined to be the residue class (j − i) (mod p). A node (i, λ_{i}) ∈ λ is called removable if λ_{i} > λ_{i+1}. A node (i, λ_{i} + 1) is called addable if i = 1 or i > 1 and λ_{i} < λ_{i−1}. We say a node (i, j) is above (resp. below) (i′, j′) if i < i′ (resp. i > i′). A removable node A of λ is called normal if for every addable node B above A with res B = res A there exists a removable node C(B) above A and below B with res C(B) = res A, and such that B ≠ B′ implies C(B) ≠ C(B′).

Now we can state our main result.

Main Theorem. Let p > 3, n ≥ 5, E be an irreducible F A_{n}-module of dimension greater than one, and G < A_{n} be a proper subgroup. Then E↓_{G} is irreducible if and only if one of the following happens:

(i) E = E^{λ} for λ ≠ λ^{M}, and (G, D^{λ}) is as in [3], Main Theorem. (ii) E = E_{±}^{λ} for λ = λ^{M}, and one of the following happens:

(a) λ is a JS-partition, and G ≥ A_{n−2}, the stabilizer in A_{n} of two points;

(b) λ has exactly two normal nodes A and B, res A ≠ res B, and G = A_{n−1}, the stabilizer in A_{n} of one point;

(c) λ = (3^{3}), p > 5, n = 9, and G is PΓL(2, 8) embedded via its permutation representation of degree 9 or AGL(3, 2) embedded into A_{8} < A_{9} via its permutation representation of degree 8;

(d) λ = (3^{2}, 2), p > 5, n = 8, and G is AGL(3, 2) embedded via its permutation representation of degree 8.

1. Preliminaries

Let G be a group, and L, M be F G-modules. We denote by M^{G} the space of G-invariants in M. The space Hom_{F} (L, M) has a natural structure of an F G-module, with Hom_{F G}(L, M) = Hom_{F} (L, M)^{G}. If D_{1}, …, D_{k} are irreducible F G-modules then the notation M = D_{1}| … |D_{k} means that M is a uniserial F G-module with composition factors D_{1}, …, D_{k} counted from bottom to top. The trivial F G-module is denoted by 1_{G} or simply 1.

We record two well-known general facts (for the first one see the explanations in [2, 5.1]).

Lemma 1.1. Let G be a group, M be an F G-module and H ⊲ G be a normal subgroup with the index [G : H] prime to p. Then the socle series for M and M↓_{H} coincide.

Lemma 1.2. Let X, Y, Z be F G-modules. Assume that Z ⊆ X ⊕Y and that Z has a simple socle. Then either X or Y (or both) contains an isomorphic copy of Z as a submodule.

Proof. Let f : Z → X ⊕ Y be an embedding, and π_{X}, π_{Y} be projections of X ⊕ Y to X, Y, respectively. Then π_{X} ∘ f or π_{Y} ∘ f must be an injection, since otherwise both maps annihilate the simple socle of Z, which is impossible as π_{X} ∘ f + π_{Y} ∘ f = f is injective.

Let λ be a partition of n. As in [13], we denote by Σ_{λ}, M^{λ} = (1_{Σ_{λ}})↑^{Σ_{n}} and S^{λ} the corresponding Young subgroup, permutation module and Specht module, respectively. We will often use the description of the blocks of Σ_{n}, known as the ‘Nakayama Conjecture’, see [14, 2.7.41, 6.1.21].

Let σ ∈ Σ_{n} be a transposition. Then A_{n} → A_{n}, g ↦ σgσ^{−1} is an (outer) automorphism of A_{n}. If M is an F A_{n}-module we can use the automorphism to twist the action of A_{n} on M:

g · m = σgσ^{−1}m, g ∈ A_{n}, m ∈ M.

This defines a new F A_{n}-module denoted by ^{σ}M. One can easily check for the dual module: (^{σ}M)^{∗} ≅ ^{σ}(M^{∗}). The following lemma and corollaries are well known and follow immediately from Clifford theory and definitions:

Lemma 1.3. Let λ be a p-regular partition of n with λ = λ^{M}. Then ^{σ}E_{±}^{λ} ≅ E_{∓}^{λ}.

Corollary 1.4. Let λ be a p-regular partition of n with λ = λ^{M}, and G ≤ A_{n} be a subgroup. Then E_{+}^{λ}↓_{G} is irreducible if and only if E_{−}^{λ}↓_{G} is irreducible.

Corollary 1.5. Let λ be a p-regular partition of n with λ = λ^{M}. Then we have isomorphisms of F A_{n}-modules:

σ Hom_{F} (E_{±}^{λ}, E_{±}^{λ}) ≅ Hom_{F} (E_{∓}^{λ}, E_{∓}^{λ}),
σ Hom_{F} (E_{±}^{λ}, E_{∓}^{λ}) ≅ Hom_{F} (E_{∓}^{λ}, E_{±}^{λ}).

The following known fact often helps to prove that some module is reducible:

Lemma 1.6. Let G be a finite group. Then dim D ≤ \sqrt{|G|} for any irreducible F G-module D.

Another trivial tool which we occasionally employ, is the following

Lemma 1.7. Suppose that E is an irreducible F A_{n}-module which is a reduction modulo p of some ℂA_{n}-module E_{ℂ}. Then E↓_{G} is irreducible if and only if the restriction E_{ℂ}↓_{G} is an irreducible ℂG-module whose reduction modulo p is also irreducible.

Proof. It suffices to note that restriction to G and reduction modulo p commute.

A typical situation where Lemma 1.7 is useful is as follows. Recall that a partition is called a p-core if none of its hook lengths is divisible by p, see [14, § 2.7]. For p-cores we have λ^{M} = λ^{t}, the transpose (or conjugate) partition, see for example [3, 2.1]. Now let λ = λ^{t} be a Mullineux-fixed p-core partition. Consider the Specht module S_{ℂ}^{λ} over ℂΣ_{n}. This is an irreducible ℂΣ_{n}-module, which splits into a direct sum of two irreducible modules under the restriction to A_{n}: S_{ℂ}^{λ}↓_{A_{n}} ≅ T_{ℂ,+}^{λ} ⊕ T_{ℂ,−}^{λ}. By ‘Nakayama’s Conjecture’, reduction modulo p of S_{ℂ}^{λ} is the irreducible module D^{λ}, which splits into a direct sum of E_{+}^{λ} and E_{−}^{λ}. It follows that (for the proper choice of + and −) reduction modulo p of the ℂA_{n}-module T_{ℂ,±}^{λ} is the irreducible module E_{±}^{λ}. So the following result follows readily from Lemma 1.7.

Lemma 1.8. Let G < A_{n}, and λ = λ^{t} be a p-core. In the notation of the previous paragraph, E_{±}^{λ}↓_{G} is irreducible if and only if the restriction T_{ℂ,±}^{λ}↓_{G} is an irreducible ℂG-module whose reduction modulo p is also irreducible.

The next result shows that partitions with few rows are usually not Mullineux-fixed. The proof uses the Mullineux algorithm a detailed description of which can be found e.g. in [12]. If λ = (λ_{1}, λ_{2}, …, λ_{h}) is a partition with λ_{h} > 0 then we write h(λ) for h and call it the height of λ.

Lemma 1.9. Let n ≥ 5 and λ be a partition of n.

(i) If p > 2 and h(λ) ≤ 2 then h(λ^{M}) ≠ h(λ). In particular, λ ≠ λ^{M}. (ii) If p > 3 and h(λ) = 3 then λ^{M} ≠ λ, except for the cases where p > 5 and λ is one of

the following: (3, 1^{2}), (3, 2, 1), (3^{2}, 2), (3^{3}). (iii) If p > 3 and h(λ) = 4 then λ^{M} ≠ λ, except for the cases where p > 7 and λ is one of

the following: (4, 1^{3}), (4, 2, 1^{2}), (4, 3, 2, 1), (4, 3^{2}, 1), (4^{2}, 2^{2}), (4^{2}, 3, 2), (4^{3}, 3), (4^{4}) or p = 5, and λ is one of the following: (4, 1^{3}), (4, 2, 1^{2}), (5, 3, 1^{2}), (6, 3, 1^{2}).

(iv) Let p > 5 and h(λ) = 5. Then λ^{M} ≠ λ unless n ≤ 25. Moreover, the only Mullineuxfixed partition for n = 16 is (5^{2}, 2^{3}) for p > 7.

Proof. Let a be the number of nodes in the p-rim of λ. Set ε = 0 if p|a and ε = 1 otherwise.

(i) If h(λ) = 1, use [13, 24.5(iii)]. Let h(λ) = 2. Then h(λ^{M}) = a + ε − 2, see [3, 2.4]. Assume h(λ^{M}) = 2. If ε = 0 then a = 4, which is a contradiction since p > 2 does not divide 4. Otherwise a = 3 and p > 3. But in this case the p-rim of λ coincides with its rim, which contradicts the assumption n ≥ 5.

(ii) The height of λ^{M} is a + ε − 3. Assume this height is 3. If ε = 0 then a = 6, which is a contradiction since p > 3. So a = 5 and p > 5. In this case the p-rim of λ coincides with its rim and λ is a p-core, which gives the required list of exceptions.

(iii) As above we show that a = 7 and p ≠ 7. Now the case p > 7 is similar to the case p > 5 in (ii). Let p = 5. Then either the Mullineux symbol of λ has just one column, which implies λ = (4, 1^{3}), or the second column of the Mullineux symbol is \binom{a′}{r′} with r′ < 4. This shows that λ^{(1)} := λ \ {5-rim} is a Mullineux-fixed partition with at most three rows. By (i) and (ii), λ^{(1)} is a partition of n′ ≤ 4. Going through all possibilities for λ^{(1)} we get the desired list.

(iv) As above, a = 9, and λ \ {p-rim} is a Mullineux-invariant partition with at most four rows. The result now follows from (i)-(iii).

Now we describe the structure of some modules associated with two row partitions. Recall that the Specht module S^{λ} is by definition a submodule of the permutation module M^{λ}. Denote by Y^{(n−k,k)} the block component of M^{(n−k,k)} containing S^{(n−k,k)}.

Lemma 1.10. [3, 3.3] Let p > k ≥ 0 and n ≥ 2k. If there exists l with 0 ≤ l < k and n ≡ k + l − 1 (mod p), then

S^{(n−k,k)} = D^{(n−l,l)}|D^{(n−k,k)},
Y^{(n−k,k)} = D^{(n−l,l)}|D^{(n−k,k)}|D^{(n−l,l)}, and
Y^{(n−k,k)}/D^{(n−k,k)} ≅ (S^{(n−k,k)})^{∗}.

Otherwise Y^{(n−k,k)} = S^{(n−k,k)} = D^{(n−k,k)}.

Lemma 1.11. Let k ≥ 0, p > max(2, k), and n ≥ max(2k, 5). If there exists l with 0 ≤ l < k and n ≡ k + l − 1 (mod p), then

S^{(n−k,k)}↓_{A_{n}} = E^{(n−l,l)}|E^{(n−k,k)},
Y^{(n−k,k)}↓_{A_{n}} = E^{(n−l,l)}|E^{(n−k,k)}|E^{(n−l,l)}, and
(Y^{(n−k,k)}↓_{A_{n}})/E^{(n−l,l)} ≅ (S^{(n−k,k)})^{∗}↓_{A_{n}}.

Otherwise Y^{(n−k,k)}↓_{A_{n}} = S^{(n−k,k)}↓_{A_{n}} = E^{(n−k,k)}.

Proof. This follows from Lemmas 1.10, 1.1 and 1.9(i).

Now, we state some results about dimensions of irreducible F A_{n}-modules.

Lemma 1.12. Let p > 3, n ≥ 5, and λ = λ^{M}. Then dim E_{±}^{λ} ≥ (n^{3} − 9n^{2} + 14n)/12. Moreover, dim E_{±}^{λ} ≥ 35, 21, 21, 10, 8 and 3 for n = 10, 9, 8, 7, 6 and 5, respectively.

Proof. If n ≥ 13 then, by [3, 1.20], either λ has at most 2 nodes outside of the first row or dim D^{λ} > (n^{3} − 9n^{2} + 14n)/6. As dim E_{±}^{λ} = (dim D^{λ})/2, the result for n ≥ 13 now follows from Lemma 1.9(i)(ii). For n ≤ 12 use e.g. [7, 16].

Lemma 1.13. Let p > 2, and λ be a p-regular partition of n = 22, 23 or 24 with λ = λ^{M}. Then, dim E_{±}^{λ} > 10395.

Proof. This is a computer calculation involving a Gap program written by J. Brundan, see [3, 1.23] for more details.

We need one result on the cohomology of symmetric groups.

Lemma 1.14. Let p > 3, n ≢ 0 (mod p), and D^{μ}, D^{ν} be irreducible modules from the set

{1_{Σ_{n}}, sgn_{Σ_{n}}, D^{(n−1,1)}, D^{(n−1,1)} ⊗ sgn, D^{(n−2,1^{2})}, D^{(n−2,1^{2})} ⊗ sgn}.

Then Ext_{Σ_{n}}^{1}(D^{μ}, D^{ν}) = 0.

Proof. If λ ∈ X := {(n), (n − 1, 1), (n − 2, 1^{2})} then D^{λ} is a direct summand of M^{λ}, thanks to ‘Nakayama’s conjecture’. So it suffices to prove that for any partitions μ, ν ∈ X we have

Ext_{Σ_{n}}^{1}(M^{μ}, M^{ν}) = 0 = Ext_{Σ_{n}}^{1}(M^{μ}, M^{ν} ⊗ sgn).

For S = 1 or sgn, using the self-duality of M^{ν} we get

Ext_{Σ_{n}}^{1}(M^{μ}, M^{ν} ⊗ S) ≅ Ext_{Σ_{n}}^{1}(M^{μ} ⊗ M^{ν}, S).

Note that M^{μ} ⊗ M^{ν} is a direct sum of certain permutation modules M^{λ}. But by Shapiro’s lemma, Ext_{Σ_{n}}^{1}(M^{λ}, S) ≅ Ext_{Σ_{λ}}^{1}(1_{Σ_{λ}}, S↓_{Σ_{λ}}), which is zero since H^{1}(Σ_{k}, 1) = H^{1}(Σ_{k}, sgn) = 0 for p > 3, see e.g. [5, 2.2(i)(iii)].

Finally, we state a criterion for the trivial module to be a submodule of a Specht module. Let a = a_{0} + a_{1}p + ⋯ + a_{r}p^{r} be the p-adic expansion of a positive integer a, with a_{r} > 0. Then define l_{p}(a) := r + 1. Also define l_{p}(0) := 0.

Proposition 1.15. [13, 24.4] Let λ = (λ_{1}, λ_{2}, …) be a partition of n, and set z(i) := l_{p}(λ_{i}). Then 1_{Σ_{n}} is a submodule of S^{λ} if and only if λ_{i} ≡ −1 (mod p^{z(i+1)}) for every i ≥ 1.

2. Main technical results

Throughout this section G is a (fixed) subgroup of Σ_{n}, and D^{λ} is a (fixed) irreducible F Σ_{n}-module. For k ≥ 0 we set

r_{k}(G) := dim(M^{(n−k,k)})^{G} and
d_{k}(λ) := dim Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})).

By agreement, r_{−1}(G) := 0 and d_{−1}(λ) := 0. Note that we always have r_{0}(G) = 1 and d_{0}(λ) = 1. In fact,

r_{k}(G) = ♯{G-orbits on unordered k-element subsets of {1, 2, …, n}},  (1)

see e.g. [25, 1.5]. Also note that by Frobenius reciprocity,

d_{k}(λ) = dim End_{F [Σ_{n−k}×Σ_{k}]}(D^{λ}↓_{Σ_{n−k}×Σ_{k}}).  (2)

The following lemma concerns with G-fixed points in Y^{(n−k,k)} and (S^{(n−k,k)})^{∗}.

Lemma 2.1. Let k ≥ 0, p > k, n ≥ 2k, and r_{k−1}(G) < r_{k}(G). Then

(Y^{(n−k,k)})^{G} ≠ 0 and ((S^{(n−k,k)})^{∗})^{G} ≠ 0.

If, moreover, 0 ≤ l < k for some l with n ≡ k + l − 1 (mod p) and r_{l−1}(G) < r_{l}(G) then

dim(Y^{(n−k,k)})^{G} ≥ 2.

Proof. For k = 0 we have Y^{(n−k,k)} = S^{(n−k,k)} ≅ 1 and the result follows. Let k > 0. By Lemmas 3.1 and 3.2 in [3], we have the exact sequence

0 ⟶ S^{(n−k,k)} ⟶ M^{(n−k,k)} ⟶^{f} M^{(n−k+1,k−1)} ⟶ 0.  (3)

Now the assumption r_{k−1}(G) < r_{k}(G) implies (S^{(n−k,k)})^{G} ≠ 0. By definition, S^{(n−k,k)} is contained in Y^{(n−k,k)}, so (S^{(n−k,k)})^{G} is contained in (Y^{(n−k,k)})^{G}, hence (Y^{(n−k,k)})^{G} ≠ 0.

By dualizing the exact sequence above we get another exact sequence:

0 ⟶ (M^{(n−k+1,k−1)})^{∗} ⟶ (M^{(n−k,k)})^{∗} ⟶ (S^{(n−k,k)})^{∗} ⟶ 0.

As (M^{μ})^{∗} ≅ M^{μ} for any μ, the assumption r_{k−1}(G) < r_{k}(G) yields ((S^{(n−k,k)})^{∗})^{G} ≠ 0. Finally, let 0 ≤ l < k and n ≡ k + l − 1 (mod p). Then, by Lemma 1.10, we have

Y^{(n−k,k)} = D^{(n−l,l)}|D^{(n−k,k)}|D^{(n−l,l)} = S^{(n−k,k)}|D^{(n−l,l)} and Y^{(n−l,l)} = D^{(n−l,l)}.

Now, from (3), M^{(n−k,k)} = Y^{(n−k,k)} ⊕ N, where N ≅ M^{(n−k+1,k−1)}/D^{(n−l,l)}. Therefore

dim(Y^{(n−k,k)})^{G} = dim(M^{(n−k,k)})^{G} − dim(M^{(n−k+1,k−1)}/D^{(n−l,l)})^{G}.  (4)

Using Lemma 1.10 and the ‘Nakayama conjecture’ we deduce that D^{(n−l,l)} = Y^{(n−l,l)} is a block component and hence a direct summand of M^{(n−k+1,k−1)}. By the first part of the lemma applied to k = l, we see that r_{l−1}(G) < r_{l}(G) implies (D^{(n−l,l)})^{G} ≠ 0. Therefore

dim(M^{(n−k+1,k−1)}/D^{(n−l,l)})^{G} = r_{k−1}(G) − dim(D^{(n−l,l)})^{G} ≤ r_{k−1}(G) − 1 ≤ r_{k}(G) − 2.

By (4), this implies dim(Y^{(n−k,k)})^{G} ≥ 2.

Recall the definition of JS-partitions from the introduction. The following result was first proved in [20] (cf. also [17, 10, 21]).

Theorem 2.2. [20, Theorem D] The restriction D^{λ}↓_{Σ_{n−1}} is irreducible if and only if λ is a JS-partition.

We need some relations between various d_{k}(λ).

Proposition 2.3. Let 0 ≤ l < k < p, n ≥ 2k, and D^{λ} be an irreducible F Σ_{n}-module with h(λ) ≥ k and h(λ^{M}) ≥ k. Then d_{l}(λ) < d_{k}(λ) unless (l, k) = (0, 1) and λ is a JS-partition, in which case d_{0}(λ) = d_{1}(λ) = 1.

Proof. By [3, 3.5], we have d_{l}(λ) ≤ d_{l+1}(λ) ≤ ⋯ ≤ d_{k}(λ). Moreover, by [3, Corollary 3.9], d_{k}(λ) > d_{k−1}(λ), provided k > 1. Finally, d_{1}(λ) = 1 if and only if D^{λ}↓_{Σ_{n−1}} is irreducible, which by (2), Theorem 2.2 and self-duality of irreducible F Σ_{n}-modules, is equivalent to λ being a JS-partition.

Lemma 2.4. Let 1 ≤ k < p, n ≥ 2k and V be an F Σ_{n}-module. If

dim Hom_{F Σ_{n}}(M^{(n−k,k)}, V) > dim Hom_{F Σ_{n}}(M^{(n−k+1,k−1)}, V).

then V contains either Y^{(n−k,k)} or (S^{(n−k,k)})^{∗} (or both) as a submodule.

Proof. Consider the exact sequence (3). By assumption, there exists an F Σ_{n}-homomorphism θ : M^{(n−k,k)} → V which does not factor through f. If the restriction θ|Y^{(n−k,k)} is injective then Y^{(n−k,k)} is a submodule of V. Otherwise, in view of Lemma 1.10, the kernel of θ|Y^{(n−k,k)} is certain D^{(n−l,l)} and Y^{(n−k,k)}/D^{(n−l,l)} ≅ (S^{(n−k,k)})^{∗} embeds into V.

The next result is our main tool to test the restrictions of F A_{n}-modules for reducibility.

Theorem 2.5. Let 1 ≤ l < k < p, n ≥ max(5, 2k), G ≤ A_{n}, and a p-regular partition λ ⊢ n satisfy λ^{M} = λ and h(λ) ≥ k. Assume that

r_{l−1}(G) < r_{l}(G), r_{k−1}(G) < r_{k}(G), and d_{l−1}(λ) < d_{l}(λ), d_{k−1}(λ) < d_{k}(λ).

Then E_{±}^{λ} ↓ G is reducible.

Proof. As D^{λ} ↓_{A_{n}}= E_{+}^{λ} ⊕ E_{−}^{λ}, we get an isomorphism of F A_{n}-modules

End_{F} (D^{λ})↓_{A_{n}} ≅ End_{F} (E_{+}^{λ}) ⊕ End_{F} (E_{−}^{λ}) ⊕ Hom_{F} (E_{+}^{λ}, E_{−}^{λ}) ⊕ Hom_{F} (E_{−}^{λ}, E_{+}^{λ}). (5)

Let X denote either Y^{(n−l,l)}↓_{A_{n}} or (S^{(n−l,l)})^{∗}↓_{A_{n}}, and Z denote Y^{(n−k,k)}↓_{A_{n}} or (S^{(n−k,k)})^{∗}↓_{A_{n}}. By Lemma 2.4 and the assumption d_{l}(λ) > d_{l−1}(λ), X is a submodule of End_{F} (D^{λ})↓_{A_{n}}.

Moreover, X has a simple socle by Lemma 1.11, so Lemma 1.2 implies that X is a submodule of at least one of the four summands in the right hand side of (5). Moreover, ^{σ}(X) ≅ X as X ‘lifts’ to Σ_{n}. Now, by Corollary 1.5, X is a submodule of both End_{F} (E_{+}^{λ}), End_{F} (E_{−}^{λ}) or X is a submodule of both Hom_{F} (E_{+}^{λ}, E_{−}^{λ}), Hom_{F} (E_{−}^{λ}, E_{+}^{λ}). The same argument shows that a similar statement holds for Z. So we may assume that one of the following holds:

(a) X ⊆ End_{F} (E_{+}^{λ}), Z ⊆ End_{F} (E_{+}^{λ});

(b) X ⊆ End_{F} (E_{+}^{λ}), Z ⊆ Hom_{F} (E_{+}^{λ}, E_{−}^{λ});

(c) X ⊆ Hom_{F} (E_{+}^{λ}, E_{−}^{λ}), Z ⊆ End_{F} (E_{+}^{λ});

(d) X ⊆ Hom_{F} (E_{+}^{λ}, E_{−}^{λ}), Z ⊆ Hom_{F} (E_{+}^{λ}, E_{−}^{λ}). Moreover, 1_{A_{n}} ⊆ End_{F} (E_{±}^{λ}). Suppose E_{±}^{λ}↓_{G} is irreducible (see Corollary 1.4). Then End_{F G}(E_{±}^{λ}↓_{G}) ≅ End_{F} (E_{±}^{λ}↓_{G})^{G} is 1-dimensional by Schur’s lemma. We consider the following two cases.

Case 1: (a), (b), or (c) holds. First assume n ≢ l−1 (mod p) and n ≢ k−1 (mod p). Then by Lemma 1.11, 1_{A_{n}} is not in the socle of X and Z. So 1_{A_{n}} ⊕ X ⊆ End_{F} (E_{+}^{λ}) or 1_{A_{n}} ⊕ Z ⊆ End_{F} (E_{+}^{λ}). By Lemma 2.1, this implies dim End_{F} (E_{+}^{λ})^{G} ≥ 2, giving a contradiction.

Now assume that n ≡ l − 1 (mod p). Then n ≢ k − 1 (mod p) as 0 < k − l < p by assumption. By Lemma 1.11, X = E^{(n−l,l)}|1_{A_{n}} or X = 1_{A_{n}}|E^{(n−l,l)}|1_{A_{n}}. In the former case, either 1_{A_{n}} ⊕ X ⊆ End_{F} (E_{+}^{λ})^{G} or 1_{A_{n}} ⊕ Z ⊆ End_{F} (E_{+}^{λ})^{G}, giving a contradiction as above. In the latter case, dim X^{G} ≥ 2 by Lemma 2.1 (applied with k = l and l = 0). If (a) or (b) holds, this shows dim End_{F} (E_{+}^{λ})^{G} ≥ 2, a contradiction. If (c) holds, we have dim Hom_{F} (E_{+}^{λ}, E_{−}^{λ})^{G} ≥ 2, but this is impossible as Hom_{F} (E_{+}^{λ}, E_{−}^{λ})^{G} ≅ Hom_{F G}(E_{+}^{λ}↓_{G}, E_{−}^{λ}↓_{G}) and E_{±}^{λ}↓_{G} is irreducible by assumption. The case n ≡ k − 1 (mod p) is considered similarly.

Case 2: (d) holds. If n ≢ k + l − 1 (mod p) then X and Z are in different blocks by the ‘Nakayama Conjecture’. So X ⊕ Z ⊆ Hom_{F} (E_{+}^{λ}, E_{−}^{λ}). By Lemma 2.1, X^{G} and Z^{G} are both non-trivial, hence dim Hom_{F} (E_{+}^{λ}, E_{−}^{λ})^{G} ≥ 2, which is impossible as explained in the previous paragraph.

Finally, let n ≡ k + l − 1 (mod p). Then by Lemma 1.10, X = E^{(n−l,l)}, and either Z = E^{(n−k,k)}|E^{(n−l,l)} or Z = E^{(n−l,l)}|E^{(n−k,k)}|E^{(n−l,l)}. In the former case we have X ⊕ Z ⊆ Hom_{F} (E_{+}^{λ}, D_{−}^{λ}), giving a contradiction as above. In the latter case, dim Z^{G} ≥ 2 by Lemma 2.1, whence dim Hom_{F} (E_{+}^{λ}, E_{−}^{λ}) ≥ 2, giving a contradiction again.

3. Reduction to 2-homogeneous subgroups

From now on we consider restrictions of an irreducible F A_{n}-module of the form E_{±}^{λ} to subgroups G < A_{n}. In particular, it is always assumed that λ^{M} = λ. Recall the constants r_{i}(G) and their characterization (1).

We first consider the case where G is intransitive. Then, up to a conjugation, G is contained in a subgroup of the form

A_{(n−k,k)} := A_{n} ∩ (Σ_{n−k} × Σ_{k})

for some 1 ≤ k ≤ n/2. The following result is straightforward.

Lemma 3.1. Let n ≥ 5, 1 < k ≤ n/2, and G = A_{(n−k,k)}. Then r_{1}(G) = 2, r_{2}(G) = 3, r_{3}(G) = 4 for k > 2, and r_{3}(G) = 3 for k = 2.

Proposition 3.2. Let p > 3, 2 < k ≤ n/2, and G ≤ A_{(n−k,k)}. Then E_{±}^{λ}↓_{G} is reducible.

Proof. We may assume that G = A_{(n−k,k)}. By Lemma 1.9(i), h(λ) ≥ 3. So Proposition 2.3 yields d_{1}(λ) < d_{2}(λ) < d_{3}(λ). Hence E_{±}^{λ}↓_{A_{(n−k,k)}} is reducible by Lemma 3.1 and Theorem 2.5.

Now we turn to the case k = 2.

Proposition 3.3. Let p > 2, n ≥ 5, G ≤ A_{(n−2,2)}, and assume that λ is not a JS-partition. Then E_{±}^{λ}↓_{G} is reducible.

Proof. We may assume that G = A_{(n−2,2)}. By Lemma 1.9(i), h(λ) ≥ 3. So Proposition 2.3 gives d_{0}(λ) < d_{1}(λ) < d_{2}(λ). By Lemma 3.1, we have 1 = r_{0}(G) < r_{1}(G) < r_{2}(G). Now apply Theorem 2.5.

Proposition 3.3 shows in particular that for A_{n−2} ≤ G ≤ A_{(n−2,2)}, the restriction E_{±}^{λ}↓_{G} is irreducible only if λ is a JS-partition. We now work towards proving Theorem 3.7 below which establishes the converse of this fact. We will use results from [22]. Recall the combinatorial notions defined in the Introduction, in particular the notion of a normal node. A removable node is called good if it is the lowest among the normal nodes of a fixed residue. By definition there is at most one good node of each residue. If A = (i, λ_{i}) is a removable node for λ = (λ_{1}, λ_{2}, …) then

λ_{A} := λ \ {A} = (λ_{1}, …, λ_{i−1}, λ_{i} − 1, λ_{i+1}, …)

denotes the Young diagram of n−1 obtained from λ by removing A. We will use the following version of the modular branching rule.

Theorem 3.4. [22, Theorem 0.6] Let D^{λ} be an arbitrary irreducible F Σ_{n}-module. The restriction D^{λ}↓_{Σ_{n−1}} is completely reducible if and only if every normal node of λ is good. In this case

D^{λ}↓_{Σ_{n−1}} ≅ ⊕D^{λ_{A}},

where the sum is over all good nodes of λ.

We also need the following relation between the Mullineux bijection and good nodes:

Lemma 3.5. [23, 4.8] Let λ be a p-regular partition, and A be a good node for λ of residue α. Then there exists a good node B for λ^{M} of residue −α such that (λ_{A})^{M} = (λ^{M})_{B}.

Now we apply the results above to prove the following

Lemma 3.6. Let p > 2, n ≥ 5, λ be a Mullineux-fixed JS-partition, and D^{λ} be the corresponding irreducible F Σ_{n}-module. Then

D^{λ} ↓_{Σ_{n−2}×Σ_{2}}= (D^{μ} ⊗ 1_{Σ_{2}}) ⊕ (D^{ν} ⊗ sgn_{Σ_{2}}),

where μ, ν are distinct p-regular partitions of n − 2, obtained from λ by consecutive removal of two good nodes, and μ^{M} = ν.

Proof. Let n = k(p−1)+d, with 0 ≤ d < p−1, and ε_{n} := ((k+1)^{d}, k^{p−1−d}). By [13, 24.5(iii)], D^{ε_{n}} ≅ sgn, so ε_{n} is not Mullineux-fixed. Hence λ ≠ ε_{n}.

It follows from definitions and Theorem 3.4 that the top removable node A of λ is its only normal node and D^{λ}↓_{Σ_{n−1}} = D^{λ_{A}}. Moreover,

(D^{λ} ⊗ sgn)↓_{Σ_{n−1}} ≅ (D^{λ}↓_{Σ_{n−1}}) ⊗ sgn  (6)

implies λ_{A} = (λ_{A})^{M}. So res A = 0 in view of Lemma 3.5. Similarly, (D^{λ}↓_{Σ_{n−2}}) ⊗ sgn ≅ D^{λ}↓_{Σ_{n−2}}. So if we prove that D^{λ}↓_{Σ_{n−2}} ≅ D^{μ} ⊕ D^{ν} and μ^{M} ≠ μ then it will follow that μ^{M} = ν. Moreover, the whole lemma will follow as by [3, 1.5], both 1_{Σ_{2}} and sgn_{Σ_{2}} must appear in D^{λ}↓_{Σ_{2}}.

We have the following four possibilities for λ :

B C D A E (a) λ =  (b) λ =  B C A D

B A C D E (c) λ =  (d) λ =  B A C D E F

Consider the case (a). Note that C is a normal node for λ_{A}. Moreover, D is a normal node for λ_{A} if and only if res D ≠ res B. If this is the case, then E is not normal for λ_{A} as λ is JS and so res E = res B. Similarly all lower removable nodes are not normal. Moreover, res C ≠ res D as p > 2. Thus C and D are the only normal nodes of λ_{A} and they have different residues. By Theorem 3.4 we now have D^{λ}↓_{Σ_{n−2}} ≅ D^{(λ_{A})_{C}} ⊕ D^{(λ_{A})_{D}}. Next, observe that res C ≠ res A = 0, so by Lemma 3.5 we have (λ_{A})_{C} ≠ ((λ_{A})_{C})^{M}, as needed.

Now assume res D = res B. Then E must exists as λ ≠ ε_{n}. Moreover, in this case res E = res D = res B, and so E is normal. Moreover, res E ≠ res C, and C, E are the only two normal nodes of λ_{A}. Now apply Theorem 3.4 and Lemma 3.5 as above.

Case (b). Here C and D are the only normal nodes for λ_{A}, and we use Theorem 3.4 and Lemma 3.5 as above. (Note that D exists as λ ≠ (n).)

Case (c). As λ is JS, we have res D = res B and res E = res C (note that E exists as λ ≠ ε_{n}). It follows that λ_{A} is a p-regular partition whose only normal nodes are D and E, with res D ≠ res E. Now apply Theorem 3.4 and Lemma 3.5 as above.

Case (d). As λ is JS, we have res E = res B and res F = res D (note that F exists since λ ≠ ε_{n}). Now, C and F are the only normal nodes for λ_{A} and they have different residues since p > 2. It remains to use Theorem 3.4 and Lemma 3.5 as above.

Theorem 3.7. Let p > 2, n ≥ 5 and A_{n−2} ≤ G ≤ A_{(n−2,2)}. Then E_{±}^{λ}↓_{G} is irreducible if and only if λ is a JS-partition.

Proof. The ‘only-if’ part follows from Proposition 3.3, and the ‘if’ part from Lemma 3.6.

Finally, we consider the subgroup A_{n−1}.

Proposition 3.8. Let p > 2 and n ≥ 5. The restriction E_{±}^{λ}↓_{A_{n−1}} is irreducible if and only if one of the following happens:

(i) λ is a JS-partition, in which case E_{±}^{λ}↓_{A_{n−1}} ≅ E_{±}^{λ_{A}} or E_{∓}^{λ_{A}}, where A is the top removable node of λ;

(ii) λ has exactly two normal nodes A, B, and res A ≠ res B, in which case E_{±}^{λ}↓_{A_{n−1}} ≅ E^{λ_{A}} ≅ E^{λ_{B}}.

Proof. ‘Only-if’ part. By Corollary 1.4, E_{+}^{λ}↓A_{n−1} is irreducible if and only if E_{−}^{λ}↓A_{n−1} is irreducible. So D^{λ}↓_{A_{n−1}} ≅ E_{+}^{λ}↓_{A_{n−1}} ⊕ E_{−}^{λ}↓_{A_{n−1}} is semisimple. Now by Lemma 1.1, the restriction D^{λ}↓_{Σ_{n−1}} is semisimple. Moreover, if λ is not JS, then D^{λ}↓_{Σ_{n−1}} must have exactly two composition factors. Now use Theorem 3.4.

‘If’ part. If λ is JS then E_{+}^{λ}↓_{A_{n−1}} is irreducible by Theorem 3.7. Moreover, in this case D^{λ}↓_{Σ_{n−1}} ≅ D^{λ_{A}} where A is the top removable node of λ, see Theorem 3.4. This proves that E_{±}^{λ}↓_{A_{n−1}} ≅ E_{±}^{λ_{A}} or E_{∓}^{λ_{A}} as λ_{A} must be Mullineux-invariant by (6). Now, let λ be as in (ii). As λ^{M} = λ, we get from Lemma 3.5 that res A = − res B, and so res A, res B ≠ 0. By Lemma 3.5 again, λ_{A} and λ_{B} are not Mullineux-invariant. Hence D^{λ_{A}}↓_{A_{n−1}} and D^{λ_{B}}↓_{A_{n−1}} are irreducible, and one of them must be E_{+}^{λ}↓_{A_{n−1}}, while the other must be E_{−}^{λ}↓_{A_{n−1}}.

Remark 3.9. Bessenrodt and Olsson [2] described the partitions satisfying (i) and (ii) in Proposition 3.8 in terms of Mullineux symbols.

Now we consider the case where G is a transitive subgroup of A_{n}. We will show in Theorem 3.14 that the restriction E_{±}^{λ} ↓_{G} is reducible unless G is 2-homogeneous. Recall that for 1 ≤ k ≤ n/2 a group G is called k-homogeneous if r_{k}(G) = 1. The following relation between various r_{k}(G) is well known, see, for example, [8, 9.4A(i)]:

Lemma 3.10. Let G ≤ Σ_{n}. If 0 ≤ k ≤ l and k + l ≤ n, then r_{k}(G) ≤ r_{l}(G).

The proof of Theorem 3.14 relies on the next result of Cameron, Neumann and Saxl.

Theorem 3.11. [6, §5, Corollary] Let n > 5, G < Σ_{n} be transitive, and r_{2}(G) = r_{3}(G). Then one of the following holds:

(i) n is even and G has 2 blocks of imprimitivity of size n/2. (ii) n is even and G has n/2 blocks of imprimitivity of size 2. (iii) G is 3-homogeneous.

If n is even, set m = n/2. In the cases (i) and (ii) of Theorem 3.11 the group G is contained in the wreath products Σ_{m} ≀ Σ_{2} and Σ_{2} ≀ Σ_{m} , respectively (see e.g. [8, 2.6]). So we need to study restrictions to such wreath products or, more precisely, their intersections with A_{n}. The next result if straightforward.

Lemma 3.12. Let n = 2m ≥ 8, and G = (Σ_{m} ≀ Σ_{2}) ∩ A_{n} or G = (Σ_{2} ≀ Σ_{m}) ∩ A_{n}. Then r_{1}(G) = 1, r_{2}(G) = r_{3}(G) = 2, and r_{4}(G) ≥ 3.

Now we show that restrictions to the wreath products above are reducible.

Proposition 3.13. Let p > 3, n = 2m ≥ 8, and G = (Σ_{m} ≀ Σ_{2}) ∩ A_{n} or G = (Σ_{2} ≀ Σ_{m}) ∩ A_{n}. Then the restriction E_{±}^{λ}↓_{G} is reducible.

Proof. If h(λ) ≥ 4 the result follows from Lemma 3.12, Proposition 2.3 and Theorem 2.5. So by Lemma 1.9, λ = (3^{2}, 2), which is a p-core. Moreover the group G does not appear in the list of [27, Theorem 2] where all irreducible restriction in characteristic 0 are given. So for the corresponding module in characteristic p the result follows from Lemma 1.8.

Now we are able to reduce the transitive case to the 2-homogeneous case:

Theorem 3.14. Let p > 3, n ≥ 7, and G < A_{n} be transitive. If E_{±}^{λ}↓_{G} is irreducible then G is 2-homogeneous.

Proof. Assume G is not 2-homogeneous, i.e. r_{2}(G) > 1 = r_{1}(G). By Theorem 3.11, either r_{3}(G) > r_{2}(G) or n is even and G is contained in one of the subgroups (Σ_{m} ≀ Σ_{2}) ∩ A_{n}, G = (Σ_{2} ≀ Σ_{m}) ∩ A_{n}. In the latter case E_{+}^{λ}↓_{G} is reducible by Proposition 3.13. In the former case, we have r_{1}(G) < r_{2}(G) < r_{3}(G). Since h(λ) ≥ 3 by Lemma 1.9(i), we can apply Proposition 2.3 and Theorem 2.5 to conclude that E_{+}^{λ}↓_{G} is reducible.

4. 2-homogeneous subgroups

Now we consider 2-homogeneous subgroups G < A_{n}. Our goal is to prove the following

Theorem 4.1. Let p > 3, n ≥ 7, and G < A_{n} be 2-homogeneous. Then E_{±}^{λ}↓_{G} is irreducible if and only if p > 5 and either λ = (3^{2}, 2) and G is AGL(3, 2) or λ = (3^{3}) and G is PΓL(2, 8).

The proof will occupy the rest of the section. We are going to use the classification of 2-homogeneous groups. The list of 2-transitive groups convenient for our purposes can be found in [18]. There are ‘very few’ 2-homogeneous groups which are not 2-transitive, and all of them are subgroups of certain AΓL(1, q)’s, see [19]. As in Theorem 4.1 we have n ≥ 7, we will keep this assumption throughout the section.

On representations of AGL(m, q) and exceptional cases in small heights. Assume that p = char F does not divide a prime power q. We describe a parametrization of irreducible F AGL(m, q)-modules, referring the reader to [29, 4, 3] for more details. Let V_{m} be the mdimensional 𝔽_{q}-vector space with basis {v_{1}, v_{2}, …, v_{m}} and the natural action of GL(m, q). We have AGL(m, q) = V_{m} ⋊ GL(m, q). Denote by Irr V_{m} the set of irreducible characters V_{m} → F^{∗} and by ¯0 ∈ Irr V_{m} the trivial character. Fix a non-trivial character χ_{0} : (𝔽_{q}, +) → F^{×}. For any element θ of the 𝔽_{q}-linear dual V_{m}^{∗}, let ε_{θ} denote the character sending v ∈ V_{m} to χ_{0}(θ(v)) ∈ F^{∗}. We let θ_{1}, …, θ_{m} denote the basis of V_{m}^{∗} dual to v_{1}, …, v_{m}, and set ε_{i} := ε_{θ_{i}} ∈ Irr V_{m}. The group GL(m, q) acts on Irr V_{m} by

(gε)(v) = ε(g^{−1}v), g ∈ GL(m, q), v ∈ V_{m}, ε ∈ Irr V_{m}.

Under this action, the trivial character ¯0 ∈ Irr V_{m} is fixed by GL(m, q) and all other characters are permuted transitively. The centralizer C_{GL(m,q)}(ε_{m}) is isomorphic to AGL(m − 1, q), embedded into GL(m, q) as the subgroup of matrices of the form

[ \begin{matrix}& ∗ & & ∗ \\ 0 & … & 0 & 1 \end{matrix} ] .

We have a chain of subgroups

GL(1, q) ⊂ AGL(1, q) ⊂ GL(2, q) ⊂ AGL(2, q) ⊂ ⋯ ⊂ GL(m, q) ⊂ AGL(m, q).

Given an F AGL(m, q)-module M and ε ∈ Irr V_{m}, let

M_{ε} = {m ∈ M | vm = ε(v)m for all v ∈ V_{m}}

be the corresponding ‘weight space’. Since F V_{m} is a semisimple algebra, M decomposes as a direct sum of such weight spaces. Observe that the action of GL(m, q) on M induces an action on the weight spaces, so that gM_{ε} = M_{gε} for all g ∈ GL(m, q), ε ∈ Irr V_{m}. In particular, M_{0} is stable under the action of GL(m, q), while M_{ε_{n}} is stable under the action of the subgroup AGL(m − 1, q), since AGL(m − 1, q) centralizes ε_{m}.

Now, let L be any irreducible F AGL(m − 1, q)-module. Consider the restriction L↓_{V_{m}}. If V_{m} acts trivially on L, the action of AGL(m, q) factors through the surjection AGL(m, q) → GL(m, q) to give an irreducible GL(m, q)-module. If this module is X, we denote L by L(X; 0). Next, assume that V_{m} does not act trivially on L. Then it not hard to see that L_{ε_{m}} is an irreducible F AGL(m − 1, q)-module in a natural way, and

L ≅ (L_{ε_{n}})↑_{V_{m}. AGL(m−1,q)}^{AGL(m,q)}

If V_{m−1} acts trivially on L_{ε_{m}}, this module factors through to give an irreducible F AGL(m − 1, q)-module, say X. In this case we denote our L by L(X; 1). Otherwise iterate by taking (L_{ε_{m}})_{ε_{m−1}}, etc. In this way the module L will be labelled by L(X; j), where 0 ≤ j ≤ m − 1 and X is an irreducible F GL(m − j, q)-module. If we do the same construction over ℂ we will get a similar parametrization. To distinguish from the modular case we use the notation L_{ℂ}(X_{ℂ}; j).

Now we can deal with the exceptional cases in Lemma 1.9(ii).

Lemma 4.2. Let λ and p be as in one of the exceptional cases in Lemma 1.9(ii), and G be a transitive subgroup of A_{n}. Then E_{±}^{λ}↓_{G} is irreducible if and only λ = (3^{2}, 2) and G is AGL(3, 2) or λ = (3^{3}) and G is PΓL(2, 8).

Proof. Note that λ is a p-core, so by Lemma 1.8 we only have to deal with the subgroups G for which the corresponding modules restrict irreducibly in characteristic 0. Checking the list in [27, Theorem 2], we see that λ = (3^{2}, 2), G = AGL(3, 2) and λ = (3^{3}), G = PΓL(2, 8) are the only two cases. By the hook formula, in both cases we have dim E_{±}^{λ} = 21. Now one checks using [7, p. 6] and [16, p. 6] that the 21-dimensional ℂ PΓL(2, 8)-module is irreducible modulo p > 5. By Lemma 1.8, the restriction to PΓL(2, 8) is indeed irreducible in characteristic p. Finally, a 21-dimensional irreducible ℂ AGL(3, 2)-module must be of the form L_{ℂ}(X_{ℂ}; j), where either j = 1 and dim X_{ℂ} = 3 or j = 2 and dim X_{ℂ} = 1. Let X be a reduction modulo p of X_{ℂ}. Then X is irreducible as the order of the corresponding group is not divisible by p. It follows that reduction modulo p of L_{ℂ}(X_{ℂ}; j) is the irreducible module L(X; j). So, by Lemma 1.8, the restriction to AGL(3, 2) is indeed irreducible.

It is now convenient to establish a special case of the ‘if’ part of the Main Theorem:

Lemma 4.3. Let p > 5, λ = (3^{3}), and G = AGL(3, 2) be embedded into A_{8} < A_{9} via its permutation representation of degree 8. Then E_{±}^{λ}↓_{G} is irreducible.

Proof. It follows from Theorem 3.4 that E_{±}^{λ}↓_{A_{8}} ≅ E_{±}^{(3^{2},2)} or E_{∓}^{(3^{2},2)}. Now use Lemma 4.2.

Lemma 4.2 treats the cases where λ = λ^{M} and h(λ) = 3. Moreover λ ≠ λ^{M} for h(λ) ≤ 2 by Lemma 1.9(i). So from now on we may assume that

h(λ) ≥ 4.  (7)

Groups with a regular normal subgroup. A 2-homogeneous group G in this class is always a subgroup of the group AΓL(m, q) of all non-degenerate semilinear affine transformations of the affine space V = 𝔽_{q}^{m} acting naturally on the q^{m} points of V. We have n = q^{m}, and | AΓL(m, q)| = fq^{m(m+1)/2}(q^{m} − 1)(q^{m−1} − 1) … (q − 1), if q = ℓ^{f} for a prime ℓ.

Assume first that m = 1. Then q = n ≥ 7. We have |G| ≤ | AΓL(1, q)| = fq(q − 1). By Lemma 1.6, if E_{±}^{λ}↓_{G} is irreducible then dim E_{±}^{λ} ≤ \sqrt{fq(q − 1)}, which contradicts Lemma 1.12. So from now on we may assume that m ≥ 2.

If, moreover, q > 2 and (m, q) ≠ (2, 3), then for G = AΓL(m, q) we have r_{4}(G) > r_{3}(G) > r_{2}(G), see [3, §5]. Hence E_{±}^{λ}↓_{G} is reducible by (7), Theorem 2.5 and Proposition 2.3. Now, let (m, q) = (2, 3). Then |G| ≤ | AΓL(2, 3)| = 432, and so E_{±}^{λ}↓_{G} is reducible by Lemmas 1.6 and 1.12.

It remains to prove that E_{±}^{λ}↓_{G} is reducible for G = AGL(m, 2). If m = 3 then n = 8, and λ = λ^{M} implies that either λ = (3^{2}, 2) and p ≠ 5 or λ = (4, 2, 1^{2}) and p ≠ 7. The former case is treated in Lemma 4.2. In the latter one, λ is a p-core and we find using the hook formula that dim E_{±}^{λ} = 45, so E_{±}^{λ}↓_{G} is reducible by virtue of Lemma 1.6.

So let m ≥ 4. Then it is shown for example in [6, §4, p.166] that r_{4}(G) > r_{3}(G) and r_{6}(G) > r_{5}(G). Now, if p > 5 and h(λ) ≥ 6, we may apply Proposition 2.3 and Theorem 2.5. Otherwise we may assume that m = 4, p > 7, and λ = (4^{4}) or (5^{2}, 2^{3}), thanks to Lemma 1.9. Now, λ is a p-core, and E_{±}^{λ}↓_{G} is reducible by dimensions, using the hook formula and Lemma 1.6.

The case p = 5 for AGL(m, 2) turns out to be harder so we consider it separately in the next subsection.

The group AGL(m, 2) in characteristic 5. Throughout the subsection, p = 5, G = AGL(m, 2), n = 2^{m}, k = n/2. Until Corollary 4.10 we do not assume that λ = λ^{M}.

We will use the following fact proved in [3, 1.11].

Lemma 4.4. [3] Let μ and ν be p-regular partitions of k. Then μ + ν (coordinate-wise sum) is a p-regular partition, and D^{μ} ⊗ D^{ν} appears as a composition factor of the restriction D^{μ+ν}↓_{Σ_{k}×Σ_{k}} with multiplicity 1.

The following characteristic-free version of the Littlewood-Richardson rule was obtained by James and Peel. Let c_{μν}^{λ} denote the classical Littlewood-Richardson coefficient, i.e. c_{μν}^{λ} is equal to the multiplicity of S_{ℂ}^{λ} in (S_{ℂ}^{μ} ⊗ S_{ℂ}^{ν})↑_{Σ_{k}×Σ_{k}}^{Σ_{n}}.

Lemma 4.5. [15, 3.1,5.5] Let λ be a partition of m + n. Then the restriction S^{λ}↓_{Σ_{m}×Σ_{n}} has a filtration with factors of the form S^{μ} ⊗ S^{ν}, with S^{μ} ⊗ S^{ν} appearing precisely c_{μν}^{λ} times.

Corollary 4.6. Let μ, ν and λ be p-regular partitions of m, n and m + n, respectively. If D^{μ} ⊗ D^{ν} appears in D^{λ}↓_{Σ_{m}×Σ_{n}} then μ ⊵ α and ν ⊵ β for some partitions α and β with c_{αβ}^{λ} ≠ 0.

Proof. This follows from Lemma 4.5 as D^{λ} is a quotient of S^{λ}, and D^{κ} is a composition factor of S^{γ} only if κ ⊵ γ.

Now we prove a special combinatorial lemma on the Littlewood-Richardson rule.

Lemma 4.7. Let m ≥ 4, λ be a 5-regular partition of n = 2^{m}, k = n/2, α = ((k/4)^{4}), β = (k − 1, 1) or (k − 2, 1^{2}), and the Littlewood-Richardson coefficient c_{αβ}^{λ} ≠ 0. Then the restriction D^{λ}↓_{Σ_{k}×Σ_{k}} has a composition factor D^{μ} ⊗D^{ν}, where μ and ν are distinct 5-regular partitions of k, with μ, ν ∉ {((k/4)^{4}), (k)}.

Proof. We want to use Lemma 4.4. For this, we need to decompose λ as a sum μ + ν with μ and ν as required. Assume first that β = (k − 1, 1). By the Littlewood-Richardson rule, λ has one of the following forms (where there are precisely k − 1 1’s in each picture).

1 1  1 2  . . . . . . . . . . . . . . . (a)  1  1 2  . . . . . . . . . . . . . . . (b)

1 1  1 2 1  1 …  . . . . . . . . . . . . . . . (c)  1  1 1  1 2  . . . . . . . . . . . . . . . . . . (d)

1  1 1  1 2  . . . . . . . . . . . . . . . . . . (e)

In case (a) we can take μ = (k − 3, 1^{3}) and ν = ((k/4) + 2, (k/4), ((k/4) − 1)^{2}). In case (b), take

μ = (k − 4, 1^{4}) and ν = ((k/4) + 3, ((k/4) − 1)^{3}).

In cases (c), (d) and (e) assume that there are x 1’s in the fifth row of λ. Then there are k − 1 − x ones in the first row. We will distinguish between the cases where x = 2r is even or x = 2r + 1 is odd. Now in (c) take

μ = (k − 4r, r^{4}), ν = ((k/4) − 1 + 2r, (k/4) + 1 − r, ((k/4) − r)^{2}, r),
if x = 2r, and
μ = (k − 4(r + 1), (r + 1)^{4}), ν = ((k/4) + 2r + 2, (k/4) − r, ((k/4) − r − 1)^{2}, r),
if x = 2r + 1.

In (d) take

μ = (k − 4r, r^{4}), ν = ((k/4) − 1 + 2r, ((k/4) − r)^{3}, r + 1),
if x = 2r, and
μ = (k − 4(r + 1), (r + 1)^{4}), ν = ((k/4) + 2r + 2, ((k/4) − r − 1)^{3}, r + 1),
if x = 2r + 1 < (k/4) − 1.

If x = 2r + 1 = (k/4) − 1, we have two subcases: m > 4 and m = 4. If m > 4, then k − 4(r + 2) ≥ (r + 2), and we can take

μ = (k − 4(r + 2), (r + 2)^{4}) and ν = ((k/4) + 2r + 6, ((k/4) − r − 2)^{3}, r).

If m = 4, we have λ = (8, 2^{4}). Then λ^{M} = (9, 2^{3}, 1) = (5, 1^{3}) + (4, 1^{4}), and by Lemma 4.4, D^{(5,1^{3})} ⊗ D^{(4,1^{4})} is a composition factor of D^{λ^{M}}↓_{Σ_{8}×Σ_{8}}. Therefore D^{(5,1^{3})^{M}} ⊗ D^{(4,1^{4})^{M}} = D^{(4,1^{4})} ⊗ D^{(5,1^{3})} is a composition factor of D^{λ}↓_{Σ_{8}×Σ_{8}}, which proves the claim.

Finally, in (e) take

μ = (k − 4r − 1, r^{4}, 1), ν = ((k/4) + 2r, ((k/4) − r)^{3}, r),
if x = 2r > 2, and
μ = (k − 4(r + 1) − 1, (r + 1)^{4}, 1), ν = ((k/4) + 2r + 3, ((k/4) − r − 1)^{3}, r),
if x = 2r + 1 > 1.

For the exceptional cases λ = (k + (k/4) − 3, (k/4)^{3}, 2, 1) and λ = (k + (k/4) − 2, (k/4)^{3}, 1^{2}) we can decompose

λ = (k + (k/4) − 3, (k/4)^{3}, 2, 1) = (k − 9, 2^{4}, 1) + ((k/4) + 6, ((k/4) − 2)^{3}) and
λ = (k + (k/4) − 2, (k/4)^{3}, 1^{2}) = (k − 8, 2^{3}, 1^{2}) + ((k/4) + 6, ((k/4) − 2)^{3}),

respectively, if m > 4. Finally, the exceptional cases for m = 4 are λ = (7, 2^{4}, 1) and λ = (8, 2^{3}, 1^{2}).

For the latter case, we use GAP to establish that the composition factors of S^{(8,2^{3},1^{2})} are

D^{(8,2^{3},1^{2})}, D^{(8,3^{2},2)}, D^{(9,2^{2},1^{3})}, D^{(9,3^{2},1)}.

Now, S^{(2^{2},1^{4})} ⊗ S^{(6,1^{2})} and hence D^{(2^{2},1^{4})} ⊗ D^{(6,1^{2})} appears in S^{(8,2^{3},1^{2})}↓_{Σ_{8}×Σ_{8}} by Lemma 4.5. Moreover, D^{(2^{2},1^{4})} ⊗ D^{(6,1^{2})} may not appear in the restrictions of D^{(8,3^{2},2)}, D^{(9,2^{2},1^{3})}, D^{(9,3^{2},1)} by Corollary 4.6, so it must appear in D^{(8,2^{3},1^{2})}↓_{Σ_{8}×Σ_{8}}.

For the former case, we use GAP to establish that the composition factors of S^{(7,2^{4},1)} are

D^{(7,2^{4},1)}, D^{(7,4,3,2)}, D^{(8,2^{3},1^{2})}, D^{(8,3^{2},2)}.

Now, D^{(2^{2},1^{4})} ⊗ D^{(5,1^{3})} appears in S^{(7,2^{4},1^{2})}↓_{Σ_{8}×Σ_{8}} as above.

Now assume β = (n−2, 1^{2}). By the Littlewood-Richardson rule, λ has one of the following forms (where there are precisely k − 2 1’s in each picture).

1 1  1 2 3  . . . . . . . . . . . . . . . (A)  1 1 2  1 3  . . . . . . . . . . . . . . . (B)

1  1 2 3  . . . . . . . . . . . . . . . (C)  1 2 3 1  1 1  1  . . . . . . . . . . . . . . . . . . (D)

1 2 3  1  1 1  1  . . . . . . . . . . . . . . . . . . (E)  1 2 3  1  1 1  1  . . . . . . . . . . . . . . . . . . (F)

1 3  2  1  1 1  1  . . . . . . . . . . . . . . . . . . (G)  1 3 2  1  1 1  1  . . . . . . . . . . . . . . . . . . (H)

In case (A) take μ = (k − 3, 1^{3}) and ν = ((k/4) + 1, (k/4)^{2}, (k/4) − 1). The cases (B), (C), (E), and (G) are special cases of the cases (c), (e), (c), and (e) above, respectively. In the remaining cases assume that there are x 1’s in the fifth row of λ. Then there are k − 2 − x ones in the first row. We distinguish between the cases where x = 2r is even and x = 2r + 1 is odd. Now in (D) take

μ = (k − 4r, r^{4}), ν = ((k/4) − 2 + 2r, ((k/4) + 1 − r)^{2}, (k/4) − r, r),
if x = 2r, and
μ = (k − 4(r + 1), (r + 1)^{4}), ν = ((k/4) + 2r + 1, ((k/4) − r)^{2}, (k/4) − r − 1, r),
if x = 2r + 1.

In (F), take

μ = (k − 4r − 2, r + 1, r^{3}, 1), ν = ((k/4) + 2r, ((k/4) − r)^{3}, r),
if x = 2r, and

μ = (k − 4(r + 1) − 2, r + 2, (r + 1)^{3}, 1), ν = ((k/4) + 2r + 3, ((k/4) − r − 1)^{3}, r), if x = 2r + 1.

In (H), take

μ = (k − 4r − 2, r^{4}, 1^{2}), ν = ((k/4) + 2r, ((k/4) − r)^{3}, r),
if x = 2r > 2, and
μ = (k − 4(r + 1) − 2, (r + 1)^{4}, 1^{2}), ν = ((k/4) + 2r + 3, ((k/4) − r − 1)^{3}, r),
if x = 2r + 1 > 1.

For the exceptional cases λ = (k + (k/4) − 4, (k/4)^{3}, 2, 1^{2}) and (k + (k/4) − 3, (k/4)^{3}, 1^{3}) we can decompose

(k + (k/4) − 4, (k/4)^{3}, 2, 1^{2}) = (k − 10, 2^{4}, 1^{2}) + ((k/4) + 6, ((k/4) − 2)^{3}) and
(k + (k/4) − 3, (k/4)^{3}, 1^{3}) = (k − 9, 2^{3}, 1^{3}) + ((k/4) + 6, ((k/4) − 2)^{3}),

respectively, if m > 4. Finally, the exceptional cases for m = 4 are λ = (6, 2^{4}, 1^{2}) and λ = (7, 2^{3}, 1^{3}). But

(6, 2^{4}, 1^{2})^{M} = (9, 3, 1^{4}) = (2^{2}, 1^{4}) + (7, 1) and
(7, 2^{3}, 1^{3})^{M} = (8, 3, 2, 1^{3}) = (2^{2}, 1^{4}) + (6, 1^{2}).

Since (2^{2}, 1^{4})^{M} = (6, 2), (7, 1)^{M} = (2^{3}, 1^{2}) and (6, 1^{2})^{M} = (3, 2, 1^{3}), we conclude that D^{(6,2)} ⊗ D^{(2^{3},1^{2})} and D^{(6,2)} ⊗D^{(3,2,1^{3})} are composition factors of D^{(6,2^{4},1^{2})}↓_{Σ_{8}×Σ_{8}} and D^{(7,2^{3},1^{3})}↓_{Σ_{8}×Σ_{8}}, respectively.

The following technical result on restrictions from Σ_{n} to Σ_{k} × Σ_{k} develops Lemma 5.2 of [3]. Set

T_{n} := {1_{Σ_{n}}, sgn_{Σ_{n}}}

and

U_{n} := {D^{(n−1,1)}, D^{(n−1,1)} ⊗ sgn, D^{(n−2,1^{2})}, D^{(n−2,1^{2})} ⊗ sgn}.

Lemma 4.8. Let p = 5, m ≥ 3, n = 2^{m}, k = n/2, and λ be a p-regular partition of n. Assume that every composition factor of D^{λ}↓_{Σ_{k}×Σ_{k}} has one of the following forms:

(1) D^{μ} ⊗ D^{μ} for an arbitrary p-regular partition μ of k.

(2) D^{μ} ⊗ D^{ν} with one of μ, ν being in U_{k} ∪ T_{k}, and the other being in T_{k}.

Moreover, assume at least one factor of type (2) with one of μ, ν being in U_{k} and the other being in T_{k} occurs. Then D^{λ} ∈ U_{n}.

Proof. Since D^{μ} ⊗ D^{ν} appears in D^{λ}↓_{Σ_{k}×Σ_{k}} if and only if D^{ν} ⊗ D^{μ} does, we can pick a composition factor D^{μ} ⊗ D^{ν} of type (2) with μ ∈ U_{k} and ν ∈ T_{k}. Tensoring D^{λ} with sgn if necessary, we may assume without loss of generality that D^{μ} = D^{(n−1,1)} or D^{(n−2,1^{2})}.

By Lemma 4.5, the restriction S^{λ}↓_{Σ_{k}×Σ_{k}} has a filtration with factors of the form S^{α} ⊗ S^{β} with multiplicities given by the Littlewood-Richardson rule. As D^{λ} is a quotient of S^{λ}, the restriction D^{λ}↓_{Σ_{k}×Σ_{k}} has a filtration with factors being quotients of the modules S^{α} ⊗ S^{β}. Let us pick one such quotient (S^{α} ⊗ S^{β})/K containing D^{μ} ⊗ D^{ν} as a composition factor. Since S^{α} ⊗ S^{β} is the outer tensor product, there exist submodules K^{α} ⊂ S^{α} and K^{β} ⊂ S^{β} such that K = K^{α} ⊗ K^{β}, and (S^{α} ⊗ S^{β})/K ≅ (S^{α}/K^{α}) ⊗ (S^{β}/K^{β}). If S^{α}/K^{α} contains some composition factor D^{κ} different from D^{μ} then D^{κ} ⊗ D^{ν} appears in (S^{α} ⊗ S^{β})/K, and hence in D^{λ}↓_{Σ_{k}×Σ_{k}}. So by our assumptions, D^{κ} must belong to T_{k} ∪ U_{k}. Similarly any composition factor of S^{β}/K^{β} must belong to T_{k} ∪ U_{k}. So, by Lemma 1.14, S^{α}/K^{α} and S^{β}/K^{β} are semisimple. In particular, D^{μ} is in the head of S^{α} and D^{ν} is in the head of S^{β}. Since 5 does not divide k we have D^{μ} = S^{μ}, see [13, 24.1(i)]. Now by [13, 11.3], Hom_{Σ_{n}}(S^{α}, S^{μ}) ≠ 0 implies α ⊵ μ, whence α = μ. Now we consider two cases: D^{ν} = 1 and D^{ν} = sgn.

Case 1: D^{ν} = 1_{Σ_{k}}. Then, as in the previous paragraph we deduce that β = (n). By the Littlewood-Richardson rule, either h(λ) ≤ 3 or λ is of the form (λ_{1}, λ_{2}, 1^{2}). If h(λ) ≤ 3 then by Corollary 4.6 we have h(μ), h(ν) ≤ 3 for any D^{μ} ⊗D^{ν} appearing in D^{λ}↓_{Σ_{k}×Σ_{k}}. This allows one to apply Lemma 5.2 of [3]. If λ = (λ_{1}, λ_{2}, 1^{2}), then λ_{1} + λ_{2} + 2 = 2^{m} implies that λ can be decomposed as a sum γ + δ of 5-regular partitions γ and δ so that γ ≠ δ and D^{γ} ⊗ D^{δ} is not of type (2). In view of Lemma 4.4, D^{γ} ⊗ D^{δ} appears in D^{λ}↓_{Σ_{k}×Σ_{k}}, which contradicts our assumptions.

Case 2: D^{ν} = sgn_{Σ_{k}}. So we have sgn in the head of S^{β}, or, equivalently, 1 in the head of S^{β}⊗sgn. By [13, 8.15], S^{β}⊗sgn ≅ (S^{β^{t}})^{∗}, so 1 is in the socle of S^{β^{t}}. Moreover, λ is p-regular, and related to α, β by the Littlewood-Richardson rule. Since α = (n − 1, 1) or (n − 2, 1^{2}), it follows that no part of β is repeated more than p+2 = 7 times. Using Proposition 1.15, we now conclude that β = ((k/4)^{4}). So, if m ≥ 4, then by Lemma 4.7, D^{λ}↓_{Σ_{k}×Σ_{k}} has a composition factor D^{γ} ⊗ D^{δ} with γ ≠ δ and not of type (2), which contradicts our assumptions. If m = 3 then by [3, 5.2] we only have to consider the cases where h(λ), h(λ^{M}) > 3, so we arrive to the partition (4, 2, 1^{2}). But this partition is a 5-core, hence D^{(4,2,1^{2})} = S^{(4,2,1^{2})}, and so D^{(4,2,1^{2})}↓_{Σ_{4}×Σ_{4}} contains for example S^{(2,1^{2})} ⊗ S^{(3,1)} = D^{(2,1^{2})} ⊗ D^{(3,1)}.

The following result is crucial. Recall that AGL(m, 2) = V_{m} ⋊ GL(m, 2). So, in particular, V_{m} < Σ_{n}.

Proposition 4.9. Let m ≥ 2, and D^{λ} be an irreducible F Σ_{n}-module. Then the fixed point space (D^{λ})^{V_{m}} is trivial if and only if D^{λ} ∈ U_{n} or n = 8 and D^{λ} ≅ D^{(5,3)} or D^{(5,3)} ⊗ sgn.

Proof. The ‘if’ part is proved in [3, 5.5]. Conversely, assume that D^{λ} ∉ U_{n} and D^{λ} ≅ D^{(5,3)}, D^{(5,3)} ⊗ sgn. We have to prove that (D^{λ})^{V_{m}} ≠ 0. For h(λ) ≤ 3 this is proved in [3, 5.5]. We will use that result. Moreover, the argument here follows the proof of [3, 5.5] closely.

Assume first that the required result is proved for m = 4 and use induction on m = 4, 5, … to prove it for m ≥ 4. Let m ≥ 5 and assume (D^{λ})^{V_{m}} = 0.

Let {v_{1}, v_{2}, …, v_{m}} be the standard basis of V_{m}. Denote our embedding V_{m} ↪ Σ_{n} by φ_{m}. Note that V_{m} = Y ⊔Z, where Y and Z are the sets of all vectors in V_{m} whose v_{m}-coordinate is 0 and 1, respectively. Then Y is a subgroup of V_{m}, isomorphic to V_{m−1}. Let us identify V_{m−1} with Y. Note that V_{m−1} acts regularly on both Y and Z. Denote the symmetric groups on the sets Y and Z by Σ(Y) and Σ(Z), respectively. Of course, Σ(Y) ≅ Σ(Z) ≅ Σ_{k}. Moreover H := Σ(Y) × Σ(Z) ≅ Σ_{k} × Σ_{k} is a Young subgroup of Σ_{n} in a natural way, and V_{m−1} is embedded into H diagonally via φ_{m−1} × φ_{m−1}. Set c := φ_{m}(v_{m}). Then c is an element of order 2 which permutes Y and Z. Let C = {1, c} < Σ_{n} be the subgroup of order 2 generated by c. Then V_{m} < H.C ≅ Σ_{k} ≀ C.

We next describe the irreducible representations of H.C. Let μ and ν be p-regular partitions of k. For any F H-module of the form D^{μ} ⊗ D^{ν} define a map

τ : D^{μ} ⊗ D^{ν} → D^{ν} ⊗ D^{μ}, d ⊗ f ↦ f ⊗ d, d ∈ D^{μ}, f ∈ D^{ν}.

For μ ≠ ν, we consider an F H-module D^{μ} ⊗ D^{ν} ⊕ D^{ν} ⊗ D^{μ}. Let c act on it via c · (x, y) = (τ(y), τ(x)) (for x ∈ D^{μ}⊗D^{ν}, y ∈ D^{ν}⊗D^{μ}). One can easily see that this defines an irreducible F [H.C]-module, which we denote by D(μ, ν). Also consider an F H-module D^{μ} ⊗ D^{μ}. Let c act on it via c · x = τ(x) or c · x = −τ(x). This defines an irreducible F [H.C]-module, which we denote by D(μ, +1) or D(μ, −1), respectively. It follows from the Clifford theory (see e.g. [9], section III.2) that any F [H.C]-module looks like D(μ, ±1) or D(μ, ν) for μ ≠ ν.

We now claim that D^{V_{m}} ≠ 0, for an irreducible F [H.C]-module D, unless one of the following happens:

(1) D ≅ D(μ, −1);

(2) D ≅ D(μ, ν), D^{ν} ∈ T_{k}, and (D^{μ})^{V_{m−1}} = 0;

(3) D ≅ D(μ, ν), D^{μ} ∈ T_{k}, and (D^{ν})^{V_{m−1}} = 0. Indeed, assume first that that D is of the form D(μ, +1). If D^{μ} ≅ 1_{Σ_{k}} or sgn_{Σ_{k}} then D↓_{V_{m}} ≅ 1_{V_{m}}, and so D^{V_{m}} ≠ 0. Otherwise, D_{χ}^{μ} ≠ 0 for every χ ∈ Irr V_{m−1} \ {¯0}. Pick a non-zero vector f ∈ D_{χ}^{μ}, and consider the vector f ⊗ f ∈ D = D^{μ} ⊗ D^{μ}. As χ ⊗ χ ≅ 1_{V_{m−1}}, we see that f ⊗f is V_{m}-invariant. Now assume that D is of the form D(μ, ν). If D^{μ}, D^{ν} ∉ T_{k}, then arguing as above, we can pick χ ∈ Irr V_{m−1} \ {¯0} and non-zero vectors d ∈ D_{χ}^{μ}, f ∈ D_{χ}^{ν}, and observe that d ⊗ f + f ⊗ d ∈ D = D^{μ} ⊗ D^{ν} ⊕ D^{ν} ⊗ D^{μ} is V_{m}-invariant. Finally, assume that (D^{μ})^{V_{m−1}} ≠ 0 and D^{ν} ∈ T_{k} (or vice versa). In this case V_{m} acts trivially on the vector d ⊗ f + f ⊗ d, where d ∈ (D^{μ})^{V_{m−1}} ≠ 0 and f is a generator of D^{ν}.

Now consider the restriction D^{λ}↓_{H.C}. By inductive hypothesis, (D^{μ})^{V_{m−1}} ≠ 0 unless D^{μ} ∈ U_{k}. By the previous paragraph, we can assume that every composition factor of D^{λ}↓_{H.C} has one of the forms:

(1) D(μ, −1);

(2) D(μ, ν) with D^{ν} ∈ T_{k} and D^{μ} ∈ U_{k};

(3) D(μ, ν) with D^{μ} ∈ T_{k} and D^{ν} ∈ U_{k}. Moreover, not all composition factors have form (1), as this would mean that φ_{m}(c) has only the eigenvalue −1 on D^{λ}, which contradicts [3, 1.5]. It follows that composition factors of D^{λ}↓_{H} satisfy the assumptions of Lemma 4.8, whence D^{λ} ∈ U_{n}, giving a contradiction.

Now we consider the cases m = 2, 3, and 4. For m = 2, we only need to consider D^{(2,2)}, and it is shown in [3, 5.4] that V_{2} acts trivially on D^{(2,2)}. For m = 3, in view of [3, 5.5], we only need to consider the partitions (5, 1^{3}), (4, 2, 1^{2}) and (4, 1^{4}). But they are 5-cores, so for such a partition λ we have S^{λ} = D^{λ}. Hence, by Lemma 4.5, D^{(5,1^{3})}↓_{Σ_{4}×Σ_{4}} and D^{(4,1^{4})}↓_{Σ_{4}×Σ_{4}} contain 1 ⊗ sgn and D^{(4,2,1^{2})}↓_{Σ_{4}×Σ_{4}} contains D^{(3,1)} ⊗ D^{(2,1,1)} as a composition factor, which allows us to argue as for the case m ≥ 5 above.

Now let m = 4. Arguing as for m ≥ 5 above, it suffices to prove that for λ with h(λ) > 3 and λ ∉ U_{16} ∪ T_{16}, the restriction D^{λ}↓_{Σ_{8}×Σ_{8}} does not satisfy the condition that (a) all of its composition factors are of the form D^{μ} ⊗ D^{ν} or D^{ν} ⊗ D^{μ}, where either μ = ν or D^{μ} ∈ T_{8} and D^{ν} ∈ U_{8} ∪ {D^{(3^{2},2)}, D^{(3^{2},2)} ⊗ sgn = D^{(5,3)}}, and (b) at least one composition factor with μ ≠ ν appears.

By Lemma 4.8, we may assume without loss of generality that D^{λ}↓_{Σ_{8}×Σ_{8}} contains D^{(5,3)}⊗1 or D^{(3^{2},2)} ⊗ 1. Arguing as in the proof of Lemma 4.8, we find a subfactor in D^{λ}↓_{Σ_{8}×Σ_{8}} of the form (S^{α}/K^{α}) ⊗ (S^{β}/K^{β}) containing D^{(5,3)} ⊗ 1 or D^{(3^{2},2)} ⊗ 1 as a composition factor, and such that c_{αβ}^{λ} ≠ 0. However, D^{(5,3)} appears only in S^{(5,3)} and S^{(3^{2},2)}, D^{(3^{2},2)} appears only in S^{(3^{2},2)} and S^{(2^{3},1^{2})}, and 1 appears only in S^{(8)} and S^{(4^{2})}. Any of these Specht modules S^{γ} corresponds to a p-regular partition γ, so it has a simple head D^{γ}. So if, say, D^{(5,3)}⊗1 appears in (S^{(3^{2},2)}/K^{(3^{2},2)}) ⊗ (S^{(4^{2})}/K^{(4^{2})}), then D^{(3^{2},2)} ⊗ D^{(4^{2})} and D^{(5,3)} ⊗ D^{(4^{2})} must appear in it, too. It follows that c_{μν}^{λ} ≠ 0 for μ = (5, 3), ν = (8) or μ = (3^{2}, 2), ν = (8). Since h(λ) > 3 we conclude that λ is one of the following partitions: (10, 3, 2, 1), (9, 3^{2}, 1), (9, 3, 2^{2}), (8, 3^{2}, 2). But for such λ, the restriction D^{λ}↓_{Σ_{8}×Σ_{8}} does not satisfy the conditions (a) and (b) above, thanks to Lemma 4.4.

Corollary 4.10. Let λ = λ^{M}. Then E_{±}^{λ}↓_{AGL(m,2)} is reducible.

Proof. It follows from Proposition 4.9 and Lemma 1.9(i)(ii) that (D^{λ})^{V_{m}} ≠ 0, so (E_{±}^{λ})^{V_{m}} ≠ 0. Note that (E_{±}^{λ})^{V_{m}} ≠ 0 is GL(m, 2)-invariant. So if E_{±}^{λ}↓_{AGL(m,2)} is irreducible then (E_{±}^{λ})^{V_{m}} = E_{±}^{λ}, which contradicts the faithfulness of the representation.

Groups G with PSL(m, q) ≤ G ≤ PΓL(m, q), m ≥ 2. The group PΓL(m, q) acts naturally on the points of ℙ^{m−1}(𝔽_{q}), yielding a permutation representation of degree n = (q^{m} − 1)/(q − 1). If we twist this permutation representation with an outer automorphism of G, we get another permutation representation, non-equivalent to the first one. However, in both cases we get the same subgroup of A_{n}, and for the problem we are considering we may assume without loss of generality that the action is natural.

We consider the case m = 2 first. We have n = q + 1 and |G| ≤ | PΓL(2, q)| = fq(q^{2} − 1), where q = ℓ^{f} for a prime ℓ. Using Lemmas 1.6 and 1.12, we may assume that q = 9 or q = 8.

If q = 9 we have PΓL(2, 9) ≅ PSL(2, 9) .2^{2}, see [7]. By [7, 16], the dimensions of irreducible F PSL(2, 9)-modules are among 1, 5, 8, 9, 10. It follows that the dimensions of irreducible F PΓL(2, 9)-modules, exceeding 34, can only be 4 · 10 = 40 and 4 · 9 = 36. Now we can apply Lemma 1.12 and [7, 16].

If q = 8, then n = 9 and so λ = λ^{M} implies that either λ = (5, 1^{4}) or λ = (3^{3}) and p > 5. The latter case has been treated in Lemma 4.2. For the former case, note that (5, 1^{4}) is a p-core for any p > 5, so using the hook formula we get dim E_{±}^{λ} = 35. On the other hand, by [7, 16], the group PΓL(2, 8) does not have irreducible modules of dimension 35.

Let m = 3 and q = 2. By [7, 16], the irreducible modules over PΓL(3, 2) ≅ GL_{3}(2) have dimension at most 8, and we can apply Lemma 1.12.

If m ≥ 3 and (m, q) ≠ (3, 2), then for the group G = PΓL(m, q) we have r_{4}(G) > r_{3}(G) > r_{2}(G), see [3, §5]. Hence E_{±}^{λ}↓_{G} is reducible by (7), Theorem 2.5 and Proposition 2.3.

The group G = Sp(2m, 2), m ≥ 3. In this case G has two 2-transitive representations Ω^{0} and Ω^{1} of degrees 2^{m−1}(2^{m} + 1) and 2^{m−1}(2^{m} − 1), respectively. It is proved in [3, 5.10,5.11] that r_{4}(G) > r_{3}(G) > r_{2}(G). So we can apply Theorem 2.5 and Proposition 2.3 as above.

The groups M_{23} and M_{24}. The result follows by dimensions using Lemma 1.13 and [7, 16].

The remaining 2-homogeneous groups. In the remaining cases one can easily check the dimensions by using Lemmas 1.12, 1.6 and [7, 16]. (One may consult [3, §5] for a similar problem for symmetric groups).

5. Proof of the Main Theorem

If λ ≠ λ^{M} then E^{λ} = D^{λ}↓_{A_{n}}, and the result follows from [3, Main Theorem]. So we may assume that λ = λ^{M}. Now the ‘if’ part follows from Theorem 3.7, Proposition 3.8 and Lemmas 4.2, 4.3. For the ‘only-if’ part, suppose that E_{±}^{λ}↓_{G} is irreducible. We may assume that n ≥ 8. Indeed, we only have to deal with the cases where p ≤ n, as otherwise one can use the characteristic 0 result [27]. For 5 ≤ n ≤ 7 and p ≤ n, the only Mullineux-fixed partition is (4, 1^{3}) for p = 5, see Lemma 1.9. But this partition is a 5-core and so this case can be dealt with using Lemma 1.8 and [27] or just using [7, 16]. Now, by Theorem 3.14 and Proposition 3.2, either G is 2-homogeneous or G ≤ A_{(n−k,k)} for k ≤ 2. If G is 2-homogeneous we apply Theorem 4.1. So let G ≤ A_{(n−k,k)} for k ≤ 2.

Assume first that k = 2. Recall that A_{(n−2,2)} = A_{n} ∩ (Σ_{n−2} × Σ_{2}). Let σ ∈ Σ_{n} be the transposition which generates the subgroup Σ_{2} in Σ_{n−2} × Σ_{2}. For g ∈ Σ_{n−2} let ε(g) be 1 if g is odd and 0 otherwise. Then

φ : Σ_{n−2} → A_{(n−2,2)}, g ↦ (g, σ^{ε(g)})

is an isomorphism. As G < A_{(n−2,2)}, the restriction E_{±}^{λ} ↓_{A_{(n−2,2)}} is irreducible and via the isomorphism φ it can be considered as an irreducible F Σ_{n−2}-module, D^{γ} say, where γ is a pregular partition of n−2. By Theorem 3.7, λ is a JS-partition. It follows from Lemma 3.6 that γ is obtained from λ by consecutively removing two good nodes. Applying [3, Main Theorem] to the group Σ_{n−2} we see that one of the following happens: (a) G contains A_{n−2}, (b) γ or γ^{M} belongs to the set of partitions X := {(n−3, 1), (n−4, 2), (n−4, 1^{2}), (21, 2, 1), (21, 1^{3}), (5, 3)} or (c) G < A_{(n−3,3)}. But (c) is impossible by Proposition 3.2. If (b) happens then using Lemma 3.5 we may assume that λ is obtained from a partition γ in X by adding two nodes. Using Lemma 1.9, we conclude that there are no such Mullineux-fixed Jantzen-Seitz partitions λ. So (b) is also impossible.

Now let k = 1, i.e. G ≤ A_{n−1}. We may assume that G ≰ A_{(n−2,2)}. Then we have to prove that G = A_{n−1}, unless G = AGL(3, 2) embedded as in Main Theorem (ii)(c) and λ = (3^{3}). Since the restriction E_{±}^{λ}↓_{A_{n−1}} is irreducible, Proposition 3.8 implies that either (1) λ is a JS-partition, in which case E_{±}^{λ}↓_{A_{n−1}} ≅ E_{±}^{λ_{A}} or E_{∓}^{λ_{A}}, where A is the top removable node of λ, or (2) λ has exactly two normal nodes A,B, and res A ≠ res B, in which case E_{±}^{λ}↓_{A_{n−1}} ≅ E^{λ_{A}} ≅ E^{λ_{B}}. As G ≰ A_{(n−2,2)}, Proposition 3.2 implies that G is transitive in A_{n−1}. Consider the case (1). By Theorems 3.14 and 4.1, either λ_{A} = (3^{2}, 2) and G is AGL(3, 2) or λ_{A} = (3^{3}) and G is PΓL(2, 8). By Theorem 3.4, only the former case is possible, in which case we get λ = (3^{3}) and G = AGL(3, 2), as required. Finally, in case (2) it follows from [3, Main Theorem] that λ_{A} or λ_{B} belongs to the set X defined in the previous paragraph. Using Lemma 1.9, we conclude that there are no Mullineux-fixed partitions λ which satisfy (2) and have λ_{A} in X.

References

[1] C. Bessenrodt and J.B. Olsson, On residue symbols and the Mullineux conjecture, J. Algebraic Combin. 7 (1998), 227–251.

[2] C. Bessenrodt and J.B. Olsson, Branching of modular representations of the alternating groups, J. Algebra 209 (1998), 143–174.

[3] J. Brundan and A. S. Kleshchev, Representations of the symmetric group which are irreducible over subgroups, preprint, University of Oregon, 1999.

[4] J. Brundan, R. Dipper, and A. S. Kleshchev, Quantum linear groups and representations of GL_{n}(𝔽_{q}), preprint, University of Oregon, 1999.

[5] V.P. Burichenko, A.S. Kleshchev, and S. Martin, On cohomology of dual Specht modules, J. Pure Appl. Alg. 112(1996), 157-180.

[6] P.J. Cameron, P.M. Neumann, and J. Saxl, An interchange property in finite permutation groups, Bull. London Math. Soc. 11(1979), 161-169.

[7] J.H. Conway, R. T. Curtis, S. P. Norton, R. A Parker, R. A. Wilson, Atlas of Finite Groups, Clarendon Press, Oxford, 1985.

[8] J. D. Dixon and B. Mortimer, Permutation Groups, Graduate Texts in Mathematics 163, Springer-Verlag, Berlin, Heidelberg, New York, 1996.

[9] W. Feit, The representation theory of finite groups, North-Holland Publ. Comp., Amsterdam, New York, Oxford, 1982.

[10] B. Ford, Irreducible restrictions of representations of the symmetric groups, Bull. Lond. Math. Soc. 27(1995), 453–459.

[11] B. Ford, Irreducible representations of the alternating group in odd characteristic. Proc. Amer. Math. Soc. 125 (1997), 375-380.

[12] B. Ford and A. S. Kleshchev, A proof of the Mullineux conjecture, Math. Z. 226(1997), 267–308.

[13] G. D. James, The representation theory of the symmetric groups, Springer Lecture Notes 682, Berlin, Heidelberg, New York, 1978.

[14] G. James and A. Kerber, The representation theory of the symmetric group, Addison-Wesley, London, 1981.

[15] G. D. James and M. H. Peel, Specht series for skew representations of symmetric groups, J. Algebra, 56 (1979), 343–364.

[16] C. Jansen, K. Lux, R. Parker, R. Wilson, An Atlas of Brauer Characters, Clarendon Press, Oxford, 1995.

[17] J. C. Jantzen and G. M. Seitz, On the representation theory of the symmetric groups, Proc. London Math. Soc. 65 (1992), 475–504.

[18] W. M. Kantor, Homogeneous designs and geometric lattices, J. Combin. Theory Ser. A 38 (1985), 66–74.

[19] W. M. Kantor, k-homogeneous groups, Math. Z. 124 (1972), 261–265.

[20] A. S. Kleshchev. On restrictions of irreducible modular representations of semisimple algebraic groups and symmetric groups to some natural subgroups I, Proc. London Math. Soc. 69 (1994), 515–540.

[21] A. S. Kleshchev, Branching rules for modular representations of symmetric groups, I, J. Algebra 178 (1995), 493–511.

[22] A. S. Kleshchev, Branching rules for modular representations of symmetric groups, II, J. reine angew. Math. 459 (1995), 163–212.

[23] A. S. Kleshchev, Branching rules for modular representations of symmetric groups III: some corollaries and a problem of Mullineux, J. London Math. Soc. 54 (1996), 25–38.

[24] A. S. Kleshchev, On decomposition numbers and branching coefficients for symmetric and special linear groups, Proc. London Math. Soc. (3) 75 (1997), 497–558.

[25] A. S. Kleshchev and J. Sheth, Representations of the symmetric group are reducible over singly transitive subgroups, Math. Z., to appear.

[26] G. Mullineux, Bijections of p-regular partitions and p-modular irreducibles of the symmetric groups. J. London Math. Soc. (2), 20(1979), 60–66.

[27] J. Saxl, Irreducible characters of the symmetric groups that remain irreducible in subgroups, J. Algebra, 111(1987), 210–219.

[28] M. Schönert et. al., Gap: groups, algorithms and programming, 3.4.3, RWTH Aachen, 1996.

[29] A. Zelevinsky, Representations of finite classical groups, Lecture Notes in Mathematics 869, Springer-Verlag, Berlin, Heidelberg, New York, 1981.

Department of Mathematics, University of Oregon, Eugene, OR 97403, USA

 E-mail address: klesh@math.uoregon.edu

Department of Mathematics, Saint Louis University, St. Louis, MO 63103, USA E-mail address: sheth@slu.edu