REPRESENTATIONS OF THE SYMMETRIC GROUP ARE REDUCIBLE OVER SINGLY TRANSITIVE SUBGROUPS

A. S. KLESHCHEV AND J. K. SHETH

Introduction

Let F be an algebraically closed field of characteristic p ≥ 0 and Σ_{n} = Sym(Ω) be the symmetric group on an n-element set Ω. We are interested in the following

Problem. For any n describe all pairs (G, D) where G ≤ Σ_{n} is a subgroup and D is a simple F Σ_{n}-module of dimension greater than 1 such that the restriction D_{G} is irreducible.

If the characteristic of F is zero, this problem has been solved by Saxl [16]. An important feature of Saxl’s result is that a group G as in the problem above is either 2-transitive or fixes a point, i.e. is contained in some Σ_{n−1}. The case G ≤ Σ_{n−1} can then be settled using the branching rule and induction. On the other hand, an explicit list of k-transitive groups (for k ≥ 2) is available, which can be used to complete the proof in characteristic zero, see [16] for more details.

From now on we assume that p > 0. In this case the problem is important for determining maximal subgroups of finite classical groups [1],[13]. However, the situation is now more complicated. For example, to determine the pairs (G, D) as above with G = A_{n} one needs the Mullineux conjecture [7], [2].

If G is intransitive then, up to a conjugation, it is contained in a standard Young subgroup of the form Σ_{n−k} × Σ_{k}. The irreducible restrictions from Σ_{n} to Σ_{n−1} have been described in [14], see also [11], [6]. In [11] it is also shown for p > 2 that the Young subgroups Σ_{n−k} × Σ_{k} with 1 < k < n − 1 never act irreducibly on a simple F Σ_{n}-module of dimension > 1 (this fact is also proved in this paper, see Theorem 3.10). This reduces the intransitive case to the subgroups of Σ_{n−1}. The main result here is

Main Theorem. Let p > 2, n ≥ 4, G ≤ Σ_{n}, and D be a simple F Σ_{n}-module with dim D > 1. If the restriction D_{G} is irreducible then either G ≤ Σ_{n−1} or G is 2-transitive on Ω.

We refer the reader to Theorem 3.10 for a partial result in the case p = 2.

For p > 2 the problem above is thus reduced to 2-transitive groups. This turns out to be a very important step in a complete solution of the problem, which is obtained in [3] (for the case p > 3).

1. Preliminaries

Let Ω be a finite set with |Ω| = n, and k ≤ n/2. A group G ≤ Sym(Ω) is called khomogeneous (resp. k-transitive ) on Ω if G acts transitively on the unordered (resp. ordered) k-element subsets of Ω. Moreover, G is called k*-homogeneous on Ω if G is khomogeneous but not k-transitive on Ω.

1991 Mathematics Subject Classification. 20C30, 20J06.

The first author partially supported by the NSF, grant DMS-9600124.

Let F be an algebraically closed field of characteristic p > 0, G be a group, and V be an F G-module. Denote by V^{G} the space of G-fixed points in V. We write V ≅ V_{1}| ⋯ |V_{k} if V admits a unique filtration with sections isomorphic to V_{j}, 1 ≤ j ≤ k, counted from bottom to top. In particular, if the sections are irreducible this means that V is uniserial. If H ≤ G is a subgroup, then V_{H} or V ↓_{H} denotes the restriction of V to H. If W is an F H-module then W ↑^{G} denotes the induced module. If V ≅ V^{∗} as F G-modules then V_{H} ≅ (V_{H})^{∗} as F Hmodules. We let 1=1_{G} denote the trivial representation of G. If V and W are F G-modules we write Hom_{G}(V, W) (resp. Hom(V, W)) for the space of all F G-homomorphisms (resp. F -linear maps) from V to W. Note that Hom(V, W) is an F G-module with Hom_{G}(V, W) ≅ Hom(V, W)^{G}.

Basic facts on the representation theory of the symmetric group can be found in [8]. For λ = (λ_{1}, …, λ_{r}), a composition of n with non-zero parts, we let Σ_{λ} = Σ_{λ_{1}} × ⋯ × Σ_{λ_{r}} denote the corresponding Young subgroup of Σ_{n}. We may identify Σ_{n−1} with Σ_{(n−1,1)}. We let Y^{λ}, S^{λ} and M^{λ} = (1_{Σ_{λ}})↑^{Σ_{n}} denote the Young, Specht, and permutation modules labelled by λ, respectively. If λ is p-regular we write D^{λ} for the unique irreducible quotient of S^{λ}. It is well known that M^{λ}, Y^{λ}, and D^{λ} are all self-dual F Σ_{n}-modules. Let sgn denote the sign representation of Σ_{n}.

We will need the structure of the permutation modules M^{(n−1,1)} and M^{(n−2,2)}. The proof of the next three lemmas is obtained by applying [8, 17.17, 24.15] and the ‘Nakayama Conjecture’ [10, 6.1.21, 2.7.41].

Lemma 1.1. The module M^{(n−1,1)} is isomorphic to D^{(n−1,1)} ⊕ 1 if n ≢ 0 (mod p), and 1|D^{(n−1,1)}|1 ≅ S^{(n−1,1)}|1 ≅ 1|(S^{(n−1,1)})^{∗} otherwise.

Lemma 1.2. Let p > 2 and n ≥ 4.

(i) If n ≢ 1, 2 (mod p) then M^{(n−2,2)} ≅ Y^{(n−2,2)} ⊕ M^{(n−1,1)} where Y^{(n−2,2)} ≅ S^{(n−2,2)} ≅ D^{(n−2,2)}.

(ii) If n ≡ 1 (mod p) then M^{(n−2,2)} ≅ Y^{(n−2,2)} ⊕ D^{(n−1,1)} where Y^{(n−2,2)} ≅ 1|D^{(n−2,2)}|1 ≅

1|(S^{(n−2,2)})^{∗}. (iii) If n ≡ 2 (mod p) then M^{(n−2,2)} ≅ Y^{(n−2,2)} ⊕ 1 where

Y^{(n−2,2)} ≅ D^{(n−1,1)}|D^{(n−2,2)}|D^{(n−1,1)} ≅ D^{(n−1,1)}|(S^{(n−2,2)})^{∗}.

We will only need the case p = 2 when n is odd:

Lemma 1.3. Let p = 2 and n ≥ 4 be odd.

(i) If n ≡ 1 (mod 4) then M^{(n−2,2)} ≅ Y^{(n−2,2)} ⊕ D^{(n−1,1)} where Y^{(n−2,2)} ≅ 1|D^{(n−2,2)}|1 ≅ 1|(S^{(n−2,2)})^{∗}.

(ii) If n ≡ 3 (mod 4) then M^{(n−2,2)} ≅ Y^{(n−2,2)} ⊕ M^{(n−1,1)} where Y^{(n−2,2)} ≅ D^{(n−2,2)}.

The following easy result is well known.

Lemma 1.4. Let X be a G-set, and M be the corresponding permutation F G-module. Then dim M^{G} equals the number of G-orbits on X.

Proof. It suffices to consider the case where G is transitive on Ω. Then M = (1_{H})↑^{G} where H is a point stabilizer. But dim Hom_{G}(1_{G}, (1_{H})↑^{G}) ≅ dim Hom_{H}(1_{H}, 1_{H}) = 1 by Frobenius reciprocity.

For 1 ≤ k ≤ n denote by d_{k} the number of G-orbits on the unordered k-element subsets of Ω. The following lemma characterizes d_{k} in terms of the corresponding permutation module.

Lemma 1.5. Let 1 ≤ k ≤ n and G ≤ Σ_{n} be a subgroup. Then dim(M^{(n−k,k)})^{G} = d_{k}.

Proof. Note that M_{G}^{(n−k,k)} is the permutation F G-module on the k-element subsets. Now we can use Lemma 1.4.

Lemma 1.6. If G ≤ Σ_{n} and n ≢ 0 (mod p) then dim(D^{(n−1,1)})^{G} = d_{1} − 1.

Proof. This follows from Lemmas 1.1 and 1.5.

Lemma 1.7. Let n ≥ 4, G ≤ Σ_{n} and n be odd if p = 2. Suppose d_{1} < d_{2}. Then (Y^{(n−2,2)})^{G} and ((S^{(n−2,2)})^{∗})^{G} are non-zero. Moreover if p > 2 and n ≡ 1 (mod p), or p = 2 and n ≡ 1 (mod 4), then dim(Y^{(n−2,2)})^{G} ≥ 2.

Proof. By Lemmas 1.2 and 1.3 (or [8, 17.17]), there is an exact sequence

0 → M^{(n−1,1)} → M^{(n−2,2)} → (S^{(n−2,2)})^{∗} → 0,

which implies that ((S^{(n−2,2)})^{∗})^{G} ≠ 0, by assumption and Lemma 1.5. By 1.2(i) and 1.3(ii), Y^{(n−2,2)} = (S^{(n−2,2)})^{∗}, unless p > 2, n ≡ 1, 2 (mod p) or p = 2, n ≡ 1 (mod 4). If p > 2 and n ≡ 2 (mod p) then by Lemma 1.2(iii), dim(Y^{(n−2,2)})^{G} = d_{2} − 1 ≥ d_{1} > 0. In the remaining cases, using Lemma 1.6, we get dim(Y^{(n−2,2)})^{G} = d_{2} − (d_{1} − 1) ≥ 2.

Lemma 1.8. Let G ≤ Σ_{n} be transitive and n ≢ 0 (mod p). Then G is 2-transitive if and only if dim End_{G}(D_{G}^{(n−1,1)}) = 1.

Proof. Note that G is 2-transitive on Ω if and only if G has exactly two orbits on Ω × Ω. However, the corresponding permutation module is isomorphic to M_{G}^{(n−1,1)} ⊗ M_{G}^{(n−1,1)}. By Lemma 1.4, we now have that G is 2-transitive if and only if dim(M_{G}^{(n−1,1)} ⊗ M_{G}^{(n−1,1)})^{G} = 2. By Lemmas 1.6 and 1.1, this is equivalent to dim(D^{(n−1,1)} ⊗ D^{(n−1,1)})^{G} = 1, which by selfduality of irreducible modules is equivalent to our claim.

2. 2*-homogeneous groups

The main result of this section is Theorem 2.5 which shows that the restriction of a simple F Σ_{n}-module D of dimension > 1 to a 2*-homogeneous subgroup is reducible. We will use a theorem of Kantor [12] (see also [4, 9.4B]) which describes 2*-homogeneous groups. Let Ω = 𝔽_{q} be a finite field of order q = r^{e} for a prime r. Let Aut_{𝔽_{r}}(𝔽_{q}) denote the Galois group, which is a cyclic group C_{e} of order e. Let AΣL_{1}(q) (resp. ASL_{1}(q)) be the group of all transformations of 𝔽_{q} of the form x ↦ a^{2}x^{σ} + b (resp. x ↦ a^{2}x + b) with a ∈ 𝔽_{q}^{∗}, b ∈ 𝔽_{q} and σ ∈ Aut_{𝔽_{r}}(𝔽_{q}). These are permutation groups on the elements of 𝔽_{q}.

Proposition 2.1. [12] Let G be a 2*-homogeneous group. Then, up to permutation isomorphism, ASL_{1}(q) ≤ G ≤ AΣL_{1}(q) with q ≡ 3 (mod 4), q > 3.

Let R_{n}(m) denote the class of simple F Σ_{n}-modules D such that for some p-regular partition λ = (λ_{1}, λ_{2}, …) with λ_{1} ≥ n − m, we have D ≅ D^{λ} or D ≅ D^{λ} ⊗ sgn . We need the following results from [9] and the main results in [17],[18].

Proposition 2.2. [9, Theorem 7] If n ≥ 15 and D is a simple F Σ_{n}-module, then either dim D > \frac{1}{2}(n − 1)(n − 2) or D ∈ R_{n}(2).

Proposition 2.3. [9, p.420] If D ∈ R_{n}(2) \ R_{n}(1) then dim D ≥ \frac{1}{2}n(n − 5). If D ∈ R_{n}(1) then dim D ≥ n − 2.

Proposition 2.4. [17], [18], [9, Theorem 6(i)] If n ≥ 7 and D is a simple F Σ_{n}-module, then either dim D > n − 1 or D ∈ R_{n}(1).

Proposition 2.5. Let G be a 2*-homogeneous subgroup of Σ_{n} and D be a simple F Σ_{n}-module with dim D > 1. Then the restriction D_{G} is reducible.

Proof. In view of Proposition 2.1 we may assume G = AΣL_{1}(q) < Σ_{q}, q ≡ 3 (mod 4), and q > 3. In particular, q ≥ 7 and r > 2. Let D_{G} be irreducible. Then

dim D ≤ \sqrt{|G|} = \sqrt{q(q − 1)e/2}. (1)

Moreover, e = log_{r} q < q − 5 as q ≥ 7 and r > 2. So

dim D < q\sqrt{(q − 5)/2} ≤ \frac{1}{2}q(q − 5). (2)

If q = 7 or q = 11 then e = 1 and we get a contradiction by (1) and Propositions 2.4, 2.3. Let q ≥ 15. Then from (2) and Propositions 2.2, 2.3 we conclude that D ∈ R_{n}(1), i.e. D ≅ D^{(n−1,1)} or D ≅ D^{(n−1,1)} ⊗ sgn . If D_{G} is irreducible, we have by Schur’s lemma

1 = dim End_{G}(D_{G}) = dim End_{G}(D_{G}^{(n−1,1)}).

Now, if p does not divide q then G is 2-transitive on 𝔽_{q} by Lemma 1.8, giving a contradiction. Otherwise the restriction of D to 𝔽_{q} ⊲ G is semisimple by Clifford’s theorem. But the trivial module is the only simple module over the group 𝔽_{q}, as 𝔽_{q} is a p-group. Hence 𝔽_{q} acts trivially on D. This contradicts the fact that D is faithful over Σ_{q}.

3. Main Results

Denote by σ the transposition (n−1, n) ∈ Σ_{n}. We will write Σ_{n−2,2} for the Young subgroup Σ_{(n−2,2)} ≅ Σ_{n−2} × Σ_{2}. For an F Σ_{n}-module V and a homomorphism θ ∈ End_{Σ_{n−1}}(V ↓_{Σ_{n−1}}) define a map θ̂: V → V by setting

θ̂(v) := θ(v) + σθ(σv) (v ∈ V).

Lemma 3.1. Let p > 2 and n ≥ 4. Then the map θ ↦ θ̂ is an injective map from End_{Σ_{n−1}}(V ↓_{Σ_{n−1}}) to End_{Σ_{n−2,2}}(V ↓_{Σ_{n−2,2}}).

Proof. It is routine that θ̂ ∈ End_{Σ_{n−2,2}}(V ↓_{Σ_{n−2,2}}). Assume θ̂ = 0. Then θ(σv) = −σθ(v) for all v ∈ V. But θ is also an F Σ_{n−1}-homomorphism. So

θ(v) = θ((σ(1, n − 1))^{3}v) = −(σ(1, n − 1))^{3}θ(v) = −θ(v),

whence θ(v) = 0 for any v ∈ V.

Lemma 3.2. Let p > 2, and W be a F Σ_{4}-module with ((1, 4) + (2, 3) − (1, 3) − (2, 4))W = 0. Then A_{4} acts trivially on W.

Proof. By assumption we have ((1, 3)+(2, 4))v = ((1, 4)+(2, 3))v for all v ∈ W. Conjugating by (2, 3) we get

((1, 2) + (3, 4))v = ((1, 3) + (2, 4))v = ((1, 4) + (2, 3))v. (3)

The group algebra F [Σ_{2}×Σ_{2}] is semisimple and has four irreducible modules: 1_{Σ_{2}}⊗1_{Σ_{2}}, 1_{Σ_{2}}⊗ sgn_{Σ_{2}}, sgn_{Σ_{2}} ⊗1_{Σ_{2}}, and sgn_{Σ_{2}} ⊗ sgn_{Σ_{2}}. We claim that only 1_{Σ_{2}} ⊗ 1_{Σ_{2}} and sgn_{Σ_{2}} ⊗ sgn_{Σ_{2}} may appear in the restriction W ↓_{Σ_{2}×Σ_{2}}. Indeed, assume for example that 1_{Σ_{2}}⊗sgn_{Σ_{2}} appears in this restriction. Pick a non-zero vector v in the corresponding isotypic component. Then (1, 2)v = v and (3, 4)v = −v. So (1, 2)(3, 4)v = −v and ((1, 2) + (3, 4))v = 0. By (3), we have (1, 3)v = −(2, 4)v and (1, 4)v = −(2, 3)v. Hence

(1, 3)(2, 4)v = −v and (1, 4)(2, 3)v = −v. (4)

But (1, 2)(3, 4) = (1, 3)(2, 4)(1, 4)(2, 3). So (4) implies (1, 2)(3, 4)v = v, and we have arrived to a contradiction. The case of sgn_{Σ_{2}} ⊗1_{Σ_{2}} is considered similarly.

As the element (1, 2)(3, 4) acts trvially on both 1_{Σ_{2}} ⊗ 1_{Σ_{2}} and sgn_{Σ_{2}} ⊗ sgn_{Σ_{2}}, it must act trivially on W. By conjugation, we conclude that the whole Klein group

H := {1, (1, 2)(3, 4), (1, 3)(2, 4), (1, 4)(2, 3)}

acts trivially on W. As (1, 2) ≡ (3, 4) (mod H) and (2, 3) ≡ (1, 4) (mod H), the equation (3) implies (1, 2)v = (2, 3)v. Hence (1, 2, 3)v = v, and so A_{4} acts trivially on W.

Theorem 3.3. Let p > 2 and n ≥ 4. Assume that V is an F Σ_{n}-module such that the alternating group A_{n} < Σ_{n} does not act trivially on V. Then

dim End_{Σ_{n−1}}(V ↓_{Σ_{n−1}}) < dim End_{Σ_{n−2,2}}(V ↓_{Σ_{n−2,2}}).

Proof. By Lemma 3.1 it is enough to demonstrate an endomorphism ψ ∈ End_{Σ_{n−2,2}}(V ↓_{Σ_{n−2,2}}) which is not in the image of θ ↦ θ̂. Set ψ(v) := σv, v ∈ V. Assume that ψ = θ̂ for some θ ∈ End_{Σ_{n−1}}(V ↓_{Σ_{n−1}}). Then

θ(σv) = v − σθ(v) (v ∈ V). (5)

For i = 1, 2, …, n − 2 denote π_{i} := (i, n − 1) and τ_{i} := (i, n). Then, using (5), we get

θ(π_{i}σπ_{i}v) = π_{i}θ(σπ_{i}v) = π_{i}(π_{i}v − σθ(π_{i}v)) = v − π_{i}σπ_{i}θ(v).

Note that π_{i}σπ_{i} = τ_{i} so we have proved that

θ(τ_{i}v) = v − τ_{i}θ(v), i = 1, 2, …, n − 2 (v ∈ V). (6)

Using (5), (6) and the equality π_{i} = τ_{i}στ_{i}, we get for any v ∈ V and 1 ≤ i ≤ n − 2

π_{i}θ(v) = θ(π_{i}v) = θ(τ_{i}στ_{i}v) = στ_{i}v − τ_{i}θ(στ_{i}v)
= στ_{i}v − τ_{i}(τ_{i}v − σθ(τ_{i}v)) = στ_{i}v − v + τ_{i}σ(v − τ_{i}θ(v))
= στ_{i}v − v + τ_{i}σv − τ_{i}στ_{i}θ(v) = στ_{i}v − v + τ_{i}σv − π_{i}θ(v).

Solving for π_{i}θ(v) and multiplying by π_{i}, we find

θ(v) = (1/2)(σv − τ_{i}στ_{i}v + τ_{i}v), i = 1, 2, …, n − 2 (v ∈ V). (7)

Taking i = 1 and i = 2, we have

(1/2)(σv − τ_{1}στ_{1}v + τ_{1}v) = (1/2)(σv − τ_{2}στ_{2}v + τ_{2}v) (v ∈ V).

Conjugating by (3, n − 1)(4, n), we get

((1, 4) + (2, 3) − (1, 3) − (2, 4))V = 0.

By Lemma 3.2, the natural subgroup A_{4} < Σ_{n} acts trivially on V, which implies by conjugation that A_{n} acts trivially on V, giving a contradiction.

Now we study the case p = 2. If n = 2l is even we write S for the irreducible module D^{(l+1,l−1)} and if n = 2l +1 is odd we write S for D^{(l+1,l)}. We call S the spinor representation of Σ_{n}. The following result is proved in [15], see also [5] for a generalization.

Lemma 3.4. Let p = 2, n ≥ 3, and D be an irreducible F Σ_{n}-module different from the spinor representation S. Then for any 3-cycle γ ∈ Σ_{n} there exists a non-zero vector v ∈ D such that γv = v.

Proof. It is proved in [15] that the only irreducible representations of the alternating group A_{n} for which 3 cycles act fixed-point-freely come from the restriction of the spinor module to A_{n}. This implies the result as these irreducible A_{n}-modules do not appear in the restriction of D^{λ} to A_{n}, unless D^{λ} ≅ S.

Lemma 3.5. Let p = 2, n ≥ 3, b, c ∈ F, and D be an irreducible F Σ_{n}-module. If b ≠ 0 and

(b(1, 2) + c(1, 2, 3) + c(1, 3, 2) + c)v = 0

for all v ∈ D then D is the trivial module 1_{Σ_{n}}.

Proof. By conjugating with (2, 3) we also get

(b(1, 3) + c(1, 2, 3) + c(1, 3, 2) + c)D = 0,

so (1, 2)v = (1, 3)v, hence (1, 2, 3)v = v for all v ∈ D. By conjugating we also get (1, 3, 2)v = v for any v. Now it follows from the assumption that (1, 2) acts trivially on D, whence any transposition acts trivially.

Theorem 3.6. Let p = 2, n ≥ 4, and D be a non-trivial irreducible F Σ_{n}-module. Then

dim End_{Σ_{n−1}}(D↓_{Σ_{n−1}}) < dim End_{Σ_{n−2,2}}(D↓_{Σ_{n−2,2}}),

unless n is odd and D is the spinor module S.

Proof. We first note that the kernel of the linear map θ ↦ θ̂ defined in the beginning of this section is 1-dimensional and is spanned by the identity map id_{D}. Indeed, θ̂(v) = 0 is equivalent to θ(σv) = σθ(v). But θ is also an F Σ_{n−1}-homomorphism. Hence θ is an F Σ_{n}-endomorphism of D so it must be proportional to id_{D}, by Schur’s lemma.

Let id_{D}, θ_{1}, …, θ_{k} be a basis of the vector space End_{Σ_{n−1}}(D↓_{Σ_{n−1}}). Define a map ψ : D → D by setting ψ(v) = σv, v ∈ D. Then ψ is an element of End_{Σ_{n−2,2}}(D↓_{Σ_{n−2,2}}). We claim that

θ̂_{1}, θ̂_{2}, …, θ̂_{k}, id_{D}, ψ

are linearly independent elements of End_{Σ_{n−2,2}}(D↓_{Σ_{n−2,2}}). This of course implies the theorem. A linear dependence between our endomorphisms looks like

a_{1}θ̂_{1} + a_{2}θ̂_{2} + ⋯ + a_{k}θ̂_{k} + bid_{D} + cψ ≡ 0 (8)

with a_{i}, b, c ∈ F. We may assume that a_{i} ≠ 0 for some i, since otherwise we would get a linear dependence between id_{D} and ψ, which is only possible if D is trivial. Let θ := a_{1}θ_{1}+⋯+a_{k}θ_{k}. Then a_{1}θ̂_{1} + ⋯ + a_{k}θ̂_{k} = θ̂, and we may rewrite (8) as

θ̂ + b id_{D} + cψ ≡ 0.

By the first paragraph of the proof, we have (b, c) ≠ (0, 0). The last equality is equivalent to

θ(σv) = σθ(v) + bσv + cv. (v ∈ D)

Let π_{i} = (i, n − 1), τ_{i} = (i, n) ∈ Σ_{n}. Then τ_{i} = π_{i}σπ_{i}. So for any v ∈ D we have

θ(τ_{i}v) = θ(π_{i}σπ_{i}v) = π_{i}θ(σπ_{i}v) = π_{i}(σθ(π_{i}v) + bσπ_{i}v + cπ_{i}v)
= π_{i}σθ(π_{i}v) + bπ_{i}σπ_{i}v + cv = τ_{i}θ(v) + bτ_{i}v + cv.  (v ∈ D)

As π_{i} = τ_{i}στ_{i} we also have for any v ∈ D

π_{i}θ(v) = θ(π_{i}v) = θ(τ_{i}στ_{i}v) = τ_{i}θ(στ_{i}v) + bτ_{i}στ_{i}v + cστ_{i}v
= τ_{i}(σθ(τ_{i}v) + bστ_{i}v + cτ_{i}v) + bτ_{i}στ_{i}v + cστ_{i}v
= τ_{i}σθ(τ_{i}v) + bτ_{i}στ_{i}v + cv + bτ_{i}στ_{i}v + cστ_{i}v
= τ_{i}σ(τ_{i}θ(v) + bτ_{i}v + cv) + cv + cστ_{i}v
= π_{i}θ(v) + bπ_{i}v + cτ_{i}σv + cv + cστ_{i}v,

whence bπ_{i}v + c(τ_{i}σ + στ_{i} + 1)v = 0. In particular, c ≠ 0. If b ≠ 0 we have

(bπ_{i} + cτ_{i}σ + cστ_{i} + c)v = 0. (v ∈ D)

Conjugating we get

(b(1, 2) + c(1, 2, 3) + c(1, 3, 2) + c)v = 0. (v ∈ D)

Lemma 3.5 now leads to a contradiction as D ≅ 1̸_{Σ_{n}}. So we may assume that b = 0. Then we get the identity

(τ_{i}σ + στ_{i} + 1)v = 0. (v ∈ D)

But τ_{i}σ =: γ is a 3-cycle and στ_{i} = γ^{2}. So it suffices to show that there is a non-zero v ∈ D such that γv = v. But this follows from Lemma 3.4, provided D ≅ S̸. Finally if D ≅ S and n = 2l is even, then S = D^{(l+1,l−1)}. By [14], D^{(l+1,l−1)}↓_{Σ_{n−1}} ≅ D^{(l,l−1)} and D^{(l,l−1)}↓_{Σ_{n−2,2}} is reducible (and self-dual). So the Hom-spaces under consideration have dimensions 1 and 2, and the theorem is true.

Corollary 3.7. Let n ≥ 4 and D be an irreducible F Σ_{n}-module with dim D > 1. If p = 2 and n is odd, assume that D ≅ S̸. Then

dim Hom_{Σ_{n}}(M^{(n−2,2)}, End(D)) > dim Hom_{Σ_{n}}(M^{(n−1,1)}, End(D)).

Proof. Follows from Theorems 3.3, 3.6 and the isomorphisms

Hom_{Σ_{n}}(M^{ν}, End(D)) ≅ Hom_{Σ_{ν}}(1_{Σ_{λ}}, End(D)) ≅ End_{Σ_{λ}}(D↓_{Σ_{λ}}).

Lemma 3.8. Let n ≥ 4 and D be an irreducible F Σ_{n}-module with dim D > 1. If p = 2, assume additionally that n is odd and D ≅ S̸. Then End(D) contains either Y^{(n−2,2)}, (S^{(n−2,2)})^{∗} or both as submodules.

Proof. Assume first that p > 2, n ≢ 1, 2 (mod p) or p = 2, n ≡ 3 (mod 4). Then the result follows from Lemmas 1.2(i), 1.3(ii) and Corollary 3.7. Now let p > 2 and n ≡ 1 (mod p). Then M^{(n−1,1)} splits as 1 ⊕ D^{(n−1,1)}. By Lemma 1.2(ii), there is a surjection M^{(n−2,2)} → M^{(n−1,1)}, and by Corollary 3.7, there must exist a homomorphism θ : M^{(n−2,2)} → End(D) which does not factor through this surjection. If the restriction θ|Y^{(n−2,2)} is injective then Y^{(n−2,2)} is a submodule of End(D). Otherwise, in view of Lemma 1.2(ii), the kernel of θ|Y^{(n−2,2)} is 1. But Y^{(n−2,2)}/1 ≅ (S^{(n−2,2)})^{∗}, so (S^{(n−2,2)})^{∗} embeds into End(D). The cases p > 2, n ≡ 2 (mod p) and p = 2, n ≡ 1 (mod 4) are considered similarly to the case n ≡ 1 (mod p) using 1.2(iii), 1.3(i) and 3.7.

Recall that d_{1} and d_{2} denote the number of G-orbits on Ω and the 2-subsets of Ω, respectively.

Theorem 3.9. Let n ≥ 4, G ≤ Σ_{n} be a subgroup, and D be a simple F Σ_{n}-module with dim D > 1. If p = 2, assume additionally that n is odd and D ≅ S̸. Then the restriction D_{G} is reducible whenever d_{1} < d_{2}.

Proof. Suppose D_{G} is irreducible and d_{1} < d_{2}. Then End_{G}(D_{G}) ≅ End(D)^{G} is 1-dimensional by Schur’s lemma. We consider the following two cases.

Case 1: p > 2, n ≢ 1 (mod p) or p = 2, n ≡ 3 (mod 4). Then by Lemmas 1.2(i),(iii), 1.3(ii) and 3.8, either 1_{Σ_{n}} ⊕ Y^{(n−2,2)} ⊆ End(D) or 1_{Σ_{n}} ⊕ (S^{(n−2,2)})^{∗} ⊆ End(D) (or both). By Lemma 1.7, End(D)^{G} is at least 2-dimensional, giving a contradiction.

Case 2: p > 2, n ≡ 1 (mod p) or p = 2, n ≡ 1 (mod 4). By Lemma 3.8 again, either Y^{(n−2,2)} ⊆ End(D) or (S^{(n−2,2)})^{∗} ⊆ End(D). In the first case Lemma 1.7 implies that End(D)^{G} is at least 2-dimensional. In the second case (S^{(n−2,2)})^{∗} = D^{(n−2,2)}|1 implies that (S^{(n−2,2)})^{∗} ⊕ 1_{Σ_{n}} ⊆ End(D). Now apply Lemma 1.7 as in Case 1.

Theorem 3.10. Let n ≥ 4, G ≤ Σ_{n} be a subgroup, and D be a simple F Σ_{n}-module with dim D > 1. If p = 2, assume additionally that n is odd and D ≅ S̸. If the restriction D_{G} is irreducible then either G ≤ Σ_{n−1} or G is 2-transitive.

Proof. Let D_{G} be irreducible. Assume first that G is intransitive. Then, up to a conjugation, G is contained in a standard Young subgroup Σ_{n−k} ×Σ_{k} for some 1 ≤ k ≤ n/2. If k > 1, then d_{1} = 2 < d_{2} = 3 with respect to the action of Σ_{n−k} × Σ_{k}. Hence D↓_{Σ_{n−k}×Σ_{k}} is reducible by Theorem 3.9 applied to the subgroup Σ_{n−k} ×Σ_{k}. This gives a contradiction. Therefore k = 1 and so G ≤ Σ_{n−1}. We may now assume that G is transitive. If G is not 2-homogeneous, then d_{1} = 1 < d_{2} and we get a contradiction by Theorem 3.9. Thus, G is 2-homogeneous. Now, if G is not 2-transitive apply Theorem 2.5.

Finally, observe that the Main Theorem is a special case of Theorem 3.10.

References

[1] M. Aschbacher, On the maximal subgroups of the finite classical groups, Invent. Math. 76 (1984), 469–514.

[2] C. Bessenrodt and J.B. Olsson, On residue symbols and the Mullineux conjecture, J. Algebraic Combin. 7 (1998), 227–251.

[3] J. Brundan and A.S. Kleshchev, Representations of the symmetric group which are irreducible over subgroups, preprint, University of Oregon, 1999.

[4] J. D. Dixon and B. Mortimer, Permutation groups. Graduate Texts in Mathematics, 163, Springer-Verlag, New York, 1996.

[5] P. Fleischmann, W. Lempken, A.E. Zalesskii, Linear groups over GF (2^{k}) generated by a conjugacy class of a fixed point free element of order 3, in preparation.

[6] B. Ford, Irreducible restrictions of representations of the symmetric groups, Bull. London Math. Soc. 27 (1995), 453–459.

[7] B. Ford and A. Kleshchev, A proof of the Mullineux Conjecture, Math. Z. 226 (1997), 267–308.

[8] G. D. James, The Representation Theory of the Symmetric Groups, volume 682 of Lecture Notes in Mathematics, Springer, New York/Heidelberg/Berlin, 1978.

[9] G. D. James, On the minimal dimensions of irreducible representations of symmetric groups, Math. Proc. Camb. Phil. Soc. 94 (1983), 417–424.

[10] G. James and A. Kerber, The representation theory of the symmetric group, Addison-Wesley, London, 1981.

[11] J. C. Jantzen and G. M. Seitz, On the representation theory of the symmetric groups, Proc. London Math. Soc. (3) 65 (1992), 475–504.

[12] W. M. Kantor, k-homogeneous groups, Math. Z. 124 (1972), 261–265.

[13] P. Kleidman and M. Liebeck, The subgroup structure of the finite classical groups, London Mathematical Society Lecture Note Series, 129, Cambridge, Cambridge University Press.

[14] A. S. Kleshchev, On restrictions of irreducible modular representations of semisimple algebraic groups and symmetric groups to some natural subgroups, I Proc. London Math. Soc. (3) 69 (1994), 515–540.

[15] G. Mullineux, Representations of A_{n} in characteristic 2 with the 3-cycles fixed-point-free, Quart. J. Math. Oxford (2) 29(1978), 199–212.

[16] J. Saxl. The complex characters of the symmetric groups that remain irreducible in subgroups, J. Algebra 111 (1987), 210–219.

[17] A. Wagner, On the faithful linear representations of least degree of S_{n} and A_{n} over a field of characteristic 2, Math. Z. 151 (1976), 127–137.

[18] A. Wagner, The faithful linear representations of least degree of S_{n} and A_{n} over a field of odd characteristic, Math. Z. 154 (1977), 103–114.

Department of Mathematics, University of Oregon, Eugene OR 97403, USA

E-mail address: klesh@math.uoregon.edu,  sheth@noether.uoregon.edu