We’re all familiar with Cayley’s theorem: every group $G$ is isomorphic to a subgroup of $\text{Sym}(G).$ In the finite case, every finite group of order $n$ is isomorphic to a subgroup of $S_n$. Now let’s see some applications of Cayley’s idea.

Problem 1. Let $G$ be a group and let $H$ be a subgroup of $G$ with $[G:H]=n < \infty.$ Prove that there exists a normal subgroup $N$ of $G$ such that $N \subseteq H$ and $[G:N] \mid n!.$

Solution. Let $X = \{x_1H, \cdots , x_n H \}$ be the set of all left cosets of $H$ in $G$. Define

$\varphi: G \longrightarrow \text{Sym}(X) \simeq S_n$

by $\varphi(g)(x_jH)=gx_jH, \ \forall g \in G, \forall \ 1 \leq j \leq n.$ Then see that $\varphi(g) \in \text{Sym}(X).$ Thus $\varphi$ is well-defined. It is easy to show that $\varphi$ is a group homomorphism and $N=\ker \varphi \subseteq H.$ So $G/N$ is isomorphic to a subgroup of $\text{Sym}(X).$ Thus $[G:N] \mid |\text{Sym}(X)|=n!. \ \Box$

Problem 2. Let $G$ be a finite group and let $p$ be the smallest prime divisor of $|G|.$ Let $H$ be a subgroup of $G$ with $[G:H]=p.$ Prove that $H$ is a normal subgroup of $G.$

Solution. By Problem 1, there exists some normal subgroup $N \subseteq H$ such that $[G:N] \mid p!.$ Therefore $p!|N|=kp|H|,$ for some integer $k$. Thus $(p-1)!|N|=k|H|.$ But $\gcd(|H|,(p-1)!)=1,$ because no prime divisor of $|H|$ is smaller than $p.$ So $|H| \mid |N|$ and hence $N=H$ because $N \subseteq H. \ \Box$

A trivial result of Problem 2 is that if $H$ is a subgroup of a group $G$ and $[G:H]=2,$ then $H$ is normal in $G.$ Also if $p$ is a prime number and $G$ is a group of order $p^n,$ then every subgroup of $G$ of order $p^{n-1}$ is normal.

Problem 3. Prove that if $G$ is an infinite simple group and $H$ is a proper subgroup of $G,$ then $[G:H]=\infty.$

Solution. Suppose to the contrary that $[G:H]=n < \infty$. Then, by Problem 1, $H$  contains some normal subgroup $N$ with $[G:N] \mid n!.$ In particular $[G:N] < \infty.$ But $G$ is simple and $N \neq G$, because $N \subseteq H.$ Hence $N = \{1\}$ and so $[G:N]=|G|=\infty,$ which is a contradiction! $\Box$

Problem 4. Let $n \geq 5$ and let $H$ be a proper subgroup of $A_n.$ Prove that $[A_n:H] \geq n.$

Solution. Let $[A_n:H]=m.$ By Problem 1, $A_n$ has a normal subgroup $N$ which is contained in $H.$ But $A_n$ is simple because $n \geq 5.$ Thus $N = \{1\}$ and hence, by Problem 1, $[A_n:N]=|A_n|=\frac{n!}{2} \mid m!,$ which is not possible unless $m \geq n. \ \Box$

Example. By Problem 4, $A_6$ has no subgroup of orders $72, \ 90, \ 120, \ 180.$