I'm trying to understand the link between MO theory and group theory, particularly why MOs must span a particular irreducible representation of the point group to which the single electron Hamiltonian belongs.
I have two approaches. If a molecular orbital is a linear combination of symmetry adapted linear combinations of atomic orbitals (SALCs), then the energy can be evaluated as follows.
[itex]\phi_i = \sum_j \chi_j c_{ji}, E=\sum_{jj'}c_{ji}^*c_{j'i}\langle \chi_j | H | \chi_{j'}\rangle [/itex]
If the matrix elements [itex]\langle \chi_j | H | \chi_{j'} \rangle [/itex] are non-zero if and only if the two symmetry adapted orbitals span the same irrep of the point group, which implies the MO spans the same irrep.
Alternately because the point group operations and the hamiltonian commute [itex] [H,R]=0 [/itex] then [itex] HR\phi = RH\phi = E R\phi [/itex] which implies the transformed state is a linear combination of states of the same energy, and to preserve the inner product the action of [itex]R[/itex] must be unitary. I don't think this is enough to show it spans an irrep though.