Degenerate orbitals means that the orbitals all appear at the same energy level. For example, in the Hydrogen atom the three 2p orbitals are degenerate because they all appear at the same energy level. Hence an electron can go into any of the three orbitals it chooses given that certain quantity of energy. However, for polyelectron atoms differences in energy levels can appear so that the 2p-orbitals are not all of the same energy (this is due to the magnetism of individual electrons). Degenerate does not mean to fill the electron in one at a time (that is called Hund's rule). I agree that the name sucks (whenever I hear "degenerate orbitals" I think of rude teenagers for some reason...).
"Penetrate the core" is a way of describing how close the electrons in a given orbital can get towards the positively charged nucleus. Orbitals that are highly penetrating allow the electron to get close to the nucleus while those with low penetration have electrons present a good distance away from the nucleus.
Atomic orbitals are the collection of all the orbitals on an atom. The valence orbitals are only those orbitals associated with the outermost electron shell that participate in chemical reactions. With sodium for instance, the atomic orbitals with electrons in them in the ground state are 1s,2s,2p,3s whereas the valence orbital is only the 3s. This is because sodium only uses its lone 3s electron to participate in chemical reactions.