Cl has three lone pairs around it in addition to the bond.
In order to get a three-dimensional orbital structure with just s and p orbitals, you must have all three sets of p orbitals hybridized to form an sp3 set.
In other words, if you proposed a bonding structure where chlorine had a naked p orbital, we'll call it pz bonding with an sp3 orbital from carbon, that would limit the other orbitals around chlorine to being no more than sp2 hybridized. In other words, all the other electrons would be stuck in the x-y plane, so you'd have three lone pairs all crammed into the same equatorial plane around chlorine and an empty space up top.
This is obviously not an ideal arrangement thanks to repulsion between electron clouds, so instead the pz orbital mixes with the others and they all spread out to nearly-equidistant spacing around the chlorine. That's where the well-known approximately 109.5° bond angle comes from.