Depends on the atom. For O, say, the 3p orbitals would be very large and diffuse, while for S, the 2p orbitals would be much smaller and in the core of the atom. So in P-O you would get 3p-2p interactions, whereas in P-S you would get 3p-3p interactions. Broadly speaking, valence orbitals interact with valence orbitals (surprise surprise).