Consider the changes in polarisation of charge in the starting materials vs the transition state.
If you consider the transition state for e.g. displacement of iodide from methyl iodide with cyanide ion:
Starting materials (SM):
-CN + CH3-I
Transition state (TS):
[NCd-...CH3...d-]-
While both SM an TS have an overall charge of -1, that charge is more dispersed in the TS where it is shared between the CN and I ligands. In the SM the charge is localised on cyanide.
One can argue that: Since stabilisation of the TS relative to the SM will result in a lower activation energy, a solvent that stabilises the TS more effectively than the SM will result in a higher reaction rate. As the charge is more dispersed in the TS than the SM, we can say the TS is less polar than the SM, and that a solvent of lower polarity will result in an enhanced reaction rate.
Note: As has been mentioned, in a practical setting the solvent must still be polar enough to dissolve the starting materials.
If you follow the same steps for the reaction:
Ph3P + CH3I --> Ph3P+CH3 I-
You should be able to argue that a more polar solvent will give higher reaction rates. Give it a try.