It's best understood if you think of what the benzene ring looks like, essentially a donut of pi electrons above and below the sigma network.
Nucleophiles cannot approach this plane, unless of course, there is a group somehwere on the ring that can actively withdraw those pi electrons away from the site where you want your nucleophile to create a new bond.
It turns out the deactivating group must be in the o or p positions. At the m position, pi electrons cannot "move" onto your deactivating group.
Vinyl halides follow the same unreactivity in substitution reactions.