Solubility is complicated, and as soon as you start to generalize, you'll almost just as soon run into exceptions that defy your explanations.
That said, if I was forced to try to explain it quickly and simply I'd point to entropy. When an ion dissolves in water, water molecules will tend to orient themselves around the ion. This is entropically unfavorable. An ion that is more highly charged, or has more concentrated charge, will exacerbate this effect. Thus we might predict that ions that have smaller, more diffuse charge density will also tend to be more soluble. Nitrates have a sweet spot for solubility because they are both low-charge (only -1) and that charge is delocalized over a large volume (4 atomic centers). This will minimize the unfavorable entropy change due to water molecule reorientation during dissolution of the anion.
We can turn to some standard entropies to see if this bears out. If the Gibbs energy for dissolution of a salt into ions is given by ΔH - TΔS, then a larger (more positive) ΔS will give rise to more driving force for solution. Since the ions are the products, a larger standard entropy will result in a lower ΔG.
Standard entropies of nitrate, chloride, bromide, hydroxide, and sulfate offer some instructive examples. The values for these anions are (in J/K mol) 116.5, 56.5, 82.4, and -10.3, 20.1. (I found these values through a Google search, then neglected to copy the link. Sorry!).
You can see that of these ions, nitrate has by far the highest standard entropy, in agreement with my argument above. Chloride, the salts of which are decently soluble, has a more modest value. You might say that sulfate is also a large polyatomic ion - why isn't it quite so soluble as nitrate, usually? Well, sulfate has a formal charge of -2. And this is reflected in its far lower standard entropy value. Hydroxide also offers a nice comparison. It, too, has a formal charge of -1, but it is quite small compared to nitrate, and actually has a negative standard entropy.
So, my guess is that the very large positive standard entropy of nitrate has quite a bit to do with the fact that nitrate salts are always more soluble than similar salts featuring the same cations but different anions. Even silver nitrate is fairly soluble! (For the same reason, ammonium salts tend to be highly soluble: large, diffuse cation.) Of course, solubility depends on more than just the thermodynamics of the anion - there is the cation and the starting salt to worry about, so as I said, it is hard, and probably unwise, to generalize too much.
(The fact that nitrate is both lowly charged and large/diffuse not only plays a role in favorable water orientation/entropy; it also means that the bonds in the solid salt that hold the cations and anions together are fairly weak, which favors thermodynamics of dissolution from this angle as well. I.e., this is a lattice energy concept - salts with weaker inter-ion bonds will tend to dissolve more easily.)
Anyway, that's my 5-minute hand-waving argument.