I think the usual argument goes something like this: F is so electronegative that we'd expect HF to be a very strong acid, since HF should be really polarized. Yet HF is weak compared to HCl, HBr, and HI. The Cl-, Br-, and I- are simply larger anions and thus the charge density is decreased. Therefore the conjugate base of HCl, HBr, and HI are much more stable than the conjugate base of HF.
Other reasons I've read as to why acidity increases going down the halogen column is that the bond strength of HX gets weaker as we move down the column due to the large differences in atomic size between H and X. For example, I is much much larger than F, and the resultant orbital overlap between I and H is not nearly as great as the orbital overlap in F and H.
I think orgopete may have some interesting comments on this...