Because size is not the only thing that matters. When it comes to F and Cl, although F is more electronegative, the fact that F is very small means more interelectronic repulsion when you try to put another electron in, as compared to the more roomy Cl. But the same comparison based on just size, cannot be made for the other halogens as well. It is both electronegativity and size that go hand in hand. Cl is able to handle the excess electron because the repulsion is lesser as in the case of F. F is too small, that the energy due to repulsion is greater than for Cl. It's just an observed fact, and we try to make sense of it.
*Astatine, btw