I agree with you, that argument has always bugged me too because you would think that an EWG next to the halide would make an SN2 transition state less favorable.
The way that I have rationalized this observation to myself has been to assume that the C bearing the Br has a partial positive charge and the C of the C=O also has a positive charge. Since these two charges conflict, the result is that you have a higher energy starting material which is therefore more reactive. So, the assumption would be that the transition state for the two variants (regular alkyl-Br and alpha-bromo carbonyl) would be comparable in energy, but in the latter case you start out at a higher E and therefore the activation energy is lowered.
You could also make a slightly hand-wavy argument about the orbitals: the pi* and the C-Br sigma* could be aligned and have the right symmetry for some overlap of these two vacant orbitals, thereby lowering the energy of anti-bonding MO that gets attacked. I kinda like that explanation, but it seems a little convoluted.