This is an interesting question, and not one that I have a great answer for.
There are some comments I do want to add as far as the kinetics go. The RDS is almost certainly the attack of the aromatic system onto the electrophile. However, this step is bimolecular so it will depend both on the concentration of the aromatic piece (usually very high concentration, sometimes even the solvent) and the concentration of the electrophile. Remember the concentration of the electrophile is the active carbocation or acylium ion, not the alkyl or acyl halide itself, so the cation concentration is likely very low, although that concentration will depend on the equilibrium between the cation and the alkyl/acyl halide, which will not be the same for each of these electrophiles.
The way I see it, there are two possibilities here: first, the equilibrium between alkyl/acyl halide and free cation is significantly different in these cases and the difference is enough to bear out in the observed rate despite the very large concentration of aromatic nucleophile; second, the rate at which the nucleophile adds to the electrophile may not be the same in each case. The latter could be due to a close proximity of the counterion to the cationic intermediate.
It is probably a lot more complicated than that. I haven't come across any extensive kinetic studies of the F–C reaction, but I haven't really looked either!