Bond energy is a function of (1) orbital overlap, (2) Coulombic forces - attraction and repulsion, and (3) electron exchange, which is a purely quantum mechanical effect. Fluorine has a lot more electron density surrounding it than carbon, and therefore a lot more charge repulsion. For the same reason, O-O and N-N single bonds are pretty unstable.