This is one of those really good questions and I don't know that I can give a really good answer. When you compare the acidity of the haloacids, you are comparing a thermodynamic property. HI is a stronger acid, and iodide is a weaker base. That suggests its electrons are not available for reaction.
The formation of a C-I bond is essentially irreversible. That it, its formation is a kinetic process. In protic solvents, iodide reacts faster. A common explanation is hydrogen bonding to the other possible halide anions slows their reaction. That would be consistent with the acidity data as well. However, iodide is also relatively immune to a solvent effect. In an aprotic solvent, fluoride must just over iodide in rates. While iodide does slow somewhat, it is still relatively reactive in aprotic solvents.
On that account, I have come to think that iodide, with its 54 valence electrons, has two properties. One is that the electrons are relatively unavailable, thermodynamically. The effect of the protons pulling the electron toward it. The other is that this shell of electrons may also move or jump out to other positive centers, a kinetic process. This would be the so-called polarizability feature. I think of this as a feature that all heavier atoms have. As more electrons are added to the shell, the easier it is for them to react kinetically. That is consistent with sulfur, phosphorous, etc.