My opinions only, but I think it can be misleading to focus on the interactions between the substance (analyte) and the mobile phase. Misleading in the sense that it is possible to make wrong predictions about how Rf changes with changes in polarities.
It is better to focus on the interactions between the substance and the stationary phase versus the solvent and the stationary phase when predicting how Rf values change with changes in the polarity of the solvent or the polarity of the analyte. In other words, I like to think of the situation as being a competition between solvent vs. analyte for binding to the stationary phase. When the analyte wins, it moves slowly up the TLC plate.