Higher boiling points mean MORE intermolecular forces, because the molecules are "holding on to each other" more, making it harder for them to enter the gas phase.
In this case, if you say o-vanilin boils at a lower temperature, then it is less polar than vanilin.
TLC plates don't directly translate into boiling point info. Was it a silica gel or alum plate? I'm guessing silca gel.
Your solvent was hexane and ethyl acetate? In that case, the solvent was more non-polar than it was polar. Smearing can be due to impurities, too high of a concentration in your beginning dots, or a stronger affinity to the solid face (silica gel) than to the solvent. People usually mess with concentrations of the solvent a lot before they find one that works well, so remember that on the TLC Plate you dont only have intermolecular forces at play, you have affinity to solid phase, liquid phase, and affinity to itself--so a lot more going on
How many times did you run the plate? Should always run at least 2 or 3 plates to rule out abnormalities.