What's a hydrogen bond? - It is just a electrostatic force between hetero-atom and hydrogen. It is just a kind of vanderwaal's force,. A dipole-dipole interaction, but what puts it apart is it's high bond energy compared to other intermolecular forces.
Both N-H bond and Cl-H bond are almost equally polar because the electro-negativity difference is almost the same, So it wont be a factor in explaining the weak intermolecular force.
How size affects.. Due to polar nature of the bond, partial charges δ+ on H and δ- on Cl/N develop. As Cl size is more, the charge density is less, the electrostatic force depends on charge density and not exclusively on charge. So that's why even though the two bonds are equally polar, the intermolecular forces are high in case of N-H than Cl-H.