a simplified explanation:
to force basic orbitals into hybrid orbitals requires energy
this energy needs to be compensated for by better bonds , else it won't happen
the "sphere" of the central atom becoming larger, repulsive effects of substituents (located on the surface of named sphere) will become smaller. this is one important factor if we'd judge the quality of bonds.
with less repulsive effects amongst the substituents, the need for extra favourable geometry becomes less important, and even (from a geometrical point of view) small-angle geometries (which, due to no hybridization required, are better with respect to energy) become competitive (or: winner)
as a result, the higher hydrides of group V and VI tend to be little hybridized, and will show bond angles near to 90° , i.e. pure p-s-sigma bonds
regards
Ingo