oxygen is more electronegative than nitrogen. On the pauling scale, nitrogen is measured as 3.04 whereas oxygen is measured as 3.44. The difference of 0.4 represents about 20% of that oxygen. I would not consider that as negligible difference.
The only reason why there is no dative bond between N and O is that the octet rule does not allow it to happen.