just memorise the exceptions. there are very few exceptions to remember.
an oxygen atom cant withdraw electron density from another oxygen atom because both atoms exhibit equal electronegativity. so when u have H-O-O-H, effectively, each oxygen atom is withdrawing electron density from each hydrogen atom. Hence, the oxidation number of O is -1, while that of H is +1.