Well, I don't think there is a strict rule for defining the reactivity series but one way it can be done is by comparing standard reduction potentials (how easy an element is reduced compared to a reference element). Standard reduction potentials are measured under standard conditions and under these conditions, magnesium will not be able to displace sodium from a solution of sodium hydroxide. However, when you have non-standard conditions, the electrode potentials change and the reaction, as you have shown, can occur.