There is an equilibrium between oxygen being produced by plants and being used for CO2 production. Kill all plants, wait some time (no idea how long, but oxygen cycle is probably known well enough), problem of abundant oxygen dissappears.
The other answer to your question (highly related) is that while oxygen is highly reactive kinetics of the oxidation in low temperatures (0-20 deg C) is low enough to allow oxygen level build-up in the atmosfere.