Ok, here's what I got for this.
You have 9.6g of O reacting, so you can take
9.6g/ (16 g/mol) = 0.6mol O
The reaction is:
M + 2O <--> MO2
So there's 2 moles of O per mole of product, so you have
0.6/2 = 0.3 mol product. There's a 1:1 ratio between product and M, so there's 0.3mol M
The mass of M is 16.5g, and you can find the molar mass by
16.5g/0.3 mol = 55g/mol, which is Mn.
you were on the right track, but you made the mistake of using the molar mass of O2 being 16, that's the molar mass of O.