Certainly a chemical formula should contain integers only. I believe you solved the problem correctly: you simply forgot that, in this case, you always have to approximate to the closest integer (but perhaps it's not your fault: no one has ever told you it).
Hence, I bet the answer is just FeO.