This is just a guess, but I'd say good news and bad news. I doubt the ester is going to react. Ketones are simply more reactive. The good news is that you want to remove the t-butyl ester, good. Treat it with trifluoroacetic acid and you will have the acid. Convert it to an acid chloride and react that with a suitable reagent to introduce your nucleophile. Since the acid chloride should be more reactive, you should be able to find a solution.