A pKa issue? Presumably you're relying on formation of HBr as a by-product, but this is a much stronger acid than the AcOH you are trying to exchange. Sulfuric is quite a bit more acidic than AcOH (-3 vs 5), but still quite a way off the pKa of HBr (-8ish) though, so not sure this is a very good explanation. I know virtually nothing about this kind of ion exchange stuff though, so perfectly willing to be shouted down on this one!
I think a method people use for these types of exchange is to make the silver salt of the acetic acid. Then after salt metathesis the AgBr precipitates out, and you don't have to rely on thermodynamics/pkas any more.