This is an equilibrium process - in low pH solubility of calcium sulfate goes up, but we are still talking about salt that is mostly insoluble (or - at best - weakly soluble).
Calcium carbonate - when treated with any acid strong enough - will react, loosing carbon dioxide and leaving well soluble salt. Calcium sulfate will be not converted to the soluble salt, it will be temporarily dissolved, but it can precipitate back once the pH goes up high enough.
OTOH its solubility is high enough that several weeks of rain can wash it out (depending on how much of the plaster was present).