Barium sulfate is extremely insoluble. Because of that, the body doesn't absorb the barium, so the sulfate is harmless.
Othwerise, barium is indeed very poisonous.
My guess is that the high atomic number/weight of barium makes it show up well on x-rays.