In a normal 9v or 1.5v battery etc. does the voltage decrease over time? I assume yes
Yes, every single battery works this way.
and is this because there is a less amount of atoms on the cathode side to pull the electrons from the anode side?
Number of atoms doesn't matter (if it did, large batteries would have larger voltage than the small ones). What matters is the ratio between different forms of the substance undergoing reaction. The longer the battery works, the more products are present and hence the voltage change.
Do electrons move throughout the wire, or does each electron get pushed along to the next atom?
You know that electrons occupy orbitals? In the solid orbitals of the neighboring atoms overlap, creating kind of a combined orbital (called a "band"), on which electrons can freely move. This is actually more complicated, as it works a bit differently for conductors, semiconductors and isolators, but in each case the underlying idea is the same. Check
http://hyperphysics.phy-astr.gsu.edu/hbase/solids/band.html (although I can't see the page at the moment, hyperphysics is usually a good source).