The walls of the container don't really have much to do with it. You can use an imaginary force field rather than a physical wall if it is helpful to think of it that way. The change in pressure/volume from ideality is an intrinsic property of the gas, due to forces strictly between particles.
Here is an easy way to think about it that is mostly correct. At low pressures, attractive forces dominate, which is why you get compressibility factors less than 1. At high pressures repulsive forces dominate, which is why you get compressibility factors greater than 1. The reason repulsive forces dominate at high pressures is because at high pressures, the number of collisions increases, and when particles get close enough to collide, the nuclear-nuclear repulsion force is exceptionally strong. Since repulsive forces become very important, the net volume is greater than the ideal case (the particles are repulsed and so like to be farther apart), where there are no repulsive forces. At low pressures collisions are rare and electron-nuclear attractive forces are more important, because these operate at slightly larger length scales. Under these circumstances, the particles like to be closer together than the ideal case, and so the volume retracts.
There are some interesting temperature effects as well, but we can leave those alone unless you're really interested.