Gold, as you've already pointed out, has an unpaired 6s electron in it which you would think would make it quite reactive. (Since Cesium also has an unpaired 6s electron). The difference is that gold also has the 4f and 5d shells completely filled. The order of filling electron shells is 1s->2s->2p->3s->3p->4s->3d->4p->5s->4d->5p->6s->4f->5d->6p-7s->5f->6d etc. etc. So the lone 6s electron of gold is 'blocked' by the full shells of 4f and 5d electrons. Gold also has a very large nucleus with 79 protons in there trying to grab that electron. The combination of the strong pull from the nucleus (remember, effective nuclear charge goes up as you move left to right), and the shielding effect that gold gets from the higher energy 4f and 5d electrons make it very difficult to pull away that 6s electron. You need a very good oxidizer in order to grab that electron away. If you look at virtually all of the elements to the right of lutetium and to the left of thallium, you'll see that they really aren't that reactive.
If you compare gold to cesium, you'll see that cesium also has a lone 6s electron. The thing with cesium is that it doesn't have any 4f or 5d electrons to shield that lone 6s electron, and it also has a much weaker effective nuclear charge. Therefore, Cesium's lone 6s electron is very exposed and very prone to reacting. Mercury, just to the right of gold, has a full 6s shell so even it was exposed you wouldn't expect it to react. With mercury being a liquid, however, it means that its internal bonding isn't all that strong so although mercury is somewhat nonreactive, it will still oxidize pretty easily. As we move one over to thallium, we start to fill in the outer 6p subshell with one electron, thus increasing the reactivity as the high energy 6p electron is now fully exposed.