A simple answer is that the carbon atoms in diamond are arranged in almost perfect tetrahedral lattice. This makes them very stable. C-C bonds are (in isolation) also reasonably strong.
A more nuanced explanation would require examination of the diamond surface. Solids predominantly react at their surfaces, so it is the surface that really determines the speed with which a hunk of something will change into something else. As you may imagine, even though carbon atoms are arranged in perfect tetrahedral arrangement, that situation ends at the edge of the diamond. Atoms on the surface have what are sometimes called "dangling bonds". This is a spot where an atom on the surface wants to be bonded to another atom, but (because it's on the edge) isn't. That can make for an unhappy situation, and it is usually where reactions take place. (If you've seen the videos of penguins in Antarctica huddling in a big mass to keep warm, imagine for a moment how the ones on the edge feel and that should given you a nice idea of what's going on.)
Surface science is complicated and the reactivity depends on a lot of things, not least of which is the specific crystal plane that is exposed to the environment. I am no expert in the surface science of diamond, but a brief look at the literature seems to indicate that the "dangling bonds" on the surface carbon atoms of diamond are usually passivated by hydrogen atoms. C-H bonds are also incredibly strong. With a strong enough base you could probably rip those off and start to do some chemistry at the dangling bond sites on the diamond surface - most likely reacting with atmospheric oxygen, but bearing in mind that any such chemistry leading to destruction of bulk diamond would also require breaking a lot of C-C bond underneath, so such decomposition would still be extraordinarily slow.
Which brings up one other point worth mentioning: the distinction between kinetic and thermodynamic stability. We think of diamond as very stable, but it's not the most stable allotrope of carbon, thermodynamically speaking (at standard temperature and pressure). The standard state of carbon is actually graphite. What this means is that carbon atoms in diamond are actually happier as graphite than they are as diamonds. By implication, if you left a diamond on your table and waited long enough, You'd come back to find a lump of graphite instead.
Why doesn't this just happen, then? Well, rearranging the carbon atoms in diamond to their more preferred graphite orientation would require breaking lots of carbon bonds so that you can make new carbon bonds in a different orientation. That breaking of carbon bonds takes a lot of energy. So even if you get that energy back (and more), it is still a (very) slow process to get things going. Which is why diamonds are forever.
One might also wonder why diamonds don't just react with oxygen in the atmosphere. Consider the reaction of complete oxidation of bulk diamond to carbon dioxide in the presence of oxygen:
C (diamond) + O
2 (g) --> CO
2 (g)
At room temperature, the standard molar change in enthalpy for this reaction is -395.4 kJ/mol. This is how much heat is either released to or sucked up from the environment during the reaction. The negative value tells us the reaction is quite exothermic, that a lot of heat is released during the reaction, and that the products are more thermodynamically stable than the reactants.
The standard molar entropy change is +6.2 J/K/mol. The reaction results in a slight gain of entropy. Pretty much a wash here because one molecule of gas is produced for every molecule of gas that's consumed.
And the standard Gibbs energy change is -397.3 kJ/mol. The negative value here tells us that the reaction is spontaneous.This is particularly relevant, because it signals that under standard conditions, diamond is quite happy to be obliterated and converted into carbon dioxide. The driving force is indeed pretty large, given how exothermic the reaction is. And yet, we don't see this happen because* the requirement of breaking lots of very stable carbon-carbon bonds is just too steep a hill to climb. There's not enough energy hanging around at room temperature in order to get this to happen efficiently. I guess that if you had some magic microscope where you could zoom in and examine each carbon atom on the surface of a diamond, you'd probably occasionally see one with an oxygen atom bonded to it, but it just doesn't happen enough to manifest a significant change to the macroscopic properties of the diamond.
Ultimately diamonds ARE chemically stable, but not for the reason you might think. Carbon atoms in diamond would rather be somewhere nicer, but they're just too lazy to go there. In that way, they are very much NOT like penguins.
*There's another reason for the slow speed as well. The electrons in oxygen in its most stable form are arranged in such a way that their spins are unpaired. Because they are unpaired, a molecule of oxygen has a relatively large total electron spin angular momentum. None of the other reactants or products have unpaired electrons. So, the reaction violates, in a way, the conservation of momentum. Getting the reaction to go also requires enough energy to force the electrons in oxygen to pair so they have no net angular momentum, which creates a kinetic barrier. This is, by the way, why all things made of carbon don't just spontaneously combust in Earth's atmosphere.
References: thermodynamic data for oxidation of diamond was calculated from values reported here:
https://www2.chem.wisc.edu/deptfiles/genchem/netorial/modules/thermodynamics/table.htm