Which fabrics? Only the plant based fabrics, such as cotton and linen are cellulose based. Animal based fabrics such as wool are keratin protein based, and insect based fibers such as silk are based on a very different protein (called appropriately enough, silk protein) which is very high in glycine, with some alanine and serine. These natural fabrics also contain a wide variety of other chemical entities such as proteins and lipids that are not entirely removed by the cloth making process. Synthetic fabrics such as nylon and polyester have more regular structures with none of the extra proteins and lipids and other natural biologics, but may also contain a number of synthetic dyes, softeners, and other additives. And as far as I know, there is no such thing as a truly white natural fiber. All white cloth is bleached and usually treated with white colorants to appear white. Blue colorants may also be present to counteract the normal yellow appearance of the material and make it appear whiter. The bleaching is a chemical process, and given the right conditions, is reversible. The colorants can be lost due to decomposition, wear, or washing, and the chemical bonds making up the fibers can decompose.
For cellulosic fibers, slow oxidative dehydrolysis appears to be one major cause of yellowing. If you look at the structure of cellulose
http://en.wikipedia.org/wiki/Cellulose you will see that it is composed of chains of cyclic sugar molecules, with lots of pendant -OH groups. These can be protonated and eliminated to leave behind double bonds, frequently conjugated double bonds, which have electronic transitions in the visible range - hence, color. Bleach, used appropriately, can reoxidize these double bonds and remove the color. Over bleaching breaks the glycosidic bonds between glucose molecules and oxidizes at C-2 and C-3 to form ketones, which can be further oxidize to break the C-2/C-3 bond and form carboxylates.
Lignins are polyphenolic molecules that are present in plant fibers, but not in animal fibers. Bleaching is used for delignification, breaking down the lignin structure so it can be washed away from the fibers, and oxidizing the chromophores that remain to remove their color. This oxidation, however, is a reversible process, and over time and especially in the presence of light, any remaining lignin will eventually return to its colored state. Wood contains much more lignin than cotton or flax, and wood pulp papers especially are much more sensitive to residual or environmental acids and oxidative decomposition than cotton. I am not aware of wood being used for a source of cloth, except through the process of dissolving and reforming the cellulose to make materials like rayon, which are neither truly natural nor truly synthetic.
Rather non-specific, but I hope some of this information helps.