Most of the colors you see are not caused by the kind of photon emission you are describing (what you might term fluorescence). Rather most colors you see are due to absorption and scatter events. A good example is a leaf. Sun light is (approximately) white, meaning in the visible range (about 400-700 nm), there are roughly equivalent* amounts of photons spanning wavelengths from 400 nm (blue, say) all the way to 700 nm (red). Leaves have a lot of chlorophyll, which is a molecular pigment that pretty strongly absorbs photons ranging from 350-450 nm and from about 600-700 nm. So when white light falls on a leaf, most of the photons in those ranges are absorbed (where their energy contributes to photosynthesis) and the rest of them - from 450 nm to 600 nm - are scattered and reflected. Guess what color most photons in that range look like to your eye? Yeah, green. So, the color of MOST objects is due to what wavelengths are left over and scattered to your eye after molecules in the surface of the object absorb a selrct fraction of the white ambient light around you.
Keep in mind, though, that the perception of color depends not only on what energy photons reach your eye (which depends on how many of each wavelength started from the various light sources around and the number of photons that are absorbed/scattered by various surfaces) but also on how efficient your eye is as picking up the photons that make it through your lenses.
*let us pretend, anyway