Well it's certainly possible to extract materials from silicate rocks, it just isn't cost effective. For instance, why go through the horribly complex process of breaking down, for instance, the silicate mineral mica for its magnesium, when you can simply take the otherwise near-worthless stone dolomite ([Mg,Ca]CO3) and dissolve away the carbonate for it? Or why try extracting iron from silicate minerals when there's so much hematite, magnetite, limonite, ilmenite, chromite, goethite, and other iron-bearing ores aplenty that require one tenth the effort to process?
One day if Earth's mineral resources become depleted enough, we may find ourselves needing to extract metals from bodies with mere trace amounts of metal and nothing you could call ore--but we're not there yet, and won't be for another century or two.