Hi,
First, screen for basic compounds like alkaloids, glycosides etc... using a classical qualitative method.
Second, run a TLC with know compounds, this is an infinite process so deriving the data from the application of the herbal formulation is helpful.
Once, you have a hit, you are done. The best method is to use LC-MS or MS-MS and compare it with existing libraries.