A good starting point is to check the regulatory methods used to test your products. For example, if you are in the US, see how the USP tests the active ingredients. Then try and develop your methods around these tests.
Most OTC are still required to meet Fda requirements, which means validation. Validation can be tedious. Often, you can cut down on a step or two by using a previously recognized and approved method.
Having said that, hplc and GC are very common for quant work. For identity, IR is most common, although I like EDXRF better for mineral fillers.
Consider gcms. It is more expensive and easier to contaminate. But when trying to ID an unknown in a liquid, it is hard to argue with gcms results ( with a good library and the right software). A regular gc- say GC/fid will be much less expensive.
LCMS is good, but more expensive than gcms. It takes more training, also.
My thoughts. There are many others on this forum with more experience.
Good luck.