Have you taken any general chemistry courses in your time at university? I think that gen chem is a pretty good survey-style course, and if you like and/or are proficient at gen chem, then you probably like chemistry as a whole. The math needed for gen chem is pretty basic algebra, but it does get harder as you move to upper level courses. Likewise, organic chemistry only requires VERY basic arithmetic; its probably the least amount of math you can do in a hard science (besides biology).
About not being a math person, trust me, I know the feeling. When I was in high school learning algebra II, I was struggling like hell. It was the hardest stuff I had to learn until that point in my life. I was about to give up on my scientific education because I thought I couldn't continue in math. However, my teacher was extremely helpful and she actually made me believe I COULD do math. Once I believed in myself, things just clicked and I started getting it! I then had that same teacher for the following year in pre-calculus and I rose to the top of the class. Then, senior year, she recommended me for AP Calculus BC (with another teacher) and I ended up getting a 5 on the test, allowing me to skip the calculus I sequence at my university.
In short, my teacher gave me the adequate mathematical foundation to continue in science, one of the greatest gifts I have ever received. To this day (four years later) she is still the best teacher I had in ANY subject, easily. I really think without her guidance, I would never have "understood" math and my life would be drastically different today.