I think it depends more on whether or not you have a more mathematical mind. Chemistry is great in that it encompasses a great many different fields, including those with a more physical inclination; however, physics is rich in deeper explainations if you're patient enough to go through the math.