From the point of view of pure mathematics:
Numbers are parts of an algebra which we have adopted to describe physical phenomena quantitatively. In mathematics, algebras are sets of elements (numbers) equipped with one or more binary operations (e.g. addition and multiplication) and a set of axioms which completely define the properties of the algebra. These abstract algebras can then be applied to describe the quantitative aspects of physical phenomena. For example, the natural numbers (1,2,3,...) are a good algebra to describe counting (e.g. how many fingers am I holding up, how many apples are on the tree). Similarly, the real numbers are a good algebra for describing continuous measurements and the complex numbers are good for describing waves (e.g. electromagnetic radiation). Algebras can also be used to describe phenomena that we do not normally associate with "numbers", for example, we use groups to describe symmetry operations and we use vector spaces to describe quantum mechanical operators.
Of course, here's a good philosophical question: did mathematicians invent these algebras to describe physical phenomena, or did these algebras exist a priori and mathematicians discovered them?