Mostly you know the products of the reaction by knowing what types of reactions the reactants can undergo, and what is required for them to undergo that type of reaction.
For example, we know that ammonia (NH3) is a base. That means it can pick up a proton to form an ammonium cation (NH4+). What else is in the reaction that might give up a proton? Water, which can act as an acid or a base. If the ammonia takes a proton from water, what is left behind? A hydroxide ion (OH-).
But maybe you don't remember that ammonia is a base (or never knew it to begin with). You can look at it's structure - it is forming three bonds to hydrogen atoms, and since it has five electrons in it's outer shell, that leaves a lone pair of electrons. Lone pairs of electrons can be shared with protons, so you know the molecule at least has the potential to act as a base. From there, you can look up the pKb to see if it is a strong enough base to react with water in an acid-base reaction.
You might also realize that nitrogen is more electronegative than hydrogen, and that the nitrogen-hydrogen bonds in ammonia are polar, with the hydrogens being somewhat more positive than the nitrogen. So possibly the ammonia could act as an acid as well. Then you would look up the pKa to see if it was a strong enough acid to react with water, and you will find that it isn't.
The key to predicting reaction products is knowing what types of reactions can occur, what types of structures will undergo those reactions, and what conditions are required. After that, you just need to practice to recognize those structures.