Experienced chemists can predict the behavior of new molecules by analogy to known molecules which have similar functional groups. For example, I could take acetic acid (a smelly compound which is acidic, relatively high boiling, miscible with water) and react it with an alcohol, say ethanol (also miscible with water, not a strongly scented, polar, not acidic or basic to an appreciable extent) and react the together to make ethyl acetate, an ester which is not miscible with water, boils lower than acetic acid, but slightly higher than ethanol, has a sweet smell, and has no appeciable acidity or basicity. I could predict the behavior of ethyl acetate because I am familiar with esters; I can predict the behavior of ethanol because I am familiar with alcohols; I can predict the behavior of acetic acid because I have dealt with carboxylic acids before. Since I have experience with all these, I can predict the reactions they will undergo and how I could potentially isolate the product. Once you know how to deal with molecules containing certain functional groups then you can extend that knowledge to molecules that haven't been made before. You just need to know how certain functional groups behave.
For example, I recently made a compound that no one had ever made before. It contained a ketone, an alkene, and a nitrile. Things get a little more confusing with multiple functional groups, but in this case it wasn't too difficult. I knew this compound would not be very volatile because the alkene and the carbonyl group were conjugated (these enone compounds are typically much higher boiling than their ketone counterparts) and I knew that the molecule would be relatively polar because of the nitrile functionality. So even now I have no idea what the boiling point of this compound is, nor any of its biological activity, but I do know how to make it and how I can isolate it from the by-products of the reaction. It's all about having experience dealing with compounds.
It's not always this easy. Sometimes functional groups undergo unexpected reactions with one another. This can't be avoided and it's always a possibility. You have to learn what kinds of functional groups are definitely not compatible. This also comes from experience. For example, I would never propose a molecule that contained an unprotected primary amine and an acid chloride. The two functional groups would reaction rapidly. Again, experience is the name of the game.
Bear in mind that even experienced chemists don't always know what to expect. Some compounds that look perfectly reasonable on paper might be impossible to make in the lab. Paper chemistry can often be deceptively easy! You can predict a lot based on what you know about other compounds, but the level of certainty is invariably low.