There are far too many different mechanisms of toxicity to be able to predict in general whether any structure might be toxic, in the absence of other information. There are some general principles and some calculations that can help predict whether a structure might be toxic by a particular pathway.
It is very hard to predict, for example, that benzene would cause cancer while toluene doesn't - the only difference is a methyl group on a benzene ring. However, that methyl group provides a site for metabolism that allows the body to get rid of the toluene, before a second mechanism which makes the benzene ring into a toxic metabolite can take place.
It is hard to predict that drinking methanol can cause blindness and death, but that drinking isopropanol doesn't - because drinking isopropanol causes you to throw up so rapidly that it is almost impossible for toxic amounts to be absorbed.
It is fairly easy to predict, knowing what we now know of enzyme structure, that compounds which can rapidly alkylate other compounds will show toxicities. In fact, almost any compound that is chemically very reactive is also toxic, because chemical reactions in the body must be very carefully controlled to keep the body alive.
It is fairly easy to predict that any material which binds very tightly to iron will be toxic, because it will bind with the iron in the blood and prevent it from carrying oxygen. Unless, of course, that material is rapidly removed from the body by some other transport mechanism.
Certain basic structural motifs are known to interact with the functional groups in RNA and DNA. It is fairly easy to predict that these will cause genetic toxicity if they can reach the nucleus of the cells.
And finally, almost any compound, including water, will show toxicity if it is taken in a high enough dose - "the dose makes the poison". Again, the body functions on a very precarious balance, and too much of any material can disrupt that balance. That is one reason we have so many redundant mechanisms for removing materials that don't belong in the body, and for balancing the ones that do.