I think this has been a general problem people have tried to solve for some time. It isn't simple. I cannot do better. The description so far is petty good. What I will try to do is describe a model which might explain why reactions favor one mechanism over another. While the model may not be entirely accurate, it may be useful.
For an SN2 reaction, at the extreme, the electrons of a nucleophile must attack the carbon before anything can happen. Good electron withdrawing groups are best, especially hydrogen. The more that electrons a pulled away from the carbon, the greater the H-C-H bond angle becomes over sp3 hybridized's 109°. This decreases electrostatic repulsion on the side opposite of the leaving group favoring backside attack. The more electron withdrawing, the faster the rate, I>Br>Cl>>>F.
For an SN1, no reaction can take place until the C-X bond has been broken. Bulky groups retard the reaction because carbon is an electron donor (it is next to boron, as BH4(-)). Electron donation provides an electrostatic barrier to electrons being attracted to a carbon. In order to break a C-X bond, I reason that hydrogen bonding may be the important factor. Water is notably hydrogen bonded and contains a high proportion of protons available for bonding. Therefore, if water or other proton sources can hydrogen bond with the leaving group, even iodine, this will weaken the C-X bond. The combination of a electron withdrawing and hydrogen bonding result in bond cleavage. Bond cleavage precedes bond formation. Neighboring pi or non-bonded electrons increase bond cleavage. Heat weakens a C-X bond and favors cleavage.
Elimination reactions follow the same pattern as substitution reactions, except the electrons that become donated are from neighboring atoms. If a carbocation is formed and the neighboring atom contains a tertiary or quaternary carbon, the electron attached to the carbon are very weakly held because carbon is a weak electron donor. Carbon-carbon bonds can donate electrons. If a C-H bond is present, the electrons can be attracted to the carbocation. This will give an intermediate which is the reverse of the electrophiic addition reaction. Removal of the proton will result in a net elimination reaction. Addition will result in a substitution reaction. It is difficult to predict which should be favored and generally mixtures occur.
For an E2 reaction, you will find elements of an E1 reaction. When I described SN1 and SN2 models, those were the extremes. Most reactions fall somewhere in between. Some bond cleavage begins before bond formation in SN2 and some bond formation may begin before bond cleavage in SN1. In a polar protic solvent, the neighboring carbon can donate electrons from the most substituted carbon and lead to the most substituted alkene, the Zaytsev product. If the reaction is more SN2-like, then acidity of a CH-bond will determine the product, the Hoffman product.
It is easiest to recognize that primary carbons should react in an SN2 fashion while tertiary by an SN1 reaction. Secondary halides can reaction by either and present the greatest ambiguity and may actually react by several mechanisms in selected instances.
As a rule of thumb, I assume a reaction should be an SN2 reaction if a nucleophile that is not the solvent is present. Therefore the rate should increase with its concentration which should favor SN2/E2. SN2 cannot occur with tertiary halides, secondary can occur mixed with elimination, and it is favored for primary halides. The higher the temperature, the greater the elimination. For a secondary halide, conjugate bases whose acid has a pKa greater than ~11 favor elimination (Brown, Foote, and Iverson).
This describes a general model. It is based upon principle, but it may not explain more nuanced examples. Your mileage may vary.