So there are actually a plethora of factors.
To name a few:
Ligand choice and concentration: What are you currently using for a capping ligand? I assume this is in the organic phase. If so, a long, aliphatic chain will help prevent aggregation (which *might* be your problem in the first place - you could be synthesizing small particles, but then improperly capping them and thus allowing them to aggregate into big hunking clusters). Some good choices are carboxylic acids and primary/secondary amines. The concentration of ligand:nanoparticle (NP) is also important - you have to cap the whole surface or - again - AGGREGATION! Typically, the larger the ligand:NP ratio, the smaller the particles.
Reaction time/temperature: Both extremely important in most syntheses. High temperature "burst nucleation" (for a II-VI quantum dot like CdS, this is typically above 280°C) occurs, followed by a slower growth process at lower temperatures. If you spend too much time reacting your crystals at the higher temperature, you're going to get more growth and larger crystals. Try using an airgun to facilitate quick cooling to thermally quench your reaction.
Precursors: Size control and distribution might start at a fundamental level with your choice of reagents! Some salts work better than others in a given solvent. Are you using cadmium acetate? This salt seems to be the precursor of choice over something else like cadmium stearate (do try to stick to metal-carboxylate precursors). Especially soluble in phosphine solvents like TOP and TOPO. What is your sulfur source? Is it in the reduced form? - If not, you can try adding a reducing agent like Diphenylphosphine (DPP) or 1,2-hexadecanediol (HDD). Play around with it a bit to find a good match.
Hope this helps!