i couldn't get the expected amine or the cyclized product by removing the mesyl LG.
So you have something like this?
N
3-[chain]-OMs
pyrrolidine/piperidine etc.
I have done several of these reactions with H
2, Pd/C and have never had a problem. They are usually very clean reactions. Purification in my cases was performed by ion exchange chromatography on acidic resins because the products were polyhydroxylated pyrrolidines and far too polar for chromatography on silica.
Can you provide any other structural information? I take it you expect it to be non-polar enough for silica gel chromatography?
I would be wary of using Staudinger reduction as the phosphine may react with the alkyl mesylate.
did you not remove the MsOH
This is an important point. The product of hydrogenation/cyclisation will be the ammonium mesylate salt of the cyclic amine, which is unlikely to be mobile on silica (is this your stationary phase for the column you did?) and may also be water soluble (you might have lost it in the workup). You will need to treat the crude salt with a base (e.g. use sat. bicarbonate in the extraction rather than water) to generate the free base form of the amine before attempting chromatography on silica. If you can't move it on a silica TLC, you can't move it on a silica column.
Did you try running TLC on triethylamine doped silica and triethylamine doped solvent? This reduces streaking problems.
For the hydrogenation method, I would advise filtering, concentrating and then getting a crude NMR before attempting any purification. This way you can tell whether the problem is the reaction or the workup. If it looks OK, take a small amount of the crude and try adding sat. bicarb and extracting with an organic solvent. Check the organic fractions by TLC to see if you can find anything and get an NMR to see if the free base of the product is in there.
If it's there, then you can safely use this method of isolating the free base on the bulk of the crude. Find TLC conditions that give appropriate mobility and low streaking, then column.
If it's not there, this means the free base is water soluble and you will have to use an alternative method. By testing the workup on a small proportion of material, you have not lost it all. Alternative purification techniques include ion exchange chromatography and recrystallisation.
Good luck.