There may be many reasons that I've missed but I think Phosphorus is more reactive because it exists as molecules of P
4 (shape of a trigonal pyramid), in which there is a considerable amount of bond strain, making the bonds easy to break and therefore it will be more reactive. This applies for white phosphorus; red phosphorus is basically a polymer of white phosphorus and the bonds are under less strain making it less reactive.
Sulfur, however, normally exists as a ring of sulfur atoms of 8, although rings of 7 etc. exist. There is much less bond strain in this and it is considerably stable.
I think the main reasons for the difference for this are structure, bond length and bond angle.