Phosphoresence is slow because the light-emitting transition corresponds to a "forbidden" transition. When you excite a molecule, excitation proceeds through an allowed transition and the electron is now in an excited state. This molecule can fall back to the ground state through the same allowed transition (except this time the transition is from the excited state to the ground state rather than from the ground state to the excited state). This produces fluorescence.
In phosphoresence, however, the molecule relaxes to an intermediate excited state through a process known as intersystem crossing that has a different spin multiplicity as the ground state. In this new excited state, transition back to the ground state is forbidden (because of the difference in spin multiplicity), so any radiative relaxation back to the ground state is phosphoresence.
Therefore, because the intersystem crossing step leads to relaxation of the electrons to a lower energy state, the phosphorescent excited state is lower than the fluorescent excited state.