In general, conjugation/delocalization of electrons lowers their potential energies, making them more stable. If you remove an p-orbital from the conjugated pi system, you are lowering the amount of delocalization and thus raising the energies of the pi-system orbitals, which is unfavorable.
In this case it isn't really the stability of the conjugate acid that matters, but the stability of the acid relative to the base. Because the base has more delocalization than its conjugate acid, the conversion from base -> acid will be disfavored relative to say aminocyclohexane where no such loss of delocalization occurs.