Well, there is no such thing as "non-weapons grade plutonium." All isotopes of plutonium that I know of, and which are formed in a breeder reactor, are fissionable. That is what makes plutonium so nasty. It dishes out neutrons like there's no tomorrow, and it is horribly poisonous chemically. It has to be the most lethal element on the periodic table in terms of chemical lethalness and radiological lethalness. It's kind of funny because I was watching Back To The Future today on TV, and when I was a kid it never really bothered me that they just cavalierly carted around plutonium. But now that I've grown up and learned about the stuff, the way they handled the Pu and how they moved it around would have exposed them to lethal amounts of radiation. In the body, plutonium mimics calcium in every manner. As a result, it builds up in your bones and stays there emitting neutrons at high rates. This leads to cancer of every form imaginable forming. That is why plutonium in the hands of a terrorist is a horribly frightening idea. Even if they don't have a nuclear blast, which is difficult not to do when you have plutonium, spreading Pu over a wide area is just intensely scary.
Another problem with having plutonium around is an accidental meltdown. It doesn't take a helluva lot of plutonium to reach a critical mass. In breeder reactors over in the former U.S.S.R., they have had "incidents" where too much generated plutonium was stored in one area and the Pu went critical. It would have been a whole lot worse if the plutonium was stored in too close of a proximity to itself. (There was enough separation between the fuel to prevent a chain reaction explosion). Instead, the pile just went critical and started heating up and outputting radiation at an alarming rate. While people did get very sick and die from it, there was no nuclear explosion. There's just too much danger when you're dealing with Pu that you don't have to deal with working with Uranium. (If someone gave me a rod of pure elemental uranium, I'd hold it for a second before throwing it back in a lead container. If someone offered to let me hold a rod of plutonium, I'd say no without hesitation).