Its an interesting problem, but its going to be hard for us to pin down without more specifics. (You given us some good specific information, but I feel we need more.)
If I follow you correctly, you've selected matrix conditions, and reproducibility is poor.
OK, are the matrix conditions good? Is pH too close the the enzyme's pI? It may be changing its structure throughout the reaction. Are you using reasonable concentration for substrate and enzyme? I know you should get reproducibility first, then work on linearity, but maybe you should try 2 or 3 points to see if you're at a reasonable concentration.
Furthermore, how far are you from published references on how this reaction is usually run? It would be worthwhile just coping something like that, to try and pin down if your matrix is OK-- pH is really what you think it is, there aren't contaminants in your water or buffers, that your enzyme isn't bad, etc.