Your system will pull over 300 but under 400W under load. That's PSU output; it might break 400W at the wall plug due to PSU inefficiency. (Extrapolated from this video card review
using generous fudge factors to allow for AMD's processors running hotter.)
So there's no way that the PSU is being outright overloaded. And although Antec has a somewhat checkered quality history, I think they all have basic overload protection, which shuts down the PSU.
That said, there still could be PSU issues. Ripple and transient stability are the big factors that it's hard to evaluate without an oscilloscope.
But this has nothing
to do with going from a 650W PSU to an 850W one. The recently reviewed Seasonic 660XP
would be even better. But heck, a Corsair CX430 would probably do fine. (Corsair agrees with me.
It's possible that caps have degraded with time (standard electrolytic capacitors have a liquid electrolyte inside that slowly evaporates at high temperatures; the details are one of the big quality differences between brands), or that the PSU was only so-so to begin with.
The fact that people look only at the big number is a huge problem for manufacturers. People think that, for the same price, the higher-wattage PSU is obviously better, and often it's obviously *worse*. Because the manufacturer spent their parts budget on quantity rather than quality of output.
I don't know about your particular situation; without an oscilloscope connected to your system it's awfully hard to tell. But I can say certainly that your old PSU is not at all underpowered, and is in fact slightly overpowered
for what you were asking of it.