I think the specs you provide in your signature should be sufficient (it's the Windows-machine, I assume?) - the only info missing at the moment are the detailed specifications of the graphics card. It might be that you're facing an OpenGL-related issue.
... a wild guess: Does the graphics card use shared memory? What was the RAM usage when you were seeing the problem?
Kind regards,
/Dietz - Vienna Symphonic Library