I replaced the stock video card with the RX580. There is one out there that is specfically reccomended by Apple, SapphirePulse. Google around for that info, I will try to find it later. There is an RX590 that has come out since then which may have superceded it, and might be better, not sure. Some people has also used the RX560 for less money. My RX580 was about $130 as I recall.
With my RX580 video card in place, I still have 3 more slots for whatever and I'm using them all for SSD and soundcard.
I think you said you already upgraded to high Sierra, which included an important firmware update in order to support Metal cards. For that upgrade, you could not use the RX580 for the firmware update part, because the RX580 does not have boot screen. What that means is that the screen is blank until its almost fully booted up and then only shows stuff right before seeing your complete desktop. So prior to High Sierra, you could not perform the firmeware update with this new GPU, you had to keep your old video card installed for that OSX upgrade. Its not a bad idea to keep your old video card in a box on a shelf for the future just in case. But apple did improve the Mojave updater, such that it will work with the RX580 and includes yet an additional firmware update...but Mojave actually REQUIRES a metal capable video card such as the RX580.
Anyway, the Mojave update should work with the new card in place, in fact Apple requires it.
My experience has been that Mojave is faster and that VePro has also been optimized for Metal in some ways, it has been more efficient, I did performance comparisons between VePro6 and VePro7 at some point and found this to be true.
I do not know whether this will solve your problem with the crashes, but in general I think you will be happy upgrading your video card and upgrading to Mojave honestly. And maybe that will solve the problem with the crashing. If not, then we can try to figure out what is different about your system that causes VePro7 to crash under MRD.
Like I said before, make sure you're using the very latest version of MRD available on your Mac.
One other thing to point out, when you move to the Metal card, the mac boots up with a totally black screen until the last minute, which be a little disconcerting. It still can respond to all the key commands for safe mode, restore more, pram reset, smc reset, etc..all that still works. Apple made sure the Mojave installer would do its firmware update while looking at a black screen, for example. It all works, but seems like I had to boot twice or something to get the mojave firmware update to work. you can google around. Apple is not supporting our macs for catalina forward. Some people will figure out how to do it anyway though. But I doubt there will ever be any more firmware updates for the 5,1 MacPro because of that. So once you update to Mojave, probably you will not bother going past that ever. Metal is an improvement for sure, I thnk it was worth the price and I plan to use my 5,1 for at least 3 more years.