There have been some issues with M-Audio drivers-- Not everyone has had this trouble, but your experience sounds similar to mine and others.
I'm running DP 5.11 with OS 10.4.8. There are reports on different forums of those using 10.4.9 with DP 5.11 and 5.12 citing some issues with crash on save or crash during editing, etc. Again, not everyone is experiencing this.
Firewire busses are still an issue as well. Some users who have firewire interfaces as well as firewire hard drives on the same bus are experiencing sporadic dropouts, pops/clicks, etc. The solution is to have your sample data on one or more internal drives-- or to get a PCIe firewire card. Another option would be to get a PCIe eSATA card and to move your samples to eSATA drives, leaving the firewire bus completely open for an audio interface. The PCIe slots thankfully all use independent busses whereas the firewire ports still share a single 33Mhz bus.
While the latest multi-core MacPros rarely seem to reach or sustain the limits of their CPU power, the same issues remain with 32-bit apps and the associated memory limits. Everything that gets done inside of any DAW is done *mostly* within 4GB or less.
As for memory, there are some reports that confirm Apple's recommendation that better performance can be had on MacPros taking 16GB RAM if the RAM is installed in fours. One member here reported that having 4 x 2GB = 8GB proved to be better performance-wise than 4 x 2GB + 2 x 512, which is what I'd been using. I've only this morning removed the 2 x 512 and am waiting to see if there are noticeable improvements.
Note: so far, I see no difference in performance using 1GB less RAM, but I've not 1) pushed my MacPro to its limits quite yet and 2) I've only been working with the lesser amount of RAM for a day.
Whether or not this is an Apple issue or a MOTU issue or an M-Audio issue, there are a few things that might help:
1. Disable all automated features such as Software Update, screensavers, and any utilities that use the system drive for anything at all on a schedule.
2. 'iSightAudio.plugin' has been known in the past to interfere with audio issues. If you don't use iSight, delete this file from:
Library> Audio> Plugins> HAL
3. Disable Widgets. These usually are not a problem, but disabling them removes one more possibility of interference.
4. As a test, disable the M-Audio interface and use Apple's Built-In Audio. If the problems go away, you know that the M-Audio driver (or even the box itself) is causing the problem.
5. Check with M-Audio for updated drivers. One problem with M-Audio is that they are slow with getting drivers released and de-bugged. About this time last year I need a driver for one of their audio cards I was using temporarily while my interface was being replaced. At the time, they didn't even have any drivers for the card compatible with Tiger! They even suggested I wipe my hard drive clean and install Panther. (Grrrrrrrr.) Their point was not well taken.
6. Check your buffer settings in DP. You made need to raise them for the sake of pre-loading enough samples at the risk of adding latency. I wish I could work @256 or lower on larger projects, but for now I'm stuck with 512 for most things and 1024 for the rest.
7. Look in your DP5 PDF addendum for features called Work Quanta, Max Work Percent, and Prime Seconds (all part of Studio Settings).
Work Quanta: This calculates the intervals DP does its number crunching. Raising the number can help divert CPU resources to real-time audio processing at the expense of some graphics functions-- like the smoothness of the wiper. This may not be as big of a concern on the MacPro 8-Core.
Max Work Percent: This assigns the amount of CPU to DP for the purpose of pre-rendering audio for playback-- with a default of 80%. Raising the number can improve audio performance, but again some graphics processing may or may not be compromised.
Prime Seconds: This determines how far in advance DP looks for and prepares playback data. This can introduce latency with playback controls, but it might take some trial and error to find the right setting that improves your performance.
8. Disable things like convolution reverbs or other memory-hungry fx until later. If you prefer, you can put a single instance of a reverb on the master fader if you do not like to work with 'dry' samples. When you're ready to mix you can add as many of these fx as your system can handle and raise your buffers as high as they can go.
9. Keep your disk permissions repaired and run a third-party disk utility from time to time, such as DiskWarrior.
10. Shut down your computer and let it rest for a while if used longer than 10-12 hours. The RAM gets a lot hotter in the MacPro than it did in the G5's-- I've actually burned myself to the point of raising a blister. Keeping your computer as cool and as well ventilated as possible will help keep it running more smoothly.