I've discussed this in the past. It is a real hindrance with my 1.8 Dual, because the CPU is so pre-occupied with loaded sounds that the headroom for actual playback is diminished.
Though a user may post Activity Monitor totals that max out memory with VI stand-alones and VI within Logic, plus VSL PE loads in EXS, the *real world* truth is a lesser powered CPU must devote resources just to accomodate the samples, and as soon as you hit play, you've lost about 50 to 60% of your CPU even with no MIDI tracks.
Because Pierre's and anton's computers are so powerful, it's not an issue (but apparently it's still noticeable).
The non-technical explanation is that, with all those samples loaded, VI and Logic have to prepare for what you MIGHT play. And the scope of the notes you MIGHT play gets larger with every instrument you add.
But I take encouragement from this thread, because both posters are saying that they can see no real demand on the CPU with MIDI performance. So they're getting a baseline CPU loss due to samples, and then the actual performance hit is negligable.
On my 1.8, I can see the performance toll as well, compounding my CPU sample loss. Because of this, though I have 6.5 GB total in the machine, I really can't load more than about 1.75 GB per EXS and VI (3.5 GB total samples) without overloading on playback. Indeed, when I start overloading, I go in Environment layer by layer and see if I can unload samples, not because I'm out of memory but because I need the CPU.