I've been battling this combination myself.. and here's what i've found.
If you have a DAW that has Automatic Delay Compensation (ADC) then the Server is your best bet. I use Digital Performer which has ADC and, with VEP Server, latency is not noticeable (even at 256 and 2 buffer) and playback is sample accurate. However, the more plugin instances of the server you have, it seems to add CPU usage, so idk if 128 will work with 4+ server instances.. again 256 is working just fine for me.
Now, If you use the standalone, this means you need some sort of "piped back" audio. For me I have MOTU's 2408's using 3 banks of ADAT. Not sure what you're using, but either way you go, you'll have to have audio coming back somehow... and this, my friend, adds latency that no DAW can "compensate" for.
So, for latency sake, the server is your only real option.
For me, i kind of use a hybrid. I use VE Server for all my perc / fast attacking samples that I get caught up on if there's latency, and for pad stuff and strings etc., i use the standalone with ADAT coming back. It saves my DP template from having 20 VEP Server plugins and with pads, latency is not noticeable..
But with DP i'm limited to the 16 midi channel Bull S&^# AU Standard... if there was a MAS version.. things would be waaay different in my setup...
Hope that helps...
J