Hello maavsl.
I have virtually the same configuration as you. I had sent a message to tech support and they responded with the following answer, which worked great. I have copied and pasted the entire exchange below (from most recent to least). Be aware, first try loading multiple instances of VE as an AU plug-in within Sibelius, and then try the stand-alone option if you max out the total RAM of the plug-in.
--Forwarded message--
Hello Philip!
Due to limitations in a 32-bit system, it is not possible for a single
application to load more than 3 GB RAM. Depending on the configuration of
the system this value may even be lower (in your case about 2 GB). It is
indeed possible to load additional stand-alone instances of Vienna
Ensemble, which will allow you to load the same amount of sample data
additionally. If you want to open more than one instances of VE stand
alone, you have to copy the Vienna Ensemble application and name it
differently. The stand alone instances are not recognised from within
Sibelius. You have to open them from the applications folder.
In order to communicate from Sibelius with stand-alone VE you have to use
IAC or a different virtual MIDI-port. Be aware that you cannot load the
sound set if you choose IAC or a hardware MIDI-out. However, there is also
a way to solve this problem. Open the sound set with the sound set editor
or a text editor and change the entry PatchLoadMechanism from "Fixed" to
"MIDI". Save the sound set with a different name. Now you can communicate
from Sibelius over the IAC-Port with the Stand-alone Vienna Ensemble and
the modified sound set takes care of the articulation changes.
Best wishes,
Andreas
> Hello Andreas.
>
> Thank you for your message. Sorry if I was not clear. I am able to open
> 2 instances of the VE plug-in in Sibelius. The problem is that I am not
> able to load more than about 2 GB total between the two instances. When
> I try to do so, VE crashes. Is there a way to load more samples into VE
> without it crashing? If I loaded a third instance of the VE plug-in,
> would I be able to access more RAM?
>
> I have the latest version of VE, from Feb. 4 2008.
>
> Thank you,
> Philip
>