I run into the same problem at low latencies and/or as soon as I use more than four or five instances.
"Audio Input" into VE Pro has always been some kind of experimental feature. Most DAWs are not ready to send audio to an external application from anything else than their actual outputs; abusing a plug-in like VE Pro's "Audio Input" for this task comes for a price ... please keep in mind that VE Pro has been primarily invented to act as a MIDI host that reacts to incoming MIDI data, not audio.
Kind regards,
/Dietz - Vienna Symphonic Library