After a while now and on some of my previous posts, I thought VSL was at fault for the stuttering but with further testing I have found out that voicemeeter is actually the cause of this. Using it with the output being directly to the headphones will make it stutter far less and allow you to have more voices playing at the same time before it starts to stutter.
I'm using Voicemeeter potato to route the audio how I like, I have kept the buffer size to 512.
In VSL I have set the streaming and loading threads to 12 (the amount of threads my CPU has).
Has anyone got any solution to using VSL with Voicemeeter nicely?