Well, I see you're using a standalone via the IAC. I think that is not going to perform as well as the server without that extraneous moving part, using a low-latency audio interface on the DAW end.
I don't get why you believe that a single instance of the standalone is that much more convenient than a single instance of the decoupled server.
One expectation you have is based here, you want to have every known thing all loaded at once and low latency. "Makes life easier", evidently not, you are having issues that could be managed better. You have something in mind that doesn't seem to be working so maybe you need to adjust to your situation some. The server is what's made for low latency DAW work. It should perform a lot better than just plugging instruments into a DAW. Although I have not used it with Logic.
With the machine in my signature, Cubase 5.5.3, I don't have crackles at 512 in the buffer with 'not a lot going on'. I start projects often at 128 and latency I can do drums by, or even 64, and add latency as the project's demands grow organically, as the load grows. Horses for courses.
Cubase is definitely lower latency on windows. I bought Logic and DP behind this [Cubase/Core Audio] latency problem but for me VE Pro has obviated any need to use those hosts.