Michael,
I fought this problem for hours, trying various approaches I found online. I had the problem when I first installed VEP 5.0, and somehow fixed it. But when I upgraded to 5.1, the problem was even more persistent.
I found a solution...the problem is that VEP somehow lost its proper permissions signature. You can verify this by opening a command prompt and typing...
codesign -vvv "/Applications/Vienna Ensemble Pro/Vienna Ensemble Pro Server.app"
...or if you are running the 64-bit version...
codesign -vvv "/Applications/Vienna Ensemble Pro/Vienna Ensemble Pro Server (64-bit).app"
...my response for this was...
/Applications/Vienna Ensemble Pro/Vienna Ensemble Pro Server (64-bit).app: code or signature modified
That is a bad answer. It means that something changed and now OSX can't authenticate the App. I tried uninstalling/reinstalling VEP (I think this is what fixed it in 5.0) with no success.
I found this fix, and it worked perfectly...
http://silvanolte.com/blog/2011/01/18/do-you-want-the-application-to-accept-incoming-network-connections/
Basically, you create a new Certificate, then assign that certificate to VEP (overriding the broken Authentication). As a novice UNIX guy, the only bit I tripped up on was the need to use quotes in the command line because the file path had spaces in it. Other than that, the solution worked perfectly for me.
Hope this works for you!
Cheers!
Randy