I have written up these notes which I hope will help someone. I don't know if this method will work on another system and I don't know about the security aspects of setting up your PC with these settings, so please exercise caution and let me know if there are any errors or bad practices here. I have also posted it as a 'permanent' page here: http://www.drumasters.com/vep.php
HOW TO SET UP A PC SLAVE FOR VIENNA ENSEMBLE PRO (or Plogue Bidule etc)
I don't know if all these steps are necessary, but this is what worked for me. I'm using one PC running Windows 7 64 bit Pro with 24GB RAM to run all my orchestral sounds (about 400 midi channels so far) with a Mac Pro 8-Core running OS 10.6 Snow Leopard and Cubase 5. I looked into Bidule as an alternative, hence including some info at the end, but decided to use VEP for now. The VEP manual covers how to get it working in your DAW - this tutorial is just to help get your Mac and PC talking to each other.
SET UP SHARING ON THE PC
Click Start, Computer, Local Disk (C:). Right-click, Share With, Advanced Sharing. Click 'Share'. Click Network and Saring Center and set Home or Work and Public profiles as follows:
Turn on network discovery
Turn on file and printer sharing
Turn on sharing so anyone with network access can read and write files in the Public folders
Use 128-bit encryption
Turn on password protected sharing
HOW TO SET UP A GIGABIT LAN (Local Area Network) between a Mac and a PC*.
* I have and ASUS P6T ATX motherboard with built-in ethernet running a Realtek PCIe GBE Family Controller driver. If you have a separate ethernet card or a different driver the PC instructions may differ slightly, although the principals will be the same.
First, check the ethernet cable! Some Cat5e cables are suitable for Gigabit connections, some aren't - it should have the word 'Gigabit' on the cable if it is. To be certain, spend a little more and get a Cat6 cable, then you'll be sure. I've never had a problem with cables before, but this one tripped me up for a while.
If you're only going to use one slave, you only need one cable (regular, not crossover). If you think you may expand to more slaves, get a Gigabit switch (or router, but don't use your cable modem that you also use for wifi - you'll see why later) and an extra cable. The Netgear GS series works (I have the GS108); I have heard that D-Link switches don't always work, but I haven't tried them, so that may not be true.
SWITCH OFF WIFI ON BOTH COMPUTERS!
This is where I went wrong for a long time - I thought I was connected via LAN but they were actually communicating via wifi. You can switch it back on later, but let's get the LAN working first.
Connect the Mac and PC to the Gigabit switch. If you're lucky it will all work. But it probably won't, so...
SETTINGS ON THE MAC (credit to Chris White for this)
Go to System Preferences, Network, Ethernet 1 (if you have a 2-port machine and assuming you've used port 1):
Configure IPv4: Manually
IP Address: 169.254.1.2
Subnet Mask: 255.255.255.0
[everything else blank]
Click Advanced and then Ethernet:
Configure: Manually
Speed: 1000baseT
Duplex: full-duplex, flow-control
MTU: Standard (1500)
SETTINGS ON THE PC
Click Start, Control Panel, Network and Internet, Network and Sharing Center [sic], Change Adapter Settings (in the left side-bar). Double-click on Local Are Connection - this should bring up the LAN status panel where you can see if you're connected and at what speed (you can also use the Windows Task Manager to monitor this, but you can't change any of the settings there).
Ideally you'd see something like this:
IPv4 Connectivity: No Internet Access
IPv6 Connectivity: No Internet Access
Media State: Enabled
Duration: [time]
Speed: 1.0 Gbps
Click on Properties and in the Networking window (which should have just popped up) double-click on Internet Protocol Version 4 (TCP/IPv4) (it's in the list of ticked items).
Now we're going to fix the PC's IP address, so the Mac (which we fixed when we set the IPv4 address to manual) will always know where it is.
Click 'Use the following IP address':
IP address: 169.254.1.3
Subnet Mask: 255.255.255.0
[everything else blank]
Click 'OK'. Now you should be back to the previous window (Local Area Connection Properties). Under 'Connect using:' click Configure which will launch the driver panel. Click Advanced, scroll down to (and click) Speed and Duplex and set the value to 1.0 Gbps Full Duplex.
You may need to restart everything (and check all the settings have been saved if it doesn't work). Now when you click Start, Control Panel, Network and Internet, Network and Sharing Center you should see your PC connected to an Unidentified Network via the Local Area Connection (in blue). If you click on that, the status box pops up and you should see a connection speed of 1.0 Gbps.
IF YOU ALSO WANT TO USE THE MAC'S BUILT-IN AIRPORT FOR WIFI
Go to System Preferences, Network, Airport.
Click Advanced then TC/IP:
Configure IPv4: Using DHCP
If there's no IP address click Renew DHCP Lease.
Make sure the IP address starts with 192.168 - if it doesn't you might want to start by setting the Airport and then go back to the ethernet settings above.
What we're trying to do is get the wifi (internet) set up on the 192 address and the ethernet on the 169 address so there's no confusion.
It's best to make sure VEP is working before turning any wifi back on.
TO RUN PLOGUE BIDULE ON A SLAVE
You can't use rewire if you're using Bidule on a slave, so you'll need MIDIoverLAN on both machines. You can download a free trial to see if it works for you. ipMIDI is an alternative but you can only get 20 sets of 16 midi channels with that, as opposed to 64 in MIDIoverLAN.
When you install MIDIoverLAN you'll need to find the control panel on both machines (probably in Applications, MusicLab, MolCp3, MolCp3CtrlPanel on the Mac and C: Program Files, MusicLab, MolCp III, molcpcfg on the PC) and set up every port to accept incoming and outgoing midi, as required. It's a bit laborious, but self-explanatory.
You'll (obviously?) need an audio card or device in order to hear anything. Most people I know send ADAT outs to their Pro Tools rig on a separate Mac. If MIDIoverLAN is set up and an audio device is connected, Bidule will automatically give you ins and outs when you open a new session. It's then pretty easy to add Kontakts with all your samples. Bidule is great if you want to get more creative or if you want your audio to go somewhere other than back to your composing DAW. VEP is great if you just want everything to play within the DAW via ethernet.