I too have a strong interest in Renaissance and Baroque instruments -- even Medieval instruments.
Most of what's out there is low quality, and to my knowledge there is NO library with viola da gamba or viola d'amore.
Although most of their libraries (except a few of the pianos) are low res, the Sampletekk Recorders library is fairly detailed, if not terribly expressive. Precisionsound has one too, but they chose to bias theirs towards pop music, with no access to dry samples or ones without vibrato.
I think MOTU Symphonic Instrument included a Lute. I keep hoping for an update on that VI.
Precisionsound has a detailed Mandolin library but it's not one that's suited for classical work. There are quite a few other mandolins out there in various ethnic and guitar libraries, but I won't mention any of them here as NONE are appropriate for classical use.
There is a brand-new Renaissance and Baroque library from Soniccouture that is MAGNIFICENT, very detailed and expressive. It covers several holes that have been neglected by other sample library developers. NONE are part of your list, however.
Nevertheless, here's the link, which could come in handy for others. I bought it immediately, and am pleased as punch as I have been wanting a quality Theorbo and Hurdy-Gurdy for some time.
http://www.soniccouture.com/en/products/20-experimental--ancient--rare/g30-the-conservatoire-collection/
At least the Theorbo is a type of Lute, which you would probably need to match to a standard Lute in many works anyway.