The missing elements for realism are the noises that often cause re-takes in the studio environment. In otherwords the extraneous sounds the individual and ensemble players make performing the score. For a sampled library it is sensible to filter this out but in the real world all these sounds exist to some degree so would be expected to be present to some extent by the listener.
I found that a room tone can prevent the drop off to dead silence at the end on a piece but would not normally be heard until that point, whereas small sounds of breaths, bow movements floor creaks are the missing realism from a sampled orchestra.
Julian.