Achievable latency is always going to reflect the specifications of your system and the complexity of the project you're running within Mir.
For example, I'm running Mir at 128ms hardware buffer and have no problems whatsoever with what I consider to be decent sized arrangements - say 50 VI instances containing fully scripted matrices, with up to 25-30 of them playing simultaneously. If I had to do something much more involved I would increase the buffer size if I saw CPU load issues. That said, I have a very fast quad-core Xeon system which was specified primarily for Mir.
In essence, you're absolutely right, you can always run at a lower buffer setting whilst recording indivdual parts, mapping out ideas and basic arrangements etc. and if you find further down the line that the full arrangement is stressing the system, increase the buffer size when you render the final mix.
Jules