I don't use Calibre for this (but I do for epub and pdf ebooks). I use Plex and use Plexamp for my music collection. For audio books I used to have Chronicle Audiobook Player installed (which is open source, but not actively maintained) but am now trying out Listen Audiobook Player (which is not open source as a heads up). Both of those allow me to download books wherever I am.
If you have no interest in running a Plex server neither of them are suitable for you, but I just wanted to mention this solution in case you're not dead set on using Calibre. Plex has a cost factor which may come into play (an obvious non starter for some). I have had Plex Pass for a long time so it was natural for me to use it, but might not be for you.