In recent years, many geodata providers have been exploited by multiple libraries - that use their REST APIs - which are usually written in #Python. That's the case of multiple #Copernicus archives. Most of them use the same multi-threaded '#requests' and '#tqdm' packages below. I wonder how all of them may suffer the same problem of hanging forever on massive downloads. Come on, guys, it is not rocket science. That's really annoying... #rant
Feature request: could #pandas and #tqdm library work together so I can see estimated runtime in my browser tab title? So I can easily see how much time I have for other tasks?
I am looking for a parallelized pipeline system in #Python. Basically a build system lile #SCons but without the files as intermediary step, all in memory. So for example I'd like to read some data files, extract metadata from them, then save that metadata (with :gitannex: #gitAnnex). Along the way there might be other branches of logic that could need parallelization.
Ideally with progress visualisation.
Is there something like this in #Python or do I have to roll my own?
@folkerschamel@birnim Yes in such cases the ETA estimation is useless, sure. But take #tqdm or #pythonRich for example. Both just try their best at estimating an eta while iterating over a sequence (or more complex structure. If ETA it's wrong, okay. But it's a huge difference between knowing if theres 10000 steps or 100 needed. Any kind of progress display is better than none. Unfortunately you can't even query SCons reliably while it's running how many steps are left... No watching wc -l...