nextstrain
[aws batch] Progress meter for build dir upload/download
It'd be nice to add a tqdm progress meter for build dir upload/download when running nextstrain build --aws-batch. I think this is only worth it if relatively easy (which it seems like it would be). ~Might be easier after implementing #83.~
(Three years later) this would still be incredibly useful. I'm currently uploading data for a job and have been looking at the same line for an hour now...
zipping: path/to/5/Gb/file
Nod. Feel free to implement this!
A few questions for you about what would be most useful.
Do you want per-file progress? total workdir progress? both? There are different considerations and challenges for each.
For total workdir progress, the easiest unit is "files" (e.g. "uploaded 13/42 files") not "bytes". Is "files" useful, or is "bytes" what really matters because it better correlates with total upload time?
For per-file progress, it'll be harder to add a progress meter because writing happens in library code outside of our direct control. Should be doable, but needs some thinking.