NCEAS
Limit the number of files a user can add at once
Describe the feature you'd like In the metadata editor a user is allowed to upload as many files and they can select in the upload dialog. We had a user try to upload 1000 files and their upload stalled and failed. Tested this on a development environment with 189 files chosen and I can see that all files are being uploaded concurrently (but very slowly)
Is your feature request related to a problem? Please describe. The problem is that this creates a negative user experience due to the fact that if there are failures uploading there is no way to recover the edited dataset from this. What ends up happening if they are able to save is that the metadata has contains references to data files that failed during upload.
Also, if the upload continues overnight, their token could expire during that time at which point they will not be able to recover.
Additional context Is it possible to change the logic or make configurable the following:
- limit how many files can be uploaded at once in the metadata editor
- limit the concurrent uploads to something reasonable. This means if a user chooses 50 files then maybe only allow 10 (or fewer) concurrent uploads. This will actually make it look like somethingis happening because the uploads wil seem like they are moving along and not stalling.
Example
Here I uploaded 189 files top files are stuck on 100% but many of the others have completed but you can't tell unless you scroll.
All great points. I think allowing them to add unlimited numbers of files is still good, but I agree we should limit the number of concurrent transfers, and provide means for dealing with resuming when a large number is requested but times out. As we use the underlying browser file selection box for choosing the files to be uploaded, I'm not sure how much control we have of the number selected. But once we get the selected list, we should be able to more intelligently handle the selection (e.g., only start 10 transfers at a time, and work though all of them in batches). @robyngit @rushirajnenuji thoughts on what might be possible here?
I think addressing this issue could help with missing resource maps that are the result of network instability as seen in #2134
From slack:
we had another PI who mentioned that his upload of about 488 data files was stuck in loading when he was trying to submit a dataset. Anywhere around 300+ files at a time starts to give PIs some trouble
