aleph icon indicating copy to clipboard operation
aleph copied to clipboard
verified publisher icon alephdata

BUG: unable to upload a large amount of files through the UI

Open stchris opened this issue 2 years ago • 3 comments

Describe the bug Uploading a large amount of files through the UI can make an Aleph installation freeze and stop processing.

To Reproduce Steps to reproduce the behavior:

  1. Go to an investigation
  2. Click on 'Upload documents'
  3. Select a large amount of files (thousands)
  4. Aleph will choke and stop processing them.

Expected behavior Aleph is either able to show and enforce a limit to the number of files being uploaded or it is able to queue the files so that they are processed correctly.

Aleph version 3.15.1

Screenshots

Additional context The current workaround is to use alephclient

stchris avatar Oct 10 '23 13:10 stchris

Thanks for helping us with this issue, Chris! Extra context can be found in https://github.com/alephdata/aleph/issues/3383 . "Uploading a folder with 8K files worked, but not another with 300K files"

andresmrm avatar Oct 10 '23 14:10 andresmrm

Options here:

  1. Document a limit (1,000 - 5,000?) files max, possibly size could have an impact.
  2. Try to fix the UI to handle larger file sets (harder, more time consuming).
  3. Limit uploads to single files at a time

Rosencrantz avatar Oct 24 '23 10:10 Rosencrantz

If 2 is too complex for now, I would prefer 1. Including a warning in the upload dialog about this limitation and a link to crawldir doc or tutorial as an alternative.

andresmrm avatar Oct 24 '23 10:10 andresmrm