agent icon indicating copy to clipboard operation
agent copied to clipboard

Downloading a large number of artifacts from S3 is significantly slower than uploading

Open keithduncan opened this issue 4 years ago • 0 comments

Presently, ArtifactDownloader creates an S3Downloader (and thus an s3.S3 client) per artifact.

This adds an s3:ListObjects request in the path of each artifact download as a preflight test for the credentials resulting in higher latency between real download requests.


TODO:

  • [ ] Examine whether this preflight request is needed or if we can catch the credential errors elsewhere from the real request and apply the same error handling
  • [ ] Investigate whether the S3Downloader instance (or at least the underlying s3.S3 client) can be safely shared between multiple download requests

keithduncan avatar Sep 20 '21 02:09 keithduncan