agent
agent copied to clipboard
Downloading a large number of artifacts from S3 is significantly slower than uploading
Presently, ArtifactDownloader creates an S3Downloader (and thus an s3.S3 client) per artifact.
This adds an s3:ListObjects request in the path of each artifact download as a preflight test for the credentials resulting in higher latency between real download requests.
TODO:
- [ ] Examine whether this preflight request is needed or if we can catch the credential errors elsewhere from the real request and apply the same error handling
- [ ] Investigate whether the S3Downloader instance (or at least the underlying s3.S3 client) can be safely shared between multiple download requests