s5cmd icon indicating copy to clipboard operation
s5cmd copied to clipboard

ExpiredToken error when `cp` large dataset

Open MqllR opened this issue 2 years ago • 1 comments

Some context:

  • large dataset: > 4TB, million of files
  • s5cmd running in AWS EKS with assumeRole

I'm facing intermittent problem to upload or download with the following error:

MultipartUpload: upload multipart failed upload id: xxx-- caused by: ExpiredToken: The provided token has expired. status code: 400

After looking at the code, there is an explicit deny for retrying this kind of error.

The PR introducing this code seems to tackle another kind of issue. So I'm wondering if that would make sense to revisit the s3 session management.

MqllR avatar Nov 09 '23 16:11 MqllR

boomp

tooptoop4 avatar Sep 27 '24 09:09 tooptoop4