Support the Multi-part-upload API in create s3object to allow the upload of files >5GB

Open rhmoult opened this issue 7 years ago • 1 comments

Hello! I'm excited to try awless, but one of my early use cases is failing. I tried to upload a 6.6 GB file to an S3 bucket in us-east-1 and got the following message: KO create s3object EntityTooLarge: Your proposed upload exceeds the maximum allowed size

Grepping through your code didn't reveal this message in your git repo. Any idea what sets the maximum allowed size? I ask because I didn't run into similar issues with aws-cli with similar sized files in the past.

Jun 20 '18 18:06 rhmoult

Hi, Thanks for your feedback. This is not an error throw by awless but by AWS servers. Indeed, they do not support uploads larger than 5GB in a single upload (what we are currently doing in awless). When you have a file from 5GB to 5TB, you need to use the multi-part-upload API (cf https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html)

Thus as this is not yet implemented in awless, for now you can not upload files bigger than 5GB.

Jun 21 '18 07:06 fxaguessy