Alias blobstore breaks multipart uploads to S3

[jrtcppv]

Using the following s3proxy.conf for an AWS S3 bucket:

s3proxy.endpoint=http://0.0.0.0:8085
s3proxy.authorization=none
jclouds.provider=aws-s3
jclouds.identity=####################
jclouds.credential=####################################
jclouds.region=us-east-2

I am able to complete a multipart upload successfully with aws s3 cp ~/big_file.mp4 s3://test-bucket --endpoint-url http://localhost:8085

However, when I enable alias-blobstore and bucket-locator:

s3proxy.endpoint=http://0.0.0.0:8085
s3proxy.authorization=none
jclouds.provider=aws-s3
jclouds.identity=####################
jclouds.credential=####################################
jclouds.region=us-east-2
s3proxy.alias-blobstore.project-1=test-bucket
s3proxy.bucket-locator.1=project-1

And attempt to upload with aws s3 cp ~/big_file.mp4 s3://project-1 --endpoint-url http://localhost:8085 I receive the following error after uploading seemingly all parts:

upload failed: ../../big_file.mp4 to s3://project-1/big_file.mp4 An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation: The specified key does not exist.

Not sure if this is relevant but I am invoking s3proxy using the docker container with the latest tag, like so:

docker run --rm -v ~/s3proxy.conf:/s3proxy.conf --publish 8085:8085 --entrypoint /bin/bash andrewgaul/s3proxy -c "java -jar /opt/s3proxy/s3proxy --properties /s3proxy.conf"

[jrtcppv]

I just tried with the bucket locator removed, so just using the alias-blobstore is sufficient to reproduce:

s3proxy.endpoint=http://0.0.0.0:8085
s3proxy.authorization=none
jclouds.provider=aws-s3
jclouds.identity=####################
jclouds.credential=####################################
jclouds.region=us-east-2
s3proxy.alias-blobstore.project-1=test-bucket

[gaul]

@timuralp could you look into this?

[msayan-vcbs]

hey, we are currently having this issue. we are trying to use s3proxy to help in a migration from one bucket to another, and aliasing was a key to that.

we turned on log = trace and were able to identify where this is going wrong. the multipart upload works ok, but when it tries to ListMultiParts, the code gets a big mangled.

As you can see, the converted ListMultiParts is using the Alias of the bucket in the HTTP GET call, not the actual bucket. For all the other operations, like PUT, it's using the real bucket. This obviously will fail and cause the entire upload to be Aborted. Any ideas on how to fix? I'd love to contribute and get it done.

S3Proxy-Jetty-83 o.j.l.s.f.RegionIdToURIFromConfigurationOrDefaultToProvider:56 |::] config key jclouds.region.us-east-1.endpoint not present, defaulting to MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier{delegate={endpoint=https://s3.us-east-1.amazonaws.com}, duration=60, unit=SECONDS}                                                                                                                                                                                                                            

S3Proxy-Jetty-83 o.j.r.i.RestAnnotationProcessor:47 |::] adding filter org.jclouds.aws.s3.filters.AWSRequestAuthorizeSignatureV4@7002db2d from annotation on org.jclouds.aws.s3.AWSS3Client                                                          

S3Proxy-Jetty-83 o.j.r.i.InvokeHttpMethod:47 |::] << converted ListMultipartParts to GET https://**ALIAS**.s3.us-east-1.amazonaws.com/chunk?uploadId=foo- HTTP/1.1```

[gaul]

It looks like AliasBlobStore should override listMultipartUpload. You should try modifying AliasBlobStoreTest.testAliasMultipartUpload to reproduce the symptom to confirm the problem before including the suggested fix.