TransferManager copy not using endpoint override in source object metadata request's Host header

[bpiper]

Describe the bug

When copying from and to a single bucket hosted on a local Minio server (e.g. localhost:9100) using S3TransferManager.copy, it's getting a 404 when trying to get the source object metadata, which is due to an incorrect Host header in the request.

Expected Behavior

Metadata for the source object should be successfully retrieved and followed by a successful PUT to create the destination object.

Current Behavior

The SDK sends something like the following request (auth/signing headers not included as they're irrelevant) to Minio:

HEAD /74123c17-e8b5-413d-9b20-386dc4afbdef HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: CRTS3NativeClient/0.1.x
X-Amz-Date: 20220821T233250Z

This results in a 404, presumably because Minio doesn't identify with the domain in the Host header, which should really have been my-bucket.localhost:9100.

Reproduction Steps

Just a standard within-bucket object copy using S3TransferManager.copy and overriding the endpoint to a local Minio server using http (behaviour may or may not differ if using https).

Possible Solution

I couldn't track down the code that actually executes the operation, so no clue currently. This just seems like an oversight with the addition of endpoint overriding (i.e. not honouring that override in the Host header and instead defaulting to the standard S3 endpoint for the default region).

Additional Information/Context

No response

AWS Java SDK version used

2.17.236

JDK version used

OpenJDK 64-Bit Server VM Temurin-11.0.16+8 (build 11.0.16+8, mixed mode)

Operating System and version

Ubuntu 20.04

[yasminetalby]

Hello @bpiper ,

Thank you very much for your submission. I saw your comment in #3350 which is currently under investigation. Is this behavior something you experience in more recent version of the SDK? Best,

Yasmine.

[bpiper]

I don't think I can test it in a later version because of that bug, i.e. it wouldn't even be able to connect to a local Minio server, let alone send an object metadata request. I can definitely re-test this when that issue is fixed though. I may also make another attempt to locate the code that's actually responsible for the metadata request.

[bpiper]

Ok, I think I tracked down the problem in the S3 CRT client code and raised this issue - https://github.com/awslabs/aws-c-s3/issues/206... so I guess the resolution here will be to update to a new version of that when the fix is published.

[yasminetalby]

Hello @bpiper ,

Thank you very much for your collaboration and research work. We have reached out to the CRT team about this behavior as well. I will update you here and on the submission regarding the connection issue with MinIO.

Best,

Yasmine

[yasminetalby]

Hello @bpiper ,

Could you please provide a reproducible code sample that leads to the above behavior? Does using the S3TransferManager with S3 CRT client prior to the change introduced in #166 that you have mentioned in your ticket to the CRT team changes the experienced behavior?

Best,

Yasmine

[bpiper]

@yasminetalby the issue can be reproduced with the following code. This assumes that Minio is running at localhost:9100, the correct credentials are filled in, and that a bucket named test-bucket exists.

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.transfer.s3.CopyRequest;
import software.amazon.awssdk.transfer.s3.S3ClientConfiguration;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.UploadRequest;

import java.net.URI;

public class TransferManagerCopyTest {

    public static void main(String... args) throws Exception {
        String testBucketName = "test-bucket";
        String accessKeyID = "******";
        String secretAccessKey = "******";
        S3TransferManager tm =
                S3TransferManager.builder().s3ClientConfiguration(S3ClientConfiguration.builder()
                        .region(Region.US_EAST_1)
                        .endpointOverride(URI.create("http://localhost:9100"))
                        .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKeyID, secretAccessKey)))
                        .build()).build();
        tm.upload(UploadRequest.builder()
                        .putObjectRequest(PutObjectRequest.builder().bucket(testBucketName).key("test-file").build())
                        .requestBody(AsyncRequestBody.fromBytes("Hello World".getBytes()))
                        .build())
                .completionFuture()
                .get();
        tm.copy(CopyRequest.builder().copyObjectRequest(CopyObjectRequest.builder()
                                .sourceBucket(testBucketName)
                                .sourceKey("test-file")
                                .destinationBucket(testBucketName)
                                .destinationKey("copy-of-test-file")
                                .build())
                        .build())
                .completionFuture()
                .get();
    }
}

I should note that I also tried reproducing this by using an actual Amazon S3 bucket in us-east-1, but I got a segfault from the C code, probably due to some unrelated bug that I don't fancy chasing up right now.

There was no way (that I know of) to override the S3 endpoint prior to that PR, as the endpoint override support was added after that, so I doubt it's possible to run the above code successfully on any version of the SDK.

[yasminetalby]

Hello @bpiper ,

Thank you very much for providing the code sample and for your collaboration. We have brought this up to the CRT Team and are working in collaboration with them to solve this issue. I will post an update here once the behavior is fixed.

Best,

Yasmine.

[zoewangg]

We are working on writing our own implementation of copy in the SDK, and this should be fixed via https://github.com/aws/aws-sdk-java-v2/pull/3403

[zoewangg]

Just a quick update, the PR has been merged but we have not released it yet. We will post it here once the change is released

[opera-pengel]

Hello AWS team! I have the similar issue like @bpiper. I use LocalStack for testing and copy files within same bucket. And the code is the same as above except port number.

The used versions:

• AWS Java SDK 2.17.276
• JDK 17.0.2, 64-Bit
• macOS Big Sur 11.6

The error:

 software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 404, Request ID: LHSUAYM580FQB2MUEHCFOW6JRM770EX4S3DYLGXSZ5LTLW4T5TSM, Extended Request ID: MzRISOwyjmnupA2EEBCF47931F8A67/JypPGXLh0OVFGcJaaO3KW/hRAqKOpIEEp)
            at app//software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
            at app//software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
            at app//software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
            at app//software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
            at app//software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:264)
            at app//software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
            at [email protected]/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
            at [email protected]/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
            at [email protected]/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
            at app//software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
            at app//software.amazon.awssdk.services.s3.internal.crt.S3CrtDataPublisher.handleTerminalEvent(S3CrtDataPublisher.java:110)
            at app//software.amazon.awssdk.services.s3.internal.crt.S3CrtDataPublisher.flushBuffer(S3CrtDataPublisher.java:135)
            at app//software.amazon.awssdk.services.s3.internal.crt.S3CrtDataPublisher.notifyStreamingFinished(S3CrtDataPublisher.java:68)
            at app//software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:93)
            at app//software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:77)
            at app//software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:20)

The additional info:

Also couldn't track down the code that actually executes the operation. However noticed that in S3MetaRequestResponseHandlerNativeAdapter.onFinished(...) return errorCode = 14343. Haven't found this code in libs, possibly return from native method.

Would you have a chance to write does fix described above cover this case also?

[zoewangg]

We have released a fix in #3724. Could you try with the latest version?

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[bpiper]

@zoewangg yes, I can confirm that it appears to be working now on 2.19.31. Thanks a lot for getting this fixed.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.