Structure Of Presigned S3 URLS Changed.

[GEverding]

Describe the bug

Some point between 2.17 and 2.23 the format of presigned s3 changed. The new urls have the bucket in the hostname the old version of the sdk has it as a url path. In particular this breaks OCI S3 Compat layer. I don't know if both are valid and if they are should we be able to choose which structure to use? I also don't know if this is just a OCI issue and I need to get them to fix it.

Expected Behavior

I want a way of choosing the structure of presigned urls. Either the bucket in the hostname or bucket in the url path.

Current Behavior

The sdk always generates it as https://bucket.{endpoint}/file it used to be https://{endpoint}/{bucket}/{file}

Reproduction Steps

Generate normal presigned url.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.x

JDK version used

openjdk version "21" 2023-09-19 LTS

Operating System and version

Linux

[debora-ito]

These different styles are called:

• virtual-hosted style - Like https://bucket.s3.us-west-2.amazonaws.com/object
• path style - Like https://s3.us-west-2.amazonaws.com/bucket/object

By default, the SDK will use virtual-hosted style endpoints in regular requests and presigned urls, but will change to path style when (1) the bucket name contains dots (due to issues with SSL certification validation) or (2) the forcePathStyle option is enabled in the ~S3 Client~ S3Presigner#serviceConfiguration. This is not a new behavior, so it's unlikely it was caused by the version upgrade.

If you're still not sure how this affects your use case, please share a sample code showing how you're generating the presigned url and the generated url, and I can take a look.

You can read more about the reason behind the style change in Jeff Barrs's blog post and get more details in the S3 Dev Guide.

[GEverding]

Thanks. So this changed between 2.17 and 2.23 in a really weird way. 2.17 gave me both styles with the same code. 1 client was configured for b2 and another client was configured for oci. Same code just different config (creds and endpoint override). Oci presign generated path and b2 client generated virtual host. No dots in bucket name. Just dashes.

I'll try that config. I didn't find it documented.

[debora-ito]

You may be hitting some weird logic because of the use of a third-party endpoint. The SDK only guarantees a consistent behavior when AWS endpoints are used directly.

Can you share a sample code?

[GEverding]

  val presignerOCI = S3Presigner
    .builder()
    .region(Region.of(config.getString("oci-bucket")))
    .credentialsProvider(
      StaticCredentialsProvider.create(
        AwsBasicCredentials.create(
          config.getString("oci.access_key"),
          config.getString("oci.secret_key")
        )
      )
    )
    .endpointOverride(URI.create(config.getString("oci-endpoint")))
    .build()

    val objectRequest =
      PutObjectRequest.builder.bucket(bucket).key(filename).contentType(mime).build
    val presignRequest = PutObjectPresignRequest.builder
      .signatureDuration(JDuration.ofMinutes(60))
      .putObjectRequest(objectRequest)
      .build
    val presignedRequest = presignerOCI.presignPutObject(presignRequest)

    (presignedRequest.url.toString, presignedRequest.expiration().toString)

Hers the code snippets to create the client and generate the token.

[debora-ito]

Can you give values for config.getString("oci-endpoint") and bucket to exemplify the behavior, and the equivalent url that is generated, for both oci and b2? I'm curious about why the SDK is choosing different styles, and the logic of which style to use is dependent on the actual values of the endpoint and bucket.

A correction of my previous comment: for S3Presigner, the path style configuration needs to be set in S3Presigner#serviceConfiguration, not in the S3 client:

S3Presigner presigner = S3Presigner.builder()
                .serviceConfiguration(S3Configuration.builder()
                        .pathStyleAccessEnabled(true)
                        .build())
                .region(Region.US_WEST_2)
                .build();

According to the OCI documentation, virtual host-style is not supported, so you should use the pathStyleAccess setting. I couldn't find any docs for b2.

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.