Using query string as payload when generating content hash is incompatible with Elasticsearch service

[senojj]

https://github.com/aws/aws-sdk-java/blob/ffdf66cf932e60d9375a19967ad30d0ab48ae84f/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AbstractAWSSigner.java#L341

When signing a request, if the request method is "POST", and the request content is null, and the request has existing query parameters, then the Signer algorithm will use the encoded query string as the content segment of the generated canonical request string.

It would appear that the Elasticsearch service does not do this on the service side when generating the hash for comparison. Instead, it appears that the Elasticsearch service will leave the content segment empty, and just populate the query string segment of the generated canonical request string. I can only speculate though, as this portion of the code is not visible to end users.

[senojj]

To elaborate further, when signing a request such as:

POST /some-index/_refresh?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true HTTP/1.1
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_232)
Host: some-endpoint.us-east-1.es.amazonaws.com
Accept-Encoding: gzip,deflate

After sending the actual signed request, the response received from the Elasticsearch service is the following:

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

[debora-ito]

@jjware is this the same case of https://github.com/aws/aws-sdk-java/issues/2078?

[senojj]

@debora-ito yes, this is the same issue that the user in #2078 was experiencing. However, this issue has nothing to do with the Elasticsearch Java client, as I have described above. I can create a SignableRequest using the Java AWS SDK directly, and reproduce the issue that I have described.

[debora-ito]

Can you send us a reproducible sample code?

[senojj]

Following is a minimum viable example that will produce an error response from the Elasticsearch service with the message "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.":

import com.amazonaws.DefaultRequest;
import com.amazonaws.auth.AWS4Signer;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.http.HttpMethodName;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.NameValuePair;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;

import java.net.URI;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;

public class Main {
    private static String SERVICE = "es";
    private static String REGION = "us-east-1";
    private static HttpMethodName METHOD = HttpMethodName.POST;
    private static String ENDPOINT = "https://vpc-endpoint.us-east-1.es.amazonaws.com/ndx/_refresh?ignore_throttled=false&ignore_unavailable=false";

    public static void main(String[] args) {
        try {
            URIBuilder uriBuilder = new URIBuilder(ENDPOINT);
            AWSCredentialsProvider awsCredentialsProvider = new DefaultAWSCredentialsProviderChain();

            AWS4Signer signer = new AWS4Signer();
            signer.setServiceName(SERVICE);
            signer.setRegionName(REGION);

            DefaultRequest<?> signableRequest = new DefaultRequest<>(SERVICE);

            signableRequest.setHttpMethod(METHOD);
            signableRequest.setEndpoint(URI.create(new HttpHost(uriBuilder.getHost()).toURI()));
            signableRequest.setResourcePath(uriBuilder.build().getRawPath());

            Map<String, List<String>> parameters = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);

            for (NameValuePair nvp : uriBuilder.getQueryParams()) {
                parameters.computeIfAbsent(nvp.getName(), k -> new ArrayList<>()).add(nvp.getValue());
            }
            signableRequest.setParameters(parameters);

            signer.sign(signableRequest, awsCredentialsProvider.getCredentials());

            HttpPost request = new HttpPost(ENDPOINT);

            request.setHeaders(signableRequest.getHeaders().entrySet().stream()
                    .map(es -> new BasicHeader(es.getKey(), es.getValue())).toArray(Header[]::new));

            CloseableHttpClient client = HttpClients.custom().build();

            CloseableHttpResponse response = client.execute(request);

            if (response.getStatusLine().getStatusCode() != 200) {
                response.getEntity().writeTo(System.out);
            } else {
                System.out.println("SUCCESS");
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

The issue can be "fixed" by simply adding the line:

signableRequest.setContent(new ByteArrayInputStream(new byte[0]));

This works because as long as the content of the SignableRequest is not null, the query string parameters will be used in the query segment of the canonical request string produced within the Signer's signing process. If the content of the SignableRequest is null, the query string parameters will be encoded and used as the contentSha256 segment of the canonical request string.

Based on this finding, I assume that the AWS Elasticsearch service is not following the same algorithm when generating the signature on its side for comparison. It would appear that the Elasticsearch service never follows the null content path when generating the signature.

[senojj]

My primary concern here, is that the algorithm used by the Java SDK Signer does not follow the AWS Signature Version 4 specification when the request method is "POST" and a null value is encountered for the SignableRequest::getContent() value and the request has existing query parameters. I've looked at AWS SDKs in other languages, and they do not take the Java SDK's approach in this specific condition. Does this path exist in the Java SDK as a relic of the past? If so, can it be removed entirely?

[debora-ito]

Hi @jjware I don't have context on why there's that specific condition in the signer. I'll discuss this case with the team, will update here after the discussion.

[peter-ponzel]

hi, it's been a while and I wanted to ask if there is any update on this topic? best regards

[zoewangg]

Apologies for losing track of this issue.

As a workaround, you try specifying an empty byte array as content.

signableRequest.setContent(new ByteArrayInputStream(new byte[0]));

[debora-ito]

We don't have plans to change this in v1 before going into Maintenance Mode, so I'll go ahead and close this issue.

A workaround is to set an empty byte array as request content, see an example in the previous comment.

Reference:

• Announcing end-of-support for AWS SDK for Java v1.x effective December 31, 2025 - blog post

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.