[BUG] Issue with Getting Cluster Stats Using AwsSdk2Transport, throws MissingRequiredPropertyException: Missing required property 'ClusterOperatingSystemName.name'

[victorekpo]

What is the bug?

There is an issue when trying to get cluster stats using the cluster().stats() method while utilizing the AwsSdk2Transport. The following exception is thrown: org.opensearch.client.util.MissingRequiredPropertyException: Missing required property 'ClusterOperatingSystemName.name'

How can one reproduce the bug?

1. Obtain an instance of the OpenSearch client using the AwsSdk2Transport.
2. Execute the code example provided below.

Code Example:

final ClusterStatsResponse stats = opClient.cluster().stats();  // <- Exception here

### What is the expected behavior?
The code should execute successfully and return parts of the information from the JSON response of the cluster stats API.

### What is your host/environment?
Java 17, Spring Boot 3.3.5, OpenSearch Java Client 2.13.0, AWS SDK v2 2.21.21

### Do you have any additional context?
This only happens when using AwsSdk2Transport and not with a regular Apache http client. 

[Xtansia]

The spec still shows the name field as required. It's likely that the Amazon OpenSearch Service or Serverless doesn't return this field. So the spec will also need updated. As far as fixing this in the client this is a relatively simple fix of removing the relevant required check. Eventually these classes will be generated from the spec.

[victorekpo]

Thank you @Xtansia, I did notice that the cluster name does come back when I use a regular Apache HTTP client with a request interceptor, it returns the ClusterResponse accordingly. Thanks for looking into this, please let me know when you can work on a fix.

[Xtansia]

@victorekpo Just to confirm, are you saying it returns correctly when connecting to the exact same cluster?

[victorekpo]

@Xtansia yes, it returns correctly when not using AwsSdk2Transport, the error is specific to using that transport.

[victorekpo]

@Xtansia Let me know if you need the full stack trace, when I tested locally I saw that AwsSdk2Transport tries to decode the response and it fails, here is part of it:

o.o.c.u.MissingRequiredPropertyException: Missing required property 'ClusterOperatingSystemName.name' at o.o.c.u.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:89) at o.o.c.o.c.s.ClusterOperatingSystemName.<init>(ClusterOperatingSystemName.java:60) at o.o.c.o.c.s.ClusterOperatingSystemName.<init>(ClusterOperatingSystemName.java:50) at o.o.c.o.c.s.ClusterOperatingSystemName$Builder.build(ClusterOperatingSystemName.java:137) at o.o.c.o.c.s.ClusterOperatingSystemName$Builder.build(ClusterOperatingSystemName.java:107) at o.o.c.j.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:98) at o.o.c.j.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:60) at o.o.c.j.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:343) at o.o.c.j.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:308) at o.o.c.j.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) at o.o.c.j.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:81) at o.o.c.j.ObjectDeserializer.deserialize(ObjectDeserializer.java:185) at o.o.c.j.ObjectDeserializer.deserialize(ObjectDeserializer.java:146) at o.o.c.j.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) at o.o.c.j.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:91) at o.o.c.j.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:55) at o.o.c.j.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:81) at o.o.c.j.ObjectDeserializer.deserialize(ObjectDeserializer.java:185) at o.o.c.j.ObjectDeserializer.deserialize(ObjectDeserializer.java:146) at o.o.c.j.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) ... 69 frames truncated

[Xtansia]

@victorekpo That is strange, could you provide a couple snippets of how you configure the two versions of the client?

[victorekpo]

Sure

With the AWS Sdk2 Transport

@Bean
    public OpenSearchClient openSearchJavaClient() throws Exception {
        LOGGER.info("Initializing Java client for AWS OpenSearch");

        if (StringUtils.isBlank(url)) {
            LOGGER.info("Configuring AWS OpenSearch Java Client by resolving route53={} lookup", route53Name);
           url = AWSOpenSearchDNSResolverTask.resolveDNSName(route53Name, awsOpenSearch());
        } else {
            LOGGER.info("Configuring AWS OpenSearch Java Client with URL {}", url);
        }
        SdkHttpClient sdkHttpClient = ApacheHttpClient.builder().build();
        OpenSearchTransport transport = new AwsSdk2Transport(
                sdkHttpClient,
                Objects.requireNonNull(extractHostnameFromStringUrl(url)),
                AWS_ES_SERVICE_NAME,
                Region.US_EAST_1,
                AwsSdk2TransportOptions
                        .builder()
                        .addHeader("app-name", headerAppName)
                        .setCredentials(DefaultCredentialsProvider.create())
                        .build()
        );
        return new OpenSearchClient(transport);
    }

With Apache Transport

@Bean
    public OpenSearchClient openSearchJavaClient() throws Exception {
        LOG.info("Initializing OpenSearch Java client");
        String scheme = "http";

        PoolingAsyncClientConnectionManagerBuilder connectionManagerBuilder = PoolingAsyncClientConnectionManagerBuilder.create();

        if (Boolean.TRUE.equals(httpsEnabled)) {
            LOG.debug("SSL enabled for OpenSearch Java client");

            scheme = "https";
            TlsStrategy tlsStrategy = buildTlsStrategy();
            connectionManagerBuilder.setTlsStrategy(tlsStrategy);
        }

        PoolingAsyncClientConnectionManager connectionManager = connectionManagerBuilder.build();

        HttpHost host = new HttpHost(scheme, url, port);
        ApacheHttpClient5TransportBuilder builder = ApacheHttpClient5TransportBuilder.builder(host);

        builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
                .setConnectionManager(connectionManager)
                .addRequestInterceptorFirst(openSearchRequestInterceptor())
        );

        builder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setResponseTimeout(Timeout.ofMilliseconds(socketTimeout)));

        OpenSearchTransport transport = builder.build();
        LOG.info("OpenSearch Java client created for {} with host={} and port={}", url, host, port);

        return new OpenSearchClient(transport);
    }

[Xtansia]

@victorekpo Thanks for providing those, could you also provide the openSearchRequestInterceptor() configuration and what version of OpenSearch cluster you're connecting to or if it's Amazon Serverless?

[victorekpo]

Sorry for the late reply on this, for some reason im not seeing my notifications. @Xtansia , we are on the latest version of Opensearch for the cluster, which is 2.17. The openSearchRequestInterceptor can be ignored since its just adding an extra header, we are going to remove this on our end as well.