[WIP] Fixing composite aggregations with correct parameters

[VachaShah]

Description

Introduced classes for composite aggregations for terms, datehistogram, histogram and geotilegrid. I looked in the server code from https://github.com/opensearch-project/OpenSearch/tree/main/server/src/main/java/org/opensearch/search/aggregations/bucket/composite.

Issues Resolved

Fixes #957

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[VachaShah]

I am working on fixing/improving the test I added in this PR.

[VachaShah]

I am wondering if I missed something in the code or I am using it incorrectly in tests 🤔 Looking through server code, it looks like the code is matching but could use another pair of eyes. @dblock @reta

[dblock]

I am wondering if I missed something in the code or I am using it incorrectly in tests 🤔 Looking through server code, it looks like the code is matching but could use another pair of eyes. @dblock @reta

Add a working sample to samples, maybe start an aggregations guide if we don't have anything n that, this way you know whether the code works?

Also do check what's available in https://github.com/opensearch-project/opensearch-api-specification for this, good opportunity to fix/add as we have an incoming code generator in #366.

[reta]

Thanks @VachaShah , I think it looks good model wise

[VachaShah]

I am wondering if I missed something in the code or I am using it incorrectly in tests 🤔 Looking through server code, it looks like the code is matching but could use another pair of eyes. @dblock @reta

Add a working sample to samples, maybe start an aggregations guide if we don't have anything n that, this way you know whether the code works?

Also do check what's available in https://github.com/opensearch-project/opensearch-api-specification for this, good opportunity to fix/add as we have an incoming code generator in #366.

I will make sure to add samples and guides and will look through the specs as well!

[VachaShah]

Thanks @VachaShah , I think it looks good model wise

Thanks @reta! Its weird why the test that I added doesn't fail here, it fails for me on my local 🤔

[reta]

Thanks @reta! Its weird why the test that I added doesn't fail here, it fails for me on my local 🤔

Hm ... I am not sure, could look later on if you need a hand!

[VachaShah]

Thanks @reta! Its weird why the test that I added doesn't fail here, it fails for me on my local 🤔

Hm ... I am not sure, could look later on if you need a hand!

Thank you @reta! Yeah I still have test failures on my local but the workflows here are green, my failure is:

org.opensearch.client.opensearch.integTest.restclient.SearchRequestIT > shouldReturnSearchResultsWithCompositeAgg FAILED
    org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [x_content_parse_exception] [1:95] [composite] failed to parse field [sources]
        at __randomizedtesting.SeedInfo.seed([8D5592150CC0C93:3514B0F383EF841C]:0)
        at app//org.opensearch.client.transport.Endpoint.exceptionConverter(Endpoint.java:102)
        at app//org.opensearch.client.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:312)
        at app//org.opensearch.client.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:148)
        at app//org.opensearch.client.opensearch.OpenSearchClient.search(OpenSearchClient.java:1381)
        at app//org.opensearch.client.opensearch.integTest.AbstractSearchRequestIT.shouldReturnSearchResultsWithCompositeAgg(AbstractSearchRequestIT.java:88)
        at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)

This is against OpenSearch 2.11.0. I will continue debugging into this.

[dblock]

@VachaShah One possible explanation would be that we're not actually running this integration test in CI/CD. Will you please try to make a change that shows which/how many tests are run? I see Task :java-client:integrationTest in the logs, but it doesn't show more detail.

[VachaShah]

@VachaShah One possible explanation would be that we're not actually running this integration test in CI/CD. Will you please try to make a change that shows which/how many tests are run? I see Task :java-client:integrationTest in the logs, but it doesn't show more detail.

Yeah thats what is happening I think. Running ./gradlew :java-client:integrationTest --info:

> Task :java-client:integrationTest
Caching disabled for task ':java-client:integrationTest' because:
  Build cache is disabled
Task ':java-client:integrationTest' is not up-to-date because:
  Value of input property 'systemProperties' has changed for task ':java-client:integrationTest'
Starting process 'Gradle Test Executor 1'. Working directory: /home/ubuntu/opensearch-java/java-client Command: /usr/lib/jvm/jdk-11.0.13+8/bin/java -Dhttps=true -Dorg.gradle.internal.worker.tmpdir=/home/ubuntu/opensearch-java/java-client/build/tmp/integrationTest/work -Dorg.gradle.native=false -Dpassword=admin -Dtests.awsSdk2support.domainHost -Dtests.awsSdk2support.domainRegion=us-east-1 -Dtests.awsSdk2support.serviceName=es -Dtests.security.manager=false -Duser=admin @/home/ubuntu/.gradle/.tmp/gradle-worker-classpath1009533041375880968txt -Xmx512m -Dfile.encoding=UTF-8 -Duser.country -Duser.language=en -Duser.variant -ea worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1'
Successfully started process 'Gradle Test Executor 1'

org.opensearch.client.opensearch.integTest.aws.AwsSdk2BulkRequestIT > testBulkRequest SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testAsyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testSyncAsyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testAsyncAsyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testSyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testDoubleWrappedException SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SecurityIT > testUnAuthorizedException SKIPPED
Finished generating test XML results (0.011 secs) into: /home/ubuntu/opensearch-java/java-client/build/test-results/integrationTest
Generating HTML test report...
Finished generating test html results (0.016 secs) into: /home/ubuntu/opensearch-java/java-client/build/reports/tests/integrationTest

Ah I think the tests are only running from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java/org/opensearch/client and not from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java11/org/opensearch/client/opensearch/integTest

[reta]

Ah I think the tests are only running from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java/org/opensearch/client and not from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java11/org/opensearch/client/opensearch/integTest

Thanks @VachaShah , I used to always run ./gradlew check locally, which runs all tests, but indeed GA workflows use separate tasks, should be fixed by https://github.com/opensearch-project/opensearch-java/pull/968

[VachaShah]

@dblock @reta This is ready for review. I added samples and guides as well.

[opensearch-trigger-bot[bot]]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-967-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 cc6c273ded052be68810bc2907bceabda6572eb1
# Push it to GitHub
git push --set-upstream origin backport/backport-967-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-967-to-2.x.

[dblock]

will need a manual backport pls