build: Kotlin DSL / Convention Plugins / Java Modules

[jjohannes]

Description:

This PR does the full package of Gradle + Java Module updates similar to what we did in hedera-services:

• All Gradle config is now in Kotlin DSL
• Gradle config done in convention plugins
• Latest Gradle Version 8.8
• Update io.grpc:* dependencies to 1.64.0 (to be Java Module compatible)
• Configuration Cache enabled
• Integration tests and example classes are moved into packages to make them Java Module compatible
• Integration tests and samples run on Module Path when executed via Gradle (locally or on CI)
• module-info.java files added to sdk/main, sdk/integrationTest and examples
• Dependencies are declared in module-info.java and scope is defined according to checkAllModuleInfo
• Versions are centrally declared in separate block in sdk-dependency-versions/build.gradle.kts
• Building the sdk-full variant is now fully done in Gradle through a separate sdk-full project that has a separate module-info (exchanging requires grpc.protobuf.lite for requires grpc.protobuf) but otherwise re-uses sources and proto buffs from the sdk/main sources folder.
• The sdk and sdk-full are now both published by the Gradle build in the one publishing run
• Cleanup: remove config files for no longer used build/ci solutions (Built, GitLab)

Manual Testing (for things not covered by the automatic tests):

• I tested that the Android app example still builds and runs
• I tested the publishing by temporarily adding a local repository

Follow ups:

• https://github.com/hashgraph/hedera-sdk-java/issues/1794
• https://github.com/hashgraph/hedera-services/issues/13505 We should adjust the code style rules and tools to what ist done in hedera-services as well. E.g. remove sonar, introduce spotless, decide if you want to keep error prone etc. Since there are some decissions to be made, I would do that as a later follow up - maybe once this is "stable" in hedera-services.

[nathanklick]

@jjohannes Please rebase this PR when you get a chance. The team is ready to review and merge.

[jjohannes]

@nathanklick rebased and ready for review

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[thenswan]

I merged PR related to the docs update (recommend to check it out), and we have a doc on creating a fat/uber JAR based on the user's request. Would like to ask if is it possible for jar creation to be done as a convention plugin (like in hedera-services)?

[thenswan]

Proposing change com.hedera.hashgraph.sdk.integration.test to com.hedera.hashgraph.sdk.test.integration.

[thenswan]

I am seeing a following error, when running an example via Intellij (works fine when running via cmd, maybe it can be fixed by updating gradle, see this discussion):

FAILURE: Build failed with an exception.

* What went wrong:
Configuration cache problems found in this build.

1 problem was found storing the configuration cache.
- Task `:examples:TransferCryptoExample.main()` of type `org.gradle.api.tasks.JavaExec`: cannot serialize object of type 'java.io.PipedInputStream', a subtype of 'java.io.InputStream', as these are not supported with the configuration cache.
  See https://docs.gradle.org/8.6/userguide/configuration_cache.html#config_cache:requirements:disallowed_types

See the complete report at file:///Users/hedera-sdk-java/build/reports/configuration-cache/4ektp07xks1k5f40qn2zxqbfv/2hvheo6cndbpywiqj8ir9e180/configuration-cache-report.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.

BUILD FAILED in 23s

[thenswan]

Running integration tests using configuration file is not working anymore.

[thenswan]

I am not able to run an Android app (but was able to run it from the main branch). Build fail with:

Could not determine the dependencies of task ':example-android:app:compileDebugJavaWithJavac'.
> Could not resolve all task dependencies for configuration ':example-android:app:debugCompileClasspath'.
   > Could not find com.google.protobuf:protobuf-javalite:.
     Required by:
         project :example-android:app > project :sdk
   > Could not find com.esaulpaugh:headlong:.
     Required by:
         project :example-android:app > project :sdk
   > Could not find io.grpc:grpc-stub:.
     Required by:
         project :example-android:app > project :sdk
   > Could not find io.helidon.grpc:io.grpc:.
     Required by:
         project :example-android:app > project :sdk
   > Could not find org.bouncycastle:bcprov-jdk18on:.
     Required by:
         project :example-android:app > project :sdk

[thenswan]

Would be great to add the docs and update current docs which is in the repo. Given the new setup it can be hard for new collaborators to catch up, so docs and guides should be super helpful, for example:

• What is the algorithm for adding new dependency, considering potential changes to gradle/plugins/src/main/kotlin/com.hedera.gradlebuild.patch-modules.gradle.kts
• etc.

[thenswan]

Can you please share how did you perform manual testing (Android app and release process)? 🙏🏻

[thenswan]

Please let me know when there will be a time to re-visit having a separate project for the versions only. Now I am thinking about simplicity (in the context of this project) and not having a separate project for the versions only.

[thenswan]

We should probably adjust the code style rules and tools to what ist done in hedera-services as well. E.g. remove sonar, introduce spotless, decide if you want to keep error prone etc. Since there are some decissions to be made, I would do that as a later follow up - maybe once this is "stable" in hedera-services.

Sounds good, looking forward to it after merging current PR!

[thenswan]

Will get back later on the com.google.code.findbugs:jsr305.

[nathanklick]

@jjohannes Please see comments and questions from the Limechain team. Please rebase this PR. The team is ready to review and merge as soon as it is rebased and comments addressed.

[jjohannes]

@thenswan thank you for the thorough review of this beast. I addressed the issues you found and answered your questions. Hope I did not miss anything. The PR is also rebased to latest main now.

I think many of the answers I gave should move into documentation. I would like to do that as a follow up step. I created https://github.com/hashgraph/hedera-sdk-java/issues/1794.

Please have another look and feel free to test things again. I might still have missed something.

---

Proposing change com.hedera.hashgraph.sdk.integration.test to com.hedera.hashgraph.sdk.test.integration.

Done.

Please let me know when there will be a time to re-visit having a separate project for the versions only.

I think we can keep it as it is now. Feels right to me.

I am seeing a following error, when running an example via Intellij ...

Should work now with the Gradle update.

Running integration tests using configuration file is not working anymore.

Thanks for pointing me at the docs. I didn't find them and did not know how it was supposed to work. Should be fixed now. Can you please test this again.

I am not able to run an Android app (but was able to run it from the main branch)...

Thank you. I am sure I tested the Android app, but seems like I did not do that with the final PR state. It's fixed now.

Can you please share how did you perform manual testing (Android app and release process)? 🙏🏻

• Make sure Gradle tasks assemble, test and checkAllModuleInfo run
• Running some examples
• Running the Android app (but apparently did not test it enough)
• For publishing, I tested running ./gradlew closeAndReleaseSonatypeStagingRepository --dry-run to see which tasks run in which order
• To check which artifacts publishing would produce I temporarily added publishing.repositories.maven("../test-repo") { name = "test"} in com.hedera.gradle.publish.gradle.kts (defines a local folder as repository) and then run ./gradlew publishAllPublicationsToTestRepository

---

I would like to move the the following items to a follow-up if that is okay.

Would be great to add the docs and update current docs which is in the repo. Given the new setup it can be hard for new collaborators to catch up, so docs and guides should be super helpful, for example:

• Add more docs for the Gradle build (in particular for modifying/adding dependencies)

...we have a doc on creating a fat/uber JAR...

• Think about making uber Jar creation easier

mergeJar("com.google.code.findbugs:jsr305")

• See what can be done to get rid of this

Follow up: https://github.com/hashgraph/hedera-sdk-java/issues/1794

[thenswan]

Taskfile.yml: publish command is irrelevant -- we can remove it.

[thenswan]

How can we address this:

> Configure project :sdk
The 'sonarqube' task depends on compile tasks. This behavior is now deprecated and will be removed in version 5.x. To avoid implicit compilation, set property 'sonar.gradle.skipCompile' to 'true' and make sure your project is compiled, before analysis has started.
The 'sonar' task depends on compile tasks. This behavior is now deprecated and will be removed in version 5.x. To avoid implicit compilation, set property 'sonar.gradle.skipCompile' to 'true' and make sure your project is compiled, before analysis has started.

> Configure project :sdk-full
The 'sonarqube' task depends on compile tasks. This behavior is now deprecated and will be removed in version 5.x. To avoid implicit compilation, set property 'sonar.gradle.skipCompile' to 'true' and make sure your project is compiled, before analysis has started.
The 'sonar' task depends on compile tasks. This behavior is now deprecated and will be removed in version 5.x. To avoid implicit compilation, set property 'sonar.gradle.skipCompile' to 'true' and make sure your project is compiled, before analysis has started.

[thenswan]

A couple of questions based on analysis of the output of ./gradlew sdk:dependencies --scan:

1. Does it make sense for compileProtoPath and testIntegrationCompileProtoPath to have all of those dependencies?
2. It is OK that project :sdk (*) is included twice in: testIntegrationCompileClasspath, testIntegrationCompileProtoPath and testIntegrationRuntimeClasspath?
3. Is it OK to have FAILED here:

mainRuntimeClasspath
+--- project :sdk (*)
+--- com.google.guava:guava FAILED
+--- com.google.protobuf:protobuf-javalite FAILED
+--- com.esaulpaugh:headlong FAILED
+--- io.grpc:grpc-stub FAILED
+--- io.helidon.grpc:io.grpc FAILED
+--- org.bouncycastle:bcprov-jdk18on FAILED
+--- org.slf4j:slf4j-api FAILED
+--- com.google.code.gson:gson FAILED
+--- io.grpc:grpc-protobuf-lite FAILED
\--- org.bouncycastle:bcpkix-jdk18on FAILED

4. and here:

errorprone - Error Prone dependencies, will be extended by all source sets' annotationProcessor configurations
+--- com.google.errorprone:error_prone_core:2.21.1
|    +--- com.google.errorprone:error_prone_annotation:2.21.1
|    |    \--- com.google.guava:guava:32.1.1-jre
|    |         +--- com.google.guava:guava-parent:32.1.1-jre
|    |         |    +--- com.google.code.findbugs:jsr305:3.0.2 (c)
|    |         |    +--- org.checkerframework:checker-qual:3.33.0 (c)
|    |         |    \--- com.google.errorprone:error_prone_annotations:2.18.0 -> 2.21.1 (c)
|    |         \--- com.google.guava:failureaccess:1.0.1
|    +--- com.google.errorprone:error_prone_type_annotations:2.21.1
|    +--- com.google.errorprone:error_prone_check_api:2.21.1
|    |    +--- com.google.errorprone:error_prone_annotation:2.21.1 (*)
|    |    +--- com.google.code.findbugs:jsr305:3.0.2
|    |    +--- io.github.eisop:dataflow-errorprone:3.34.0-eisop1
|    |    +--- io.github.java-diff-utils:java-diff-utils:4.0
|    |    |    \--- org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
|    |    +--- com.google.auto.value:auto-value-annotations:1.9
|    |    +--- com.google.errorprone:error_prone_annotations:2.21.1
|    |    +--- com.github.kevinstern:software-and-algorithms:1.0
|    |    +--- com.github.ben-manes.caffeine:caffeine:3.0.5
|    |    |    +--- org.checkerframework:checker-qual:3.19.0 -> 3.33.0
|    |    |    \--- com.google.errorprone:error_prone_annotations:2.10.0 -> 2.21.1
|    |    \--- com.google.inject:guice:5.1.0
|    |         +--- javax.inject:javax.inject:1
|    |         +--- aopalliance:aopalliance:1.0
|    |         \--- com.google.guava:guava:30.1-jre -> 32.1.1-jre (*)
|    +--- org.pcollections:pcollections:3.1.4
|    +--- com.google.guava:guava:32.1.1-jre (*)
|    +--- com.google.auto:auto-common:1.2.1
|    |    \--- com.google.guava:guava:31.0.1-jre -> 32.1.1-jre (*)
|    +--- com.google.code.findbugs:jsr305:3.0.2
|    +--- io.github.eisop:dataflow-errorprone:3.34.0-eisop1
|    +--- com.google.auto.value:auto-value-annotations:1.9
|    +--- com.google.errorprone:error_prone_annotations:2.21.1
|    +--- com.google.protobuf:protobuf-java:3.19.6
|    +--- com.google.auto.service:auto-service-annotations:1.0.1
|    +--- javax.inject:javax.inject:1
|    \--- javax.annotation:javax.annotation-api FAILED
+--- com.uber.nullaway:nullaway:0.10.14
|    +--- org.checkerframework:dataflow-nullaway:3.38.0
|    \--- com.google.guava:guava:24.1.1-jre -> 32.1.1-jre (*)
\--- io.grpc:grpc-java-api-checker:1.1.0

[thenswan]

Android example still not working for me.

• When I am trying to build it as is - I get a lot of Duplicate class io.grpc errors:

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':app:checkDebugDuplicateClasses'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.CheckDuplicatesRunnable
   > Duplicate class io.grpc.Attributes found in modules grpc-api-1.61.1.jar -> grpc-api-1.61.1 (io.grpc:grpc-api:1.61.1) and io.grpc-3.2.1.jar -> io.grpc-3.2.1 (io.helidon.grpc:io.grpc:3.2.1)
     Duplicate class io.grpc.Attributes$1 found in modules grpc-api-1.61.1.jar -> grpc-api-1.61.1 (io.grpc:grpc-api:1.61.1) and io.grpc-3.2.1.jar -> io.grpc-3.2.1 (io.helidon.grpc:io.grpc:3.2.1)
     Duplicate class io.grpc.Attributes$Builder found in modules grpc-api-1.61.1.jar -> grpc-api-1.61.1 (io.grpc:grpc-api:1.61.1) and io.grpc-3.2.1.jar -> io.grpc-3.2.1 (io.helidon.grpc:io.grpc:3.2.1)
     Duplicate class io.grpc.Attributes$Key found in modules grpc-api-1.61.1.jar -> grpc-api-1.61.1 (io.grpc:grpc-api:1.61.1) and io.grpc-3.2.1.jar -> io.grpc-3.2.1 (io.helidon.grpc:io.grpc:3.2.1)
     Duplicate class io.grpc.BinaryLog found in modules grpc-api-1.61.1.jar -> grpc-api-1.61.1 (io.grpc:grpc-api:1.61.1) and io.grpc-3.2.1.jar -> io.grpc-3.2.1 (io.helidon.grpc:io.grpc:3.2.1)
     Duplicate class io.grpc.BindableService found in modules grpc-api-1.61.1.jar -> grpc-api-1.61.1 (io.grpc:grpc-api:1.61.1) and io.grpc-3.2.1.jar -> io.grpc-3.2.1 (io.helidon.grpc:io.grpc:3.2.1)
     Duplicate class io.grpc.CallCredentials found in modules grpc-api-1.61.1.jar -> grpc-api-1.61.1 (io.grpc:grpc-api:1.61.1) and io.grpc-3.2.1.jar -> io.grpc-3.2.1 (io.helidon.grpc:io.grpc:3.2.1)
     ...

• When I am commenting implementation(platform("com.hedera.hashgraph:sdk")) ... and trying to use implementation("com.hedera.hashgraph:sdk") (btw can I use it with version here?):

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':app:compileDebugJavaWithJavac'.
> Could not resolve all task dependencies for configuration ':app:debugCompileClasspath'.
   > Could not find com.google.protobuf:protobuf-javalite:.
     Required by:
         project :app > project :hedera-sdk-java:sdk

[thenswan]

Integration tests

I get following error once, when changing some code and then trying to run integration test. After trying to run integration test the second time - nothing really happens. Sample output:

> Task :sdk:compileTestIntegrationJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sdk:compileTestIntegrationJava'.
> Could not resolve all files for configuration ':sdk:testIntegrationCompileClasspath'.
   > Failed to transform sdk-2.32.0-beta.1.jar (project :sdk) to match attributes {artifactType=jar, javaModule=true, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.version=17, org.gradle.libraryelements=jar, org.gradle.usage=java-api}.
      > Execution failed for ExtraJavaModuleInfoTransform: /Users/nikitalebedev/Work/LimeChain/projects-workzone/hedera-sdk-java/sdk/build/libs/sdk-2.32.0-beta.1.jar.
         > Not a module and no mapping defined: sdk-2.32.0-beta.1.jar

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.

BUILD FAILED in 7s
19 actionable tasks: 6 executed, 13 up-to-date
Configuration cache entry stored.

[thenswan]

Docs

• Can you please update following docs based on the changes you introduced 🙏🏻
  • Integration tests
  • Android
• Also would like to ask to make following doc items included in this PR (basically doc of work done in this PR):

Would be great to add the docs and update current docs which is in the repo. Given the new setup it can be hard for new collaborators to catch up, so docs and guides should be super helpful, for example:

I think many of the answers I gave should move into documentation. I would like to do that as a follow up step. I created https://github.com/hashgraph/hedera-sdk-java/issues/1794.

[jjohannes]

How can we address: The 'sonarqube' task depends on compile tasks. This behavior is now deprecated ...

By not using sonarqube anymore or waiting for them to update their Gradle plugin. 😄 I would suggest to remove sonarqube in the follow up https://github.com/hashgraph/hedera-services/issues/13505 and use the other solutions we now use in 'hedera-services' (Codecov and Codacy).

[thenswan]

I would suggest to remove sonarqube in the follow up https://github.com/hashgraph/hedera-services/issues/13505 and use the other solutions we now use in 'hedera-services' (Codecov and Codacy).

Agree with aligning it with hedera-services 👍🏻

[thenswan]

@jjohannes kindly pinging to figure out if you saw this comment and others I left on May 17th 🙂

[jjohannes]

@thenswan yes, saw your comments. This is next on my list when I have a bit more time again for Hedera. Moste likely in the beginning of next week.

[jjohannes]

@thenswan worked on this today, and it was more than expected. The first point below is something we need to think about carefully. The other things should be addressed. I have two open TODOs, that I'll do later (could also be done as follow up). You can have a look at everything else. See details below.

Issue with Android App / Open Question

The problem with the Android App made me realize something we have not discussed and which might be a problem: In the new setup, we replace io.grpc:grpc-core with io.helidon.grpc:io.grpc. Otherwise, things do not work as Java Module due to split package problems. We will then publish the dependency to io.helidon.grpc:io.grpc. Which is a difference to the current release. Is this acceptable at all?

In any case, I made sure that the publishing is sound now so that if you depend on com.hedera.hashgraph:sdk, you will only get io.helidon.grpc:io.grpc transitively and not the overlapping io.grpc:* modules. This is done using exclude which is written to the pom/module metadata. The Android sample now works.

If we do not want to do this change, we cannot deliver a working module-info. If you are unsure about this, maybe we need to discuss this directly together with @nathanklick.

We can maybe also adjust the setup in a way that we build a correct "module-info" with requires io.helidon.grpc:io.grpc but keep publish the dependency to io.grpc:grpc-core in POM. Then when the SDK is used as normal Jar on the classpath (module-info is ignored) things work as before. And when someone want's to use it as Module, they would need to fix up the dependencies on their own... this is a bit ugly and can be risky – we build and test against one things, but the publish another. But it would allow us to not change things for now. (Not sure I like this idea.)

Documentation (guides)

I updated the docs on integration tests and Android.

Integration Tests issue

I am not able to reproduce this problem:

> Task :sdk:compileTestIntegrationJava FAILED
...
   > Failed to transform sdk-2.32.0-beta.1.jar (project :sdk) to match attributes {artifactType=jar, javaModule=true, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.version=17, org.gradle.libraryelements=jar, org.gradle.usage=java-api}.
      > Execution failed for ExtraJavaModuleInfoTransform: /Users/nikitalebedev/Work/LimeChain/projects-workzone/hedera-sdk-java/sdk/build/libs/sdk-2.32.0-beta.1.jar.
         > Not a module and no mapping defined: sdk-2.32.0-beta.1.jar

I know this issue in general, but don't see it in this project. Also works on CI. Can you try again, and if you still get it, run again with --scan and send me the link.

Questions about dependencies

1. Does it make sense for compileProtoPath and testIntegrationCompileProtoPath to have all of those dependencies?

It's the same as on main. This setup comes from Google's protobuf plugin and it inherits all the dependencies that the "compileClasspath" has. If this is necessary, I don't know. But I won't touch this standard setup if there is no good reason to change it. Note that on this branch, everything is shown twice, because the version (tagged with constraint or (c)) is shown as a separate entry.

2. It is OK that project :sdk (*) is included twice in: testIntegrationCompileClasspath, testIntegrationCompileProtoPath and testIntegrationRuntimeClasspath?

It's not a problem if things are duplicated. But it's interesting why this is. Looks like the first entry is added by Gradle automatically when we setup the tests by register<JvmTestSuite>("testIntegration") (by this I am surprised, I'll do some more research is it can be turned off). The second is added, because the "intgration tests" run as Java Module and we have an explicit dependency defined through the requires com.hedera.hashgraph.sdk;.

3./4. Is it OK to have FAILED here: mainRuntimeClasspath / errorprone

Does not seem to be a problem, but it's not nice. Put it on my TODO list to check.

Broken integration tests

I disabled two integration tests on this branch, because they are failing. When I test locally, they also fail on "main" for me. I am not sure if this has something to do with the changes on the branch or if they are already broken and somehow did not run on CI with the old setup. Looks to me, as if the assertions in the tests need to be a bit different. Can you or someone else from the team check if there is already an issue with the tests on main?

These are the tests: https://github.com/hashgraph/hedera-sdk-java/pull/1732/commits/7880fd20233b4f278cbbe88aedf9e93dbb132462

TODOs

• [ ] TODO, More comments (docs) in .gradle.kts filesThis is independent of everything else.
• [x] TODO, check why "mainRuntimeClasspath FAILED" and "errorprone FAILED" and improve
• [x] TODO, maybe do version definition different (Gradle standard format) to not break Dependabot - see https://github.com/hashgraph/hedera-services/pull/13132

[jjohannes]

I moved the versions into a separate file sdk-dependency-versions/build.gradle.kts – in alignment with what we do now in hedera-services. Sorry for the back-and-forth on this, but I now think it is the better solution as the Dependabot integration keeps working flawlessly this way and the setup in Gradle is also a bit clearer. The versions are also used by the examples and the sdk-full project (and not just the sdk project).

[thenswan]

Thank you for detailed answers and updates @jjohannes I addressed tests, fix is already on the main branch 🙂

Re Android - will check it more in detail.

[thenswan]

Integration tests

Seems to be working at this point and I am not able to reproduce same error after new changes 👍🏻

[jjohannes]

@thenswan good news on the topic of:

Issue with Android App / Open Question The problem with the Android App made me realize something we have not discussed and which might be a problem: In the new setup, we replace io.grpc:grpc-core with io.helidon.grpc:io.grpc. Otherwise, things do not work as Java Module due to split package problems. We will then publish the dependency to io.helidon.grpc:io.grpc. Which is a difference to the current release. Is this acceptable at all?

We are updating 'io.grpc' in the services repo (https://github.com/hashgraph/hedera-services/compare/13679-tm-upgrade-netty) with the newest version. Turns out, the replacement stuff is no longer necessary and we can keep using grpc-api (or grpc-core) directly. Then this whole problematic change goes away. I will update this PR accordingly and let you know when I am done. Then you can have another look.