spring-cloud-stream icon indicating copy to clipboard operation
spring-cloud-stream copied to clipboard
verified publisher icon spring-cloud

Native compilation runtime error: java.lang.ClassNotFoundException: org.apache.kafka.common.security.scram.internals.ScramSaslClient$ScramSaslClientFactory

Open spulci opened this issue 1 year ago • 3 comments

Version of the framework SpringBoot 3.2.7 SpringCloud 2023.0.2 (BOM) Spring Cloud Stream: 4.1.2 GraalVM: CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30)

Describe the issue I've got a multi module maven project based on SpingBoot 3.2.x with Spring Cloud Stream 4.1.2 with Kafka Binder (as per Spring Cloud BOM 2023.0.2. I've got a reactive consumer that runs without issue on the JVM. Doing native compiling after the agent has runned for a few minutes, produces a runtime error regarding SASL security mechanism to authenticate to our Kafka Broker.

First af all, here it is our application.yml conifg:

cloud:
    refresh:
      enabled: false
      client:
        refresh-enabled: false
        config:
          default:
            logger-level: full
    stream:
      bindings:
        kafkaMessageConsumer-in-0:
          destination: devicemanager-topic        
      kafka:
        bindings:
          kafkaMessageConsumer-in-0:
            consumer:
              configuration:
                value.deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
                spring.json.value.default.type: com.minsait.phygital.resource.devicemanager.outadapter.domain.MetricsMessageBean
        binder:
            brokers: kafka-test.iiot.ourbroker.com:9098
            configuration:
              commit.interval.ms: 100
              default:
                value.deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
                spring.json.value.default.type: com.minsait.phygital.resource.devicemanager.outadapter.domain.MetricsMessageBean
              security:
                protocol: SASL_SSL
              sasl:
                mechanism: SCRAM-SHA-512
                jaas:
                  config: org.apache.kafka.common.security.scram.ScramLoginModule required username=test password=test-secret;
              ssl:
                truststore:
                  location: /path/to/truststore.pem
                  type: PEM
                endpoint:
                  identification:
                    algorithm: ""

I will omit consumer code as it has no issue in normal JRE run. I will omit also generated json descriptors produced by GraalVM native-image agent.

The runtime exception launching the executable is:

2024-07-01T16:29:13.266+02:00  WARN 10588 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Error connecting to node kafka-test.iiot.ourbroker.com:9098 (id: -1 rack: null)

java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348) ~[na:na]
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329) ~[na:na]
        at org.apache.kafka.common.network.Selector.connect(Selector.java:256) ~[na:na]
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1032) ~[na:na]
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301) ~[na:na]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1109) ~[na:na]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1369) ~[na:na]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312) ~[na:na]
        at [email protected]/java.lang.Thread.runWith(Thread.java:1596) ~[device-manager:na]
        at [email protected]/java.lang.Thread.run(Thread.java:1583) ~[device-manager:na]
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:833) ~[device-manager:na]
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:211) ~[na:na]
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:239) ~[na:na]
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338) ~[na:na]
        ... 11 common frames omitted
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism SCRAM-SHA-512
Caused by: javax.security.sasl.SaslException: Cannot instantiate service SASL/SCRAM Client Provider: SaslClientFactory.SCRAM-SHA-512 -> org.apache.kafka.common.security.scram.internals.ScramSaslClient$ScramSaslClientFactory

        at [email protected]/javax.security.sasl.Sasl.loadFactory(Sasl.java:461) ~[na:na]
        at [email protected]/javax.security.sasl.Sasl.createSaslClient(Sasl.java:432) ~[na:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslClient$0(SaslClientAuthenticator.java:219) ~[na:na]
        at [email protected]/java.security.AccessController.executePrivileged(AccessController.java:114) ~[na:na]
        at [email protected]/java.security.AccessController.doPrivileged(AccessController.java:714) ~[na:na]
        at [email protected]/javax.security.auth.Subject.doAs(Subject.java:525) ~[device-manager:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:215) ~[na:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:206) ~[na:na]
        at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:285) ~[na:na]
        at org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:228) ~[na:na]
        at org.apache.kafka.common.network.KafkaChannel.<init>(KafkaChannel.java:143) ~[na:na]
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:236) ~[na:na]
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338) ~[na:na]
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329) ~[na:na]
        at org.apache.kafka.common.network.Selector.connect(Selector.java:256) ~[na:na]
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1032) ~[na:na]
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301) ~[na:na]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1109) ~[na:na]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1369) ~[na:na]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312) ~[na:na]
        at [email protected]/java.lang.Thread.runWith(Thread.java:1596) ~[device-manager:na]
        at [email protected]/java.lang.Thread.run(Thread.java:1583) ~[device-manager:na]
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:833) ~[device-manager:na]
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:211) ~[na:na]
Caused by: java.security.NoSuchAlgorithmException: class configured for SaslClientFactory (provider: SASL/SCRAM Client Provider) cannot be found.
        at [email protected]/java.security.Provider$Service.getImplClass(Provider.java:2004) ~[device-manager:na]
        at [email protected]/java.security.Provider$Service.getDefaultConstructor(Provider.java:2020) ~[device-manager:na]
        at [email protected]/java.security.Provider$Service.newInstanceOf(Provider.java:1934) ~[device-manager:na]
        at [email protected]/java.security.Provider$Service.newInstanceUtil(Provider.java:1942) ~[device-manager:na]
        at [email protected]/java.security.Provider$Service.newInstance(Provider.java:1917) ~[device-manager:na]
        at [email protected]/javax.security.sasl.Sasl.loadFactory(Sasl.java:459) ~[na:na]
        ... 23 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.security.scram.internals.ScramSaslClient$ScramSaslClientFactory
        at [email protected]/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:52) ~[device-manager:na]
        at [email protected]/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na]
        at [email protected]/java.lang.ClassLoader.loadClass(ClassLoader.java:121) ~[device-manager:na]
        at [email protected]/java.security.Provider$Service.getImplClass(Provider.java:1991) ~[device-manager:na]
        ... 28 common frames omitted

Please let us know if more infos needed.

spulci avatar Jul 01 '24 14:07 spulci

@spulci It is likely that we are missing some native runtime hints. Since the offending class (ScramSaslClient$ScramSaslClientFactory) is part of Apache Kafka and not maintained by Spring for Apache Kafka project, we would ideally like to see this runtime hint added in the Oracle's graalvm reachability metadata repository here. Can you try adding a native hint locally in your project and see if this resolves? If so, please consider sending a PR to the repository I mentioned.

sobychacko avatar Jul 01 '24 15:07 sobychacko

@spulci It is likely that we are missing some native runtime hints. Since the offending class (ScramSaslClient$ScramSaslClientFactory) is part of Apache Kafka and not maintained by Spring for Apache Kafka project, we would ideally like to see this runtime hint added in the Oracle's graalvm reachability metadata repository here. Can you try adding a native hint locally in your project and see if this resolves? If so, please consider sending a PR to the repository I mentioned.

@sobychacko thanks Soby and accept my apologize due to the fact the the issue is not related to Spring Cloud Stream. Taking care of the fact I'm a real newbie in Java Native compilation, can you help me to elaborate the missing descriptor? I've tried this one;

{
    "name": "org.apache.kafka.common.security.scram.internals.ScramSaslClient$ScramSaslClientFactory",
    "queryAllPublicMethods": true,
    "queryAllPublicConstructors": true,
    "condition": {
      "typeReachable": "javax.security.sasl.SaslClientFactory"
    },
    "methods": [
      {
        "name": "<init>",
        "parameterTypes": [

        ]
      }
    ]
  }

added to my resource/META-INF/native-image///reflect-config.json, adding this entry to the ones generated by the agent. I've still got the same error as before. As far as I can understand, ScramSaslClientFactory is able to reach the JAVA SE type javax.security.sasl.SaslClientFactory in this way: I suppose the first needs to call the second inside its business logic to attach the Kafka implementation to the SE interface, right? This advice the native-compiler that there's a "graph edge" when ScramSaslClientFactory is reflected. Am I missing something?

spulci avatar Jul 02 '24 08:07 spulci

Can you try to add a runtime hints class in your application like this one? Then provide a corresponding aot.factories file. Start with that and see if it helps. If it doesn't, please share a sample application so we can investigate it further.

sobychacko avatar Jul 02 '24 23:07 sobychacko

Ok,

issue has been fixed locally with these two added hints:

      hints.reflection().registerType(
          org.apache.kafka.common.security.scram.internals.ScramSaslClient.ScramSaslClientFactory.class,
          builder -> builder.withMembers(MemberCategory.INVOKE_DECLARED_METHODS,
              MemberCategory.INVOKE_DECLARED_CONSTRUCTORS));
      hints.reflection().registerType(org.apache.kafka.common.security.scram.ScramLoginModule.class,
          builder -> builder.withMembers(MemberCategory.INVOKE_DECLARED_METHODS,
              MemberCategory.INVOKE_DECLARED_CONSTRUCTORS));

Related metadata in reflect-config.json is:

{
    "name": "org.apache.kafka.common.security.scram.internals.ScramSaslClient$ScramSaslClientFactory",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true
  },
  {
    "name": "org.apache.kafka.common.security.scram.ScramLoginModule",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true
  }

Please note that I've added ScramLoginModule too otherwise the app would have crashed on this class too (used in my spring cloud stream).

I will try to open a PR to the other repo soon, citing this issue too. Please feel free to track that PR when opened and of course suggest me any improvement I can do in this configuration.

Many thanks for your help!

Simone

spulci avatar Jul 03 '24 12:07 spulci

I'm glad to hear that it worked. Yes, please send a PR to the other repo. You should probably still keep your local hints until we have a release from the reachability metadata and it is incorporated with the spring native bits. Here is a recent PR I sent to that repo if you need an example: https://github.com/oracle/graalvm-reachability-metadata/pull/422.

Also, can we close this issue?

Thanks!

sobychacko avatar Jul 03 '24 13:07 sobychacko

I'm closing the issue. Hope to have the PR tomorrow.

spulci avatar Jul 03 '24 14:07 spulci