clickhouse-java icon indicating copy to clipboard operation
clickhouse-java copied to clipboard
verified publisher icon ClickHouse

[Bug] Always thrown `StreamingQueryException: No client available` when shaded jar is assembled with spark 2.4.5 (JDK1.8)

Open oliverdding opened this issue 3 years ago • 9 comments

I'm testing clickhouse-jdbc with Spark 2.4.5, but org.apache.spark.sql.streaming.StreamingQueryException: No client available is always thrown.

I swear that I added the classifier "all" like this(sbt):

"com.clickhouse" % "clickhouse-jdbc" % "0.4.0" classifier "all"

After jar -xf the assembled jar file, I could find the http client related class file:

image

Then I looked into the source code of clickhouse-http-client, and found this:

    CONNECTION_PROVIDER("http_connection_provider", HttpConnectionProvider.HTTP_URL_CONNECTION,
            "HTTP connection provider. HTTP_CLIENT is only supported in JDK 11 or above."),

I'm not sure if this is related, so I'm asking for help. Thanks for any advice.

oliverdding avatar Feb 06 '23 11:02 oliverdding

The shaded jar does not contain JDK http client, so it should work under JDK 8 - we still have CI job to check that. Is it working fine before upgrading to 0.4.0? Have you removed the old driver(0.3.2*) from your classpath?

zhicwu avatar Feb 06 '23 11:02 zhicwu

I haven't tried 0.3.2* yet, everything above is all about 0.4.0.

Below is part of the project dependencies:

Screenshot 2023-02-06 at 21 16 11

Following your advice, I tried 0.3.2-patch11 but it did not work:

image 
23/02/06 13:19:02 ERROR MicroBatchExecution: Query test.clickHouse [id = 57dc7540-0052-48b0-8d44-2a4a8a0b3db9, runId = 9d90a833-f0c3-4423-b69e-696745592086] terminated with error
java.sql.SQLException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@2085039148
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:85)
        at com.clickhouse.jdbc.SqlExceptionUtils.create(SqlExceptionUtils.java:31)
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:90)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:93)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:298)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:241)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:145)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:41)
        at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:48)
        at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
        at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:515)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:44)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:28)
        at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5.apply(MicroBatchExecution.scala:535)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:534)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.net.ConnectException: No client available
        at com.clickhouse.client.ClickHouseClientBuilder$DummyClient.execute(ClickHouseClientBuilder.java:58)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.sendOnce(ClickHouseClientBuilder.java:284)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.send(ClickHouseClientBuilder.java:296)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.execute(ClickHouseClientBuilder.java:351)
        at com.clickhouse.client.ClickHouseClient.executeAndWait(ClickHouseClient.java:824)
        at com.clickhouse.client.ClickHouseRequest.executeAndWait(ClickHouseRequest.java:1972)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:90)
        ... 50 more
org.apache.spark.sql.streaming.StreamingQueryException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@2085039148

oliverdding avatar Feb 06 '23 13:02 oliverdding

Could you add exclude("*", "*") for clickhouse-jdbc? It's to exclude transitive dependencies and only use the shaded jar.

zhicwu avatar Feb 07 '23 05:02 zhicwu

Hi @zhicwu . I have added exclude("*", "*") like this:

    libraryDependencies ++= Seq(
      ...
      "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*") classifier "all"
    )

But it doesn't seem to work yet.

23/02/07 05:47:38 ERROR MicroBatchExecution: Query test.clickHouse [id = 57dc7540-0052-48b0-8d44-2a4a8a0b3db9, runId = f50172c8-ccfa-4947-82f2-037ac98cc2a1] terminated with error
java.sql.SQLException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@-174080404
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:85)
        at com.clickhouse.jdbc.SqlExceptionUtils.create(SqlExceptionUtils.java:31)
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:90)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:124)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:333)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:274)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:157)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:41)
        at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:48)
        at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
        at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:515)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:44)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:28)
        at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5.apply(MicroBatchExecution.scala:535)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:534)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.net.ConnectException: No client available
        at com.clickhouse.client.ClickHouseClientBuilder$DummyClient.execute(ClickHouseClientBuilder.java:61)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.sendOnce(ClickHouseClientBuilder.java:287)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.send(ClickHouseClientBuilder.java:299)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.execute(ClickHouseClientBuilder.java:354)
        at com.clickhouse.client.ClickHouseClient.executeAndWait(ClickHouseClient.java:884)
        at com.clickhouse.client.ClickHouseRequest.executeAndWait(ClickHouseRequest.java:2068)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:121)
        ... 50 more
org.apache.spark.sql.streaming.StreamingQueryException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@-174080404

Additional information:

  • Spark 2.4.5
  • JDK 8
  • Scala 2.11.12

oliverdding avatar Feb 07 '23 05:02 oliverdding

Hi @zhicwu , then I manually add clickhouse-http-client like below:

    libraryDependencies ++= Seq(
      ...
      "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*"),
      "com.clickhouse" % "clickhouse-http-client" % "0.4.0"
    )

Although necessary dependencies are shown, it doesn't seem to work yet.

image

Any advice are appreciated.

oliverdding avatar Feb 07 '23 07:02 oliverdding

You don't need to add clickhouse-http-client specifically as the shaded jar did that for you. Have you tried to validate in standalone mode? I don't have a similar environment but I'll try when I get home.

zhicwu avatar Feb 08 '23 00:02 zhicwu

Hi @zhicwu , If I just use "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*") classifier "all", there is no clickhouse-*-client related jar file in the dependencies panel of the project, which I think is not expected.

image

I create a test project for this issue: https://github.com/oliverdding/clickhouse-jdbc-test

I set up clickhouse, kafka with docker, then run sparkapp in local mode.

image

Below is my running report:

  1. ❌ "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*") classifier "all" and run in local mode
image
  1. ❌ "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" classifier "all" and run in local mode
image

oliverdding avatar Feb 08 '23 05:02 oliverdding

I have the same error so FWIW, I tried to copy these jars to SPARK_HOME/jars and the app worked for me.

clickhouse-cli-client-0.3.2-patch11.jar
clickhouse-http-client-0.3.2-patch11.jar
clickhouse-jdbc-0.3.2-patch11.jar
clickhouse-client-0.3.2-patch11.jar

shahsiddharth08 avatar Feb 09 '23 17:02 shahsiddharth08

Hi @shahsiddharth08 , thanks for your reply. I tried it and it works!

THAT IS STRANGE ! 🤣

image image

I have put: 企业微信截图_63286c6f-540b-44fa-978c-f60f7b789217 into $SPARK_HOME/jars folder.

oliverdding avatar Feb 10 '23 03:02 oliverdding

Fixed in latest 0.7.2 release.

chernser avatar Jan 09 '25 20:01 chernser