clickhouse-java
clickhouse-java copied to clipboard
Table engine is not specified in CREATE query when executing df.write from pyspark to clickhouse
Hello everyone, i was trying to write a new table from spark to clickhouse (this table ddl was not created in clickhouse yet) by this piece of code:
df2.write\
.format("jdbc")\
.mode("overwrite")\
.option("driver", "com.clickhouse.jdbc.ClickHouseDriver")\
.option("url", "")\
.option("user", "")\
.option("password", "")\
.option("dbtable", "default.test_write_from_spark")\
.option("truncate", "true")\
.option("batchsize", 10000)\
.option("isolationLevel", "NONE")\
.save()
Then i encounter the error :"java.sql.SQLException: Code: 119. DB::Exception: Table engine is not specified in CREATE query. (ENGINE_REQUIRED) (version 22.10.1.61 (official build))"
I think that i need to define the DDL and engine of the table in clickhouse in advance . But the question is, if i do not define the table in clickhouse first, can i write from spark to clickhouse by df.write ? If yes, what type of options should i put in the above code in order to do that? Thanks you guys very much.