clickhouse-java icon indicating copy to clipboard operation
clickhouse-java copied to clipboard

Table engine is not specified in CREATE query when executing df.write from pyspark to clickhouse

Open Vinhbuster opened this issue 3 years ago • 0 comments

Hello everyone, i was trying to write a new table from spark to clickhouse (this table ddl was not created in clickhouse yet) by this piece of code:

df2.write\
    .format("jdbc")\
    .mode("overwrite")\
    .option("driver", "com.clickhouse.jdbc.ClickHouseDriver")\
    .option("url", "")\
    .option("user", "")\
    .option("password", "")\
    .option("dbtable", "default.test_write_from_spark")\
    .option("truncate", "true")\
    .option("batchsize", 10000)\
    .option("isolationLevel", "NONE")\
    .save()

Then i encounter the error :"java.sql.SQLException: Code: 119. DB::Exception: Table engine is not specified in CREATE query. (ENGINE_REQUIRED) (version 22.10.1.61 (official build))"

I think that i need to define the DDL and engine of the table in clickhouse in advance . But the question is, if i do not define the table in clickhouse first, can i write from spark to clickhouse by df.write ? If yes, what type of options should i put in the above code in order to do that? Thanks you guys very much.

Vinhbuster avatar Sep 29 '22 17:09 Vinhbuster