cannot import into bigint column from CSV file

[mewalig]

Hi,

Am trying to import a CSV file to Databricks (9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12) and even with a very simple 2-column example and tiny integer input, it fails with Failed to merge incompatible data types LongType and IntegerType

Here are my test commands:

dbsqlcli -e "create or replace table `datatype_test`(`bigint_col` bigint, `int_col` integer)"

dbsqlcli -e "COPY INTO datatype_test FROM '/tmp/datatype_test.csv' FILEFORMAT = CSV FORMAT_OPTIONS ('header'='true', 'inferSchema'='true') COPY_OPTIONS ('mergeSchema'='true')"

where /tmp/datatype_test.csv contains:

bigint_col,int_col
,123
123,
123,123

Any suggestions?

The error response header is:

org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Failed to merge fields 'bigint_col' and 'bigint_col'. Failed to merge incompatible data types LongType and IntegerType
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:1019)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:759)
...