datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

CSV can't parse null value for non-string type (i32, i64, float)

Open jayzhan211 opened this issue 1 year ago • 1 comments

Describe the bug

As title, I would like to store null value in the csv table but couldn't.

To Reproduce

Write a csv file to datafusion/core/tests/data/data.csv with

c1,c2
NULL,'a'
statement ok
CREATE EXTERNAL TABLE t (
c1 INT,
c2 VARCHAR,
)
STORED AS CSV
LOCATION '../core/tests/data/data.csv'
OPTIONS ('format.has_header' 'true');

query error DataFusion error: Arrow error: Parser error: Error while parsing value NULL for column 0 at line 1
select * from t;

Expected behavior

Successfully store nulls for non-string type

Additional context

String is fine

jayzhan211 avatar Oct 13 '24 09:10 jayzhan211

I think this may be something best handled in the arrow-csv crate

Omega359 avatar Oct 14 '24 14:10 Omega359

take

dhegberg avatar Nov 01 '24 14:11 dhegberg