carbondata icon indicating copy to clipboard operation
carbondata copied to clipboard

[CARBONDATA-4291] Carbon hive table supports float datatype

Open jack86596 opened this issue 4 years ago • 24 comments

Why is this PR needed?

Currently when create carbon hive table, if data type is float, will convert to double type. This means all float data will be stored as double. In CTAS secnario, if source table column is float type, the data in newly created carbon table will be incorrect. Reproduce steps: CREATE TABLE p1(f float) stored as parquet; insert into table p1 select 12.36; create table carbon1 stored as carbondata as select * from p1; select * from carbon1; Result: 5.410467587E-315

What changes were proposed in this PR?

Carbon should support store float datatype directly. Handle float type in read and write logic.

Does this PR introduce any user interface change?

  • No

Is any new testcase added?

  • Yes

jack86596 avatar Oct 08 '21 13:10 jack86596

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/429/

CarbonDataQA2 avatar Oct 08 '21 15:10 CarbonDataQA2

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6039/

CarbonDataQA2 avatar Oct 08 '21 15:10 CarbonDataQA2

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4294/

CarbonDataQA2 avatar Oct 08 '21 15:10 CarbonDataQA2

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6041/

CarbonDataQA2 avatar Oct 09 '21 06:10 CarbonDataQA2

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4297/

CarbonDataQA2 avatar Oct 09 '21 06:10 CarbonDataQA2

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/431/

CarbonDataQA2 avatar Oct 09 '21 06:10 CarbonDataQA2

retest this please

jack86596 avatar Oct 12 '21 01:10 jack86596

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6048/

CarbonDataQA2 avatar Oct 12 '21 06:10 CarbonDataQA2

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4303/

CarbonDataQA2 avatar Oct 12 '21 06:10 CarbonDataQA2

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/437/

CarbonDataQA2 avatar Oct 12 '21 06:10 CarbonDataQA2

please rebase

kunal642 avatar Oct 23 '21 06:10 kunal642

@akashrn5 please check once

kunal642 avatar Oct 23 '21 06:10 kunal642

please rebase @jack86596

vikramahuja1001 avatar Oct 27 '21 05:10 vikramahuja1001

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6115/

CarbonDataQA2 avatar Nov 14 '21 08:11 CarbonDataQA2

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4372/

CarbonDataQA2 avatar Nov 14 '21 08:11 CarbonDataQA2

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/505/

CarbonDataQA2 avatar Nov 14 '21 09:11 CarbonDataQA2

retest this please

jack86596 avatar Nov 15 '21 02:11 jack86596

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4373/

CarbonDataQA2 avatar Nov 15 '21 05:11 CarbonDataQA2

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6116/

CarbonDataQA2 avatar Nov 15 '21 05:11 CarbonDataQA2

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/506/

CarbonDataQA2 avatar Nov 15 '21 05:11 CarbonDataQA2

@vikramahuja1001 @kunal642 rebased, please review, thanks.

jack86596 avatar Nov 15 '21 06:11 jack86596

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6121/

CarbonDataQA2 avatar Nov 18 '21 19:11 CarbonDataQA2

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4378/

CarbonDataQA2 avatar Nov 18 '21 19:11 CarbonDataQA2

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/511/

CarbonDataQA2 avatar Nov 18 '21 20:11 CarbonDataQA2