datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

feat: correlation support

Open huaxingao opened this issue 1 year ago • 2 comments

Which issue does this PR close?

Supports CORR The implementation mostly is the same as the DataFusion's implementation. The reason we have our own implementation is that DataFusion has UInt64 for state_field count, while Spark has Double for count. Also adding null_on_divide_by_zero to be consistent with Spark's implementation. Closes #.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

huaxingao avatar May 21 '24 16:05 huaxingao

Codecov Report

Attention: Patch coverage is 85.71429% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 34.02%. Comparing base (14494d3) to head (fead995). Report is 26 commits behind head on main.

Files Patch % Lines
.../scala/org/apache/comet/serde/QueryPlanSerde.scala 85.71% 1 Missing and 1 partial :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##               main     #456      +/-   ##
============================================
- Coverage     34.02%   34.02%   -0.01%     
- Complexity      857      859       +2     
============================================
  Files           116      116              
  Lines         38565    38671     +106     
  Branches       8517     8564      +47     
============================================
+ Hits          13120    13156      +36     
- Misses        22691    22753      +62     
- Partials       2754     2762       +8     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar May 21 '24 18:05 codecov-commenter

cc @andygrove @viirya

huaxingao avatar May 22 '24 00:05 huaxingao

Merged. Thanks @huaxingao @andygrove @kazuyukitanimura

viirya avatar May 23 '24 15:05 viirya

Thanks @viirya @andygrove @kazuyukitanimura

huaxingao avatar May 23 '24 15:05 huaxingao