datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

remove Duplication between `ParquetFormat` and `ParquetReadOptions` is confusing

Open alamb opened this issue 3 years ago • 2 comments

Draft as it builds on https://github.com/apache/arrow-datafusion/pull/2985

Which issue does this PR close?

Closes https://github.com/apache/arrow-datafusion/issues/2987

Rationale for this change

The duplication was confusing.

However, I am not sure about this change as now the different formats are treated differently, whereas before the formats all had a uniform implementation

What changes are included in this PR?

Move shared options into a ParquetFormatOptions shared structure

Are there any user-facing changes?

This is backards incompatible for anyone who manipulated the configuration fields directly. If they used the builder API it will be fine.

alamb avatar Jul 30 '22 11:07 alamb

🤔 now that I see how this looks I am not convinced it saves much duplication

alamb avatar Jul 30 '22 11:07 alamb

Codecov Report

Merging #2988 (a0274a1) into master (3d1de15) will increase coverage by 0.00%. The diff coverage is 91.66%.

@@           Coverage Diff           @@
##           master    #2988   +/-   ##
=======================================
  Coverage   85.78%   85.78%           
=======================================
  Files         281      282    +1     
  Lines       51580    51656   +76     
=======================================
+ Hits        44246    44313   +67     
- Misses       7334     7343    +9     
Impacted Files Coverage Δ
benchmarks/src/bin/tpch.rs 39.52% <0.00%> (ø)
datafusion/core/tests/sql/mod.rs 98.26% <ø> (ø)
datafusion/core/tests/sql/parquet.rs 100.00% <ø> (ø)
datafusion/proto/src/logical_plan.rs 17.42% <0.00%> (ø)
datafusion/core/src/execution/options.rs 59.45% <77.77%> (+0.03%) :arrow_up:
...afusion/core/src/datasource/file_format/parquet.rs 85.20% <83.33%> (-0.70%) :arrow_down:
datafusion/core/tests/sql/parquet_schema.rs 98.79% <98.79%> (ø)
...tafusion/core/src/physical_plan/file_format/mod.rs 97.36% <100.00%> (ø)
datafusion/expr/src/window_frame.rs 92.43% <0.00%> (-0.85%) :arrow_down:
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

codecov-commenter avatar Jul 30 '22 11:07 codecov-commenter

I don't think this is worth pursuing at the current time

alamb avatar Aug 13 '22 11:08 alamb