remove Duplication between `ParquetFormat` and `ParquetReadOptions` is confusing
Draft as it builds on https://github.com/apache/arrow-datafusion/pull/2985
Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/2987
Rationale for this change
The duplication was confusing.
However, I am not sure about this change as now the different formats are treated differently, whereas before the formats all had a uniform implementation
What changes are included in this PR?
Move shared options into a ParquetFormatOptions shared structure
Are there any user-facing changes?
This is backards incompatible for anyone who manipulated the configuration fields directly. If they used the builder API it will be fine.
🤔 now that I see how this looks I am not convinced it saves much duplication
Codecov Report
Merging #2988 (a0274a1) into master (3d1de15) will increase coverage by
0.00%. The diff coverage is91.66%.
@@ Coverage Diff @@
## master #2988 +/- ##
=======================================
Coverage 85.78% 85.78%
=======================================
Files 281 282 +1
Lines 51580 51656 +76
=======================================
+ Hits 44246 44313 +67
- Misses 7334 7343 +9
| Impacted Files | Coverage Δ | |
|---|---|---|
| benchmarks/src/bin/tpch.rs | 39.52% <0.00%> (ø) |
|
| datafusion/core/tests/sql/mod.rs | 98.26% <ø> (ø) |
|
| datafusion/core/tests/sql/parquet.rs | 100.00% <ø> (ø) |
|
| datafusion/proto/src/logical_plan.rs | 17.42% <0.00%> (ø) |
|
| datafusion/core/src/execution/options.rs | 59.45% <77.77%> (+0.03%) |
:arrow_up: |
| ...afusion/core/src/datasource/file_format/parquet.rs | 85.20% <83.33%> (-0.70%) |
:arrow_down: |
| datafusion/core/tests/sql/parquet_schema.rs | 98.79% <98.79%> (ø) |
|
| ...tafusion/core/src/physical_plan/file_format/mod.rs | 97.36% <100.00%> (ø) |
|
| datafusion/expr/src/window_frame.rs | 92.43% <0.00%> (-0.85%) |
:arrow_down: |
| ... and 2 more |
Help us with your feedback. Take ten seconds to tell us how you rate us.
I don't think this is worth pursuing at the current time