Padma Penumarthy comments

Results 6 comments of


                                            Padma Penumarthy

DRILL-4706: Fragment planning causes Drillbits to read remote chunks when local copies are available.

Updated the JIRA with details on how current algorithm works, why remote reads were happening and the new algorithm details. https://issues.apache.org/jira/browse/DRILL-4706

DRILL-4706: Fragment planning causes Drillbits to read remote chunks when local copies are available.

Updated with all review comments taken care of. Added TestLocalAffinityFragmentParallelizer.java which has bunch of test cases with examples.

DRILL-4706: Fragment planning causes Drillbits to read remote chunks when local copies are available.

Some initial comments. The issue is regarding assigning fragments based on strict locality. So why is the parallelization logic affected, and not exclusively locality? Parallelization logic is affected because it...

DRILL-4706: Fragment planning causes Drillbits to read remote chunks when local copies are available.

Parallelization logic is affected for following reasons: Depending upon how many rowGroups to scan on a node (based on locality information) i.e. how much work the node has to do,...

DRILL-4706: Fragment planning causes Drillbits to read remote chunks when local copies are available.

Merged with latest code. All review comments taken care of. All tests pass with the option `store.parquet.use_local_affinity` = true and false, both.

DRILL-4706: Fragment planning causes Drillbits to read remote chunks when local copies are available.

Even though it is old, this PR is still very much relevant and useful feature to have in Drill for certain use cases/scenarios. I request a committer to work with...