HIVE-29322: Avoid TopNKeyOperator When Map-Side LIMIT Pushdown Provides Better Pruning for ORDER BY LIMIT Queries
What changes were proposed in this pull request?
This PR updates TopNKeyProcessor to skip creating TopNKeyOperator when ReduceSinkDesc.topN is already set by LIMIT pushdown for ORDER BY LIMIT case. This prevents TopNKey from overriding pushdown and ensures the map-side LIMIT optimization is applied correctly.
Why are the changes needed?
Currently, when a query includes ORDER BY + LIMIT, LIMIT pushdown is generated during planning but is effectively overridden by the subsequent TopNKey rewrite. As a result, TopNKey operator receives full input rather than a reduced data set, leading to worse performance (e.g., 16M rows forwarded to reducer instead of a few). In cases where global ordering uses a single reducer, LIMIT pushdown is sufficient and far more efficient. This fix prevents unnecessary TopNKey creation so that pushdown can reduce shuffle and significantly improve execution time.
Test reports:
For query: select * from table order by h limit 100; Total num of rows: 67764224
with fix:
without fix:
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manual testing + existing testcases
@kasakrisz @deniskuzZ Can you help to review this PR. Thanks
@ayushtkn @zabetak @kasakrisz @deniskuzZ Can you help to review the PR. thanks
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
I left some questions under the JIRA ticket to better understand the issue that we are trying to solve here.