hive icon indicating copy to clipboard operation
hive copied to clipboard

HIVE-29322: Avoid TopNKeyOperator When Map-Side LIMIT Pushdown Provides Better Pruning for ORDER BY LIMIT Queries

Open Indhumathi27 opened this issue 2 months ago • 4 comments

What changes were proposed in this pull request?

This PR updates TopNKeyProcessor to skip creating TopNKeyOperator when ReduceSinkDesc.topN is already set by LIMIT pushdown for ORDER BY LIMIT case. This prevents TopNKey from overriding pushdown and ensures the map-side LIMIT optimization is applied correctly.

Why are the changes needed?

Currently, when a query includes ORDER BY + LIMIT, LIMIT pushdown is generated during planning but is effectively overridden by the subsequent TopNKey rewrite. As a result, TopNKey operator receives full input rather than a reduced data set, leading to worse performance (e.g., 16M rows forwarded to reducer instead of a few). In cases where global ordering uses a single reducer, LIMIT pushdown is sufficient and far more efficient. This fix prevents unnecessary TopNKey creation so that pushdown can reduce shuffle and significantly improve execution time.

Test reports:

For query: select * from table order by h limit 100; Total num of rows: 67764224

with fix: Screenshot 2025-12-01 at 11 45 37 PM

without fix: Screenshot 2025-12-01 at 11 49 39 PM

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual testing + existing testcases

Indhumathi27 avatar Nov 18 '25 17:11 Indhumathi27

@kasakrisz @deniskuzZ Can you help to review this PR. Thanks

Indhumathi27 avatar Dec 01 '25 14:12 Indhumathi27

@ayushtkn @zabetak @kasakrisz @deniskuzZ Can you help to review the PR. thanks

Indhumathi27 avatar Dec 05 '25 05:12 Indhumathi27

I left some questions under the JIRA ticket to better understand the issue that we are trying to solve here.

zabetak avatar Dec 08 '25 14:12 zabetak