Feature: LIMIT push down for new planner

[leiysky]

In some cases, it's possible to push the LIMIT operator down to reduce the data to be processed.

Besides, LIMIT hint of ReadDataSource will affect the behavior of Random table engine.

[AngleNet]

@leiysky hi, is there any one working on this? I would like to pick this up.

[sundy-li]

Besides, LIMIT push down could affect the Sort Plan which has partial sort optimization.

#[derive(Clone, Debug)]
pub struct Sort {
    pub items: Vec<SortItem>,
+   pub limit: option<usize>
}

[AngleNet]

Besides, LIMIT push down could affect the Sort Plan which has partial sort optimization.

#[derive(Clone, Debug)]
pub struct Sort {
    pub items: Vec<SortItem>,
+   pub limit: option<usize>
}

@sundy-li Hi. What do you mean partial sort optimization?

[sundy-li]

@sundy-li Hi. What do you mean partial sort optimization?

https://github.com/datafuselabs/databend/blob/4d458914ab5e6fcf26d3c5fc74b830ef3c3a2d82/query/src/pipelines/pipeline_builder.rs#L381-L402

[leiysky]

@sundy-li Is it actually a TopN operator?

[AngleNet]

@sundy-li @leiysky Shall we implement a TopN operator and convert an order followed by limit to a TopN operator and the limit could be pushed down further?

[leiysky]

@AngleNet I've just opened a issue https://github.com/datafuselabs/databend/issues/7015 for that.

[AngleNet]

Great! I think I could implement limit push down firstly and then introduce topn operator, refactor the limit push down to support topn finally. Is that ok?

[leiysky]

Great! I think I could implement limit push down firstly and then introduce topn operator, refactor the limit push down to support topn finally. Is that ok?

That's good.