iceberg icon indicating copy to clipboard operation
iceberg copied to clipboard
verified publisher icon apache

Add filter pushdown API for contains and endsWith

Open yabola opened this issue 1 year ago • 3 comments

This PR adds filter pushing down api for endsWith (like %x) and contains (like %x%). Before this PR, iceberg only support pushdown startWith filter. Spark parquet supports pushdown endWiths and contains. This PR is the first part, the next PR will complete the related implementation of spark and fileFormat.

yabola avatar Feb 11 '24 08:02 yabola

The origin PR is https://github.com/apache/iceberg/pull/9683 , I split it into two PRs and add some UT. Please take a look @amogh-jahagirdar

yabola avatar Feb 11 '24 09:02 yabola

@amogh-jahagirdar Hi, if you have time, please take a look~

yabola avatar Feb 23 '24 13:02 yabola

Sorry for the delay @yabola , got busy with some other work, I haven't forgotten about this PR! When my time frees up I'll review this.

amogh-jahagirdar avatar Feb 23 '24 16:02 amogh-jahagirdar

@amogh-jahagirdar Hi, please take a look if you have time~

yabola avatar Mar 25 '24 03:03 yabola

@sujithjay @rdblue could you also take a look since you implements startWith~ Spark on parquet also support endsWith and contains https://github.com/apache/spark/blob/356830ada6c6ebbf54e7852c37266c32bfa137ea/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L816-L844

yabola avatar Apr 27 '24 16:04 yabola