hudi icon indicating copy to clipboard operation
hudi copied to clipboard
verified publisher icon apache

[HUDI-3907][RFC-52] RFC for Introduce Secondary Index to Improve Hudi Query Performance

Open huberylee opened this issue 4 years ago • 8 comments

What is the purpose of the pull request

RFC for Introduce Secondary Index to Improve HUDI Query Performance

Brief change log

  • Modify rfc/README.md
  • Add rfc/rfc-52 dir

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • [ ] Has a corresponding JIRA in PR title & commit

  • [ ] Commit message is descriptive of the change

  • [ ] CI is green

  • [ ] Necessary doc changes done or have another open PR

  • [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

huberylee avatar Apr 20 '22 06:04 huberylee

CI report:

  • 95ca56ff2e76f43017333195df1c40b4cfa3aa0a UNKNOWN
  • bb0c0c4323ffbf605def455ded35130b6ed39500 UNKNOWN
  • f8863fd21c46ff0cc5e422ac323d36a252125895 Azure: FAILURE
Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

hudi-bot avatar Apr 28 '22 12:04 hudi-bot

@huberylee are you still actively driving this one

vinothchandar avatar Jul 15 '22 05:07 vinothchandar

@huberylee are you still actively driving this one

Yes, it's already under development

huberylee avatar Jul 15 '22 06:07 huberylee

Great start. Overall I also think we need to think about the abstraction API more carefully here.

The current implementation provides an abstract framework upon which we can easily extend other types of secondary indexes, and this document is a little out of date, I will update it later.

huberylee avatar Oct 21 '22 02:10 huberylee

What is the state of this issue? Any plans to merge RFC to mark it under active development or there are significant changes expected?

yandooo avatar Jan 11 '23 12:01 yandooo

What is the state of this issue? Any plans to merge RFC to mark it under active development or there are significant changes expected?

Most of the development work is over, some PRs have been merged and another are under review.

Subtasks:

  • [x] https://github.com/apache/hudi/pull/5761
  • [x] https://github.com/apache/hudi/pull/5894
  • [x] https://github.com/apache/hudi/pull/5933
  • [ ] https://github.com/apache/hudi/pull/6677
  • [ ] https://github.com/apache/hudi/pull/6712

huberylee avatar Jan 11 '23 14:01 huberylee

Hi @huberylee there's "Explore other execution engines/runtimes (Ray, native Rust, Python)" on hudi roadmap, it seems like secondary index will be using Lucene (at least one implementation), so when using non-JVM client it will not be possible to maintain this index? I would like to understand how it will look like in this case, I guess the simplest would be not to use this this type of index?

kazdy avatar Jan 12 '23 16:01 kazdy

Can I search from hudi using some keywords just like elasticsearch, beaucse of the lucene secondary index?

colagy avatar Sep 12 '23 06:09 colagy

Closing this RFC due to inactivity. There will be or should be a new RFC for text search using sec indexes, on top of 1.x

vinothchandar avatar Aug 28 '25 03:08 vinothchandar