[Feature] Support lance file format

[coolderli]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

Lance format provides better support for machine learning. It can better support point lookup and wide columns to support vectorized data storage.

Do we have a plan to support this file format? Can you share your thoughts? Thanks. @JingsongLi @SteNicholas

https://lancedb.github.io/lance/format https://blog.lancedb.com/lance-v2/

Solution

Support lance file format for vectorized data storage.

Anything else?

No response

Are you willing to submit a PR?

• [X] I'm willing to submit a PR!

[liyubin117]

Is there any progress?

[Sucran]

Is there any progress?

see leaves12138:lance_support and pr #5760

[benqwu]

@Sucran When will Lance be supported approximately? Thank you.

[Sucran]

@benqwu the pr #5760 is merged.

[littleDrew]

hi @Sucran, from your branch and merge request, it seems has no direct relation to support lance in paimon, there is no key words "lance" in your implementation, can you explain in detail, thanks @Sucran