broccoliSpicy

### Is there an existing issue for the same bug? - [X] I have checked the existing issues. ### Environment ```Markdown - Version or commit-id (e.g. v0.1.0 or 8b23a93): -...

fix: more granular conversion from object_store::Error to Error

5

This PR tries to fix issue #2067 by adding a more granular conversion from object_store::Error types to Error types.

#147

https://github.com/lanterndata/lantern/blob/33e6bcfbe4336b61d11344ef5b55b88a6fd688a2/src/hnsw/utils.c#L81-L82

as @wjones127 pointed out in #2316, currently our codebase has some errors that are returned as the wrong error type, for example: [HashJoiner](https://github.com/lancedb/lance/blob/79de38027d03b049bba17c8a41f8f677dc1b110b/rust/lance/src/dataset/hash_joiner.rs#L55-60) ```rust return Err(Error::IO { message: "HashJoiner: No...

page compaction after encoding

3

For encodings like fsst, we store the decoding meta data in `PageInfo.encoding`, however, we may expect many decoding meta data in one page after compaction

lance run benchmark panic

2

steps to reproduce: 1: git clone lance(44d978be5b4e483597f3422affd7999c68) 2: run `RUST_BACKTRACE=1 cargo bench -- bench_hnsw` panic output:

fix: spawn multiple encoding task inside a batch number of rows in one column when needed

7

#2561 list and struct remain unchanged, I want to get some discussion and advise of this approach first

Currently, we spawn only one encoding task per batch number rows in a single column, which can be problematic for datasets like `MS MARCO` where the third column is significant...

discussion(encoding): The FastLanes Compression Layout

2

[The FastLanes Compression Layout](https://www.vldb.org/pvldb/vol16/p2132-afroozeh.pdf)