lancedb
add packed struct encoding
Currently struct arrays are encoded using multiple IOPS for random access, which is suboptimal.
e.g. if we have a struct array x: {1, 2, 3}, y: {2.4, 5.6, 3.8}, z: ['a', 'b', 'c'], with x of type uint64, y of type Float32, and z of type UInt8
in the packed encoding, we would encode fields of the struct close together as a "packed struct": [1, 2.4, 'a', 2, 5.6, 'b', 3, 3.8, 'c']
This is better for random access since decoding will only use 1 IOP for random access. Of course this means the fields would have to be decoded together. But often, this is fine (especially in cases where we need to access multiple fields of the struct anyway)
This would involve encoding the individual child arrays, and then packing the encoded data together afterwards.
Tasks:
- [x] Packed struct encoding + packing
- [x] This should be an array encoder (should fall under the primitive field encoding path)
- [x] Packed struct unpacking + decode
- [x] Select which encoder to use (default struct encoder vs packed struct encoder) based on flag in metadata
