lance
lance copied to clipboard
perf: benchmarking encodings (read time, write time)
Many things to add/change, but wanted to get a review on the initial structure. Some considerations -
- Can add write benchmarking within the same functions, and populate results in the same csv file. In the future if we want to benchmark query performance we can add new functions to the same file.
- Later we can extend this to compare with frameworks other than pyarrow. But for now I have just compared pyarrow and lance.
- Can extend to other datasets and encodings