lance
lance copied to clipboard
Enable setting batch size in `add_columns`
Default batch size for lance may cause OOM depending on the size of a row and available system memory
Yes, we should make this configurable. However, in the meantime, you can use LANCE_DEFAULT_BATCH_SIZE for spots where we don't make it configurable.
Is num_threads configurable via an environment variable as well?
nvm, looks like it is via LANCE_IO_THREADS
There is also a LANCE_CPU_THREADS. The difference between LANCE_IO_THREADS and LANCE_CPU_THREADS is described in more detail here.