lance icon indicating copy to clipboard operation
lance copied to clipboard

Enable setting batch size in `add_columns`

Open tonyf opened this issue 1 year ago • 4 comments

Default batch size for lance may cause OOM depending on the size of a row and available system memory

tonyf avatar Aug 28 '24 15:08 tonyf

Yes, we should make this configurable. However, in the meantime, you can use LANCE_DEFAULT_BATCH_SIZE for spots where we don't make it configurable.

westonpace avatar Aug 28 '24 17:08 westonpace

Is num_threads configurable via an environment variable as well?

tonyf avatar Aug 28 '24 17:08 tonyf

nvm, looks like it is via LANCE_IO_THREADS

tonyf avatar Aug 28 '24 17:08 tonyf

There is also a LANCE_CPU_THREADS. The difference between LANCE_IO_THREADS and LANCE_CPU_THREADS is described in more detail here.

westonpace avatar Aug 28 '24 22:08 westonpace