IGB-Datasets icon indicating copy to clipboard operation
IGB-Datasets copied to clipboard

Torch mmap support

Open msharmavikram opened this issue 2 years ago • 0 comments

Currently, IGB dataloaders use numpy mmap. numpy mmap is extremely slow and single-threaded. We should consider enabling torch mmap to enable multithreaded support when reading the data from the disk. We should also evaluate how much improvement we get with torch mmap compared to numpy mmap.

msharmavikram avatar Mar 09 '23 20:03 msharmavikram