python-plyfile icon indicating copy to clipboard operation
python-plyfile copied to clipboard

Reading ply file is too slow (1.5 GB)

Open CUBICinfinity opened this issue 6 years ago • 5 comments

I have some code that does what I need it to do, but when testing with a larger sample of data (about 1.5GB) it seems to hang on reading the file to memory. I have waited indefinitely for this to complete: plydata = PlyData.read('C:/Users/Jim/Desktop/data.ply')

CUBICinfinity avatar Jul 10 '19 20:07 CUBICinfinity

There's an efficient loader for large elements via memory-mapping, but it requires:

  • Binary mode, not ASCII
  • No list properties
  • Underlying file can be memory-mapped

If those requirements aren't satisfied, then it will fall back to one of the slow loaders.

dranjan avatar Jul 11 '19 15:07 dranjan

By the way, here's the logic for deciding which reader it can use: https://github.com/dranjan/python-plyfile/blob/9f8e8708d3a071229cf292caae7d13264e11c88b/plyfile.py#L638-L656

The middle branch, where _can_mmap(stream) and not self._have_list evaluates to True and text is False, is the only one I would expect to load your data efficiently.

dranjan avatar Jul 11 '19 20:07 dranjan

Is there a way to use the memory-mapping while the .ply file includes lists?

ttsesm avatar Aug 28 '20 13:08 ttsesm

Unfortunately elements that contain list properties can't be memory-mapped.

dranjan avatar Aug 29 '20 20:08 dranjan

I see. Thanks.

ttsesm avatar Aug 30 '20 22:08 ttsesm