Kyle Vedder

icon indicating a website link vedder.io icon indicating an email [email protected]

icon company Dyna Robotics icon location Redwood City, CA icon location Scaling on the right data is all you need. CS PhD from Penn

Results 4 comments of Kyle Vedder

`run_pipeline.py` hangs on Open3D's `iou_bev` invocation in PyTorch

I was able to reproduce this error on a friend's system using my `Dockerfile`, making me think this is not an issue with my base machine.

`run_pipeline.py` hangs on Open3D's `iou_bev` invocation in PyTorch

I tried compiling Open3D from source inside the docker container. I was able to get it to compile if I did not compile the ML library, causing the training pipeline...

`run_pipeline.py` hangs on Open3D's `iou_bev` invocation in PyTorch

As per @sanskar107's suggestion, adding `--pipeline.num_workers 0` serves as a viable workaround. As suggested, the root cause seems to be any custom op in the PyTorch dataloader, not just Open3D's...

`run_pipeline.py` hangs on Open3D's `iou_bev` invocation in PyTorch

Setting `OMP_NUM_THREADS=1` and running with the default number of workers fixes this issue and has significantly higher throughput than setting `---pipeline.num_workers 0`. I will investigate this thread more later.