concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending

[fenglonz]

Traceback (most recent call last): File "./deepwalk", line 11, in load_entry_point('deepwalk==1.0.3', 'console_scripts', 'deepwalk')() File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/main.py", line 162, in main process(args) File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/main.py", line 83, in process num_workers=args.workers) File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/walks.py", line 85, in write_walks_to_disk for file_ in executor.map(_write_walks_to_disk, args_list): File "/usr/local/python36/lib/python3.6/concurrent/futures/process.py", line 366, in _chain_from_iterable_of_lists for element in iterable: File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

[fenglonz]

84 with ProcessPoolExecutor(max_workers=num_workers) as executor: 85 for file_ in executor.map(write_walks_to_disk, args_list): 86 files.append(file)

It fails at walks.py:85. How to fix?

[GTmac]

What OS are you using? What is the exact command you ran? This question seems to be related: https://stackoverflow.com/questions/15900366/all-example-concurrent-futures-code-is-failing-with-brokenprocesspool

[NZiozis]

Yes, I too believe that it is related to the issue that you linked. The way that I run into this problem is when I try to run multiple workers, but can't keep all the walks in memory. So, when I try to run the following command on a large graph, it crashed with the original issue message above. I am using Debian Buster.

--number-walks 80 --representation-size 128 --walk-length 40 --window-size 10 --workers 20

Have you found a solution to this bug, or have any idea about how I can go about fixing it. I am not so familiar with the python ProcessPoolExecutor library, and as indicated by the article you linked the error messages aren't very helpful.