boxed
Mutmut stops running
Hi,
Mutmut stops running after a while and outputs the following error:
Exception in thread queue_mutants:
Traceback (most recent call last):
File ".../python3.7/site-packages/mutmut/__init__.py", line 712, in queue_mutants
mutants_queue.put(('mutant', context))
File "/.../python3.7/multiprocessing/queues.py", line 81, in put
assert not self._closed, "Queue {0!r} has been closed".format(self)
AssertionError: Queue <multiprocessing.queues.Queue object at 0x7fe5a0b69f50> has been closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/.../python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/.../python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/.../python3.7/site-packages/mutmut/__init__.py", line 715, in queue_mutants
mutants_queue.put(('end', None))
File "/.../python3.7/multiprocessing/queues.py", line 81, in put
assert not self._closed, "Queue {0!r} has been closed".format(self)
AssertionError: Queue <multiprocessing.queues.Queue object at 0x7fe5a0b69f50> has been closed
Traceback (most recent call last):
File "/.../python3.7/multiprocessing/queues.py", line 242, in _feed
send_bytes(obj)
File "/.../python3.7/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/.../python3.7/multiprocessing/connection.py", line 398, in _send_bytes
self._send(buf)
File "/.../python3.7/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
I am running mutmut in a conda environment with python 3.7 on Ubuntu 18.04. Mutmut version is 2.1.0.
Mutmut command used: mutmut run --with-coverage --runner="pytest ."
I tried using the installing mutmut from sources, but ran into the same error. Most times, mutmut just exits without any error message.
Any ideas what is going wrong here?
Well that's super strange! Multiprocessing queues should be pretty straight forward. I'd have to guess that the worker process dies for some reason. But it's not the same mutant that it stops on every time?
Hmm.. maybe you could try running with the -s flag? It will dump all output from the workers into the console. Hopefully you can see something there when they shut down... It's a long shot though.
I face this same issue issue non-deterministically when performing a lot of mutant skips. If I don't add a skip or I add in a long enough delay, then the issue does not come up.
@boxed Happening for me as well, and I just recently started using this tool today.
I'm on Python 3.9.6 and I installed mutmut with pip install mutmut.
My mutmut version is 2.1.0.
I'm running mutmut on a Chalice project with the following structure:
my-project/
chalicelib/
tests/
Here's the command I used, without any mutmut config files:
mutmut run --paths-to-mutate=chalicelib/
It randomly stops at around 100 or so mutants, usually with no error message. However, it did give me this recently:
Exception in thread queue_mutants:
Traceback (most recent call last):
File "c:\python39\lib\site-packages\mutmut\__init__.py", line 712, in queue_mutants
Traceback (most recent call last):
mutants_queue.put(('mutant', context))
File "c:\python39\lib\multiprocessing\queues.py", line 88, in put
File "c:\python39\lib\multiprocessing\queues.py", line 247, in _feed
send_bytes(obj)
File "c:\python39\lib\multiprocessing\connection.py", line 205, in send_bytes
self._send_bytes(m[offset:offset + size])
raise ValueError(f"Queue {self!r} is closed")
File "c:\python39\lib\multiprocessing\connection.py", line 285, in _send_bytes
ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
ValueError: Queue <multiprocessing.queues.Queue object at 0x000001617B72CCD0> is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\python39\lib\threading.py", line 954, in _bootstrap_inner
BrokenPipeError: [WinError 232] The pipe is being closed
I'm running into the same issue now. Currently running mutmut 2.1.0 (installed with python3 -m pip install mutmut), and Python 3.6.9.
Mutmut will just stop after 100 tests each time and require a restart.
I am running mutmut with the following command: mutmut run --runner="python3 -m pytest ."
@MirceaZisu Have you tried running it with the -s flag and see if that gives you any output?
Yes, I have. And it doesn't show anything out of the ordinary, other than pytest tests failing or passing, due to various reasons within my project.
⠧ 339/1228 🎉 140 ⏰ 0 🤔 0 🙁 199 🔇 0============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/mircea/asn1_parser
collecting ...
collected 137 items
tests/unit/test_hello.py . [ 0%]
tests/unit/asn1_parser/asn1/validation/test_multi_sequence_invalid.py .. [ 2%]
[ 2%]
tests/unit/multi_module/test_multi_comments.py ......................... [ 20%]
............ [ 29%]
tests/unit/multi_module/test_multi_multiimport.py . [ 29%]
tests/unit/multi_module/test_multi_sequence.py . [ 30%]
tests/unit/single_module/test_array.py ... [ 32%]
tests/unit/single_module/test_choice.py ... [ 35%]
tests/unit/single_module/test_choice_invalid.py .... [ 37%]
tests/unit/single_module/test_comments.py .............................. [ 59%]
... [ 62%]
tests/unit/single_module/test_comments_invalid.py ssssss. [ 67%]
tests/unit/single_module/test_enumerated.py ...... [ 71%]
tests/unit/single_module/test_enumerated_invalid.py ......... [ 78%]
tests/unit/single_module/test_sequence.py ...... [ 82%]
tests/unit/single_module/test_sequence_invalid.py .......... [ 89%]
tests/unit/single_module/test_simple_definition.py ...ss...ss [ 97%]
tests/unit/single_module/test_simple_definition_invalid.py .. [ 98%]
tests/unit/single_module/test_string.py .. [100%]
======================= 127 passed, 10 skipped in 6.83s ========================
⠏ 340/1228 🎉 140 ⏰ 0 🤔 0 🙁 200 🔇 0
mircea@mircea-N7x0WU ~/asn1_parser %
These are the logs at the end of the current -s run. The tests stopped at 340, as it killed lots of mutants on its own at certain points, but still ran 100 pytest runs twice.
Same issue here. It stops every 100 mutants. Made some experiments and found that if I manually change the value of cycle_process_after in create_worker() function to be equal to an integer n, then it stops every n mutants. (I have tested values between 5 and 150.)
I don't know if it's safe to arbitrarily increase the value of that parameter, I guess not.
That's an interesting finding. At least that should mean you can set it to 1 and reproduce it very fast?
Hi! I stumbled upon this problem and after some testing and debugging, I proposed a fix in #269.
As I discovered, the most probable root cause (at least in my case) was the race condition between results consumer loop (in run_mutation_tests) and mutant worker loop (in check_mutants). The former relies on the current worker of the latter being "alive", so when the worker sends cycle and terminates before the former digests the last status command, loop is over before cycle is consumed. This caused mutmut to finish prematurely, leaving one of the threads (queue_mutants) orphaned, hence the hanging.
More information can be found in PR comment.
