ThreadPool seems to be broken (python 2.7.14, billiard 3.5.0.3): "no attribute _writer"

[guyarad]

What am I missing?

Cannot instantiate ThreadPool. Exception: AttributeError: Queue instance has no attribute '_writer' Relevant output:

In [9]: from billiard.pool import ThreadPool
In [10]: import sys
In [11]: print sys.version
2.7.14 (default, Dec  3 2017, 15:45:23)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)]

In [12]: print billiard.__version__
3.5.0.3

In [13]: tp = ThreadPool()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-13-f9f264fdac57> in <module>()
----> 1 tp = ThreadPool()

/Users/guyarad/src/venvs/archive_monitors/lib/python2.7/site-packages/billiard/pool.pyc in __init__(self, processes, initializer, initargs)
   1983
   1984     def __init__(self, processes=None, initializer=None, initargs=()):
-> 1985         Pool.__init__(self, processes, initializer, initargs)
   1986
   1987     def _setup_queues(self):

/Users/guyarad/src/venvs/archive_monitors/lib/python2.7/site-packages/billiard/pool.pyc in __init__(self, processes, initializer, initargs, maxtasksperchild, timeout, soft_timeout, lost_worker_timeout, max_restarts, max_restart_freq, on_process_up, on_process_down, on_timeout_set, on_timeout_cancel, threads, semaphore, putlocks, allow_restart, synack, on_process_exit, context, max_memory_per_child, enable_timeouts, **kwargs)
   1005         self._putlock = semaphore or LaxBoundedSemaphore(self._processes)
   1006         for i in range(self._processes):
-> 1007             self._create_worker_process(i)
   1008
   1009         self._worker_handler = self.Supervisor(self)

/Users/guyarad/src/venvs/archive_monitors/lib/python2.7/site-packages/billiard/pool.pyc in _create_worker_process(self, i)
   1107             sigprotection=self.threads,
   1108             wrap_exception=self._wrap_exception,
-> 1109             max_memory_per_child=self._max_memory_per_child,
   1110         ))
   1111         self._pool.append(w)

/Users/guyarad/src/venvs/archive_monitors/lib/python2.7/site-packages/billiard/pool.pyc in __init__(self, inq, outq, synq, initializer, initargs, maxtasks, sentinel, on_exit, sigprotection, wrap_exception, max_memory_per_child)
    248         self.inq, self.outq, self.synq = inq, outq, synq
    249         self.wrap_exception = wrap_exception  # XXX cannot disable yet
--> 250         self.contribute_to_object(self)
    251
    252     def contribute_to_object(self, obj):

/Users/guyarad/src/venvs/archive_monitors/lib/python2.7/site-packages/billiard/pool.pyc in contribute_to_object(self, obj)
    252     def contribute_to_object(self, obj):
    253         obj.inq, obj.outq, obj.synq = self.inq, self.outq, self.synq
--> 254         obj.inqW_fd = self.inq._writer.fileno()    # inqueue write fd
    255         obj.outqR_fd = self.outq._reader.fileno()  # outqueue read fd
    256         if self.synq:

AttributeError: Queue instance has no attribute '_writer'

[calhamdower]

bump?

[thedrow]

ThreadPool is broken. We're not using it in Celery.

In Celery 5 we won't need billiard anymore so there's not much effort going on to fix this. Feel free to create a PR to do so if you wish.