kazoo icon indicating copy to clipboard operation
kazoo copied to clipboard

Unhandled exception in connection loop

Open FreezeIce1 opened this issue 4 years ago • 3 comments

When the service starts, kazoo success connect to zookeeper. At the same time, the node of ZK was created successfully.But there will be an error immediately, causing the session to close. `[2021-05-07 14:00:07] kazoo.client INFO [/usr/lib/python3.6/site-packages/kazoo/client.py:533] Zookeeper connection established, state: CONNECTED

[2021-05-07 14:00:07]gunicorn INFO [gunicorn_dev.py:241] register succ.path:/serversets/api-dev/0000000090api
[2021-05-07 14:00:11] kazoo.client ERROR [/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py:631] Unhandled exception in connection loop
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 602, in _connect_attempt
response = self._read_socket(read_timeout)
File "/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 453, in _read_socket
return self._read_response(header, buffer, offset)
File "/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 376, in _read_response
request, async_object, xid = client._pending.popleft()
IndexError: pop from an empty deque
[2021-05-07 14:00:11] kazoo.client INFO [/usr/lib/python3.6/site-packages/kazoo/client.py:537] Zookeeper session closed, state: CLOSED
[2021-05-07 14:00:11] kazoo.client ERROR [/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py:631] Unhandled exception in connection loop
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 602, in _connect_attempt
response = self._read_socket(read_timeout)
File "/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 435, in _read_socket
header, buffer, offset = self._read_header(read_timeout)
File "/usr/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 228, in _read_header
header, offset = ReplyHeader.deserialize(b, 0)
File "/usr/lib/python3.6/site-packages/kazoo/protocol/serialization.py", line 434, in deserialize
reply_header_struct.unpack_from(bytes, offset)), new_offset
struct.error: unpack_from requires a buffer of at least 16 bytes
[2021-05-07 14:00:11] root INFO [gunicorn_dev.py:210] zkserver session LOST`

I create the node of ephemeral, so i can't use zk.stop. I find it seem to i don't use zk.stop() cause this problem. This is the bug of kazoo ? How to fix this problem? Please kazoo:2.8.0 python:3.8

FreezeIce1 avatar May 07 '21 06:05 FreezeIce1

Hello,

I see you are using gunicorn, is it configured to use multithreading?

StephenSorriaux avatar May 07 '21 14:05 StephenSorriaux

Hello,

I see you are using gunicorn, is it configured to use multithreading?

I use prometheus multithread model, and my zookeeper verison is 3.6.0.

I check the code, in this case it set the workers =1. But in production we will set workers > 10,

FreezeIce1 avatar May 07 '21 14:05 FreezeIce1

Hi,

Can you please share a snippet (code + gunicorn configuration) that would help us understand/reproduce the issue?

StephenSorriaux avatar May 14 '21 14:05 StephenSorriaux