KeyDB icon indicating copy to clipboard operation
KeyDB copied to clipboard

[BUG] unresponsive due to a lot of dead connections in state CLOSE_WAIT

Open dup2 opened this issue 3 years ago • 5 comments

Describe the bug

The server can not be contacted anymore to dead connections in state CLOSE_WAIT

To reproduce

We are using a rails application with around 7-10 persistent connections running on a Ubuntu 18.04 LTS system with a local keydb server installation.

Since the update from 6.2.2 to 6.3.1 this starts to occur.

We see a lot of IP connections in CLOSE_WAIT with lsof -i | grep CLOSE_WAIT

keydb-ser  6382           keydb *434u  IPv4 420149223      0t0  TCP localhost:6379->localhost:58424 (CLOSE_WAIT)
keydb-ser  6382           keydb *438u  IPv4 420153874      0t0  TCP localhost:6379->localhost:58496 (CLOSE_WAIT)
keydb-ser  6382           keydb *446u  IPv4 420156237      0t0  TCP localhost:6379->localhost:58932 (CLOSE_WAIT)
keydb-ser  6382           keydb *462u  IPv4 420172264      0t0  TCP localhost:6379->localhost:60060 (CLOSE_WAIT)
keydb-ser  6382           keydb *464u  IPv4 420173888      0t0  TCP localhost:6379->localhost:60110 (CLOSE_WAIT)
keydb-ser  6382           keydb *471u  IPv4 420174527      0t0  TCP localhost:6379->localhost:60422 (CLOSE_WAIT)
keydb-ser  6382           keydb *478u  IPv4 420181046      0t0  TCP localhost:6379->localhost:60910 (CLOSE_WAIT)
keydb-ser  6382           keydb *497u  IPv4 420193081      0t0  TCP localhost:6379->localhost:33818 (CLOSE_WAIT)
keydb-ser  6382           keydb *515u  IPv4 420202001      0t0  TCP localhost:6379->localhost:34960 (CLOSE_WAIT)
keydb-ser  6382           keydb *525u  IPv4 420208097      0t0  TCP localhost:6379->localhost:35472 (CLOSE_WAIT)
keydb-ser  6382           keydb *558u  IPv4 420231285      0t0  TCP localhost:6379->localhost:37398 (CLOSE_WAIT)
keydb-ser  6382           keydb *564u  IPv4 420251080      0t0  TCP localhost:6379->localhost:37740 (CLOSE_WAIT)

Expected behavior

KeyDB should close these connections and allow for new clients to connect.

Additional information

Running KeyDB 6.3.1 standalone on Ubuntu 18.04 LTS from the keydb.dev repo.

No replication, single node.

dup2 avatar Jul 08 '22 08:07 dup2

CLOSE_WAIT means that the connection was closed from the client side but keydb has not processed that and closed the connection, I was unable to reproduce this with keydb-cli/memtier_benchmark so it is likely specific to your client setup/workload, can you provide more details about those.

msotheeswaran-sc avatar Sep 06 '23 20:09 msotheeswaran-sc