Redis Sentinel / Connection Leak

[cboudereau]

Redis Sentinel / Connection Leak

StackExchange.Redis (2.8.24) / OpenTelemetry.Instrumentation.StackExchangeRedis (1.9.0-beta.1)

Description

We are observing a connection leak in our production servers running redis sentinel.

First of all, we checked that there is no ConnectionMultiplexer instance leak in our code base and in the dump.

It turns out that when switching primary, the ConnectionMultiplexer's servers are cleared but the _serverSnapshot is not which causes duplicates (see below 1. figure) in the _serverSnapshot which pins PhysicalBridges (see below 2. and 3. figures).

1. Duplicates in _serverSnapshot

2. Leak of PhysicalBridge : 338

3. Similar Retention (max 204) with a common path:

It seems that when adding a new endpoint to the _serverSnapshot the servers field has been previously cleared and duplicates appear in the _serverSnapshot field.

How to reproduce

1. Setup a redis sentinel
2. Create a ConnectionMultiplexer with StackExchange.Redis (2.8.24)
3. Instrument with OpenTelemetry.Instrumentation.StackExchangeRedis (1.9.0-beta.1)
4. Switch primary / replica over and over
5. Observe the number of connections.