StackExchange.Redis icon indicating copy to clipboard operation
StackExchange.Redis copied to clipboard

Allow heartbeat to restart the pipe thread with only sync commands

Open frobion opened this issue 4 months ago • 0 comments

There is a thread looping in the method PhysicalConnection.ReadFromPipe to process response from Redis, match them with the sent command and signaling the completion of the message. If this thread has an exception, its catch block will call RecordConnectionFailed which will proceed to restart a new thread to continue reading Redis responses.

However, if another exception occurred in the catch before the new thread can be started (in a case of high memory pressure, OOM exceptions can happen anywhere) we are in a state where no one is reading the pipe of Redis responses, and all commands sent end in timeout.

If at least one async command is sent, the heartbeat thread will detect the timeout in the OnBridgeHeartbeat method, and if no read were perform for 4 heartbeat it will issue a connection failure. However, no such protection were in place if only sync commands are sent. In this case, they were all ending in timeout without any mechanisms to start reading their responses again.

In this PR, the heartbeat thread will check timeouts for sync commands as well. Therefore, it will be able to start the thread looping in ReadFromPipe even if only sync commands are sent, ensuring we will not reach a state were all commands end in timeout.

This PR is loosely linked to the issue #2919, as this problem and its correction were found during investigation of the issue. Moreover, it can be related to issue #2888 as the symptoms are similar to what I observed. However, I don't know if in this case only synchronous commands were sent or not.

frobion avatar Oct 01 '25 09:10 frobion