dice icon indicating copy to clipboard operation
dice copied to clipboard

BUG: Deadlock while cache eviction

Open apoorvyadav1111 opened this issue 1 year ago • 4 comments

Hi team,

This issue was first reported as per conversation on the discord community by Ritik. To replicate the issue, please follow the steps below:

  1. Update the KeysLimit in dice/config/config.go var KeysLimit int = 2 // default buffer size.
  2. Run following commands mset a b c d works fine set x y should lead to deadlock

Upon initial debugging, I found the issue is due to the locks being acquired in functions when trying to set a new value. Please find the sequence leading to deadlock below:

  1. evalSet calls store.Put()
  2. store.Put() calls PutHelper with acquiring Lock()
  3. PutHelper() validates the key limit and goes for eviction via store.evict()
  4. In populateEvictionPool() we try to get RLock() to evict, which causes deadlock as Lock() is already acquired on step 2.

cc: @JyotinderSingh

Regards, Apoorv

apoorvyadav1111 avatar Sep 10 '24 04:09 apoorvyadav1111

Hey @apoorvyadav1111 are you picking this up? If not I would like to pick this up

aashraybhandar1 avatar Sep 10 '24 05:09 aashraybhandar1

Hey @aashraybhandar1 , You can pick this up. Though, I don't have GitHub permission to assign this issue to you. Apoorv

apoorvyadav1111 avatar Sep 10 '24 05:09 apoorvyadav1111

@aashraybhandar1 Assigned, thanks for contributing.

lucifercr07 avatar Sep 10 '24 11:09 lucifercr07

@lucifercr07 @apoorvyadav1111 Created a PR: #529 . Would appreciate feedback from you guys. Thank You!

aashraybhandar1 avatar Sep 10 '24 20:09 aashraybhandar1

No longer an issue since we removed all store level locks.

JyotinderSingh avatar Sep 11 '24 13:09 JyotinderSingh