apache
feat(search): Support mutex to guarantee index update thread safety
Closes #2489
Summary
This PR fixes the concurrency issue mentioned in #2481, however, this is now pretty slow for uploading operation and bounded by the computational resources from my local laptop. I'll set up the same instance capacity as mentioned in the post and benchmark.
Tried the benchmark framework today, which took more than one hour to upload, but didn't even finish. I thought there was something wrong and killed the process, but after double-checking the redis benchmark here, seems like it also took redis 1.5 hours. Will try it again these two days, but one call-out is that kvrocks definitely doesn't fully utilize the compute resources.
@Beihao-Zhou Do you know if theres a place where you can configure your compute for kvrocks?
@Beihao-Zhou Do you know if theres a place where you can configure your compute for kvrocks?
Thanks a lot for pointing this out! I thought I was bottlenecked by lock itself so didn't try different config options, will look into them!
Apology for the late update! Was busy with relocation for the past two weeks.
Quick Update: I tried to benchmark the default dataset upload operation, but it took around 4.5 hours and didn't even finish... I can't quite estimate how long it will take, but notice that only one thread will use 100% CPU during the majority time. I'm not quite sure if this is because we used mutex here. I'll try to tune CPU usage in the config and run the benchmark again these two days; however, let me know if you have any better ideas! @PragmaTwice
I'll take this task and submit a patch for it : )
Thanks a ton!! Was too distracted by work these days.... My apology 😭