queue icon indicating copy to clipboard operation
queue copied to clipboard

utubettl: slow take

Open oleg-jukovec opened this issue 1 year ago • 0 comments

Our clients reported that utubettl:take() loads CPU too much and takes too much time when there is a utube with a big amount of tasks (+100.000).

  • We need a benchmark for the case in t/benchmark. It could be a queue with utubettl driver and 5-10 utubes with ~50.000 tasks + 5-10 parallel clients with take() + ack(). It will help to identify the problem.
  • We need an another benchmark in t/benchmark. With a big number of utubes (10.000) with a small number of tasks on each (4-5) + 5-10 parallel clients with take() + ack(). This way we can make sure that our fix doesn't degrade performance.
  • We need to fix the problem.

I suggest to make the task in the order:

  1. Create a benchmark for the problem and ensure that we have the problem.
  2. You offer solutions to the problem in the issue, which we will discuss.
  3. Fix the problem after discussion. Ensure that it helps.
  4. Create a regression benchmark and ensure that we don't degrade performance.

oleg-jukovec avatar Apr 12 '24 09:04 oleg-jukovec