cortex icon indicating copy to clipboard operation
cortex copied to clipboard

Average time to evaluate rule groups is higher when using streaming of chunks

Open pstibrany opened this issue 4 years ago • 8 comments

When using streaming of chunks as implemented in https://github.com/cortexproject/cortex/pull/3889, average time to evaluate rule groups goes up.

pstibrany avatar Mar 16 '21 10:03 pstibrany

Is it because of the chunks decoding done in the ruler (instead of spreading it across all ingesters)? Do you have any ruler trace to investigate it further?

pracucci avatar Mar 16 '21 14:03 pracucci

Is it because of the chunks decoding done in the ruler (instead of spreading it across all ingesters)?

I assume that is the case, but haven't investigated it further yet. What is worrying is that rule evaluation has doubled in average, but it could also be specific to rules.

pstibrany avatar Mar 16 '21 14:03 pstibrany

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Jun 14 '21 20:06 stale[bot]

Still valid

pstibrany avatar Jun 30 '21 06:06 pstibrany

Was this fixed by #4341 ?

bboreham avatar Sep 23 '21 14:09 bboreham

Answering my own question: no it was not fixed. That part improves, but mergeIterator seems to be so much slower that it defeats the improvement.

bboreham avatar Nov 08 '21 11:11 bboreham

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Feb 08 '22 11:02 stale[bot]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Jun 18 '23 09:06 stale[bot]