chainlink icon indicating copy to clipboard operation
chainlink copied to clipboard

Dequeue upkeeps using the current dequeue iteration

Open ferglor opened this issue 1 year ago • 1 comments

https://smartcontract-it.atlassian.net/browse/AUTO-9357

This PR consists of the following key change:

  • Calculate the number of iterations needed to dequeue the minimum number of logs from the log buffer
    • Use a custom upkeep selector to determine which upkeeps we should dequeue logs for in the current dequeue round

The goal here is that, in cases where the number of upkeeps exceeds the maximum number of results for a single call to log dequeue, we want to dequeue logs for subsets of upkeep IDs, in successive calls to dequeue, until at least the minimum number of logs have been dequeued for all upkeep IDs.

For example, if maxResults is 100, but we store logs for 200 upkeeps; even if we choose to dequeue at most 1 log for each upkeep, we could still only dequeue logs for at most 100 upkeeps in a single dequeue call. This means that in order to dequeue 1 log for all 200 upkeeps; we would need to make 2 calls to dequeue, and for each call, ensure that we dequeue for two unique, distinct sets of upkeep IDs.

To do this determinstically, we can:

  • First, calculate the total number of dequeue calls we would need to make to dequeue for all upkeep IDs. We refer to this number as iterations
  • Add a state variable to the provider, that tracks the current iteration, in the range of (0...iterations), and in each dequeue round, define a custom upkeep selector such that upkeepID % iterations == currentIteration
  • At the end of each dequeue round, increment the currentIterations counter, so that on the next call to dequeue, the upkeep selector will cause a different subset of upkeep IDs to be selected

Testing

Node upgrade test is passing here: https://github.com/smartcontractkit/chainlink/actions/runs/8871312521 Load test run here: https://github.com/smartcontractkit/chainlink/actions/runs/8989012623/job/24691139888

Additionally, some unit tests have been added that demonstrate the behaviour when the buffer has logs for a small number of upkeeps, a large number of upkeeps, and a large number of upkeeps that increases midway through the dequeue rounds.

ferglor avatar Apr 16 '24 22:04 ferglor