BackbeatConsumer : ensure ordered processing of messages with the same key

[Kerkesni]

Instead of processing all messages concurrently, we now queue messages with the same key and process them one after the other in order. Messages with no key are processed concurrently as before.

The taskScheduler was re-used for this task as it allows queuing and sequential processing of tasks based on a key. This class was enhanced to support limiting the concurrency of tasks without a key + helper function to allow flow control in BackbeatConsumer.

Issue: BB-645

[bert-e]

Hello kerkesni,

My role is to assist you with the merge of this pull request. Please type @bert-e help to get information on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	:star:
/bypass_build_status	Bypass the build and test status	:star:
/bypass_commit_size	Bypass the check on the size of the changeset TBA	:star:
/bypass_incompatible_branch	Bypass the check on the source branch prefix	:star:
/bypass_jira_check	Bypass the Jira issue check	:star:
/bypass_peer_approval	Bypass the pull request peers' approval	:star:
/bypass_leader_approval	Bypass the pull request leaders' approval	:star:
/approve	Instruct Bert-E that the author has approved the pull request.		:writing_hand:
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue BB-645 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.1.0

Please check the Fix Version/s of BB-645, or the target branch of this pull request.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 73.16%. Comparing base (df0d3fc) to head (658f6cd). Report is 15 commits behind head on development/9.1.

Additional details and impacted files

Files with missing lines	Coverage Δ
extensions/gc/GarbageCollector.js	40.90% <ø> (ø)
extensions/lifecycle/LifecycleConfigValidator.js	100.00% <100.00%> (ø)
...ecycle/bucketProcessor/LifecycleBucketProcessor.js	79.48% <ø> (ø)
...ecycle/objectProcessor/LifecycleObjectProcessor.js	73.07% <ø> (ø)
...ectProcessor/LifecycleObjectTransitionProcessor.js	50.74% <ø> (ø)
...ns/mongoProcessor/MongoProcessorConfigValidator.js	100.00% <100.00%> (ø)
extensions/mongoProcessor/MongoQueueProcessor.js	69.41% <ø> (ø)
...nsions/notification/NotificationConfigValidator.js	100.00% <100.00%> (ø)
...ions/notification/queueProcessor/QueueProcessor.js	56.86% <ø> (ø)
...tensions/replication/ReplicationConfigValidator.js	100.00% <100.00%> (ø)
... and 6 more

Components	Coverage Δ
Bucket Notification	75.40% <100.00%> (+0.03%)	:arrow_up:
Core Library	79.54% <100.00%> (+0.14%)	:arrow_up:
Ingestion	70.17% <100.00%> (+0.02%)	:arrow_up:
Lifecycle	76.88% <100.00%> (ø)
Oplog Populator	85.06% <ø> (ø)
Replication	59.27% <100.00%> (+0.01%)	:arrow_up:
Bucket Scanner	85.60% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/9.1    #2626      +/-   ##
===================================================
+ Coverage            73.08%   73.16%   +0.07%     
===================================================
  Files                  201      201              
  Lines                13413    13445      +32     
===================================================
+ Hits                  9803     9837      +34     
+ Misses                3600     3598       -2     
  Partials                10       10              

Flag	Coverage Δ
api:retry	9.50% <12.50%> (+0.01%)	:arrow_up:
api:routes	9.31% <12.50%> (+0.01%)	:arrow_up:
bucket-scanner	85.60% <ø> (ø)
ft_test:queuepopulator	9.04% <41.66%> (+0.13%)	:arrow_up:
ingestion	12.58% <16.66%> (+0.03%)	:arrow_up:
lib	7.63% <89.58%> (+0.29%)	:arrow_up:
lifecycle	18.99% <87.50%> (+0.26%)	:arrow_up:
notification	1.06% <0.00%> (-0.01%)	:arrow_down:
replication	18.89% <97.91%> (+0.20%)	:arrow_up:
unit	48.97% <70.83%> (+0.07%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Kerkesni]

/approve

[bert-e]

In the queue

The changeset has received all authorizations and has been added to the relevant queue(s). The queue(s) will be merged in the target development branch(es) as soon as builds have passed.

The changeset will be merged in:

• :heavy_check_mark: development/9.1

The following branches will NOT be impacted:

• development/7.10
• development/7.4
• development/7.70
• development/8.6
• development/9.0

There is no action required on your side. You will be notified here once the changeset has been merged. In the unlikely event that the changeset fails permanently on the queue, a member of the admin team will contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

• Any commit you add on the source branch will trigger a new cycle after the current queue is merged.
• Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a member of the admin team now.

The following options are set: approve

[bert-e]

I have successfully merged the changeset of this pull request into targetted development branches:

• :heavy_check_mark: development/9.1

The following branches have NOT changed:

• development/7.10
• development/7.4
• development/7.70
• development/8.6
• development/9.0

Please check the status of the associated issue BB-645.

Goodbye kerkesni.