Optionally limit concurrency of job execution

[patrickmann]

References Graylog2/graylog-plugin-enterprise#6313 /prd Graylog2/graylog-plugin-enterprise#6399 /nocl See Enterprise repo

Adds the ability to configure clusterwide max concurrency for specified job types. No more than this number of worker threads will be scheduled concurrently across the entire cluster.

Description

New configuration setting, e.g.: job_scheduler_max_concurrent_jobs = event-processor-execution-v1:2,notification-execution-v1:2 When nothing is specified we default to the regular behavior, i.e. no limitations.

Note that it is OK for concurrency values to add up to more than the max number of worker threads; or even for a single value to exceed that.

A meter job_scheduler_executions_deny_rate tracks the rate at which triggers are denied execution due to lock contention. Gauge job_scheduler_executions_rescheduled shows how many times jobs are being rescheduled before running.

Motivation and Context

Provides granular control of resource allocation. See linked issue for more details.

How Tested

Create a bunch of event definitions that run frequently. Observe execution deny rate is low. Configure a low maximum concurrency level and restart server. Observe execution deny rate metric increases as available threads are insufficient to handle query rate. Granular testing is really only possible via the integration test.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [x] Refactoring (non-breaking change)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [ ] I have read the CONTRIBUTING document.
• [x] I have added tests to cover my changes.