Throttle dispatcher service's redis reconnection rate

[jwestfall69]

The dispatcher service can enter a state where its trying to connect to the redis service 1000's of times a second if the redis port is returning connection refused.

This PR makes use of the python redis module's built-in backoff/retry logic to rate limit reconnections to once every 2 seconds (per thread) and will retry the current redis action 5 times before throwing an exception.

Fixes #16513

DO NOT DELETE THE UNDERLYING TEXT

Please note

Please read this information carefully. You can run ./lnms dev:check to check your code before submitting.

• [X] Have you followed our code guidelines?
• [ ] If my Pull Request does some changes/fixes/enhancements in the WebUI, I have inserted a screenshot of it.
• [ ] If my Pull Request makes discovery/polling/yaml changes, I have added/updated test data.

Testers

If you would like to test this pull request then please run: ./scripts/github-apply <pr_id>, i.e ./scripts/github-apply 5926 After you are done testing, you can remove the changes with ./scripts/github-remove. If there are schema changes, you can ask on discord how to revert.

[CLAassistant]

All committers have signed the CLA.

[jwestfall69]

Hrm, guess I need some guidance here. Running the lint check locally says there were no issues, but the PR job says there are?

librenms@scratch:~$ ./lnms dev:check
Running Python lint check... 
success (2.62s)
Style check skipped.
Unit check skipped.

[murrant]

CI uses python black which is very aggressive with python formatting. The local check does not use that.

[peejaychilds]

As someone that had once had a redis pop generate an insane amount of logging this looks awesome 🙏