BrighterCommand
[Bug] Auto-restart Performers
Is your feature request related to a problem? Please describe.
We noticed that a consumer count was going down and not recovering in Rabbit. Restarting a service brings the count back up.
From what we found in code, Brighter knows how to get to required number on start-up, but we could not find anything about some kind of a process that oversees the numbers of performers and gets them back up.
Describe the solution you'd like
Have auto-restart flag for performers, so they reconnect in case they disappear. It would also be nice to know why a performer disappeared.
Describe alternatives you've considered Restarting services. but gets a bit tedious.
Additional context Add any other context or screenshots about the feature request here.
Ian's comments on the issue from gitter:
When we lose a performer we don't auto-restart it. We could. Probably we want to make that configurable i.e. auto-restart and we would need to make sure that we were not in shutdown when we did that.
I am looking at message pumps (kick @DevJonny for me) and I could add that feature pretty fast, if you guys wanted to raise an issue
Let me grab this one, as I am playing with our pump options and management and my head is in the right space
We I fixed https://github.com/BrighterCommand/Brighter/commit/6019b7c9874b230e19533bd0f5a03801c53c770c as part of https://github.com/BrighterCommand/Brighter/pull/1067 our disappearing consumers stopped.
@holytshirt I think this one still needs attention. Try the following, spin up an RMQ sample against docker-hosted infra. Kill the infra. The consumer loops but eventually kills its performers. It then won't restart once the infra comes back online. It ought to monitor its performer count and restart them if it falls below the desired number