cloud_controller_ng
cloud_controller_ng copied to clipboard
WIP: Replace Thin with Puma
This is a first pass based on the equivalent BOSH PR. CC uses a runner.rb rather than embedding the server startup directly in the bin/ script so move stuff there.
Still needs a lot of work:
- Not feature flagged (probably should be)
- Does not allow configuration of puma workers/threads
- Check signal handling works with BOSH release/k8s as this just lets Puma handle them directly rather than intercepting them
- Periodic metrics are added on every worker but most are gauges about the individual thread (see vitals) so will clash and cause confusing numbers as each worker reports. We could probably use https://github.com/yob/puma-plugin-statsd to replace the
thread_info(EM) metrics - Starting EM in a different thread means it may not get shutdown properly but given it just does metrics now I'm not sure this really matters anymore? (If you don't spawn the new thread it blocks the main thread)
- Are we really benefitting from
preload_app!? - we don't really need fast startup time for workers as we probably won't cycle them and it means we have to disconnect the DB after loading the models. It might be simpler to just start the CC after forking which should hopefully mean the DB only connects after forking - Some config options like DB connection pool will need expressing/updating to be per-worker to avoid over-allocating
Hopefully this helps at least have a play around with it. I'm pretty stuck on what to do about metrics - maybe try and uniquely identify a worker and put that identifier in the metric? We use Telegraf for collecting our StatsD metrics which supports tagging but not every StatsD implementation does so that may not be a great option.
cc: @sethboyles