Support prometheus

[mmpei]

https://github.com/uber/kraken/issues/161 add 'http_' as prefix of metrics name. for there is a restriction in prometheus that metrics name should start with letter. and you can set backend to 'promethues' to active prometheus metrics.

metrics:
  backend: prometheus
  m3:
    service: kraken-agent
    host_port: 10740
  statsd:
    host_port: 0.0.0.0:10740
    prefix: kraken-agent
  prometheus:
    listen_address: 0.0.0.0:10740

[evelynl94]

This is going to break all of our existing monitoring. Is there a way to make this backwards compatible?

+1, maybe its better to wrap promethues metrics to add a prefix.

[evelynl94]

also http_ usually implies its related to http endpoints...you should just use kraken_

[mmpei]

This is going to break all of our existing monitoring. Is there a way to make this backwards compatible?

+1, maybe its better to wrap promethues metrics to add a prefix.

it is difficult to do. i have considered that but it will break the abstract architecture if only add prefix to promethues. i will try.

[mmpei]

also http_ usually implies its related to http endpoints...you should just use kraken_

you are right. maybe kraken_http_ to identify that it's from kraken and related to http.

[yiranwang52]

I actually think this change is fine...let's discuss a little more about this