Expose metrics from Dispatch

[kars7e]

Feature Request

Detailed Description

This issue tracks the metrics implementation in Dispatch. The goal is to collect metrics for different aspects of Dispatch and push them to some backend(ideally customizable).

Context

• Metrics require code instrumentation (much less than tracing). We need to decide what kind of metrics we want to collect.
• As an open source project, we should try to provide an option to use an opensource metrics implementation. The most popular choice seems to be Prometheus. We would also like to support other backends, notably Wavefront.
• Should be helpful in scale & performance testing.

Possible Implementation

Popular go metrics libraries:

• go-metrics - very popular, but seems maintained the least. Many open issues without any comments, missing big features like tags. Existing library to integrate with Wavefront
• Tally - less popular, seems better maintained, has tags and Prometheus integration. No Wavefront client.
• Prometheus - average popularity, client for Prometheus only (but many backends are compatible with Prometheus). Used by OpenFaas and Kubeless. Provides useful helpers for HTTP instrumentation. Important: Prometheus works in pull model (Dispatch would have to expose an endpoint that Prometheus can scrape the metrics from).

Example metrics we could expose

• Basic HTTP metrics - number of simultaneous requests in flight, number of total requests, request duration, time to write, request/response size (each metric per handler/resource type)
• Counters/Durations/Gauges for async operations - number of functions being created at the same time, number of errors, time it took to create/delete/update the function. (replace "function" with other resource types)
• Metrics for resources we use in different backends - number of images in embedded registry, resources used in kubernetes, size of the DB, etc. (some of them can be reported independently of Dispatch)

Complexity

• [ ] Low - Simple enhancement or bug fix, no architectural changes or refactoring
• [x] Medium - Change requires some thought, but is relatively isolated
• [ ] High - Significant architectural change or large refactor