audius-protocol icon indicating copy to clipboard operation
audius-protocol copied to clipboard

[CON-435] - Track metrics content stream

Open vicky-g opened this issue 3 years ago • 4 comments

Description

In this PR:

  • adds metrics to the /ipfs/<cid> and /ipfs/<dirCID>/<filename> routes
  • prometheus labels in the prometheus constants file + metric mapping for ipfs route
  • updates the prom middleware to add in the route regex into the express request object used to match on the metric associated with the route, if there is one
  • middleware to fetch the metric for a route, if it exists
  • wrapper api helper send fn that ends the metric, if it exists
  • misc. typescript changes, importantly including adding metrics properties to the request type

Tests

  • a unit test in creator-node/test/prometheus.test.js
# HELP audius_cn_stream_content_duration_seconds Time spent to stream content (seconds)
# TYPE audius_cn_stream_content_duration_seconds histogram
audius_cn_stream_content_duration_seconds_bucket{le="0.1",result="success_found_in_fs",mode="new_storage_path"} 20
audius_cn_stream_content_duration_seconds_bucket{le="0.2",result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_bucket{le="0.5",result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_bucket{le="1",result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_bucket{le="4",result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_bucket{le="15",result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_bucket{le="60",result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_bucket{le="+Inf",result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_sum{result="success_found_in_fs",mode="new_storage_path"} 1.250481766
audius_cn_stream_content_duration_seconds_count{result="success_found_in_fs",mode="new_storage_path"} 21
audius_cn_stream_content_duration_seconds_bucket{le="0.1",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_bucket{le="0.2",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_bucket{le="0.5",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_bucket{le="1",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_bucket{le="4",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_bucket{le="15",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_bucket{le="60",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_bucket{le="+Inf",result="abort_bad_cid",mode="default"} 1
audius_cn_stream_content_duration_seconds_sum{result="abort_bad_cid",mode="default"} 0.005209824
audius_cn_stream_content_duration_seconds_count{result="abort_bad_cid",mode="default"} 1

# HELP audius_cn_stream_content_dir_duration_seconds Time spent to stream content from directory (seconds)
# TYPE audius_cn_stream_content_dir_duration_seconds histogram
audius_cn_stream_content_dir_duration_seconds_bucket{le="0.1",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_bucket{le="0.2",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_bucket{le="0.5",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_bucket{le="1",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_bucket{le="4",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_bucket{le="15",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_bucket{le="60",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_bucket{le="+Inf",result="abort_cid_not_found_in_db"} 1
audius_cn_stream_content_dir_duration_seconds_sum{result="abort_cid_not_found_in_db"} 0.005693088
audius_cn_stream_content_dir_duration_seconds_count{result="abort_cid_not_found_in_db"} 1

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

Look at /prometheus_metrics to see if metrics are recorded for cids under audius_cn_stream_content_duration_seconds_bucket and audius_cn_stream_content_dir_duration_seconds_count

vicky-g avatar Oct 04 '22 17:10 vicky-g

@SidSethi could you take a look at this when you have a chance?

vicky-g avatar Oct 10 '22 15:10 vicky-g

@jonaylor89 I made some changes to the RequestWithLogger type

vicky-g avatar Oct 10 '22 15:10 vicky-g

@jonaylor89 tagging for review bc there are a lot of TS changes prob good to get your eyes on those at least

SidSethi avatar Oct 12 '22 19:10 SidSethi