MongooseIM icon indicating copy to clipboard operation
MongooseIM copied to clipboard
verified publisher icon esl

HTTP API metrics

Open arkgil opened this issue 7 years ago • 5 comments

The goal of this PR is to introduce metrics to the HTTP API ("backend", "frontend", whatever). It consists of two parts:

  • two Cowboy middleware modules responsible for recording metrics (mongoose_cowboy_metrics_mw_before and mongoose_cowboy_metrics_mw_after)
  • little code in ejabberd_cowboy responsible for assembling names of metrics and defining them (currently metrics are opt-in)

For each handler (or rather for each prefix) "types" of metrics are recorded:

  • request count per HTTP method (suffix ..., Method, request, count]
  • response count per HTTP method x status class (1XX, 2XX etc.) (suffix ..., Method, response, Class, count])
  • response latency per HTTP method x status class (suffix ..., Method, response, Class, latency])

These metrics together cover 3 of 4 golden signals.

Metrics need to be enabled by providing {metrics, true} option to ejabberd_cowboy. All defined metrics are global. By default they are named like [http] ++ lists:flatten([HandlerBaseName]) ++ MetricTypeSuffix].

The HandlerBaseName part is the handler's module name by default. The handler's definition tuple can now contain a fifth element, which is not passed to handler but is interpreted by ejabberd_cowboy. This element is a list of options, and currently the only supported one is metrics:

  • it can be set to false, in which case the metrics for the handler won't be created
  • it can be a list with {prefix, Prefix} option, in which case Prefix becomes a HandlerBaseName

There is a small issue with this design - if one configured the same handler multiple times under different paths, metrics for both these handlers will have the same name (which obviously might skew the results if the handlers are configured differently). In such cases when we have duplicated metric names we could:

  • log a warning (per @michalwski 's suggestion)
  • raise an error, in which case the listener won't be started

I haven't decided which option to choose yet.

Oh, and recording of metrics can be disabled by executing cowboy:set_env(Ref, record_metrics, false).

TODOs:

  • [ ] documentation for ejabberd_cowboy's new options
  • [ ] choose what to do in case of duplicate metric names
  • [ ] tests for ejabberd_cowboy with metrics
  • [ ] new metrics should be added when routes are being reloaded (we shouldn't delete the old ones)

arkgil avatar Jul 09 '18 09:07 arkgil

5077.1 / Erlang 19.3 / small_tests / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root / small

5077.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big OK: 445 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

5077.5 / Erlang 19.3 / ldap_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big OK: 1024 / Failed: 0 / User-skipped: 77 / Auto-skipped: 0

5077.3 / Erlang 19.3 / mysql_redis / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big OK: 2799 / Failed: 0 / User-skipped: 222 / Auto-skipped: 0

5077.2 / Erlang 19.3 / internal_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big OK: 1060 / Failed: 0 / User-skipped: 41 / Auto-skipped: 0

5077.4 / Erlang 19.3 / odbc_mssql_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big OK: 2813 / Failed: 0 / User-skipped: 208 / Auto-skipped: 0

5077.8 / Erlang 20.0 / pgsql_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big / small OK: 2845 / Failed: 0 / User-skipped: 176 / Auto-skipped: 0

5077.9 / Erlang 21.0 / riak_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big / small OK: 1282 / Failed: 1 / User-skipped: 43 / Auto-skipped: 0

mod_global_distrib_SUITE:multi_connection:test_muc_conversation_history 
{error,{{assertEqual,[{module,mod_global_distrib_SUITE},
            {line,418},
            {expression,"exml_query : attr ( AlicePresence , << \"from\" >> )"},
            {expected,<<"[email protected]/alicE4.776981">>},
            {value,<<"[email protected]/eve5.71013">>}]},
    [{mod_global_distrib_SUITE,'-test_muc_conversation_history/1-fun-1-',
                   3,
                   [{file,"mod_global_distrib_SUITE.erl"},
                  {line,418}]},
     {mod_global_distrib_SUITE,'-test_muc_conversation_history/1-fun-5-',
                   3,
                   [{file,"mod_global_distrib_SUITE.erl"},
                  {line,417}]},
     {escalus_story,story,4,
            [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {mod_global_distrib_SUITE,test_muc_conversation_history,1,
                   [{file,"mod_global_distrib_SUITE.erl"},
                  {line,390}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1545}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1063}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,995}]}]}}

Report log

5077.8 / Erlang 20.0 / pgsql_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big / small OK: 2845 / Failed: 0 / User-skipped: 176 / Auto-skipped: 0

5077.9 / Erlang 21.0 / riak_mnesia / fa7b0414eb73fbdfbd257af670bafdd51ab8cf26 Reports root/ big / small OK: 1284 / Failed: 1 / User-skipped: 43 / Auto-skipped: 0

offline_SUITE:mod_offline_tests:max_offline_messages_reached 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"bOb8.11573@localhost/res1">>,escalus_tcp,<0.8734.1>,
          [{event_manager,<0.8708.1>},
           {server,<<"localhost">>},
           {username,<<"bOb8.11573">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.8708.1>},
            {server,<<"localhost">>},
            {username,<<"bOb8.11573">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"bOb8.11573">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"bOb8.11573">>},
           {server,<<"localhost">>},
           {password,<<"makrolika">>},
           {stream_id,<<"3D08F955361D3E89">>}]},
        5000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {offline_SUITE,'-max_offline_messages_reached/1-fun-1-',6,
       [{file,"offline_SUITE.erl"},{line,112}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1545}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1063}]},...

Report log

mongoose-im avatar Jul 09 '18 09:07 mongoose-im

5078.1 / Erlang 19.3 / small_tests / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root / small

5078.2 / Erlang 19.3 / internal_mnesia / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root/ big OK: 1060 / Failed: 0 / User-skipped: 41 / Auto-skipped: 0

5078.3 / Erlang 19.3 / mysql_redis / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root/ big OK: 2799 / Failed: 0 / User-skipped: 222 / Auto-skipped: 0

5078.4 / Erlang 19.3 / odbc_mssql_mnesia / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root/ big OK: 2813 / Failed: 0 / User-skipped: 208 / Auto-skipped: 0

5078.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root/ big OK: 445 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

5078.5 / Erlang 19.3 / ldap_mnesia / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root/ big OK: 1024 / Failed: 0 / User-skipped: 77 / Auto-skipped: 0

5078.9 / Erlang 21.0 / riak_mnesia / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root/ big / small OK: 1279 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

5078.8 / Erlang 20.0 / pgsql_mnesia / b7bffef3985ffc8ba878d40e9177c529201b13bd Reports root/ big / small OK: 2845 / Failed: 0 / User-skipped: 176 / Auto-skipped: 0

mongoose-im avatar Jul 09 '18 10:07 mongoose-im

Codecov Report

Merging #1969 into master will decrease coverage by 6.58%. The diff coverage is 12.5%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1969      +/-   ##
==========================================
- Coverage   74.91%   68.32%   -6.59%     
==========================================
  Files         313      315       +2     
  Lines       28459    28468       +9     
==========================================
- Hits        21319    19450    -1869     
- Misses       7140     9018    +1878
Impacted Files Coverage Δ
src/mongoose_cowboy_metrics_mw_after.erl 0% <0%> (ø)
src/mongoose_cowboy_metrics.erl 0% <0%> (ø)
src/mongoose_cowboy_metrics_mw_before.erl 0% <0%> (ø)
src/ejabberd_app.erl 82.29% <100%> (-4.17%) :arrow_down:
src/ejabberd_cowboy.erl 64.76% <23.33%> (-23.7%) :arrow_down:
src/mod_auth_token_odbc.erl 0% <0%> (-100%) :arrow_down:
src/mam/mam_message_xml.erl 0% <0%> (-100%) :arrow_down:
src/inbox/mod_inbox_odbc_pgsql.erl 0% <0%> (-100%) :arrow_down:
src/mod_offline_riak.erl 0% <0%> (-96.43%) :arrow_down:
src/mod_last_riak.erl 0% <0%> (-95.24%) :arrow_down:
... and 112 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8dd8a1b...e18e89c. Read the comment docs.

codecov[bot] avatar Jul 09 '18 11:07 codecov[bot]

5080.1 / Erlang 19.3 / small_tests / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root / small

5080.5 / Erlang 19.3 / ldap_mnesia / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root/ big OK: 1024 / Failed: 0 / User-skipped: 77 / Auto-skipped: 0

5080.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root/ big OK: 445 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

5080.3 / Erlang 19.3 / mysql_redis / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root/ big OK: 2799 / Failed: 0 / User-skipped: 222 / Auto-skipped: 0

5080.2 / Erlang 19.3 / internal_mnesia / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root/ big OK: 1060 / Failed: 0 / User-skipped: 41 / Auto-skipped: 0

5080.4 / Erlang 19.3 / odbc_mssql_mnesia / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root/ big OK: 2813 / Failed: 0 / User-skipped: 208 / Auto-skipped: 0

5080.8 / Erlang 20.0 / pgsql_mnesia / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root/ big / small OK: 2845 / Failed: 0 / User-skipped: 176 / Auto-skipped: 0

5080.9 / Erlang 21.0 / riak_mnesia / 7be9fc14e6b6c823cae294c5ce04e23580e2bf15 Reports root/ big / small OK: 1279 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

mongoose-im avatar Jul 09 '18 11:07 mongoose-im

5164.1 / Erlang 19.3 / small_tests / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root / small

5164.3 / Erlang 19.3 / mysql_redis / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root/ big OK: 2803 / Failed: 0 / User-skipped: 222 / Auto-skipped: 0

5164.5 / Erlang 19.3 / ldap_mnesia / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root/ big OK: 1028 / Failed: 0 / User-skipped: 77 / Auto-skipped: 0

5164.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root/ big OK: 445 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

5164.2 / Erlang 19.3 / internal_mnesia / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root/ big OK: 1064 / Failed: 0 / User-skipped: 41 / Auto-skipped: 0

5164.4 / Erlang 19.3 / odbc_mssql_mnesia / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root/ big OK: 2822 / Failed: 1 / User-skipped: 208 / Auto-skipped: 0

offline_SUITE:mod_offline_tests:headline_message_is_not_stored 
{error,
  {{assertion_failed,assert,is_presence,
     {xmlel,<<"message">>,
       [{<<"from">>,<<"alicE46.330326@localhost/res1">>},
        {<<"to">>,<<"bOb46.330326@localhost/res1">>},
        {<<"xml:lang">>,<<"en">>},
        {<<"type">>,<<"headline">>}],
       [{xmlel,<<"body">>,[],[{xmlcdata,<<"msgtxt">>}]}]},
     "<message from='alicE46.330326@localhost/res1' to='bOb46.330326@localhost/res1' xml:lang='en' type='headline'><body>msgtxt</body></message>"},
   [{escalus_new_assert,assert_true,2,
      [{file,
         "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
       {line,84}]},
    {offline_SUITE,login_send_and_receive_presence,2,
      [{file,"offline_SUITE.erl"},{line,171}]},
    {offline_SUITE,'-headline_message_is_not_stored/1-fun-0-',3,
      [{file,"offline_SUITE.erl"},{line,95}]},
    {escalus_story,story,4,
      [{file,
         "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1529}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1045}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,977}]}]}}

Report log

5164.8 / Erlang 20.0 / pgsql_mnesia / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root/ big / small OK: 2849 / Failed: 0 / User-skipped: 176 / Auto-skipped: 0

5164.9 / Erlang 21.0 / riak_mnesia / 741de3b59b6749b88c553d3c34d5f6aefaf82761 Reports root/ big / small OK: 1283 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

mongoose-im avatar Jul 17 '18 11:07 mongoose-im