Telemetry issues with Honeybadger dependency

[massawho]

A very very similar issue to https://github.com/TheRealReal/hackney_telemetry/issues/4 but the solution didn't work for me somehow.

I am only getting the error when starting the release, tests and the local server (even running on prod MIX_ENV) works.

The log error I can get is as below

{"severity":"info","msg":"Application honeybadger exited: Honeybadger.start(:normal, []) returned an error: shutdown: failed to start child: Honeybadger.Client
    ** (EXIT) an exception was raised:
        ** (CaseClauseError) no case clause matching: {:error, {{:noproc, {:gen_server, :call, [:hackney_telemetry_sup, {:start_child, %{id: {:hackney_telemetry_worker, [:hackney_pool, Honeybadger.Client, :take_rate]}, start: {:hackney_telemetry_worker, :start_link, [[metric: [:hackney_pool, Honeybadger.Client, :take_rate]]]}}}, :infinity]}}, {:child, :undefined, Honeybadger.Client, {:hackney_pool, :start_link, [Honeybadger.Client, [name: Honeybadger.Client, max_connections: 20]]}, :permanent, 10000, :worker, [:hackney_pool]}}}
            (hackney) /olympian/deps/hackney/src/hackney_pool.erl:189: :hackney_pool.do_start_pool/2
            (hackney) /olympian/deps/hackney/src/hackney_pool.erl:113: :hackney_pool.start_pool/2
            (honeybadger) lib/honeybadger/client.ex:101: Honeybadger.Client.init/1
            (stdlib) gen_server.erl:374: :gen_server.init_it/2
            (stdlib) gen_server.erl:342: :gen_server.init_it/6
            (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3","appname":"olympian"}

The project is an umbrella, with two apps olympian and olympian_web and their mix files look like below

olympian_web/mix.exs
  def application do
    [
      mod: {OlympianWeb.Application, []},
      extra_applications: [:logger, :runtime_tools, :olympian]
    ]
  end

olympian/mix.exs
  def application do
    [
      mod: {Olympian.Application, []},
      extra_applications: [:logger, :runtime_tools, :ex_shards, :hackney_telemetry]
    ]
  end

Any help is much appreciated

[msramos]

Hello, @massawho! Thanks for reaching out!

Releases with umbrella projects are a bit different than regular mix projects. You need to edit mix.exs on the umbrella root, and add a releases config in the project section, something like this:

releases: [
  my_release: [
    version: "0.0.1",
    applications: [olympian: :permanent, olympian_web: :permanent]
  ]
]

The official guide and the official docs have explicit instructions for releases using umbrellas.

Could you please try it and see if it works?

[massawho]

Thanks for the answer! It's an old project, so I am not using elixir releases for this one but seems like I managed to figure it out. It's just the same issue as #4, but I solved it by adding honeybadger dep to the end of the list :smile: I was reading about extra_applications here https://www.amberbit.com/blog/2017/9/22/elixir-applications-vs-extra_applications-guide/ and I remembered there was a comment on our codebase about mariaex having to be added before ecto so I tried the same :smile:

edit: I also added runtime: false and forgot to start Honeybadger, so still trying to figure out. Just adding it to the end of the list didn't help

[massawho]

Reopening because I thought I had it figured out, but haven't... in fact I had just prevented honeybadger to be started.

I tried your suggestion @msramos, added olympian as permanent on release (only web was permanent):

release :olympian do
  set version: "0.1.0"
  set applications: [
    :runtime_tools,
    olympian: :permanent,
    olympian_web: :permanent
  ]
  set commands: []
end

still got the same result

[msramos]

I was able to successfully make hackney_telemetry work on an umbrella application.

• Add your applications into the releases sections of the umbrella's root:
• Add :hackney_telemetry as an extra application, on your umbrella app that is going to use hackney
• Add the metrics into the telemetry config (the README file has a good starting point)
• Add the config :hackney, mod_metrics: :hackney_telemetry config into the umbrella's config.exs

Let's say the we have the application like this:

example_umbrella
  -> example
  -> example_web

Here, the example will have :hackney_telemetry and :httpoison (that uses hackney). Its mixs.exs have:

...
  def application do
    [
      mod: {Example.Application, []},
      extra_applications: [:logger, :runtime_tools, :hackney_telemetry, :httpoison]
    ]
  end
...

You could also add :hackney_telemetry as an extra application on the example_web, this will ensure that it'll always be started before your umbrella apps, but it's not required.

On the mix.exs of the umbrella root we'll have:

...
  def project do
    [
      apps_path: "apps",
      version: "0.1.0",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      aliases: aliases(),
      releases: [
        example: [
          applications: [
            example: :permanent,
            example_web: :permanent
          ]
        ]
      ]
    ]
  end
...

Now you can run mix release and then execute _build/dev/rel/example/bin/example start.

It seems to me that in your project you're not configuring the dependency/order of the applications in the right order.

[massawho]

It seems to me that in your project you're not configuring the dependency/order of the applications in the right order.

yes, that's exactly what I am trying to figure out how to ensure the order. Would you also be able to try adding honeybadger as a dependency on your test project? As of now, I have both hackney_telemetry and honeybadger on both extra_applications and deps like this:

  def application do
    [
      mod: {OlympianWeb.Application, []},
      extra_applications: [:logger, :runtime_tools, :hackney_telemetry, :honeybadger]
    ]
  end

  defp deps do
    [
      {:spandex, "~> 3.0.3"},
      {:spandex_phoenix, "~> 1.0"},
      {:spandex_datadog, "~> 1.2"},
      {:cee_log_formatter, "~> 0.2"},
      {:covertool, "~> 2.0.3", only: :test},
      {:cowboy, "~> 1.0"},
      {:gettext, "~> 0.13"},
      {:junit_formatter, "~> 3.1", only: :test},
      {:mix_test_watch, "~> 0.5", only: :dev, runtime: false},
      {:olympian, in_umbrella: true},
      {:phoenix, "~> 1.3"},
      {:phoenix_html, "~> 2.10"},
      {:phoenix_live_reload, "~> 1.1", only: :dev},
      {:phoenix_pubsub, "~> 1.0"},
      {:plug_logger_json, "~> 0.6"},
      {:hackney_telemetry, "~> 0.1.1"},
      {:honeybadger, "~> 0.18"}
      # {:honeybadger, "~> 0.18", runtime: false}
    ]
  end

[massawho]

I actually created a repo with honeybadger and hackney_telemetry and it gives me the same error when i start the release. Would you mind taking a look? https://github.com/massawho/honeybadger_hackney_telemetry Feels like it could be something trivial I am missing, but at this point I am kinda lost :sweat_smile:

[massawho]

Got a "fix" for now. I set honeybadger with runtime: false and added it to included_applications. That allowed me to start Honeybadger on my application.ex (previously was getting "module doesn't exist"). I had to upgrade distillery to 2.1 tho because 1.5 wasn't loading the dep with runtime: false somehow. I am not sure if that's the most elegant solution, but seems like it works