bosh icon indicating copy to clipboard operation
bosh copied to clipboard
verified publisher icon cloudfoundry

Director API Metrics broken

Open fkittelinger opened this issue 3 years ago • 0 comments

Describe the bug The API Metrics endpoint is failing, missing the metrics.

To Reproduce Steps to reproduce the behavior (example):

  1. Deploy a bosh director on AWS, Azure or Alicloud
  2. configure your monitoring system to observe the Metrics API Endpoint (Prometheus). You can also curl to that enpoint with the correspondig certs with curl --cacert 'some-ca'.cert --cert 'some-cert'.cert --key 'some-key'.cert https://:9091/api_metrics

Expected behavior The API Metrics Endpoint should provide metrics to monitor the director behaviour.

Logs Puma caught this error: undefined method unpack' for nil:NilClass value, timestamp = @f.read(16).unpack('dd') ^^^^^^^ (NoMethodError) /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:261:in block (2 levels) in all_values' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:259:in each' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:259:in map' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:259:in block in all_values' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:295:in with_file_lock' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:258:in all_values' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:136:in block in all_values' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:133:in each' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/data_stores/direct_file_store.rb:133:in all_values' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/metric.rb:65:in values' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/formats/text.rb:30:in block in marshal' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/formats/text.rb:26:in each' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/client/formats/text.rb:26:in marshal' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/middleware/exporter.rb:69:in respond_with' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/middleware/exporter.rb:30:in call' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/middleware/collector.rb:32:in block in call' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/middleware/collector.rb:61:in block in trace' /var/vcap/data/packages/ruby-3.1/8b0d198f2a4c28350a677dc3db638d96b3fbe750/lib/ruby/3.1.0/benchmark.rb:311:in realtime' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/middleware/collector.rb:61:in trace' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/prometheus-client-2.1.0/lib/prometheus/middleware/collector.rb:32:in call' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/rack-2.2.3.1/lib/rack/common_logger.rb:38:in call' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/sinatra-2.2.0/lib/sinatra/base.rb:255:in call' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/puma-5.6.4/lib/puma/configuration.rb:252:in call' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:77:in block in handle_request' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:340:in with_force_shutdown' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:76:in handle_request' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/puma-5.6.4/lib/puma/server.rb:441:in process_client' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:147:in block in spawn_thread' /var/vcap/data/packages/director/4c00823f3ea2802c7eaa9dde6f4197481e9b9ce6/gem_home/ruby/3.1.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context'prometheus2/888c639b-3248-4454-b841-415f11dbc2df:/var/vcap/jobs/prometheus2/certs

Versions (please complete the following information):

  • Infrastructure: tested on AWS, Azure and Alicloud
  • BOSH version 272.6.0
  • BOSH CLI version 6.4.17
  • Stemcell version ubuntu-bionic/1.90
  • bosh-deployment: https://github.com/cloudfoundry/bosh-deployment/tree/0abc4c270277343ad28a3844172da43dfb8f5e79

Deployment info: For deploying the directors, we use the bosh-deployment repo. To get some metrics for the monitoring system, we use the following ops-file: -o $bosh_deployment/experimental/enable-metrics.yml \

fkittelinger avatar Aug 11 '22 11:08 fkittelinger