druid
druid copied to clipboard
apache
promethes-emitter complains of unmapped metrics
Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").
Affected Version
0.23.0
Description
The cluster is deployed with druid-operator in k8s cluster with 3 nodes and broker, coordinator, router and historicals with each having 1, 1, 1, 2 replicas each. Note that even though I have deployed druid in k8s but the steps and configuration should be same even for druid running on non k8s cluster.
- Cluster size -- 3 nodes
- Configurations in use
- Steps to reproduce the problem Provision druid cluster using druid-operator in k8s with following configurations:
druid.extensions.loadList=["prometheus-emitter", "mysql-metadata-storage", ...]
druid.emitter=prometheus
druid.emitter.prometheus.strategy=exporter
druid.emitter.prometheus.port=9100
druid.emitter.prometheus.namespace=druid
druid.emitter.prometheus.addHostAsLabel=true
druid.emitter.prometheus.addServiceAsLabel=true
druid.emitter.logging.logLevel=debug
Metrics Dimensions json should be:
metricDimensions.json: |-
{
"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer"},
"query/bytes" : { "dimensions" : ["dataSource", "type"], "type" : "count"},
"query/priority" : { "dimensions" : ["dataSource", "lane", "type"], "type" : "gauge"},
"sqlQuery/bytes" : { "dimensions" : [], "type" : "count"},
"query/node/time" : { "dimensions" : ["server"], "type" : "timer"},
"query/node/ttfb" : { "dimensions" : ["server"], "type" : "timer"},
"query/node/bytes" : { "dimensions" : ["server"], "type" : "count"},
"query/node/backpressure": { "dimensions" : ["server"], "type" : "timer"},
"query/intervalChunk/time" : { "dimensions" : [], "type" : "timer"},
"query/segment/time" : { "dimensions" : [], "type" : "timer"},
"query/wait/time" : { "dimensions" : [], "type" : "timer"},
"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"},
"query/segmentAndCache/time" : { "dimensions" : [], "type" : "timer" },
"query/cpu/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer" },
"query/count" : { "dimensions" : [], "type" : "count" },
"query/success/count" : { "dimensions" : [], "type" : "count" },
"query/failed/count" : { "dimensions" : [], "type" : "count" },
"query/interrupted/count" : { "dimensions" : [], "type" : "count" },
"query/timeout/count" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/numEntries" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/sizeBytes" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/hits" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/misses" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/evictions" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/hitRate" : { "dimensions" : [], "type" : "count", "convertRange" : true },
"query/cache/delta/averageBytes" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/timeouts" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/errors" : { "dimensions" : [], "type" : "count" },
"query/cache/total/numEntries" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/sizeBytes" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/hits" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/misses" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/evictions" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/hitRate" : { "dimensions" : [], "type" : "gauge", "convertRange" : true },
"query/cache/total/averageBytes" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/timeouts" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/errors" : { "dimensions" : [], "type" : "gauge" },
"sqlQuery/time" : { "dimensions" : ["dataSource"], "type" : "timer"},
"ingest/events/thrownAway" : { "dimensions" : ["dataSource"], "type" : "count" },
"ingest/events/unparseable" : { "dimensions" : ["dataSource"], "type" : "count" },
"ingest/events/duplicate" : { "dimensions" : ["dataSource"], "type" : "count" },
"ingest/events/processed" : { "dimensions" : ["dataSource", "taskType", "taskId"], "type" : "count" },
"ingest/events/messageGap" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"ingest/rows/output" : { "dimensions" : ["dataSource"], "type" : "count" },
"ingest/persists/count" : { "dimensions" : ["dataSource"], "type" : "count" },
"ingest/persists/time" : { "dimensions" : ["dataSource"], "type" : "timer" },
"ingest/persists/cpu" : { "dimensions" : ["dataSource"], "type" : "timer" },
"ingest/persists/backPressure" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"ingest/persists/failed" : { "dimensions" : ["dataSource"], "type" : "count" },
"ingest/handoff/failed" : { "dimensions" : ["dataSource"], "type" : "count" },
"ingest/merge/time" : { "dimensions" : ["dataSource"], "type" : "timer" },
"ingest/merge/cpu" : { "dimensions" : ["dataSource"], "type" : "timer" },
"ingest/kafka/lag" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"ingest/kafka/maxLag" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"ingest/kafka/avgLag" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"task/success/count" : { "dimensions" : ["dataSource"], "type" : "count" },
"task/failed/count" : { "dimensions" : ["dataSource"], "type" : "count" },
"task/running/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"task/pending/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"task/waiting/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"taskSlot/total/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/idle/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/busy/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/lazy/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/blacklisted/count" : { "dimensions" : [], "type" : "gauge" },
"task/run/time" : { "dimensions" : ["dataSource", "taskType"], "type" : "timer" },
"segment/added/bytes" : { "dimensions" : ["dataSource", "taskType"], "type" : "count" },
"segment/moved/bytes" : { "dimensions" : ["dataSource", "taskType"], "type" : "count" },
"segment/nuked/bytes" : { "dimensions" : ["dataSource", "taskType"], "type" : "count" },
"segment/assigned/count" : { "dimensions" : ["tier"], "type" : "count" },
"segment/moved/count" : { "dimensions" : ["tier"], "type" : "count" },
"segment/dropped/count" : { "dimensions" : ["tier"], "type" : "count" },
"segment/deleted/count" : { "dimensions" : ["tier"], "type" : "count" },
"segment/unneeded/count" : { "dimensions" : ["tier"], "type" : "count" },
"segment/unavailable/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"segment/underReplicated/count" : { "dimensions" : ["dataSource", "tier"], "type" : "gauge" },
"segment/cost/raw" : { "dimensions" : ["tier"], "type" : "count" },
"segment/cost/normalization" : { "dimensions" : ["tier"], "type" : "count" },
"segment/cost/normalized" : { "dimensions" : ["tier"], "type" : "count" },
"segment/loadQueue/size" : { "dimensions" : ["server"], "type" : "gauge" },
"segment/loadQueue/failed" : { "dimensions" : ["server"], "type" : "gauge" },
"segment/loadQueue/count" : { "dimensions" : ["server"], "type" : "gauge" },
"segment/dropQueue/count" : { "dimensions" : ["server"], "type" : "gauge" },
"segment/size" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"segment/overShadowed/count" : { "dimensions" : [], "type" : "gauge" },
"segment/max" : { "dimensions" : [], "type" : "gauge"},
"segment/used" : { "dimensions" : ["dataSource", "tier", "priority"], "type" : "gauge" },
"segment/usedPercent" : { "dimensions" : ["dataSource", "tier", "priority"], "type" : "gauge", "convertRange" : true },
"segment/pendingDelete" : { "dimensions" : [], "type" : "gauge"},
"segment/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"coordinator-segment/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"historical-segment/count" : { "dimensions" : ["dataSource", "tier", "priority"], "type" : "gauge" },
"jvm/pool/committed" : { "dimensions" : ["poolKind", "poolName"], "type" : "gauge" },
"jvm/pool/init" : { "dimensions" : ["poolKind", "poolName"], "type" : "gauge" },
"jvm/pool/max" : { "dimensions" : ["poolKind", "poolName"], "type" : "gauge" },
"jvm/pool/used" : { "dimensions" : ["poolKind", "poolName"], "type" : "gauge" },
"jvm/bufferpool/count" : { "dimensions" : ["bufferpoolName"], "type" : "gauge" },
"jvm/bufferpool/used" : { "dimensions" : ["bufferpoolName"], "type" : "gauge" },
"jvm/bufferpool/capacity" : { "dimensions" : ["bufferpoolName"], "type" : "gauge" },
"jvm/mem/init" : { "dimensions" : ["memKind"], "type" : "gauge" },
"jvm/mem/max" : { "dimensions" : ["memKind"], "type" : "gauge" },
"jvm/mem/used" : { "dimensions" : ["memKind"], "type" : "gauge" },
"jvm/mem/committed" : { "dimensions" : ["memKind"], "type" : "gauge" },
"jvm/gc/count" : { "dimensions" : ["gcName", "gcGen"], "type" : "count" },
"jvm/gc/cpu" : { "dimensions" : ["gcName", "gcGen"], "type" : "count" },
"jvm/gc/mem/max" : { "dimensions" : [], "type" : "gauge" },
"jvm/gc/mem/capacity" : { "dimensions" : [], "type" : "gauge" },
"jvm/gc/mem/used" : { "dimensions" : [], "type" : "gauge" },
"jvm/cpu/user" : { "dimensions" : [], "type" : "gauge" },
"jvm/cpu/total" : { "dimensions" : [], "type" : "gauge" },
"jvm/threads/started" : { "dimensions" : [], "type" : "count" },
"jvm/threads/finished" : { "dimensions" : [], "type" : "count" },
"jvm/threads/live" : { "dimensions" : [], "type" : "count" },
"ingest/events/buffered" : { "dimensions" : ["serviceName", "bufferCapacity"], "type" : "gauge"},
"ingest/shuffle/bytes" : { "dimensions" : ["supervisorTaskId"], "type" : "gauge"},
"sys/swap/free" : { "dimensions" : [], "type" : "gauge"},
"sys/swap/max" : { "dimensions" : [], "type" : "gauge"},
"sys/swap/pageIn" : { "dimensions" : [], "type" : "gauge"},
"sys/swap/pageOut" : { "dimensions" : [], "type" : "gauge"},
"sys/disk/write/count" : { "dimensions" : ["fsDevName"], "type" : "count"},
"sys/disk/read/count" : { "dimensions" : ["fsDevName"], "type" : "count"},
"sys/disk/write/size" : { "dimensions" : ["fsDevName"], "type" : "count"},
"sys/disk/read/size" : { "dimensions" : ["fsDevName"], "type" : "count"},
"sys/net/write/size" : { "dimensions" : [], "type" : "count"},
"sys/net/read/size" : { "dimensions" : [], "type" : "count"},
"sys/fs/used" : { "dimensions" : ["fsDevName", "fsDirName", "fsTypeName", "fsSysTypeName", "fsOptions"], "type" : "gauge"},
"sys/fs/max" : { "dimensions" : ["fsDevName", "fsDirName", "fsTypeName", "fsSysTypeName", "fsOptions"], "type" : "gauge"},
"sys/mem/used" : { "dimensions" : [], "type" : "gauge"},
"sys/mem/max" : { "dimensions" : [], "type" : "gauge"},
"sys/storage/used" : { "dimensions" : ["fsDirName"], "type" : "gauge"},
"sys/cpu" : { "dimensions" : ["cpuName", "cpuTime"], "type" : "gauge"},
"jvm/cpu/percent" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"jetty/numOpenConnections" : { "dimensions" : [], "type" : "gauge" },
"query/cache/caffeine/total/requests" : { "dimensions" : [], "type" : "gauge" },
"query/cache/caffeine/total/loadTime" : { "dimensions" : [], "type" : "gauge" },
"query/cache/caffeine/total/evictionBytes" : { "dimensions" : [], "type" : "gauge" },
"query/cache/memcached/total" : { "dimensions" : ["[MEM] Reconnecting Nodes (ReconnectQueue)",
"[MEM] Request Rate: All",
"[MEM] Average Bytes written to OS per write",
"[MEM] Average Bytes read from OS per read",
"[MEM] Response Rate: All (Failure + Success + Retry)",
"[MEM] Response Rate: Retry",
"[MEM] Response Rate: Failure",
"[MEM] Response Rate: Success"],
"type" : "gauge" },
"query/cache/caffeine/delta/requests" : { "dimensions" : [], "type" : "count" },
"query/cache/caffeine/delta/loadTime" : { "dimensions" : [], "type" : "count" },
"query/cache/caffeine/delta/evictionBytes" : { "dimensions" : [], "type" : "count" },
"query/cache/memcached/delta" : { "dimensions" : ["[MEM] Reconnecting Nodes (ReconnectQueue)",
"[MEM] Request Rate: All",
"[MEM] Average Bytes written to OS per write",
"[MEM] Average Bytes read from OS per read",
"[MEM] Response Rate: All (Failure + Success + Retry)",
"[MEM] Response Rate: Retry",
"[MEM] Response Rate: Failure",
"[MEM] Response Rate: Success"],
"type" : "count" }
}
The cluster spec in druid-operator is:
nodes:
brokers:
kind: Deployment
nodeType: "broker"
druid.port: 8088
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/query/broker"
replicas: 1
runtime.properties: |
druid.service=druid/broker
# HTTP server threads
druid.broker.http.numConnections=5
druid.server.http.numThreads=10
# Processing threads and buffers
druid.processing.buffer.sizeBytes=1
druid.processing.numMergeBuffers=1
druid.processing.numThreads=1
druid.sql.enable=true
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.server.metrics.HistoricalMetricsMonitor", "org.apache.druid.server.metrics.QueryCountStatsMonitor"]
coordinators:
# Optionally specify for running coordinator as Deployment
kind: Deployment
nodeType: "coordinator"
druid.port: 8088
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/coordinator-overlord"
replicas: 1
runtime.properties: |
druid.service=druid/coordinator
# HTTP server threads
druid.coordinator.startDelay=PT30S
druid.coordinator.period=PT30S
# Configure this coordinator to also run as Overlord
druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT30S
druid.indexer.runner.type=local
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor"]
extra.jvm.options: |-
-Xmx512M
-Xms512M
historicals:
nodeType: "historical"
druid.port: 8088
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/data/historical"
replicas: 2
runtime.properties: |
druid.service=druid/historical
druid.server.http.numThreads=5
druid.processing.buffer.sizeBytes=536870912
druid.processing.numMergeBuffers=1
druid.processing.numThreads=1
# Segment storage
druid.segmentCache.locations=[{\"path\":\"/druid/data/segments\",\"maxSize\":10737418240}]
druid.server.maxSize=10737418240
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.server.metrics.HistoricalMetricsMonitor", "org.apache.druid.server.metrics.QueryCountStatsMonitor", "org.apache.druid.indexing.worker.shuffle.ShuffleMonitor"]
extra.jvm.options: |-
-Xmx512M
-Xms512M
routers:
kind: Deployment
nodeType: "router"
druid.port: 8088
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/query/router"
replicas: 1
runtime.properties: |
druid.service=druid/router
# HTTP proxy
druid.router.http.numConnections=10
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=10
druid.server.http.numThreads=10
# Service discovery
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator
# Management proxy to coordinator / overlord: required for unified web console.
druid.router.managementProxy.enabled=true
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.server.metrics.QueryCountStatsMonitor"]
extra.jvm.options: |-
-Xmx512M
-Xms512M
- The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
Output from broker (have removed the log prefixes to keep things simple:
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/timeout/count]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/started]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/finished]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/live]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/liveDaemon]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/livePeak]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/heapAlloc/bytes]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/server/AvaticaProtobufHandler/Handler/RequestTimings]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/server/AvaticaJsonHandler/Handler/RequestTimings]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/remote/ProtobufHandler/Handler/Serialization]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/remote/JsonHandler/Handler/Serialization]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/user]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/total]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/sys]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/percent]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/delta/put/ok]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/delta/put/error]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/delta/put/oversized]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/total/put/ok]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/total/put/error]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/total/put/oversized]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/delta/requests]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/total/requests]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/delta/loadTime]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/total/loadTime]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/delta/evictionBytes]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/total/evictionBytes]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/numOpenConnections]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/total]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/idle]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/isLowOnThreads]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/min]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/queueSize]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/busy]
Output from coordinator:
Unmapped metric [query/cache/delta/put/ok]
Unmapped metric [query/cache/delta/put/error]
Unmapped metric [query/cache/delta/put/oversized]
Unmapped metric [query/cache/total/put/ok]
Unmapped metric [query/cache/total/put/error]
Unmapped metric [query/cache/total/put/oversized]
Unmapped metric [query/cache/caffeine/delta/requests]
Unmapped metric [query/cache/caffeine/total/requests]
Unmapped metric [query/cache/caffeine/delta/loadTime]
Unmapped metric [query/cache/caffeine/total/loadTime]
Unmapped metric [query/cache/caffeine/delta/evictionBytes]
Unmapped metric [query/cache/caffeine/total/evictionBytes]
Unmapped metric [jvm/cpu/user]
Unmapped metric [jvm/cpu/total]
Unmapped metric [jvm/cpu/sys]
Unmapped metric [jvm/cpu/percent]
Unmapped metric [jvm/threads/started]
Unmapped metric [jvm/threads/finished]
Unmapped metric [jvm/threads/live]
Unmapped metric [jvm/threads/liveDaemon]
Unmapped metric [jvm/threads/livePeak]
Unmapped metric [jetty/numOpenConnections]
Unmapped metric [jetty/threadPool/total]
Unmapped metric [jetty/threadPool/idle]
Unmapped metric [jetty/threadPool/isLowOnThreads]
Unmapped metric [jetty/threadPool/min]
Unmapped metric [jetty/threadPool/max]
Unmapped metric [jetty/threadPool/queueSize]
Unmapped metric [jetty/threadPool/busy]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/heapAlloc/bytes]
Unmapped metric [tier/required/capacity]
Unmapped metric [tier/total/capacity]
Unmapped metric [tier/replication/factor]
Unmapped metric [tier/historical/count]
Unmapped metric [segment/count]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/global/time]
Output from historical:
Unmapped metric [query/cache/delta/put/ok]
Unmapped metric [query/cache/delta/put/error]
Unmapped metric [query/cache/delta/put/oversized]
Unmapped metric [query/cache/total/put/ok]
Unmapped metric [query/cache/total/put/error]
Unmapped metric [query/cache/total/put/oversized]
Unmapped metric [query/cache/caffeine/delta/requests]
Unmapped metric [query/cache/caffeine/total/requests]
Unmapped metric [query/cache/caffeine/delta/loadTime]
Unmapped metric [query/cache/caffeine/total/loadTime]
Unmapped metric [query/cache/caffeine/delta/evictionBytes]
Unmapped metric [query/cache/caffeine/total/evictionBytes]
Unmapped metric [jvm/cpu/user]
Unmapped metric [jvm/cpu/total]
Unmapped metric [jvm/cpu/sys]
Unmapped metric [jvm/cpu/percent]
Unmapped metric [query/timeout/count]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/heapAlloc/bytes]
Unmapped metric [jvm/threads/started]
Unmapped metric [jvm/threads/finished]
Unmapped metric [jvm/threads/live]
Unmapped metric [jvm/threads/liveDaemon]
Unmapped metric [jvm/threads/livePeak]
Unmapped metric [jetty/numOpenConnections]
Unmapped metric [jetty/threadPool/total]
Unmapped metric [jetty/threadPool/idle]
Unmapped metric [jetty/threadPool/isLowOnThreads]
Unmapped metric [jetty/threadPool/min]
Unmapped metric [jetty/threadPool/max]
Unmapped metric [jetty/threadPool/queueSize]
Unmapped metric [jetty/threadPool/busy]
Unmapped metric [segment/count]
Output from router:
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/heapAlloc/bytes]
Unmapped metric [jvm/cpu/user]
Unmapped metric [jvm/cpu/total]
Unmapped metric [jvm/cpu/sys]
Unmapped metric [jvm/cpu/percent]
Unmapped metric [jetty/numOpenConnections]
Unmapped metric [jetty/threadPool/total]
Unmapped metric [jetty/threadPool/idle]
Unmapped metric [jetty/threadPool/isLowOnThreads]
Unmapped metric [jetty/threadPool/min]
Unmapped metric [jetty/threadPool/max]
Unmapped metric [jetty/threadPool/queueSize]
Unmapped metric [jetty/threadPool/busy]
Unmapped metric [query/timeout/count]
Unmapped metric [jvm/threads/started]
Unmapped metric [jvm/threads/finished]
Unmapped metric [jvm/threads/live]
Unmapped metric [jvm/threads/liveDaemon]
Unmapped metric [jvm/threads/livePeak]
- Any debugging that you have already done
I have tried mapping the metrics which do not exist in metricDimensions.json but it still complains. As an example, one could see that in my metricsDimensions.json above,
"jvm/threads/started" : { "dimensions" : [], "type" : "count" },
"jvm/threads/finished" : { "dimensions" : [], "type" : "count" },
"jvm/threads/live" : { "dimensions" : [], "type" : "count" },
is included and the JvmThreadsMonitor is in the list of monitors for each but I still see that it complains of unmapped metrics.
Also note that even though I tried with:
"coordinator-segment/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"historical-segment/count" : { "dimensions" : ["dataSource", "tier", "priority"], "type" : "gauge" },
for getting segment/count for coordinator and historical, but I still see coordinator and historical both have logs containing:
Unmapped metric [segment/count]
As per docs, it says, service prefix is needed in case of different dimensions to be tracked for same metrics emitted from different services, but I don't think that makes any difference.
I ran into the same problem in version 25.0.0, and found that the code is using druid.service as the prefix. In my case this was set to druid/historical, so the path to use in metricDimensions.json was druid/historical-segment/count.
I found that you can also just remove the prefix and just use 'segment/count', this causes both metrics to be reported with the same metric name but this seems to work without issue in Prometheus.
This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.