ERROR: cpufreq collector failed after 1.499322s: strconv.ParseUint: parsing \"<unknown>\": invalid syntax" source="collector.go:132"

[yangmingbai]

Host operating system: output of uname -a

Linux EM-4V8NH42 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 0.18.1 (branch: HEAD, revision: 3db77732e925c08f675d7404a8c46466b2ece83e) build user: root@b50852a1acba build date: 20190604-16:41:18 go version: go1.12.5

node_exporter command line flags

/opt/node_exporter-0.18.1.linux-amd64/node_exporter --web.listen-address=":9111"

Are you running node_exporter in Docker?

no

How to solve or avoid this problem? Thank you

[SuperQ]

This seems like a duplicate of https://github.com/prometheus/node_exporter/issues/1241, but possibly a new bug.

Can you run the node_exporter with --log.level=debug and get the output for the cpufreq collector?

[SuperQ]

It might also be worth trying out the latest 1.0 release candidate.

[yangmingbai]

[SuperQ]

This looks like an issue with the procfs parsing of the cpufreq data.

[SuperQ]

It would be useful to include the text output of this script:

for f in $(find /sys/devices/system/cpu/cpu*/cpufreq/ -type f) ; do echo "$f: $(< $f)" ; done

[yangmingbai]

for f in $(find /sys/devices/system/cpu/cpu*/cpufreq/ -type f) ; do echo "$f: $(< $f)" ; done output:

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu0/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq: 2472000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu0/cpufreq/affected_cpus: 0
/sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu0/cpufreq/related_cpus: 0
/sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu10/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu10/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu10/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu10/cpufreq/cpuinfo_cur_freq: 2016000
/sys/devices/system/cpu/cpu10/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu10/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu10/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu10/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu10/cpufreq/affected_cpus: 10
/sys/devices/system/cpu/cpu10/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu10/cpufreq/related_cpus: 10
/sys/devices/system/cpu/cpu10/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu11/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu11/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu11/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu11/cpufreq/cpuinfo_cur_freq: 1872000
/sys/devices/system/cpu/cpu11/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu11/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu11/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu11/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu11/cpufreq/affected_cpus: 11
/sys/devices/system/cpu/cpu11/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu11/cpufreq/related_cpus: 11
/sys/devices/system/cpu/cpu11/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu12/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu12/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu12/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu12/cpufreq/cpuinfo_cur_freq: 5064000
/sys/devices/system/cpu/cpu12/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu12/cpufreq/scaling_cur_freq: 2160000
/sys/devices/system/cpu/cpu12/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu12/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu12/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu12/cpufreq/affected_cpus: 12
/sys/devices/system/cpu/cpu12/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu12/cpufreq/related_cpus: 12
/sys/devices/system/cpu/cpu12/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu13/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu13/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu13/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu13/cpufreq/cpuinfo_cur_freq: 2472000
/sys/devices/system/cpu/cpu13/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu13/cpufreq/scaling_cur_freq: 1440000
/sys/devices/system/cpu/cpu13/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu13/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu13/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu13/cpufreq/affected_cpus: 13
/sys/devices/system/cpu/cpu13/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu13/cpufreq/related_cpus: 13
/sys/devices/system/cpu/cpu13/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu14/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu14/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu14/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu14/cpufreq/cpuinfo_cur_freq: 2112000
/sys/devices/system/cpu/cpu14/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu14/cpufreq/scaling_cur_freq: 2160000
/sys/devices/system/cpu/cpu14/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu14/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu14/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu14/cpufreq/affected_cpus: 14
/sys/devices/system/cpu/cpu14/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu14/cpufreq/related_cpus: 14
/sys/devices/system/cpu/cpu14/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu15/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu15/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu15/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu15/cpufreq/cpuinfo_cur_freq: 624000
/sys/devices/system/cpu/cpu15/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu15/cpufreq/scaling_cur_freq: 1920000
/sys/devices/system/cpu/cpu15/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu15/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu15/cpufreq/affected_cpus: 15
/sys/devices/system/cpu/cpu15/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu15/cpufreq/related_cpus: 15
/sys/devices/system/cpu/cpu15/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu17/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu17/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu17/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu17/cpufreq/cpuinfo_cur_freq: 2040000
/sys/devices/system/cpu/cpu17/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu17/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu17/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu17/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu17/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu17/cpufreq/affected_cpus: 17
/sys/devices/system/cpu/cpu17/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu17/cpufreq/related_cpus: 17
/sys/devices/system/cpu/cpu17/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu18/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu18/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu18/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu18/cpufreq/cpuinfo_cur_freq: 2016000
/sys/devices/system/cpu/cpu18/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu18/cpufreq/scaling_cur_freq: 1560000
/sys/devices/system/cpu/cpu18/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu18/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu18/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu18/cpufreq/affected_cpus: 18
/sys/devices/system/cpu/cpu18/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu18/cpufreq/related_cpus: 18
/sys/devices/system/cpu/cpu18/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu19/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu19/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu19/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu19/cpufreq/cpuinfo_cur_freq: 2352000
/sys/devices/system/cpu/cpu19/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu19/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu19/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu19/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu19/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu19/cpufreq/affected_cpus: 19
/sys/devices/system/cpu/cpu19/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu19/cpufreq/related_cpus: 19
/sys/devices/system/cpu/cpu19/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu1/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq: 2424000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu1/cpufreq/affected_cpus: 1
/sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu1/cpufreq/related_cpus: 1
/sys/devices/system/cpu/cpu1/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu20/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu20/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu20/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu20/cpufreq/cpuinfo_cur_freq: 2448000
/sys/devices/system/cpu/cpu20/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu20/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu20/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu20/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu20/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu20/cpufreq/affected_cpus: 20
/sys/devices/system/cpu/cpu20/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu20/cpufreq/related_cpus: 20
/sys/devices/system/cpu/cpu20/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu21/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu21/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu21/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu21/cpufreq/cpuinfo_cur_freq: 1920000
/sys/devices/system/cpu/cpu21/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu21/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu21/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu21/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu21/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu21/cpufreq/affected_cpus: 21
/sys/devices/system/cpu/cpu21/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu21/cpufreq/related_cpus: 21
/sys/devices/system/cpu/cpu21/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu22/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu22/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu22/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu22/cpufreq/cpuinfo_cur_freq: 2064000
/sys/devices/system/cpu/cpu22/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu22/cpufreq/scaling_cur_freq: 2280000
/sys/devices/system/cpu/cpu22/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu22/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu22/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu22/cpufreq/affected_cpus: 22
/sys/devices/system/cpu/cpu22/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu22/cpufreq/related_cpus: 22
/sys/devices/system/cpu/cpu22/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu2/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu2/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_cur_freq: 1872000
/sys/devices/system/cpu/cpu2/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu2/cpufreq/affected_cpus: 2
/sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu2/cpufreq/related_cpus: 2
/sys/devices/system/cpu/cpu2/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu3/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu3/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_cur_freq: 2208000
/sys/devices/system/cpu/cpu3/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu3/cpufreq/affected_cpus: 3
/sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu3/cpufreq/related_cpus: 3
/sys/devices/system/cpu/cpu3/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu4/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu4/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_cur_freq: 1944000
/sys/devices/system/cpu/cpu4/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq: 2400000
/sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu4/cpufreq/affected_cpus: 4
/sys/devices/system/cpu/cpu4/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu4/cpufreq/related_cpus: 4
/sys/devices/system/cpu/cpu4/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu5/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu5/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_cur_freq: 5064000
/sys/devices/system/cpu/cpu5/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu5/cpufreq/affected_cpus: 5
/sys/devices/system/cpu/cpu5/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu5/cpufreq/related_cpus: 5
/sys/devices/system/cpu/cpu5/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu6/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu6/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_cur_freq: 1944000
/sys/devices/system/cpu/cpu6/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq: 1440000
/sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu6/cpufreq/affected_cpus: 6
/sys/devices/system/cpu/cpu6/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu6/cpufreq/related_cpus: 6
/sys/devices/system/cpu/cpu6/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu7/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu7/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_cur_freq: 2160000
/sys/devices/system/cpu/cpu7/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu7/cpufreq/affected_cpus: 7
/sys/devices/system/cpu/cpu7/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu7/cpufreq/related_cpus: 7
/sys/devices/system/cpu/cpu7/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu8/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu8/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu8/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu8/cpufreq/cpuinfo_cur_freq: 1560000
/sys/devices/system/cpu/cpu8/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu8/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu8/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu8/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu8/cpufreq/affected_cpus: 8
/sys/devices/system/cpu/cpu8/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu8/cpufreq/related_cpus: 8
/sys/devices/system/cpu/cpu8/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu9/cpufreq/scaling_governor: conservative
/sys/devices/system/cpu/cpu9/cpufreq/cpuinfo_transition_latency: 0
/sys/devices/system/cpu/cpu9/cpufreq/scaling_driver: pcc-cpufreq
/sys/devices/system/cpu/cpu9/cpufreq/cpuinfo_cur_freq: 1968000
/sys/devices/system/cpu/cpu9/cpufreq/scaling_available_governors: conservative userspace powersave ondemand performance 
/sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq: 1200000
/sys/devices/system/cpu/cpu9/cpufreq/cpuinfo_max_freq: 2400000
/sys/devices/system/cpu/cpu9/cpufreq/cpuinfo_min_freq: 1200000
/sys/devices/system/cpu/cpu9/cpufreq/scaling_max_freq: 2400000
/sys/devices/system/cpu/cpu9/cpufreq/affected_cpus: 9
/sys/devices/system/cpu/cpu9/cpufreq/scaling_min_freq: 1200000
/sys/devices/system/cpu/cpu9/cpufreq/related_cpus: 9
/sys/devices/system/cpu/cpu9/cpufreq/scaling_setspeed: <unsupported>

[SuperQ]

That's strange, I don't see <unknown> in any of that output. Are you sure this is the same system?

[yangmingbai]

yes, I'm sure，the bug is a rare occurrence

[SuperQ]

I take it that means the error doesn't occur every scrape? How often does this error get reported?

[hoffie]

We have been continuously seeing this as well (still node_exporter 0.18.x). Across a fleet of about 500 HPE DL380 Gen8/9/10 servers running on RHEL7, there are almost always some (5-10) servers in a state where the cpufreq collector returns a failure with the log showing exactly this error message.

My gut feeling says that some machines are more affected than others (maybe CPU type / core count?), but I have not made any analyses to verify this (yet). I suspect a kernel and/or firmware/microcode bug. If I remember correctly, this problem started to increase over time which would imply that some kind of update broke it. We run latest RHEL7 while the original report is about a rather old RHEL7/CentOS7 kernel (7.0/1/2?).

Our workaround consists of accepting the gaps + setting a longer for: period for alerts on cpufreq collector scrape failures.

node_exporter/Prometheus makes this problem visible (and annoying), but I'm not sure if there's anything fixable on the node_exporter/procfs side if the kernel doesn't return valid data.

Please take all of this with a grain of salt as I have no hard evidence to back this. However, I'm not sure if we ever get such evidence as we don't have a way to reproduce this reliably at will.

[SuperQ]

Thanks @hoffie. If this was a consistent problem where it always returned <unknown>, it would be something to make an effort to patch. But if it's a random/occasional error that affects only a specific set of platform combinations, I think we should leave it as is.

It's an indicator of a real problem. Covering it up is more dangerous than the minor annoyance of some missed data and soft errors in the scrape. We already handle the collector failing in a soft way.

I do think the error message could be improved, to make it easier to see exactly what CPU/metric failed.

[hoffie]

On the affected systems, the following loop demonstrates the problem outside of node_exporter after some time:

$ while true; do sudo grep unknown /sys/devices/system/cpu/cpu*/cpufreq/*; sleep 1; done
/sys/devices/system/cpu/cpu23/cpufreq/cpuinfo_cur_freq:<unknown>

We have opened a support case with Red Hat and at least on our HPE machines running RHEL7 this issue can be explained by a broken cpufreq driver. According to Red Hat and HPE, the pcc-cpufreq driver is selected on such machines if the Power regulator setting in the BIOS is not set to OS Control Mode. This driver is considered broken and is no longer developed. Both HPE and Red Hat advise to change the BIOS setting. This will load the intel_pstate driver instead, which should not have this problem.

We have confirmed that changing the BIOS setting makes Linux load the intel_pstate driver. We have not yet run any larger tests if this fixes the problem described in this Github issue, but I'm optimistic. :)

So, the generic answer is as expected: This is a problem in the cpufreq driver and nothing which node_exporter can fix. It also seems that this cosmetic issue is one of the less important symptoms of the buggy driver (which, it appears, can lead to unbootable or performance-degraded systems).

References: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=emr_na-c04704148 https://patchwork.kernel.org/patch/10528797/ https://access.redhat.com/solutions/3421421

I do think the error message could be improved, to make it easier to see exactly what CPU/metric failed.

@SuperQ I had a quick look regarding improving the error messages. node_exporter uses procfs' sysfs package and its parseCpufreqCpuinfo function. That function uses ReadUintFromFile. So I assume this is where the error should be wrapped with a hint about the affected filename. If this is wanted, I assume that it should be done with all those helper functions which take a path name, right?

[SuperQ]

Thanks for the detailed info. Yes, improving things in the procfs code would be the right place.

[Xphobia]

On the affected systems, the following loop demonstrates the problem outside of node_exporter after some time:

$ while true; do sudo grep unknown /sys/devices/system/cpu/cpu*/cpufreq/*; sleep 1; done
/sys/devices/system/cpu/cpu23/cpufreq/cpuinfo_cur_freq:<unknown>

We have opened a support case with Red Hat and at least on our HPE machines running RHEL7 this issue can be explained by a broken cpufreq driver. According to Red Hat and HPE, the pcc-cpufreq driver is selected on such machines if the Power regulator setting in the BIOS is not set to OS Control Mode. This driver is considered broken and is no longer developed. Both HPE and Red Hat advise to change the BIOS setting. This will load the intel_pstate driver instead, which should not have this problem.

We have confirmed that changing the BIOS setting makes Linux load the intel_pstate driver. We have not yet run any larger tests if this fixes the problem described in this Github issue, but I'm optimistic. :)

So, the generic answer is as expected: This is a problem in the cpufreq driver and nothing which node_exporter can fix. It also seems that this cosmetic issue is one of the less important symptoms of the buggy driver (which, it appears, can lead to unbootable or performance-degraded systems).

References: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=emr_na-c04704148 https://patchwork.kernel.org/patch/10528797/ https://access.redhat.com/solutions/3421421

I do think the error message could be improved, to make it easier to see exactly what CPU/metric failed.

@SuperQ I had a quick look regarding improving the error messages. node_exporter uses procfs' sysfs package and its parseCpufreqCpuinfo function. That function uses ReadUintFromFile. So I assume this is where the error should be wrapped with a hint about the affected filename. If this is wanted, I assume that it should be done with all those helper functions which take a path name, right?

Thanks for the detailed info. Yes, improving things in the procfs code would be the right place.

On the affected systems, the following loop demonstrates the problem outside of node_exporter after some time:

$ while true; do sudo grep unknown /sys/devices/system/cpu/cpu*/cpufreq/*; sleep 1; done
/sys/devices/system/cpu/cpu23/cpufreq/cpuinfo_cur_freq:<unknown>

We have opened a support case with Red Hat and at least on our HPE machines running RHEL7 this issue can be explained by a broken cpufreq driver. According to Red Hat and HPE, the pcc-cpufreq driver is selected on such machines if the Power regulator setting in the BIOS is not set to OS Control Mode. This driver is considered broken and is no longer developed. Both HPE and Red Hat advise to change the BIOS setting. This will load the intel_pstate driver instead, which should not have this problem.

We have confirmed that changing the BIOS setting makes Linux load the intel_pstate driver. We have not yet run any larger tests if this fixes the problem described in this Github issue, but I'm optimistic. :)

So, the generic answer is as expected: This is a problem in the cpufreq driver and nothing which node_exporter can fix. It also seems that this cosmetic issue is one of the less important symptoms of the buggy driver (which, it appears, can lead to unbootable or performance-degraded systems).

References: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=emr_na-c04704148 https://patchwork.kernel.org/patch/10528797/ https://access.redhat.com/solutions/3421421

I do think the error message could be improved, to make it easier to see exactly what CPU/metric failed.

@SuperQ I had a quick look regarding improving the error messages. node_exporter uses procfs' sysfs package and its parseCpufreqCpuinfo function. That function uses ReadUintFromFile. So I assume this is where the error should be wrapped with a hint about the affected filename. If this is wanted, I assume that it should be done with all those helper functions which take a path name, right?

I'v met this issue with node_exporter 1.1.4. My enviroment is a centos7.6 vm guest. The output of command metioned before:

[root@localhost ~]# for f in $(find /sys/devices/system/cpu/cpu*/cpufreq/ -type f) ; do echo "$f: $(< $f)" ; done
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: performance
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency: 4294967295
/sys/devices/system/cpu/cpu0/cpufreq/scaling_driver: intel_pstate
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq: <unknown>
/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors: performance powersave
/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: 0
/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_available_preferences: default performance balance_performance balance_power power 
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq: 0
/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference: performance
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq: 0
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: 0
/sys/devices/system/cpu/cpu0/cpufreq/affected_cpus: 0
/sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq: 0
/sys/devices/system/cpu/cpu0/cpufreq/related_cpus: 0
/sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed: <unsupported>
/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor: performance
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_transition_latency: 4294967295
/sys/devices/system/cpu/cpu1/cpufreq/scaling_driver: intel_pstate
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq: <unknown>
/sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors: performance powersave
/sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq: 0
/sys/devices/system/cpu/cpu1/cpufreq/energy_performance_available_preferences: default performance balance_performance balance_power power 
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_max_freq: 0
/sys/devices/system/cpu/cpu1/cpufreq/energy_performance_preference: performance
/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_min_freq: 0
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq: 0
/sys/devices/system/cpu/cpu1/cpufreq/affected_cpus: 1
/sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq: 0
/sys/devices/system/cpu/cpu1/cpufreq/related_cpus: 1
/sys/devices/system/cpu/cpu1/cpufreq/scaling_setspeed: <unsupported>

[hoffie]

Just a quick update from my side: As expected, this issue went away for us by fixing the frequency scaling driver as recommended by the vendors (i.e. moving away from pcc-cpufreq). So, it was not a node_exporter issue, but a kernel (driver) bug in our case.

[rexagod]

Raised a PR on procfs: https://github.com/prometheus/procfs/pull/614, not sure if merging that's going to close this issue automatically, probably needs to be manually done once that's in.