node_exporter icon indicating copy to clipboard operation
node_exporter copied to clipboard

Add IO stats and ext4 FS stats through new ext4 collector

Open mshahzeb opened this issue 1 year ago • 6 comments

Fixes: https://github.com/prometheus/node_exporter/issues/3005

Adds:

# HELP node_filesystem_errors Number of filesystem errors encountered.
# TYPE node_filesystem_errors counter
node_filesystem_errors{device="/dev/vda2",device_error="",fstype="ext4",mountpoint="/boot"} 0

# HELP node_filesystem_warnings Number of filesystem warnings encountered.
# TYPE node_filesystem_warnings counter
node_filesystem_warnings{device="/dev/vda2",device_error="",fstype="ext4",mountpoint="/boot"} 0

# HELP node_filesystem_messages Number of filesystem log messages.
# TYPE node_filesystem_messages counter
node_filesystem_messages{device="/dev/vda2",device_error="",fstype="ext4",mountpoint="/boot"} 0

From

  • /sys/fs/ext4/<partition>/errors_count: number of ext4 errors (commit)
  • /sys/fs/ext4/<partition>/warning_count: number of ext4 warning log messages (commit)
  • /sys/fs/ext4/<partition>/msg_count: number of other ext4 log messages

and

# HELP node_disk_ioerr_total Number of IO commands that completed with an error.
# TYPE node_disk_ioerr_total counter
node_disk_ioerr_total{device="sda"} 3
node_disk_ioerr_total{device="sr0"} 29

# HELP node_disk_iodone_total Number of completed or rejected IO commands.
# TYPE node_disk_iodone_total counter
node_disk_iodone_total{device="sda"} 307
node_disk_iodone_total{device="sr0"} 4483

From

  • /sys/block/<disk>/device/ioerr_cnt: number of SCSI commands that completed with an error
  • /sys/block/<disk>/device/iodone_cnt: number of completed or rejected SCSI commands

Implements new ext4 collector.

Corresponding procfs changes: https://github.com/prometheus/procfs/pull/651

mshahzeb avatar Jun 10 '24 11:06 mshahzeb

Sample generated metrics file node_metrics.txt

mshahzeb avatar Jun 10 '24 11:06 mshahzeb

This would be wonderful to have as a feature

BurritoWrapped avatar Jun 14 '24 16:06 BurritoWrapped

Hi @mshahzeb, thanks for looking into this! This is a great start, we now know which files to read.

node_exporter doesn't really try to read the files directly in this codebase, but rather, we abstract the parsing here: https://github.com/prometheus/procfs

/sys/block/<disk>/device/ioerr_cnt and /sys/block/<disk>/device/iodone_cnt should be added here: https://github.com/prometheus/procfs/blob/master/blockdevice/stats.go

/sys/fs/ext4/<partition> should be added to a new ext4 folder like we did for xfs and btrfs

gouthamve avatar Jun 21 '24 09:06 gouthamve

Thank you I will be moving the code to procfs and open a PR there.

mshahzeb avatar Jun 26 '24 14:06 mshahzeb

PR in the works on procfs: https://github.com/prometheus/procfs/pull/651

mshahzeb avatar Jul 10 '24 11:07 mshahzeb

Procfs PR merged: https://github.com/prometheus/procfs/pull/651

mshahzeb avatar Sep 10 '24 05:09 mshahzeb

Waiting for new procfs release

mshahzeb avatar Nov 06 '24 10:11 mshahzeb

You can also use the unreleased version for now

discordianfish avatar Dec 14 '24 13:12 discordianfish

Closing this PR; new PR opened at: https://github.com/prometheus/node_exporter/pull/3295

mshahzeb avatar Apr 01 '25 00:04 mshahzeb