Sysstat End of system activity file unexpected

[cptcrush2]

I frequently encounter "End of system activity file unexpected" when running "sar -B -F". I have checked the FAQ and my systems are not running out of disk space. I am running Ubuntu 18.04. Ubuntu repos only contain sysstat version 11.6.1. Could "End of system activity file unexpected" problem be related to the sysstat version? I realize that the current version of sysstat is v12x. How can I resolve my "End of system activity file unexpected" issue without installing sysstat from source? Thank you for your help.

[sysstat]

No, I don't think that such an error can be related to a specific sysstat version. As written in the FAQ, this is probably because of a lack of disk space (I don't believe in the data collector sadc being interrupted when it writes its data to file). Look at the time when the datafile gets interrupted and check that you don't have another job running at the same time that could fill you filesystem. BTW I assume that you get this error even if you don't use those specific options "-B -F", don't you?

[cptcrush2]

Hello,

I have checked the time when I am encountering "End of system activity file unexpected." It seems to happen on random systems, but there is a pattern. It seems to happen a few minutes after 12 am. I can see that daily sa* files are rotated at 12am. I do not have disk space issues on any systems at 12am. Is it possible to change the time rotation from 12am to another time? Thank you. Alex

[sysstat]

Datafiles are rotated at 12am because this is when the day changes! A day change is precisely what shall trigger a file rotation, so you can't change that. Again, make sure that that you don't have a process that is started around midnight which could lead to fill your filesystem. Even if I can't exclude another reason, I presently don't see another explanation for your problem.

[cptcrush2]

Thank you for getting back to me. I do not see any issues with the file system. Perhaps I am not looking in the right place, and therefore, I am wrong. Are there any logs that I can provide to help identify this issue? How can I pinpoint a problem?

[sysstat]

Send me your saDD datafiles which are corrupted so that I can have a look at them (sysstat at orange dot fr). Did you try to configure sadc to save the datafiles in another filesystem? (Change the value for SA_DIR variable in /etc/sysconfig/sysstat file).

[cptcrush2]

Thank you for your help. I have emailed saDD files.

[sysstat]

Thanks for your saDD and crontab files. For context purpose, here is the crontab you sent me (contents of /etc/cron.d/sysstat file):

PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

# Activity reports every 10 minutes everyday
*/2 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2

I guess that two instances of sadc are colliding at midnight, the first one being run by this crontab line:

59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2

(which writes a final record at 23:59 for datafile saDD and another one at 00:00 for datafile sa(DD+1)), and a second one run by this crontab line:

*/2 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

which starts by writing a record also at 00:00 for datafile sa(DD+1).

Removing the first line above (which starts debian-sa1 at 23:59) from your crontab may fix your problem (tell me if it's the case) but statistics will be generated only from 00:00 until 23:58, and the period from 23:58 to midnight will be missing. Anyway sadc will still guess the right file name to use when the day changes (if your debian-sa1 script runs as expected by the genuine sysstat sa1 script). sysstat doesn't manage the midnight sample very well. A solution to get a sample for 00:00 has been given in sysstat's FAQ for those collecting statistics every 10 minutes:

*/10 1-22 * * * /usr/lib/sa/sa1 1 1
0,10,20,30,40 23 * * * /usr/lib/sa/sa1 1 1
50 23 * * * /usr/lib/sa/sa1 600 2
10,20,30,40,50 0 * * * /usr/lib/sa/sa1 1 1

You can use this example and modify it to suit your needs. But maybe a more robust solution should be implemented by sysstat. I will think about it.

[sysstat]

@cptcrush2 : Did you remove from your crontab the line used for file rotation as I suggested above? Do you still get the "End of system activity file unexpected" error message? Thank you.

[cptcrush2]

Hello. Yes, I have removed the crontab line as per your suggestion. I will monitor it for the next few weeks and let you know if the corruption issue is resolved. Thank you so much.

On Fri, Nov 13, 2020 at 11:33 PM GODARD Sebastien [email protected] wrote:

@cptcrush2 https://github.com/cptcrush2 : Did you remove from your crontab the line used for file rotation as I suggested above? Do you still get the "End of system activity file unexpected" error message? Thank you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sysstat/sysstat/issues/275#issuecomment-727160835, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGQ32KMMPA2M4DL66BVUAFTSPYXEFANCNFSM4SG5EABQ .

[cjl20062529]

Steps to Reproduce the corrupted sa file:

1, Mount ext4 file system to output sa file.

  [root@rhel74 ~]# mount | grep ext
  /dev/vdb1 on /ext4 type ext4 (rw,relatime,data=ordered)

2. Run the following reproduction shell. It outputs record ten times and reboots the system forcibly.

   [root@rhel74 ~]# cat sadc_ext4.sh #!/bin/sh

   for (( i = 0 ; i < 10 ; i++ )) do /usr/lib64/sa/sadc -F -L -S DISK 1 1 /ext4/sadate "+%m%d"_ext4 /usr/bin/sleep 1 done

   /usr/bin/echo b > /proc/sysrq-trigger

   [root@rhel74 ~]# ./sadc_ext4.sh

Actual results: sar files getting corrupted https://bugzilla.redhat.com/show_bug.cgi?id=1669932 @sysstat

[sysstat]

@cjl20062529 Use -f flag with sadc to make sure data are written to disk. This flag had been added in sysstat version 12.1.3 to prevent such a file corruption that a sudden system reboot could cause. Of course syncing each data sample to disk implies a (probably slight) performance penalty.