saltstack
Minion scheduler not executed
Hey,
I have added a scheduler using the schedule.present state as follow:
midnight-execution:
schedule.present:
- persist: True
- maxrunning: 1
- function: state.apply
- when: 00:02am
The state successfully run and I can see the schedule at the /etc/salt/minion.d/_schdule.conf file as follow:
schedule:
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2, minutes: 60, return_job: false, run_on_start: true}
midnight-execution: {enabled: true, function: state.apply, jid_include: true, maxrunning: 1, name: midnight-execution, when: '00:02am'}
Everything is fine until now...
The trouble come after few days when the scheduler stop working properly.
After several days/weeks the scheduler stop fully executing the midnight-execution, I can tell that using two methonds:
- The new/updated functionality is not applied
- The log
The NOT WORKING log file show the following:
[2021-08-18 23:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
[2021-08-19 00:02:00.015] [salt.utils.schedule] [INFO] Running scheduled job: midnight-execution
[2021-08-19 00:02:00.755] [salt.state] [INFO] Loading fresh modules for state activity
[2021-08-19 00:02:00.790] [salt.minion] [INFO] Returning information for job: req
[2021-08-19 00:02:00.791] [salt.minion] [INFO] Returning information for job: req
[2021-08-19 00:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
[2021-08-19 01:09:00.017] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
...
...
...
[2021-08-19 23:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
**The same routine all over again...**
This issue is not happening on all of my servers. At WORKING servers I can see the following log: This is how is suppose to be seen when working as expected
[2021-08-18 23:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
[2021-08-19 00:02:00.015] [salt.utils.schedule] [INFO] Running scheduled job: midnight-execution
[2021-08-19 00:02:00.755] [salt.state] [INFO] Loading fresh modules for state activity
[2021-08-19 00:02:00.799] [salt.template] [PROFILE] Time (in seconds) to render 'SOME TOP.SLS file path' using 'jinja' renderer: 0.0099799
[2021-08-19 00:02:00.799] [salt.template] [PROFILE] Time (in seconds) to render 'SOME TOP.SLS file path' using 'yaml' renderer: 0.0003178
...
[2021-08-19 00:02:03.799] [salt.fileclient] [INFO] Caching directory........
[2021-08-19 00:02:03.801] [salt.utils.extmods] [INFO] Syncing matchers for environment....
[2021-08-19 00:02:03.802] [salt.utils.extmods] [INFO] Loading cache from salt://_matchers, for environment ...
...
...
[2021-08-19 00:02:06.000] [salt.state] [INFO] Running state [STATE NAME] at time ....
[2021-08-19 00:02:06.005] [salt.state] [INFO] Executing state file.securse for [STATE NAME]
[2021-08-19 00:02:06.010] [salt.state] [INFO] The directory 'DIR PATH' is in the correct state
...
I'm running on Ubuntu 16.04:
- salt-master version
3000.6 - salt-minion version:
3000.2,3000.3,3002.2.
I have around ~1800 servers assign to the master and ~400 are having this issue (the scheduler is not executing the state.apply scheduled task).
The majority of the servers that are having this issue is running version 3002.2
Please help... Best regards, Amiram
Running minions newer than the master is not generally supported, especially across two major versions. You are also using versions with critical security issues. If you synchronise everything to the same version (e.g. 3000.9 or 3002.6) does everything then work correctly?
Ubuntu 16.04 is also not supported, being EOL.
I have upgraded my OS to Ubuntu Focal (20.04) and I'm still having the same issue.
In addition the minion and the master are having the same version - 3004.1 (and in test case the master have newer version - 3004.2).
Even after minion service restart the scheduler failed to execute successfully.