saltstack
[BUG] Schedule jobs deleted or cleaned up on multimaster failover
Description When multimaster set to failover, and the minion failovers to the second master, the tasks in the Salt Scheduler (conf\minion.d_schedule.conf) are gon, cleaned up. So the minion does not run the expected scheduled tasks when connected to the failover master
Here a sequence on a minion configured in multimaster failover, two masters .10 and .11. A state is executed to add the task to the scheduler, confirmed its presence in the schedule file, then the minion failover to the other master and the task is gone
PS C:\Users\adrian> salt-call status.master master=172.21.0.10
local:
True
PS C:\Users\adrian> salt-call status.master master=172.21.0.11
local:
False
PS C:\Users\adrian> Get-Content 'C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf'
schedule:
__master_alive_172.21.0.10:
enabled: true
function: status.master
jid_include: true
kwargs: {connected: true, master: 172.21.0.10}
maxrunning: 1
return_job: false
seconds: 30
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 60, return_job: false, run_on_start: true}
PS C:\Users\adrian> salt-call state.sls utils/set_salt_schedule_reboot -l quiet
local:
----------
ID: schedule_new_task
Function: schedule.present
Result: True
Comment: Adding new job schedule_new_task to schedule
Started: 12:45:42.712122
Duration: 62.501 ms
Changes:
----------
schedule_new_task:
added
Summary for local
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 62.501 ms
PS C:\Users\adrian> salt-call schedule.list
local:
schedule:
schedule_new_task:
args:
- utils.reboot_system_module
enabled: true
function: state.sls
kwargs:
queue: true
saltenv: base
maxrunning: 1
name: schedule_new_task
return_job: false
saved: true
splay: 10
when:
- '2024-04-16 12:50:42'
PS C:\Users\adrian> Get-Content 'C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf'
schedule:
__master_alive_172.21.0.10:
enabled: true
function: status.master
jid_include: true
kwargs: {connected: true, master: 172.21.0.10}
maxrunning: 1
name: __master_alive_172.21.0.10
return_job: false
run: true
seconds: 30
splay: null
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 60, name: __mine_interval, return_job: false, run: true, run_on_start: true,
splay: null}
schedule_new_task:
args: [utils.reboot_system_module]
enabled: true
function: state.sls
kwargs:
queue: true
saltenv: base
maxrunning: 1
name: schedule_new_task
return_job: false
splay: 10
when: ['2024-04-16 12:50:42']
# Master .10 is disconnected and the minion did the failover to the .11 master
PS C:\Users\adrian> salt-call status.master master=172.21.0.10
12:48:11,637 [salt.minion :187 ][WARNING ][1984] Master ip address changed from 172.21.0.10 to 172.21.0.11
local:
False
PS C:\Users\adrian> salt-call status.master master=172.21.0.11
12:48:24,072 [salt.minion :187 ][WARNING ][4280] Master ip address changed from 172.21.0.10 to 172.21.0.11
local:
True
# The schedule is empty
PS C:\Users\adrian> Get-Content 'C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf'
schedule:
__master_alive_172.21.0.11:
function: status.master
jid_include: true
kwargs: {connected: true, master: 172.21.0.11}
maxrunning: 1
return_job: false
seconds: 30
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 60, return_job: false, run_on_start: true}
PS C:\Users\adrian> salt-call schedule.list
12:48:51,702 [salt.minion :187 ][WARNING ][4740] Master ip address changed from 172.21.0.10 to 172.21.0.11
local:
schedule: {}
Setup 3006.1 Minion set in multimaster failover master:
- 172.21.0.10
- 172.21.0.11 master_type: failover master_alive_interval: 30 retry_dns: 0
Steps to Reproduce the behavior Described in the issue description
Expected behavior I would expect the schedule jobs to be retained
Screenshots If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)PASTE HERE
Additional context Add any other context about the problem here.
