[BUG] error KeyError: 'master_list' in sali-minion 3005.1 logs

[korjek]

Description Soon after minion process is started there is an error in the logs

2023-07-08 12:05:08,966 [salt.minion      :232 ][DEBUG   ][19917] Master URI: tcp://127.0.0.1:4506
2023-07-08 12:05:08,975 [salt.utils.event :315 ][DEBUG   ][19917] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_e6adef2bc4_pub.ipc
2023-07-08 12:05:08,975 [salt.utils.event :316 ][DEBUG   ][19917] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_e6adef2bc4_pull.ipc
2023-07-08 12:05:08,976 [salt.utils.event :821 ][DEBUG   ][19917] Sending event: tag = __master_failback; data = {'master': '127.0.0.1', '_stamp': '2023-07-08T12:05:08.976638'}
2023-07-08 12:05:08,977 [salt.transport.ipc:373 ][DEBUG   ][19917] Closing IPCMessageClient instance
2023-07-08 12:05:08,978 [salt.channel.client:313 ][DEBUG   ][19917] Closing AsyncReqChannel instance
2023-07-08 12:05:08,978 [salt.minion      :2690][DEBUG   ][19693] Minion of 'salt-a.domain.' is handling event tag '__master_failback'
2023-07-08 12:05:08,980 [tornado.application:356 ][ERROR   ][19693] Future <salt.ext.tornado.concurrent.Future object at 0x7f2e94f49588> exception was never retrieved: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2748, in handle_event
    if data["master"] != self.opts["master_list"][0]:
KeyError: 'master_list'
2023-07-08 12:05:08,980 [salt.minion      :2690][DEBUG   ][19693] Minion of 'salt-b.domain.' is handling event tag '__master_failback'
2023-07-08 12:05:08,980 [tornado.application:356 ][ERROR   ][19693] Future <salt.ext.tornado.concurrent.Future object at 0x7f2e94f49588> exception was never retrieved: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2748, in handle_event
    if data["master"] != self.opts["master_list"][0]:
KeyError: 'master_list'

Setup salt-minion config

master:
  - salt-a.domain
  - salt-b.domain.
master_type: str
renderer: yaml_jinja
log_level_logfile: debug
log_level: info

Steps to Reproduce the behavior Restart salt minion process

Expected behavior No errors in the log

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

salt-minion --versions-report
Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: 0.26.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: 0.26.2
        Python: 3.6.9 (default, Oct  8 2020, 12:12:24)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5

System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 5.4.0-1035-aws
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

[s0undt3ch]

@korjek can you, by any chance, test this with the latest release, 3006.1, to see if the issue still exists?

[korjek]

@korjek can you, by any chance, test this with the latest release, 3006.1, to see if the issue still exists?

coming back from vacation on July, 24th. So I'll test it till July, 28th (or earlier)

[OrangeDog]

Is the master also 3005.1?

[korjek]

Is the master also 3005.1?

yes

salt-master --version
salt-master 3005.1

[korjek]

@korjek can you, by any chance, test this with the latest release, 3006.1, to see if the issue still exists?

@s0undt3ch I got the same error on salt 3006.1

root@dev-sl-salt-master247:~# salt-minion --version
salt-minion 3006.1
root@dev-sl-salt-master247:~# salt-master --version
salt-master 3006.1

To reproduce I just sent the event with tag __master_failback to the minion:

salt 'dev-sl-salt-master247' event.fire '{"data":"my event data"}' '__master_failback'

[OrangeDog]

To reproduce I just sent the event with tag __master_failback to the minion

Was that also the case before, or is something else triggering an event with that tag?

[korjek]

To reproduce I just sent the event with tag __master_failback to the minion

Was that also the case before, or is something else triggering an event with that tag?

On 3005.1 something else triggered an event with that tag so I noticed that in the logs (but I'm able to reproduce it with event.fire). On 3006.1 I didn't notice the error in the logs until I manually sent this event to check whether this still causes an error record in logs (maybe I haven't waited enough time though. As my understanding was that an error record with a manual event is enough to consider this as a bug in the code for 3006.1)

[korjek]

we've updated to 3006.6 and issue is still here (