saltstack
[BUG] Salt master-initiated jobs no longer work after master upgraded to Fedora 39/Python 3.12 / 3006.5
Description Running Fedora 38 master with a Fedora 39 minion and a Debian 11 minion. Everything worked perfectly until I upgraded the master to Fedora 39. No sls or config changes took place.
See #65360 for background.
salt apply trace output:
[gwyn@REDACTED ~]$ sudo salt 'REDACTED' state.apply
/usr/lib/python3.12/site-packages/salt/ext/tornado/gen.py:1064: DeprecationWarning: the (type, exc, tb) signature of throw() is deprecated, use the single-arg signature instead.
yielded = self.gen.throw(*exc_info)
There is no current event loop in thread 'Thread-2 (_target)'.
[gwyn@REDACTED ~]$ sudo salt 'REDACTED' state.apply -l trace
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'warning', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'salt'}
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[TRACE ] Processing <bound method SaltfileMixIn.process_saltfile of <salt.cli.salt.SaltCMD object at 0x7f706ef52a80>>
[TRACE ] Processing <bound method ConfigDirMixIn.process_config_dir of <salt.cli.salt.SaltCMD object at 0x7f706ef52a80>>
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: REDACTED
[DEBUG ] Missing configuration file: /root/.saltrc
[TRACE ] Processing <bound method ExtendedTargetOptionsMixIn.process_pillar_target of <salt.cli.salt.SaltCMD object at 0x7f706ef52a80>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output of <salt.cli.salt.SaltCMD object at 0x7f706ef52a80>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output_file of <salt.cli.salt.SaltCMD object at 0x7f706ef52a80>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_state_verbose of <salt.cli.salt.SaltCMD object at 0x7f706ef52a80>>
[TRACE ] Processing <bound method NoParseMixin.process_no_parse of <salt.cli.salt.SaltCMD object at 0x7f706ef52a80>>
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.
@limburgher Are you installing the packages from repo.saltproject.io? The packages in fedora's repositories are not supported by us.
No, I'm the Fedora package maintainer. Not looking for package support, just the code itself.
Salt Version: Salt: 3006.5
Python Version: Python: 3.12.0 (main, Oct 2 2023, 00:00:00) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)]
Dependency Versions: cffi: 1.15.1 cherrypy: 18.8.0 dateutil: 2.8.2 docker-py: Not Installed gitdb: 4.0.10 gitpython: 3.1.40 Jinja2: 3.1.2 libgit2: 1.7.1 looseversion: 1.3.0 M2Crypto: Not Installed Mako: 1.2.3 msgpack: 1.0.5 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 23.1 pycparser: 2.20 pycrypto: 2.6.1 pycryptodome: 3.19.0 pygit2: 1.13.3 python-gnupg: 0.5.0 PyYAML: 6.0.1 PyZMQ: 25.1.0 relenv: Not Installed smmap: 5.0.0 timelib: Not Installed Tornado: 6.3.3 ZMQ: 4.3.4
System Versions: dist: fedora 39 locale: utf-8 machine: x86_64 release: 6.5.6-300.fc39.x86_64 system: Linux version: Fedora Linux 39
Sorry, We do not officially support 3.12 yet. If you'd like support you'll need to use the packages from repo.saltproject.io.
I understand that, but I assume you will eventually, and I'd like to help make that happen.
Okay, I've added this to the Argon release though I'm not really sure when we'll start shipping a newer python.
Things are much better with 3007.0rc1. Now I have the following in the minon log, and minons don't connect.
2024-01-09 11:24:40,298 [salt.transport.zeromq:279 ][ERROR ][929268] Exception while running callback Traceback (most recent call last): File "/usr/lib/python3.12/site-packages/salt/transport/zeromq.py", line 394, in consume await callback(msg) File "/usr/lib/python3.12/site-packages/salt/channel/client.py", line 486, in wrap_callback await callback(decoded) TypeError: object NoneType can't be used in 'await' expression
Just hit this issue with upgrades from Fedora 38->40. 'salt-call state.highstate' run on a minion is unable to connect to the master:
/usr/lib/python3.12/site-packages/salt/grains/core.py:2914: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC). start_time = datetime.datetime.utcnow() /usr/lib/python3.12/site-packages/salt/ext/tornado/gen.py:1064: DeprecationWarning: the (type, exc, tb) signature of throw() is deprecated, use the single-arg signature instead. yielded = self.gen.throw(*exc_info) Unable to sign_in to master: Attempt to authenticate with the salt master failed with timeout error
Trace shows this error:
[TRACE ] Failed to send msg SaltReqTimeoutError('Message timed out')
immediately after sending the public key to the server:
[TRACE ] ReqChannel send clear load={'cmd': '_auth', 'id': ...}
Network traffic is observed on the salt master (and was working just fine before the upgrade from 38->40).
Fedora 40 has the package 3006.7, which is the same version as Fedora 38. I think the big difference/issue here is the python version. If we look to install 3007.0 via this page: https://docs.saltproject.io/salt/install-guide/en/latest/topics/install-by-operating-system/fedora.html it only shows install options for Fedora 37 (EOL ~6 months ago) and Fedora 38 (to be EOL in a month). Is our only option to do the bootstrap install for Fedora 40?
Thanks.
@gregoster Same, this caused me issues when I upgraded a bunch of hosts to F39. I manually added the saltstack onedir EL9 repo to my F39 hosts to install 3007 and it seems to have been working fine. I system-upgraded a Fedora 39 host to 40 the other day and the EL9 packages still work fine.
root@test:0 ~ # cat /etc/os-release
NAME="Fedora Linux"
VERSION="40 (Forty)"
ID=fedora
VERSION_ID=40
root@test:0 ~ # cat /etc/yum.repos.d/salt.repo
[salt-repo]
name=Salt repo for RHEL/CentOS 9 PY3
baseurl=https://repo.saltproject.io/salt/py3/redhat/9/x86_64/latest
skip_if_unavailable=True
priority=10
enabled=1
enabled_metadata=1
gpgcheck=1
gpgkey=https://repo.saltproject.io/salt/py3/redhat/9/x86_64/latest/SALT-PROJECT-GPG-PUBKEY-2023.pub
root@test:0 ~ # dnf list installed | grep salt
salt.x86_64 3007.0-0 @salt-repo
salt-minion.x86_64 3007.0-0 @salt-repo
Running sh bootstrap-salt.sh on a Fedora 40 system results in:
`...
- ERROR: https://repo.saltproject.io/salt/py3/fedora/40/x86_64/latest.repo failed to download to /etc /yum.repos.d/salt.repo
- ERROR: https://repo.saltproject.io/salt/py3/fedora/40/x86_64/latest/SALT-PROJECT-GPG-PUBKEY-2023.pub failed to download to /tmp/salt-gpg-DfWiB24J.pub
- ERROR: Failed to run install_fedora_onedir_deps()!!!
Runningsh bootstrap-salt.sh git v3007.0results in:... a whole bunch of output, ending with... Processing /tmp/git/deps/yarl-1.9.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing /tmp/git/deps/zc.lockfile-3.0.post1-py3-none-any.whl Processing /tmp/git/deps/zipp-3.18.1-py3-none-any.whl INFO: pip is looking at multiple versions of salt to determine which version is compatible with other requirements. This could take a while. ERROR: Could not find a version that satisfies the requirement pywin32>=305 (from salt) (from versions: none) ERROR: No matching distribution found for pywin32>=305 - ERROR: Failed to run install_fedora_git()!!!
`
Not sure what to try next, or why it should even be thinking about pywin32, so suggestions are welcome.
Thanks.
Fedora package maintainer here. salt as packaged by Fedora doesn't work from Fedora 39 forward because Fedora 39 and up have Python 3.12, which salt does not yet support. 3007.0 cannot yet be packaged for Fedora because it requires python3-cryptography 42.0+, which a few other Fedora packages cannot yet use.
The only way I'm aware of to run a salt master on Fedora 39 and above at the moment is the onedir packaging. Then you rely on the salt project to keep that whole python stack up to date and secure. Not being able to run distro packages might be a dealbreaker for some organizations, but because of the cryptography dependency I can't even really begin checking for issues with Python 3.13, which will probably hit Fedora 42.
I don't know why the project is interested in up to date versions of one module but not the rest of the language stack, but it makes life tricky for distro packagers.
Thanks @limburgher , looks like its going to be a bit before there are 'official' packages then... Thanks @29821632 , we went with your solution, and so far it seems to be working fine here too.
@29821632 > manually added the saltstack onedir EL9 repo to my F39 hosts to install 3007 and it seems to have been working fine. I system-upgraded a Fedora 39 host to 40 the other day and the EL9 packages still work fine.
seems like a valid workaround for now. confirmed working on f40
