Salt API crashes if Python is updated

[voidmain]

Description of Issue

Last night our server running Salt (master, api, and a minion for the master) upgraded Python via unattended-upgrades. This caused the salt-api process to crash. It appears that the process was running, but become unresponsive to HTTP requests.

This was in the system log files:

Dec 16 06:29:35 saltstack-master.internal.fusionauth.io salt-api[302948]: [16/Dec/2021:06:29:35] ENGINE Restarting because /usr/lib/python3.8/email/base64mime.py changed.
Dec 16 06:29:35 saltstack-master.internal.fusionauth.io salt-api[302948]: [16/Dec/2021:06:29:35] ENGINE Stopped thread 'Autoreloader'.
Dec 16 06:29:35 saltstack-master.internal.fusionauth.io salt-api[302948]: [16/Dec/2021:06:29:35] ENGINE Bus STOPPING

Additionally, here is the Python update in the APT logs:

Start-Date: 2021-12-16  06:29:33
Commandline: /usr/bin/unattended-upgrade
Upgrade: libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), libpython3.8:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), python3.8:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), python3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2)
End-Date: 2021-12-16  06:29:36

Setup

Here's our master config:

rest_cherrypy:
  port: 8080
  thread_pool: 10
  socket_queue_size: 5
  disable_ssl: True
  debug: True

external_auth:
  file:
    ^filename: /etc/salt/users.txt
    <redacted>:
      - .*
    <redacted>:
      - test.ping

Versions Report

Salt Version:
          Salt: 3003

Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Nov 26 2021, 20:14:08)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-1047-aws
        system: Linux
       version: Ubuntu 20.04 focal

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[garethgreenaway]

@voidmain Thanks for the report. The issue above seems related to cherrypy, which is causing the restart when the underlying Python files changed. With Salt moving to onedir packages, https://docs.saltproject.io/en/master/topics/releases/3005.html#new-packages-available, and no longer relying on the system version of Python this particular issue should no longer occur.

[voidmain]

Looks like Onedir is broken on ARM64 Linux. I opened another issue (#62757) for that because we are using Graviton EC2 instances for everything and can't upgrade to Onedir until it is fixed.

[garethgreenaway]

Closing this due to age and the old version of Salt. Can you retest this with the latest release of Salt 3006.2 and if this is still an issue, please open a new issue.