salt [BUG] ldap module broken after upgrade to 3006.2

Description On a clean install of 3006.2 on AlmaLinux 9, the ldap module fails to import with the following error:

$ /opt/saltstack/salt/bin/python3 -c 'import ldap'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/ldap/__init__.py", line 34, in <module>
    import _ldap
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

It works with 3006.1, as well as the system Python.

Steps to Reproduce the behavior

$ podman run --rm -it almalinux:9
$ curl -o bootstrap-salt.sh -L https://bootstrap.saltproject.io
$ bash bootstrap-salt.sh onedir
$ rpm -qf `which salt-pip`
salt-3006.2-0.x86_64
$ dnf install gcc openldap-devel
$ rpm -q openldap-devel
openldap-devel-2.6.2-3.el9.x86_64
$ salt-pip install python-ldap
Collecting python-ldap
  Downloading python-ldap-3.4.3.tar.gz (377 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━ 377.4/377.4 kB 8.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyasn1-modules>=0.1.5
  Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 22.3 MB/s eta 0:00:00
Collecting pyasn1>=0.3.7
  Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 64.4 MB/s eta 0:00:00
Building wheels for collected packages: python-ldap
  Building wheel for python-ldap (pyproject.toml) ... done
  Created wheel for python-ldap: filename=python_ldap-3.4.3-cp310-cp310-linux_x86_64.whl size=317527 sha256=caec96d436be13a253a9e328e3cafd4257429b96f17a44210e05df38d283f479
  Stored in directory: /root/.cache/pip/wheels/46/da/1e/560bb8f6a776fb7e42af5f32f40360f4bd4674710337e1b087
Successfully built python-ldap
Installing collected packages: pyasn1, pyasn1-modules, python-ldap
  WARNING: In `rpath_only mode` but /lib64/libldap.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/liblber.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libevent-2.1.so.7 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libsasl2.so.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libcrypt.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libcom_err.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libkeyutils.so.1 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libselinux.so.1 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libpcre2-8.so.0 is not in /opt/saltstack/salt
  Do not adjust rpath of /tmp/pip-target-zabty43p/_ldap.cpython-310-x86_64-linux-gnu.so
Successfully installed pyasn1-0.5.0 pyasn1-modules-0.3.0 python-ldap-3.4.3

$ /opt/saltstack/salt/bin/python3 -c 'import ldap'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/ldap/__init__.py", line 34, in <module>
    import _ldap
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

Repeating the above with bootstrap-salt.sh stable 3006.1 works as expected. For now we're switching to the ldap3 module in our external pillars.

Aug 11 '23 20:08 gvalkov

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

Aug 11 '23 20:08 welcome[bot]

Do you have the same versions of libldap and libssl installed on both systems?

Is it python_ldap==3.4.3 that you're trying to install on both?

Aug 14 '23 09:08 OrangeDog

Hi @gvalkov, it seems that openldap is trying to use the md2 method of openssl. This is happening on 3006.2 because we switched to openssl 3.1.2, and md2 is seemingly not available normally or with the legacy provider. Oddly enough, older versions of openldap (debian 11, for instance, installs 2.4.2 I believe), don't run into this problem with relenv==0.13.4. I have looked at it, and conferred with @dwoz - He is out until friday but will look at this when he gets time. With that being said, we are in code-freeze for 3006.3 as of tomorrow, so I will be pushing to 3006.4.

Aug 16 '23 16:08 MKLeb

Hi @MKLeb. Thank you for looking into this.

I already migrated our FreeIPA extpillar to use the pure-python ldap3 module (it's somewhat confusing that salt's ldap3 module is still based on the native bindings). Before that, I had this workaround that patched EVP_md2 out of libldap (with lief):

import lief

lib = lief.parse("/lib64/libldap.so.2")
sym = next(i for i in lib.imported_symbols if i.name == "EVP_md2")
lib.remove_dynamic_symbol(sym)
lib.write("/opt/saltstack/salt/lib//libldap.so.2")

Aug 19 '23 21:08 gvalkov

As I also had this issue and based on @gvalkovr comment's, here is what I did to workaround this problem.

System is running salt version 3006.4 (Sulfur) on OS Red Hat Enterprise Linux release 9.3 (Plow)

To confirm the error, I ran the following command

/opt/saltstack/salt/bin/python3 -c 'import ldap'

As expected, it throw the following

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/ldap/__init__.py", line 34, in <module>
    import _ldap
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

So, I nstalled lief python module

salt-pip install lief

Then edited file in /opt/saltstack/salt/extras-3.10/ldap/__init__.py or /opt/saltstack/salt/lib/python3.10/site-packages/ldap/__init__.py, depend on your installation

I'v added the following on top of the line 34 before import _ldap

import lief

lib = lief.parse("/lib64/libldap.so.2")
sym = next(i for i in lib.imported_symbols if i.name == "EVP_md2")
lib.remove_dynamic_symbol(sym)
lib.write("/opt/saltstack/salt/lib/libldap.so.2")

Restarted services

systemctl restart salt-master.service salt-minion.service salt-api.service

It generated segfault errors in syslog output

segfault at 15d90 ip 0000000000015d90 sp 00007ffdb8d040e8 error 14 in python3.10[400000+308000] likely on CPU

Again, edited file in /opt/saltstack/salt/extras-3.10/ldap/__init__.py or /opt/saltstack/salt/lib/python3.10/site-packages/ldap/__init__.py, depend on your installation

Removed the lines previously added near line 34 before import _ldap

And again, restarted services

systemctl restart salt-master.service salt-minion.service salt-api.service

Re-ran the following

/opt/saltstack/salt/bin/python3 -c 'import ldap'

Then voilà, this time no output nor errors and no more error such as LDAP authentication requires python-ldap module

Dec 08 '23 14:12 iMikeG6

Also present in 3007.1

May 31 '24 00:05 zleite

Confirming 3007.1 is affected.

I need to roll back; LDAP eauth is the only enterprise-level auth option available and it is incredibly broken. My team cannot use the API whatsoever.

~~Note that the workaround above does not work in 3007.1 in my testing.~~ (See next comment)

Jun 15 '24 08:06 nf-brentsaner

Okay, I have this working in my lab environment. I've identified two things.

I had a /opt/saltstack/salt/extras-3.10/_ldap.cpython-310-x86_64-linux-gnu.so file. This does not seem to be present in fresh installs, and may have been causing issues. I have removed it.
The lief workaround above DOES work, but it does not need to be run from within Salt python. The below will also not segfault. It (should) stop and start Salt services. Works fine on Alma 9.4 once the lief python module is installed.

#!/usr/bin/env python3

import datetime
import grp
import os
import pathlib
import pwd
import shutil
##
import dbus  # dnf -y install python3-dbus
##
import lief  # https://pypi.org/project/lief/

# https://github.com/saltstack/salt/issues/64962

salt_root = pathlib.Path('/opt/saltstack')
src_lib = pathlib.Path('/lib64/libldap.so.2')
dst_lib = salt_root.joinpath('salt', 'lib', 'libldap.so.2')

uname = 'salt'
gname = 'salt'

lib = lief.parse(str(src_lib))
sym = next(i for i in lib.imported_symbols if i.name == 'EVP_md2')
if sym:
    # Get the Salt services from DBus.
    sysbus = dbus.SystemBus()
    sysd = sysbus.get_object('org.freedesktop.systemd1', '/org/freedesktop/systemd1')
    mgr = dbus.Interface(sysd, 'org.freedesktop.systemd1.Manager')
    svcs = []
    for i in mgr.ListUnits():
        # first element is unit name.
        if not str(i[0]).startswith('salt-'):
            continue
        svc = sysbus.get_object('org.freedesktop.systemd1', object_path = mgr.GetUnit(str(i[0])))
        props = dbus.Interface(svc, dbus_interface = 'org.freedesktop.DBus.Properties')
        state = props.Get('org.freedesktop.systemd1.Unit', 'ActiveState')
        if str(state) == 'active':
            svcs.append(i[0])
    # Get the user/group
    u = pwd.getpwnam(uname)
    g = grp.getgrnam(gname)
    # Modify
    print('Modifications necessary.')
    if svcs:
        # Stop the services first.
        for sn in svcs:
            mgr.StopUnit(sn, 'replace')
    if dst_lib.exists():
        # 3.10 deprecated .utcnow().
        #dst_lib_bak = pathlib.Path(str(dst_lib) + '.bak_{0}'.format(datetime.datetime.now(datetime.UTC).timestamp()))
        dst_lib_bak = pathlib.Path(str(dst_lib) + '.bak_{0}'.format(datetime.datetime.utcnow().timestamp()))
        os.rename(dst_lib, dst_lib_bak)
        print('Destination file {0} exists; backed up to {1}.'.format(dst_lib, dst_lib_bak))
    lib.remove_dynamic_symbol(sym)
    lib.write(str(dst_lib))
    os.chown(dst_lib, u.pw_uid, g.gr_gid)
    os.chmod(dst_lib, src_lib.stat().st_mode)
    # Before we restart services, we also want to remove any python caches.
    for root, dirs, files in os.walk(salt_root):
        for f in files:
            if f.lower().endswith('.pyc'):
                fpath = os.path.join(root, f)
                os.remove(fpath)
                print('Removed file {0}'.format(fpath))
        if '__pycache__' in dirs:
            dpath = os.path.join(root, '__pycache__')
            shutil.rmtree(dpath)
            print('Removed directory {0}'.format(dpath))
    # And then start the units that were started before.
    if svcs:
        for sn in svcs:
            mgr.RestartUnit(sn, 'replace')

print('Done.')

Jun 16 '24 06:06 nf-brentsaner

@nf-brentsaner Are you able to test this on a current nightly build?

https://repo.saltproject.io/salt-dev/3006.x/latest/

Jun 22 '24 10:06 dwoz

I tried with latest 3007 on RHEL9 and it still fails.

Jul 11 '24 11:07 Sxderp

This appears to be having an impact on the libvirt module as well. This issue is not present in 3006.1.

Running the script from nf-brentsaner resolves the issue.

Prior to running the script:

[root@localhost ~]# /opt/saltstack/salt/bin/python3.10
Python 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 16, in <module>
    import cygvirtmod as libvirtmod  # type: ignore
ModuleNotFoundError: No module named 'cygvirtmod'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 19, in <module>
    raise lib_e
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 13, in <module>
    import libvirtmod  # type: ignore
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

After running the script

[root@localhost ~]# /opt/saltstack/salt/bin/python3.10
Python 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>>

[root@localhost ~]# salt --versions
Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: oracle 9.2
        locale: utf-8
       machine: x86_64
       release: 5.15.0-101.103.2.1.el9uek.x86_64
        system: Linux
       version: Oracle Linux Server 9.2

Aug 26 '24 13:08 m0duspwnens