relocatable-python icon indicating copy to clipboard operation
relocatable-python copied to clipboard
verified publisher icon gregneagle

Using link to execute relocated 3.11 bin/python3 fails to find libraries

Open MagerValp opened this issue 2 years ago • 7 comments

We're using a symbolic link for munkireport-python3 to point to a relocated python framework:

 → ls -l /usr/local/munkireport/munkireport-python3 
lrwxr-xr-x  1 root  wheel  63 24 Feb 10:10 /usr/local/munkireport/munkireport-python3 -> /Library/MunkiReport/Python.framework/Versions/3.11/bin/python3

Executing /Library/MunkiReport/Python.framework/Versions/3.11/bin/python3 works fine:

 → /Library/MunkiReport/Python.framework/Versions/3.11/bin/python3
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/site-packages']
>>> ^D

Calling it with the link fails to find platform dependent libraries:

 → /usr/local/munkireport/munkireport-python3 
Could not find platform dependent libraries <exec_prefix>
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',        # ouch
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/site-packages']
>>> import socket
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 51, in <module>
    import _socket
ModuleNotFoundError: No module named '_socket'
>>> ^D

A relocated 3.10 doesn't seem to have a problem with this:

 → /usr/local/munkireport/munkireport-python3                     
Python 3.10.2 (v3.10.2:a58ebcc701, Jan 13 2022, 14:50:16) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python310.zip',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python3.10',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python3.10/lib-dynload',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python3.10/site-packages']
>>> ^D

I'm building with these options:

	./make_relocatable_python_framework.py \
		--upgrade-pip \
		--pip-requirements="../$PKGDIR/requirements.txt" \
		--os-version=11 \
		--python-version "$version" \
		--destination "../$FWROOT" \

MagerValp avatar Feb 24 '23 09:02 MagerValp

I suspect that this is the relevant upstream change: https://github.com/python/cpython/pull/29041

MagerValp avatar Feb 24 '23 10:02 MagerValp

Relevant code from getpath.py:

https://github.com/python/cpython/blob/9f3ecd1aa3566947648a053bd9716ed67dd9a718/Modules/getpath.py#L174-L182

https://github.com/python/cpython/blob/9f3ecd1aa3566947648a053bd9716ed67dd9a718/Modules/getpath.py#L598-L622

MagerValp avatar Feb 24 '23 10:02 MagerValp

Possible workaround would be to replace the symlink with a shell script that basically does

/real/path/to/Python.framework/Versions/Current/bin/python3 $additional_args

gregneagle avatar Mar 20 '23 16:03 gregneagle

I've been playing around with PYTHONPATH and PYTHONHOME as documented here which "works", but keep in mind that thse exported environment vars will apply to any other installations of Python on the system (obviously depending on how/where they're exported). Also, any PYTHONPATH exports are added to the existing path that Python itself generates, so you could end up with a precedence issue as the exported PYTHONPATH may take priority in search order, etc, so it could have unintended consequences for any possible clashing imports.

Without the exports, you get the Could not find platform dependent libraries <exec_prefix> error and can't import standard packages like subprocess, etc.

[appleseed@infiniteloop]:~ # export PYTHONPATH=""                                                                                  
[appleseed@infiniteloop]:~ # export PYTHONHOME=""
[appleseed@infiniteloop]:~ # python3             
Could not find platform dependent libraries <exec_prefix>
Python 3.11.3 (v3.11.3:f3909b8bc8, Apr  4 2023, 20:12:10) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, subprocess
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 104, in <module>
    from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages',
 '/usr/local/python/site-packages']
>>>

With the exports, the Could not find platform dependent libraries <exec_prefix> error is gone, and can now import standard packages like subprocess, etc.

[appleseed@infiniteloop]:~ # export PYTHONHOME=/Library/ManagedFrameworks/Python.framework/Versions/3.11
[appleseed@infiniteloop]:~ # export PYTHONPATH=/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload                        
[appleseed@infiniteloop]:~ # python3                                                                                                                       
Python 3.11.3 (v3.11.3:f3909b8bc8, Apr  4 2023, 20:12:10) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> from pprint import pprint
>>> import subprocess
>>> pprint(sys.path)
['',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages',
 '/usr/local/python/site-packages']
>>> 
KeyboardInterrupt
>>> ^D

This is very early testing, and understandably won't work particularly well if your Python deployments are more complicated than just installing the ManagedFrameworks version, or in the case of munki, so YMMV.

carlashley avatar May 31 '23 00:05 carlashley

With @carlashley's tip, I was able to work around this by creating a simple shim script that can be invoked (the context here was addressing https://github.com/macadmins/python/issues/48):

#!/bin/zsh

bin="/Library/ManagedFrameworks/Python/Python3.framework/Versions/3.11/bin/python3.11"

PYTHONHOME=/Library/ManagedFrameworks/Python.framework/Versions/3.11
PYTHONPATH=/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload
exec "${bin}" "$@"

This script can put anywhere (such as /usr/local/bin/managed_python or whatever), and solves the immediate issue:

% /usr/local/bin/managed_python3_fixed
Python 3.11.1 (v3.11.1:a7a450f84a, Dec  6 2022, 15:24:06) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> exit()

nmcspadden avatar Jun 20 '23 16:06 nmcspadden

Actually, this works even without the two env variables.

I used Munki's build_python_framework.sh script to build a 3.12.2 Python framework (see https://github.com/munki/munki/issues/1204 for why), and inserted this shell script:

% cat /usr/local/munki/python3.12.fixed
#!/bin/sh
# Wrapper to force arm64 execution of the python3.12 universal binary; the `arch` is probably unnecessary
exec /usr/bin/arch -arm64 /Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/bin/python3 "$@"

Then:

% /usr/local/munki/python3.12.fixed
Python 3.12.2 (v3.12.2:6abddd9f6a, Feb  6 2024, 17:02:06) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python312.zip', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python3.12', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python3.12/lib-dynload', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python3.12/site-packages']
>>> import socket
>>>

It seems to correctly utilize the path of the directory it's in, rather than looking in /Library/Frameworks.

nmcspadden avatar Feb 20 '24 23:02 nmcspadden