asdf-community
Python build failed on MacOs Big Sur 11.2.3
Relates to #93 and #94
After updating to MacOs 11.2.3 I cannot build python.
I've reinstalled all Brew dependencies:
brew reinstall bzip2 lbzip2 lzlib openssl zlib
I've set environment variables, and run asdf install with variables directly:
LDFLAGS="-L/usr/local/opt/bzip2/lib -L/usr/local/opt/zlib/lib -L/usr/local/opt/[email protected]/lib" CFLAGS="-I/usr/local/opt/bzip2/include -I/usr/local/opt/zlib/include -I/usr/local/opt/[email protected]/include -I$(xcrun --show-sdk-path)/usr/include -Wno-implicit-function-declaration" asdf install python 3.8.8
This is the current result of running with variables;
python-build 3.8.8 /Users/anders/.asdf/installs/python/3.8.8
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.8.tar.xz...
-> https://www.python.org/ftp/python/3.8.8/Python-3.8.8.tar.xz
Installing Python-3.8.8...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
BUILD FAILED (OS X 11.2.3 using python-build 1.2.23-75-g80e418ec)
Inspect or clean up the working tree at /var/folders/tl/06ddx4rj5tj_5h3gpvvtxtbm0000gn/T/python-build.20210317113113.24543
Results logged to /var/folders/tl/06ddx4rj5tj_5h3gpvvtxtbm0000gn/T/python-build.20210317113113.24543.log
Last 10 log lines:
File "/private/var/folders/tl/06ddx4rj5tj_5h3gpvvtxtbm0000gn/T/python-build.20210317113113.24543/Python-3.8.8/Lib/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "./setup.py", line 320, in build_extensions
self.detect_modules()
File "./setup.py", line 1695, in detect_modules
self.detect_readline_curses()
File "./setup.py", line 930, in detect_readline_curses
(tuple(int(n) for n in dep_target.split('.')[0:2])
AttributeError: 'int' object has no attribute 'split'
make: *** [sharedmods] Error 1
Looks like Python bug: https://bugs.python.org/issue42504
So, I don't think this is an asdf-python or python-build issue.
It was fixed in both latest Python and at least backported to 3.9. Fix exists in CPython v3.9.2: https://github.com/python/cpython/blob/v3.9.2/Lib/sysconfig.py#L22-L24
Not too sure if they plan to backport to 3.8.
EDIT: @smedegaard if you need Python 3.8 and cannot use 3.9, you may want to try manual patch. Homebrew uses patch file: https://github.com/python/cpython/commit/09a698b4.patch?full_index=1 as seen https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]#L74
ASDF supports a single patch file: https://github.com/danhper/asdf-python/blob/8ab052fced78b2c68c8fe17d5811d0ebae06eee8/bin/install#L19
So, maybe try something like
export ASDF_PYTHON_PATCHES_DIRECTORY=/tmp
curl -sSL "https://github.com/python/cpython/commit/09a698b4.patch?full_index=1" > "$ASDF_PYTHON_PATCHES_DIRECTORY/3.8.8.patch"
I'm having a related problem but for 3.7.3. Here's the information from my log:
python-build 3.7.3 /Users/me/.asdf/installs/python/3.7.3
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.3.tar.xz...
-> https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz
Installing Python-3.7.3...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
BUILD FAILED (OS X 11.2.3 using python-build 1.2.24.1-6-g132d5464)
Inspect or clean up the working tree at /var/folders/l3/plnsfnc91qd2_3dh7v5b1x180000gn/T/python-build.20210331134326.79346
Results logged to /var/folders/l3/plnsfnc91qd2_3dh7v5b1x180000gn/T/python-build.20210331134326.79346.log
Last 10 log lines:
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/errnomodule.c -o Modules/errnomodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/_sre.c -o Modules/_sre.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/_codecsmodule.c -o Modules/_codecsmodule.o
./Modules/posixmodule.c:8431:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = sendfile(in, out, offset, &sbytes, &sf, flags);
^
1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
@curtisault The error looks similar to https://bugs.python.org/issue43470, which was closed (i.e. won't fix)
From response,
While the particular error you ran into is easily worked around, the resulting Python will still not execute correctly on macOS 11 Big Sur as there are a number of other run-time issues that would need to be fixed. At the moment, only Python 3.9.2 is fully supported on Big Sur.
Looking at Python status at https://devguide.python.org/#status-of-python-branches,
- Python 3.7 will only accept security fixes, so it probably will never work on Big Sur unless you manually patch it.
- Python 3.8 accepts bugfixes, but it is still up to release manager to decide if they want to backport fixes for Big Sur.
- Python 3.9 is current stable, so it works on Big Sur
EDIT: if you seriously need an older version of Python and don't care about the fact that Python devs consider it unsupported, then you can one of following:
- try hunting down alternative libraries. Try the Homebrew versions like shown in first post (or https://github.com/danhper/asdf-python/issues/93#issuecomment-791737833) and work from there. You may also have to hunt down patches to apply during build.
- you can always try a Pre-Built. As
asdf-pythonneeds to build Python usingpython-build, it probably doesn't support this. If you are on Intel Mac, you can give Homebrew's[email protected]formula or MacPort'spython37a try
Also, see if you can use a newer minor version. I have been able to build v3.7.8+ locally with XCode 12.4 and Homebrew's readline/openssl on M1 Rosetta2 (v3.7.7 and older had similar issue).
I believe this issue may not be limited to prior versions of python, I'm also running into similar python-build errors while trying to install the latest v3.9.4 (also MacOS 11.2.3).
I'm trying to understand more what the problem is, but I believe there may be some compat issues goinig on with my xcode tooling. I've tried to do the install using latest xcode v12 and beta v12.5 via xcode-select --switch but no difference.
Last 10 log lines:
checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in '/var/folders/01/197gjn9100312t8k_61655tr0000gn/T/python-build.20210405194509.72821/Python-3.9.4':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use '--host'.
See 'config.log' for more details
I wish I could provide more details; does anyone know how to access the config.log this refers too?
I'm pretty sure this is not relevant but during asdf's install via python-build it shows that while it's recognizes the readline dep installed via homebrew, it uses zlib from my xcode sdk instead of homebrews.
Both of these homebrew packages are installed and relevant env vars are global in my shell (LDFLAGS/CPPFLAGS/PKG_CONFIG_PATH), is there any way to force python-build to use dependencies installed from Homebrew's (like zlib)?
I believe this issue may not be limited to prior versions of python, I'm also running into similar
python-builderrors while trying to install the latest v3.9.4 (also MacOS 11.2.3).
@ev-dev there are many reasons why a build/compilation can fail. Each failure will often need to be analyzed separately for root cause. Based on glance, I don't see a reason in your build errors to assume it is similar to previously discussed issues. Not sure what exact reason for failure is though.
The other failures shown in this issue have known upstream bugs in CPython that were fixed in later versions of python (or alternatively worked-around by linking other libraries / applying patches).
Your failure seems to be happening during CPython's configure script that should create Makefile: https://github.com/python/cpython/blob/v3.9.4/configure#L4067
There was at least one person who saw similar issue for python-build of 3.9.2: https://github.com/pyenv/pyenv/issues/1768#issuecomment-808604291
Someone else will need to comment if log file is still available. If log was stored inside install path, then it may be deleted during ASDF failure cleanup: https://github.com/asdf-vm/asdf/blob/b60c08acb2d7b39159eae7dfa2622289537a5304/lib/commands/command-install.bash#L5
EDIT: You might be able to find log inside directory in log lines you posted:
/var/folders/01/197gjn9100312t8k_61655tr0000gn/T/python-build.20210405194509.72821/Python-3.9.4
install via
python-buildit shows that while it's recognizes thereadlinedep installed via homebrew, it useszlibfrom my xcode sdk instead of homebrews.
In the case of zlib, python-build is coded to prioritize XCode zlib on macOS/OSX 10.14+:
https://github.com/pyenv/pyenv/blob/4f93366be8060e2b096e8136ffeedf9cbd050bd8/plugins/python-build/bin/python-build#L771
I don't think this would be an issue.
There are many factors that can impact a build.
In my case, I am able to build both 3.9.2 & 3.9.4 on M1 (arm64) macOS 11.2.3 using stable XCode 12.4 without exporting any flags and python-build automatically using Homebrew's readline & openssl.
It is hard to tell the exact state of your environment/setup.
You can try rerunning XCode CLI install command to make sure header files are all available with xcode-select --install
Otherwise, you might want to try seeing if issue occurs on pyenv and report issue there since asdf-python is just wrapping around pyenv's python-build so ASDF is often too high of a layer to analyze failures. Dropping down to python-build or even lower to direct Python source code would give more details.
Looks like Python bug: https://bugs.python.org/issue42504 So, I don't think this is an
asdf-pythonorpython-buildissue.It was fixed in both latest Python and at least backported to 3.9. Fix exists in CPython v3.9.2: https://github.com/python/cpython/blob/v3.9.2/Lib/sysconfig.py#L22-L24
Not too sure if they plan to backport to 3.8.
EDIT: @smedegaard if you need Python 3.8 and cannot use 3.9, you may want to try manual patch. Homebrew uses patch file: https://github.com/python/cpython/commit/09a698b4.patch?full_index=1 as seen https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]#L74
ASDF supports a single patch file:
https://github.com/danhper/asdf-python/blob/8ab052fced78b2c68c8fe17d5811d0ebae06eee8/bin/install#L19
So, maybe try something like
export ASDF_PYTHON_PATCHES_DIRECTORY=/tmp curl -sSL "https://github.com/python/cpython/commit/09a698b4.patch?full_index=1" > "$ASDF_PYTHON_PATCHES_DIRECTORY/3.8.8.patch"
Thanks, I needed 3.8.
I'll test again at a later point in time
Hey @cho-m really appreciate all the details, was looking forward to grasping the install process more and you provided lots of areas of research for me so thank you!
First, I just wanted to clarify the recommended pre-reqs for using asdf python plugin:
- Python build tools (ie. https://github.com/pyenv/pyenv/wiki#suggested-build-environment)
- Existing/functional
pyenvinstalled on system as a prerequisite? If yes:- Are the most common package managers per major OS the suggested resource to install
pyenvlike brew on Mac & apt on Ubuntu/Deb? - Is it best to have a bare-install of
pyenv(no existing configuration of the tool) when you wantasdfto manage your default (first in $PATH) python dist?
- Are the most common package managers per major OS the suggested resource to install
- Other pre-installed tooling? (within reasonable scope, sorry if this one is too big a rabbit hole)
I may try to submit a small update to the readme on this detail in pursuit of providing others with a bit more definitive info like is found in some other popular asdf plugin repos.
@ev-dev
- Python build tools (ie. https://github.com/pyenv/pyenv/wiki#suggested-build-environment)
You probably should be fine with those. The exact packages can vary based on what libraries you want to have available (which may require additional CFLAGS/CPPFLAGS/LDFLAGS).
The essential build dependencies on macOS are from Xcode Command Line Tools as required by official docs.
For libraries that are automatically detected by python-build (no need to add flags):
- Homebrew
opensslis good to have. If not installed,python-buildwill use a fixed version, so security issues may not be patched. - Homebrew
readlineis also good to have. Probably also defaults to fixed version. - For
zlib, as previously mentioned, on newer macOS building newer Python versions the existing zlib is usually fine. This is also what is used by Homebrew.- NOTE: For older Python versions on newer macOS, some people have had to use Homebrew
zlib/bzip2.
- NOTE: For older Python versions on newer macOS, some people have had to use Homebrew
For other packages listed in link:
- Homebrew
xzshould addlibzmalibrary (discussed in Python docs) and can also be used during decompressing source code (e.g. Python-3.9.4.tar.xz). - For
sqlite, I don't think it is used by default. May be useful if building Python with--enable-loadable-sqlite-extensionsas it seems to be missing in default macOS version.
- Existing/functional
pyenvinstalled on system as a prerequisite? If yes:
No.
asdf-python automatically pulls a copy of pyenv into "${ASDF_DATA_DIR:-$HOME/.asdf}/plugins/python/pyenv"
It really doesn't use pyenv, but the underlying python-build.
Sadly, they didn't split off repo unlike rbenv and underlying ruby-build.
- Other pre-installed tooling? (within reasonable scope, sorry if this one is too big a rabbit hole)
In general, not really much else once you have working ASDF install and XCode Tools.
Most of extra dependencies are C libraries that are used by some modules in CPython.
macOS BigSur 11.6 (20G165)
$ asdf version
v0.8.1-fe5b71d
I have successfully installed it with the following command.
CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" ASDF_PYTHON_PATCH_URL=https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1 asdf install python 3.7.3
macOS BigSur 11.6 (20G165)
$ asdf version v0.8.1-fe5b71dI have successfully installed it with the following command.
CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" ASDF_PYTHON_PATCH_URL=https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1 asdf install python 3.7.3
too hard
I had this warning, using 3.10.0 on Apple Silicon. I don't want to diverge too the conversation as I was on Monterey. But after the installing the latest version of Command Line Tools, it seems that uninstalling python through asdf and reinstalling resolved the issue for me.
