rules_python icon indicating copy to clipboard operation
rules_python copied to clipboard
verified publisher icon bazelbuild

Can't install plyvel with standalone toolchain (0.10.2)

Open dhalperi opened this issue 3 years ago • 8 comments

🐞 bug report

Affected Rule

pip_parse

Is this a regression?

No, I'm not sure this ever worked in a previous version. This is the first version of standalone I've tried.

The library does work with a custom interpreter.

Description

plyvel requires leveldb installed on the system. With brew install leveldb, I get the file at:

/usr/local/Cellar/[email protected]/1.22/include/leveldb/db.h

When resolving plyvel, the below stack trace is thrown. It seems that plyvel can't find the C headers.

🔬 Minimal Reproduction

  1. Using macOS Monterey (12.5) on a 2017 Macbook Pro (x64)
  2. Install leveldb from Homebrew
  3. Use the following simplish repo:
  4. bazel build @pip_plyvel//:pkg - will throw the given error
  5. Comment out python_interpreter_target and rerun; will succeed.

Repo: WORKSPACE:

workspace(name = "demo")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

##########################################################
## Python setup

http_archive(
    name = "rules_python",
    sha256 = "a3a6e99f497be089f81ec082882e40246bfd435f52f4e82f37e89449b04573f6",
    strip_prefix = "rules_python-0.10.2",
    url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.10.2.tar.gz",
)

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python3_9",
    # Available versions are listed in @rules_python//python:versions.bzl.
    python_version = "3.9",
)

load("@python3_9//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "pip",
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements.txt",
)

load("@pip//:requirements.bzl", "install_deps")

install_deps()

requirements.txt:

#
# This file is autogenerated by pip-compile with python 3.7
# To update, run:
#
#    pip-compile --allow-unsafe --generate-hashes --output-file=requirements.txt requirements.in
#
plyvel==1.4.0 \
    --hash=sha256:0741be98096db98929ba507c5d99be6b9e009e38fb1361e755e52e69b2a6a496 \
    --hash=sha256:4ea98bea04ebf0f44747bacdfafefc8827787106fbb787f0aedc46482b2dfd53 \
    --hash=sha256:61d6dc2edff07412be0bcfb20cd3a6cb0b2ac6e136e0d53ce91a2d9b10e89ebc \
    --hash=sha256:b4a4e25b75852a379744f14e743f17ae78513d0ba4f354be54199735487c71d7 \
    --hash=sha256:d50eb65421e40d788868943bbaa0e1b62e806a07323a21ce9a4b7bf6bea022ec \
    --hash=sha256:daef3f2076db41fa0a0fcbb3eabf1a36445739e101367356b7ad3c455b77b44d
    # via -r requirements.in

🔥 Exception or Error


ERROR: An error occurred during the fetch of repository 'pip_plyvel':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/rules_python/python/pip_install/pip_repository.bzl", line 445, column 13, in _whl_library_impl
		fail("whl_library %s failed: %s (%s)" % (rctx.attr.name, result.stdout, result.stderr))
Error in fail: whl_library pip_plyvel failed: Collecting plyvel==1.4.0
  Using cached plyvel-1.4.0.tar.gz (151 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: plyvel
  Building wheel for plyvel (setup.py): started
  Building wheel for plyvel (setup.py): finished with status 'error'
  Running setup.py clean for plyvel
Failed to build plyvel
 (  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-x86_64-3.9
      creating build/lib.macosx-10.9-x86_64-3.9/plyvel
      copying plyvel/_version.py -> build/lib.macosx-10.9-x86_64-3.9/plyvel
      copying plyvel/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/plyvel
      running build_ext
      building 'plyvel._plyvel' extension
      creating build/temp.macosx-10.9-x86_64-3.9
      creating build/temp.macosx-10.9-x86_64-3.9/plyvel
      Compiling with an SDK that doesn't seem to exist: /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
      Please check your Xcode installation
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmphaz3a_ee/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmphaz3a_ee/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmphaz3a_ee/tools/deps/include/uuid -Werror=unguarded-availability-new -g0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/python3_9_x86_64-apple-darwin/include/python3.9 -c plyvel/_plyvel.cpp -o build/temp.macosx-10.9-x86_64-3.9/plyvel/_plyvel.o -Wall -g -x c++ -std=c++11 -stdlib=libc++
      plyvel/_plyvel.cpp:703:10: fatal error: 'leveldb/db.h' file not found
      #include "leveldb/db.h"
               ^~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for plyvel
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/python3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/python3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/rules_python/python/pip_install/extract_wheels/extract_single_wheel.py", line 81, in 
    main()
  File "/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/rules_python/python/pip_install/extract_wheels/extract_single_wheel.py", line 57, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/python3_9_x86_64-apple-darwin/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_dan/b3405f8b4e31d4c991d0d779169ce930/external/python3_9_x86_64-apple-darwin/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/var/folders/zy/32xth96x23vb7bddgpscl7g40000gn/T/tmpf70uqj3w']' returned non-zero exit status 1.
)

🌍 Your Environment

Operating System:

macOS Monterey, x86-64 processor

Output of bazel version:

Bazelisk version: development
Build label: 5.2.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 7 16:11:03 2022 (1654618263)
Build timestamp: 1654618263
Build timestamp as int: 1654618263

Rules_python version:

0.10.2

Anything else relevant?

Is there a way to supply C headers in-project?

dhalperi avatar Aug 02 '22 04:08 dhalperi

It seems to me that /usr/local/include is not on the compile path. CC: @hrfuller who may know the most about this?

dhalperi avatar Aug 02 '22 04:08 dhalperi

Running into a similar problem trying to install gevent using pip_parse on macOS 12.5.1

      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fPIC -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/include -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/include/ncursesw -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/include/uuid -Werror=unguarded-availability-new -g0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -DLIBEV_EMBED=1 -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -DEV_USE_REALTIME=1 -DEV_USE_MONOTONIC=1 -DEV_USE_FLOOR=1 -I/private/var/tmp/_bazel_maarten/8c4bfcd606c4d7b95cf418885251b08c/external/python3_9_aarch64-apple-darwin/include/python3.9 -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/deps -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/src/gevent/libev -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/deps/libev -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/src/gevent/libev -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/src/gevent -I/install/include/python3.9 -c build/temp.macosx-11.0-arm64-cpython-39/gevent.libev._corecffi.c -o build/temp.macosx-11.0-arm64-cpython-39/build/temp.macosx-11.0-arm64-cpython-39/gevent.libev._corecffi.o
      clang -bundle -undefined dynamic_lookup -arch arm64 -mmacosx-version-min=11.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -L/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/lib -g0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk build/temp.macosx-11.0-arm64-cpython-39/build/temp.macosx-11.0-arm64-cpython-39/gevent.libev._corecffi.o -L/install/lib -o build/lib.macosx-11.0-arm64-cpython-39/gevent/libev/_corecffi.abi3.so
      ld: warning: directory not found for option '-L/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/lib'
      ld: warning: directory not found for option '-L/install/lib'
      building 'gevent.libuv._corecffi' extension
      creating build/temp.macosx-11.0-arm64-cpython-39/deps/libuv
      creating build/temp.macosx-11.0-arm64-cpython-39/deps/libuv/src
      creating build/temp.macosx-11.0-arm64-cpython-39/deps/libuv/src/unix
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fPIC -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/include -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/include/ncursesw -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmps5wb7dre/tools/deps/include/uuid -Werror=unguarded-availability-new -g0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -DLIBUV_EMBED=1 -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -D_DARWIN_USE_64_BIT_INODE=1 -D_DARWIN_UNLIMITED_SELECT=1 -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/deps/libuv/include -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/deps/libuv/src -I/private/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/pip-wheel-qrxquzu4/gevent_3c0ddb0a3bc74ed18e829b5f418459e4/src/gevent -I/install/include/python3.9 -c build/temp.macosx-11.0-arm64-cpython-39/gevent.libuv._corecffi.c -o build/temp.macosx-11.0-arm64-cpython-39/build/temp.macosx-11.0-arm64-cpython-39/gevent.libuv._corecffi.o -Wno-unreachable-code -Wno-deprecated-declarations -Wno-incompatible-sysroot -Wno-tautological-compare -Wno-implicit-function-declaration -Wno-unused-value -Wno-macro-redefined
      build/temp.macosx-11.0-arm64-cpython-39/gevent.libuv._corecffi.c:50:14: fatal error: 'pyconfig.h' file not found
      #    include <pyconfig.h>
                   ^~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for gevent
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_maarten/8c4bfcd606c4d7b95cf418885251b08c/external/python3_9_aarch64-apple-darwin/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/var/tmp/_bazel_maarten/8c4bfcd606c4d7b95cf418885251b08c/external/python3_9_aarch64-apple-darwin/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_maarten/8c4bfcd606c4d7b95cf418885251b08c/external/rules_python/python/pip_install/extract_wheels/extract_single_wheel.py", line 83, in <module>
    main()
  File "/private/var/tmp/_bazel_maarten/8c4bfcd606c4d7b95cf418885251b08c/external/rules_python/python/pip_install/extract_wheels/extract_single_wheel.py", line 59, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_maarten/8c4bfcd606c4d7b95cf418885251b08c/external/python3_9_aarch64-apple-darwin/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_maarten/8c4bfcd606c4d7b95cf418885251b08c/external/python3_9_aarch64-apple-darwin/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/var/folders/86/9v04xnt92zgc3djqqdy4c7_00000gn/T/tmpt5hi3lcn']' returned non-zero exit status 1.
)

mvgijssel avatar Aug 29 '22 14:08 mvgijssel

Just came across https://github.com/bazelbuild/rules_python/pull/809 which might help for me, validating if this is the case and will post back here.

mvgijssel avatar Aug 29 '22 14:08 mvgijssel

Adding the most recent rules_python solved the issue for me 🎉

http_archive(
    name = "rules_python",
    sha256 = "b593d13bb43c94ce94b483c2858e53a9b811f6f10e1e0eedc61073bd90e58d9c",
    strip_prefix = "rules_python-0.12.0",
    url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.12.0.tar.gz",
)

mvgijssel avatar Aug 29 '22 15:08 mvgijssel

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

github-actions[bot] avatar Feb 25 '23 22:02 github-actions[bot]

activity

dhalperi avatar Feb 26 '23 03:02 dhalperi

@dhalperi can you test with the latest version of rules_python?

chrislovecnm avatar Aug 07 '23 18:08 chrislovecnm

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

github-actions[bot] avatar Feb 03 '24 22:02 github-actions[bot]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

github-actions[bot] avatar Mar 04 '24 22:03 github-actions[bot]