Breaking changes to setuptools can break pip install

[smocherla-brex]

🐞 bug report

Recently there were some breaking changes to setuptools (see https://github.com/pypa/setuptools/pull/3505), and this broke pip_install for some existing packages for us. FWIW, we use the system interpreter (and not a hermetic one), but I don't believe that should matter when it comes to pinning the package that rules_python uses? It appears that setuptools is pinned here, but unclear to me how we suddenly ran into this issue with presumably a new verison of setuptools.

Affected Rule

pip_parse/pip_install.

Is this a regression?

No sure

Description

From my understanding (and reading about this here), this happens because setuptools isn't pinned. Since setuptools is an internal dep here for the pip install, it seems like rules_python should pin the version for it, since it's prone to breaking changes (I'm not entirely sure how that's configured here)?

A clear and concise description of the problem...

🔬 Minimal Reproduction

This issue is reproducible by adding this dependency (https://pypi.org/project/scs/2.1.4/) to your requirements.txt and then run a pip_parse and pip_install

I created a repo here https://github.com/smocherla-brex/pip-install-bug and if you run bazel fetch @my_deps//.. on a Linux machine, you run into the same issue.

🔥 Exception or Error

[2022-08-15T13:11:20Z] ERROR: An error occurred during the fetch of repository 'wheel_tool_deps_python_3_8_10_scs':
--
  | [2022-08-15T13:11:20Z]    Traceback (most recent call last):
  | [2022-08-15T13:11:20Z] 	File "/bazel-cache/external/rules_python/python/pip_install/pip_repository.bzl", line 371, column 13, in _whl_library_impl
  | [2022-08-15T13:11:20Z] 		fail("whl_library %s failed: %s (%s)" % (rctx.attr.name, result.stdout, result.stderr))
  | [2022-08-15T13:11:20Z] Error in fail: whl_library wheel_tool_deps_python_3_8_10_scs failed: Collecting scs==2.1.4
  | [2022-08-15T13:11:20Z]   Downloading scs-2.1.4.tar.gz (6.6 MB)
  | [2022-08-15T13:11:20Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 68.6 MB/s eta 0:00:00
  | [2022-08-15T13:11:20Z]   Installing build dependencies: started
  | [2022-08-15T13:11:20Z]   Installing build dependencies: finished with status 'done'
  | [2022-08-15T13:11:20Z]   Getting requirements to build wheel: started
  | [2022-08-15T13:11:20Z]   Getting requirements to build wheel: finished with status 'error'
  | [2022-08-15T13:11:20Z]  (  error: subprocess-exited-with-error
  | [2022-08-15T13:11:20Z]
  | [2022-08-15T13:11:20Z]   × Getting requirements to build wheel did not run successfully.
  | [2022-08-15T13:11:20Z]   │ exit code: 1
  | [2022-08-15T13:11:20Z]   ╰─> [17 lines of output]
  | [2022-08-15T13:11:20Z]       Traceback (most recent call last):
  | [2022-08-15T13:11:20Z]         File "/bazel-cache/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 363, in 
  | [2022-08-15T13:11:20Z]           main()
  | [2022-08-15T13:11:20Z]         File "/bazel-cache/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
  | [2022-08-15T13:11:20Z]           json_out['return_val'] = hook(**hook_input['kwargs'])
  | [2022-08-15T13:11:20Z]         File "/bazel-cache/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
  | [2022-08-15T13:11:20Z]           return hook(config_settings)
  | [2022-08-15T13:11:20Z]         File "/tmp/pip-build-env-80wkr9u3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 337, in get_requires_for_build_wheel
  | [2022-08-15T13:11:20Z]           return self._get_build_requires(config_settings, requirements=['wheel'])
  | [2022-08-15T13:11:20Z]         File "/tmp/pip-build-env-80wkr9u3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 319, in _get_build_requires
  | [2022-08-15T13:11:20Z]           self.run_setup()
  | [2022-08-15T13:11:20Z]         File "/tmp/pip-build-env-80wkr9u3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 474, in run_setup
  | [2022-08-15T13:11:20Z]           super(_BuildMetaLegacyBackend,
  | [2022-08-15T13:11:20Z]         File "/tmp/pip-build-env-80wkr9u3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 334, in run_setup
  | [2022-08-15T13:11:20Z]           exec(code, locals())
  | [2022-08-15T13:11:20Z]         File "", line 2, in 
  | [2022-08-15T13:11:20Z]       ModuleNotFoundError: No module named 'distutils.msvccompiler'
  | [2022-08-15T13:11:20Z]       [end of output]

🌍 Your Environment

This is on our CI machines Operating System:

  
root@8989a933eb44:/app# uname -a
Linux 8989a933eb44 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  

Output of bazel version:

  
    root@8989a933eb44:/app# bazel version
Bazelisk version: v1.10.1
Build label: 5.2.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 7 16:02:26 2022 (1654617746)
Build timestamp: 1654617746
Build timestamp as int: 1654617746
  

Rules_python version:

  
  0.10.2
  

Anything else relevant?

I also tried registering a hermetic python interpreter using python_register_toolchains, but this issue still happened.

[anuragkanungo]

I had the same issue yesterday with one of the package. For now the solution for me was to pin the setuptools version in the package.

In your case if you change https://github.com/bodono/scs-python/blob/master/pyproject.toml#L3 here from setuptools to setuptools==59.5.0 and then either point to that forked commit or new version in your requirements.txt, should work.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"