Use pip repos with different names between different python versions

[michaeljs1990]

🐞 bug report

Affected Rule

This seems to be an "issue" when you using py_library and add a dependency to it that is in a pip repo not using your default python version. Hopefully I explained that correctly but I also opened a PR to make it easy to reproduce https://github.com/bazelbuild/rules_python/pull/2529.

Is this a regression?

I don't think so I have another version of rules python in use 0.32.2 I believe and it has the same issue.

Description

https://github.com/bazelbuild/rules_python/pull/2529 described better here but when using multiple repos and python versions and trying to use the library that contains a requirement from the non default and building it things blow up and look like this.

ERROR: /private/var/tmp/_bazel_schuettm/e053d0eaf0a843e9f76bdb61766af4b7/external/rules_python++pip+pypi_3_10_only/flask/BUILD.bazel:5:12: configurable attribute "actual" in @@rules_python++pip+pypi_3_10_only//flask:pkg doesn't match this configuration: No matching wheel for current configuration's Python version.

The current build configuration's Python version doesn't match any of the Python
wheels available for this distribution. This distribution supports the following Python
configuration settings:
    //_config:is_python_3.11

To determine the current configuration's Python version, run:
    `bazel config <config id>` (shown further below)

and look for one of:
    @@rules_python+//python/config_settings:python_version
    @@rules_python+//python/config_settings:pip_whl
    @@rules_python+//python/config_settings:pip_whl_glibc_version
    @@rules_python+//python/config_settings:pip_whl_muslc_version
    @@rules_python+//python/config_settings:pip_whl_osx_arch
    @@rules_python+//python/config_settings:pip_whl_osx_version
    @@rules_python+//python/config_settings:py_freethreaded
    @@rules_python+//python/config_settings:py_linux_libc

If the value is missing, then the default value is being used, see documentation:
https://rules-python.readthedocs.io/en/latest/api/rules_python/python/config_settings

This instance of @@rules_python++pip+pypi_3_10_only//flask:pkg has configuration identifier e62ee83. To inspect its configuration, run: bazel config e62ee83.

For more help, see https://bazel.build/docs/configurable-attributes#faq-select-choose-condition.

Use --verbose_failures to see the command lines of failed build steps.
ERROR: Analysis of target '//tests:my_lib_3_10_only_test' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.197s, Critical Path: 0.02s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully
//tests:test_versions                                                 NO STATUS

Executed 0 out of 1 test: 1 was skipped.

🔬 Minimal Reproduction

https://github.com/bazelbuild/rules_python/pull/2529 and just run bezel test //... inside examples/multi_python_versions.

🔥 Exception or Error

See above

🌍 Your Environment

Operating System:

This is on a Mac but I believe it is independent of OS.

Output of bazel version:

I have hit this on 7.x and 8..x

Rules_python version:

0.32.0 -> main

[aignas]

Just what I wrote in #2529:

Just to write here what I wrote in a related thread on Slack. I think what is happening here is:

• The default python toolchain version is 3.11 and if you build any target in the repo that is not using transitions, //python/version:python_version will just use that.
• This means that the flask here will be evaluated and will fail because there is no set of dependencies for 3.11 in the deps list.

There are ways to work around it:

• Set the target_compatible_with for this target, see the suggestion below.
• Set tags = ["manual"] and create a separate target that is transitioning the my_310_only_lib to something a build configuration where python_version = 3.10.
• Add a pip.parse entry for the default python toolchain version.
• Make 3.10 the default toolchain version by correctly setting is_default = True in one of the python.toolchain invocations.

The TCW snippet:

    target_compatible_with = select({
        "@rules_python//python/config_settings:is_python_3.10": [],
        "//conditions:default": ["@platforms//:incompatible"],
    })

Not sure exactly how rules_python can help here best. The error allows to ensure that the packages are getting the versions that they expect, but I see how this might be adding unnecessary work on the consumer side.

I can only think of the following two ideas:

• Create a transitionable py_library implementation. This has a drawback that for each py_library target we would create an extra target and that would increase the total number of targets by a factor of 2-ish.
• Add a target_compatible_with automatically to the pypi hub repo targets. This means that bazel build //... would succeed without doing anything if the --//python/config_settings:python_version is not set to the same version as the python_version from the hub repo.

None of them is ideal, but I am leaning towards the second one. I am also thinking that if the #1708 is fixed, then this could be a lesser issue.

[brandonchinn178]

Stupid question: why can't we just add py_library to @python_3_X like py_binary or py_test? If that's what you mean by transitionable, do py_binary/py_test have the same downside?

py_library is a bit different from py_binary/py_test in that it might need to be compatible with multiple python versions, but I don't see why the common case can't be treated the same. Then to support multiple versions, rules_python could provide a helper like

target_compatible_with = python_versions("3.10", "3.11", "3.12")

[rickeylev]

add py_library to python_3_x, is that what transitionable means

Yes, that's what transitionable means in this context. Doing so means py_library would have a python_version attribute, which sets the python version, which affects the config state transitively.

py_test and py_binary (or py_wheel) don't have this problem because they're terminal nodes in the graph. Them applying a transition still affects things transitively, but because everything is under them, it applies everywhere, so there isn't an explosion of config states.

[brandonchinn178]

Got it. So maybe transitionable won't work here, but I don't see why we can't have a version-specific py_library that just adds the target_compatible_with clause. FWIW when I first started using rules_python, I was confused why py binary/test was version specific but py library wasn't.

[aignas]

Just to write it down some thoughts:

If we want to set target_compatible_with (and sometimes it makes things more annoying to work with), then we can do it in 2 places:

• py_library setting. We need to parse the wheel filename here to do this.
• alias in the hub repo setting. We need to basically collect all of the keys from the select statement and use that to set the value. Another way (probably easier) is to use the python_version values that are used in the hub repo.