Error trying to use a py_library with requirements from another module

[calliecameron]

🐞 bug report

Affected Rule

py_library

Description

With a py_library defined in one module using requirement, depending on the library from another module causes a build error.

🔬 Minimal Reproduction

Note that module2 doesn't use rules_python itself; it's depending on a library from module1 that uses rules_python.

$ cat ./module1/.bazelversion
7.0.2

$ cat ./module1/BUILD
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
load("@pip//:requirements.bzl", "requirement")
load("@rules_python//python:defs.bzl", "py_library")

compile_pip_requirements(
    name = "requirements",
    requirements_in = "requirements.txt",
    requirements_txt = "requirements_lock.txt",
    tags = ["requires-network"],
)

py_library(
    name = "foo",
    srcs = ["foo.py"],
    visibility = ["//visibility:public"],
    deps = [requirement("pyyaml")],
)

$ cat ./module1/MODULE.bazel
module(
    name = "module1",
    version = "0.0.0",
)

bazel_dep(
    name = "rules_python",
    version = "0.31.0",
)

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.10",
)
use_repo(python, "python_versions")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pip",
    python_version = "3.10",
    requirements_lock = "//:requirements_lock.txt",
)
use_repo(pip, "pip")

$ cat ./module1/WORKSPACE

$ cat ./module1/foo.py
print("foo")

$ cat ./module1/requirements.txt
PyYAML == 6.0.1

$ cat ./module2/.bazelversion
7.0.2

$ cat ./module2/BUILD
sh_binary(
    name = "bar",
    srcs = ["bar.sh"],
    data = ["@module1//:foo"],
)

$ cat ./module2/MODULE.bazel
module(
    name = "module2",
    version = "0.0.0",
)

bazel_dep(
    name = "module1",
    version = "0.0.0",
)
local_path_override(
    module_name = "module1",
    path = "../module1",
)

$ cat ./module2/WORKSPACE

$ cat ./module2/bar.sh
#!/bin/bash

echo bar

🔥 Exception or Error

Preparation:

cd module1
touch requirements_lock.txt
bazel run :requirements.update
cd ../module2

The error:

$ bazel build :bar
Starting local Bazel server and connecting to it...
bazel: Entering directory `/home/callie/.cache/bazel/_bazel_callie/452a97bf58a333a814222c54dff465cc/execroot/_main/'
ERROR: /home/callie/.cache/bazel/_bazel_callie/452a97bf58a333a814222c54dff465cc/external/rules_python~0.31.0~pip~pip/pyyaml/BUILD.bazel:27:6: configurable attribute "actual" in @@rules_python~0.31.0~pip~pip//pyyaml:pkg doesn't match this configuration: No matching wheel for current configuration's Python version.

The current build configuration's Python version doesn't match any of the Python
versions available for this wheel. This wheel supports the following Python versions:
    3.10

As matched by the `@rules_python~0.31.0//python/config_settings:is_python_<version>`
configuration settings.

To determine the current configuration's Python version, run:
    `bazel config <config id>` (shown further below)
and look for
    rules_python~0.31.0//python/config_settings:python_version

If the value is missing, then the "default" Python version is being used,
which has a "null" version value and will not match version constraints.


This instance of @@rules_python~0.31.0~pip~pip//pyyaml:pkg has configuration identifier 312a038. To inspect its configuration, run: bazel config 312a038.

For more help, see https://bazel.build/docs/configurable-attributes#faq-select-choose-condition.

bazel: Leaving directory `/home/callie/.cache/bazel/_bazel_callie/452a97bf58a333a814222c54dff465cc/execroot/_main/'
ERROR: Analysis of target '//:bar' failed; build aborted: Analysis failed
INFO: Elapsed time: 5.179s, Critical Path: 0.04s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

🌍 Your Environment

Operating System:

Ubuntu 22.04

Output of bazel version:

7.0.2

Rules_python version:

0.31.0

Anything else relevant?

The example is a bit contrived (having an sh_binary with a data dep on a py_library) but is the simplest case that causes the error. In practice, I've got a py_library, used by a py_binary, used in an action in a rule, all defined in one module - and the rule being instantiated in another module. Since the py_binary is defined using the versioned rule from the python extension, I'd expect the library it calls to use that version even when used in another module.

[aignas]

This is working as designed, kind of...

Right now you define in your module1 python 3.10 dependencies but you do not set the default toolchain there. What is more setting the default would not do much because only the root module can configure the default toolchains. Anyway, what you have in module 3.10 is a target that will correctly resolve the dependencies when the toolchain is 3.10 and when your build configuration is defining 3.10 as the default.

Your module2 sh_binary does not set this python configuration value, so python library does not know what do, because the dependencies are only registered for python 3.10. If you were using the python version aware py_binary in module2 I suspect that this error would go away. Another way would be to specify the default python version via the .bazelrc.

What is your usecase far sharing the modules?

[calliecameron]

I'm writing a ruleset for markdown. Internally, parts of the ruleset are implemented in python. But that's an implementation detail - the public API of the module is a bzl file that doesn't mention python at all. Users of the module won't be writing python code, and shouldn't have to know that python is being used internally.

The actual WIP code is on the 'bazel-test' branch at https://github.com/calliecameron/markdown-makefile. Specifically:

• there's a py_library 'metadata' here
• which is used by the py_binary 'version' here
• which is an implicit dependency of the rule 'md_file' here
• which is used in an action in 'md_file' here
• 'md_file' is exposed in the module's public API through a macro here
• and also through a higher-level macro 'md_document' here

Users of the module depend on the module as here and call md_document as here.

When I add pydantic as a dependency to 'metadata' using 'requirement', the md_document in the user module fails to build.

---

Another way would be to specify the default python version via the .bazelrc

How do I do this? - what do I need to set?

This is fine as a workaround, but it would be nice if it wasn't needed, e.g. by having a versioned version of py_library, like we have for py_binary and py_test.

[aignas]

Is there a reason why the py_binary version is not versioned? That is the place where you could enforce the 3.10 python version and the users then don't need to do anything.

See the multi-version or bzlmod examples of how one can do this. I actually needed to do exactly the same in #1572 just today. :)

[calliecameron]

The py_binary is versioned, or at least it's supposed to be:

• usage here using the wrapper loaded here
• wrapper definition calls the real py_binary here, which is the versioned version loaded here

Or am I doing something wrong here?

[aignas]

I think what you are doing in py_binary is correct, but it seems that the configuration might be getting dropped. Maybe you should add cfg = exec where you use the version target as the default of the attribute.

On 25 March 2024 19:04:12 GMT+09:00, Callie Cameron @.***> wrote:

The py_binary is versioned, or at least it's supposed to be:

• usage here using the wrapper loaded here
• wrapper definition calls the real py_binary here, which is the versioned version loaded here

Or am I doing something wrong here?

-- Reply to this email directly or view it on GitHub: https://github.com/bazelbuild/rules_python/issues/1824#issuecomment-2017633488 You are receiving this because you commented.

Message ID: @.***>

[calliecameron]

I tried that (and also added executable = True) but still get the same error.

[aignas]

FWIW, I asked about this on bazel Slack and the transitions should work correctly in this case. i am not sure why they are not working.

[aignas]

Could you please see if this is still not working as expected with 0.32.1?

[calliecameron]

It works now.