rules_python icon indicating copy to clipboard operation
rules_python copied to clipboard

feat(py_wheel): `deps` can provide requirement specifiers for `Requires-Dist`

Open pcj opened this issue 1 year ago • 6 comments

This feature adds two new providers (PyRequirementInfo and PyRequirementsInfo) to py_wheel.bzl.

Downstream users can use these Providers to supply additional requirement specifiers to the generated wheel metadata.

The use case is that we'd like wheels to contain files limited to transitive dependencies not including those which are available from pypi directly. If we use py_package out of the box, we get "fat wheels" that contain too many files (all of protobuf, for example). Rather, we'd like to be able to have our own "thinner" implementation of py_package that filters protobuf out, but also include Requires-Dist: protobuf>=4.25.2 in the wheel metadata.

This is similar but more limited in scope than #1182.

pcj avatar Feb 23 '24 02:02 pcj

Example py_project rule that demonstrates usage:

"Implementation of py_package rule"

load("@rules_python//python:packaging.bzl", "PyRequrirementInfo", "PyRequrirementsInfo")

PYPI_LOCK_PREFIX = "../pypi/_lock/"

def _py_package_impl(ctx):
    inputs = depset(
        transitive = [dep[DefaultInfo].data_runfiles.files for dep in ctx.attr.deps] +
                     [dep[DefaultInfo].default_runfiles.files for dep in ctx.attr.deps],
    )

    files = []
    requirements = []

    for input_file in inputs.to_list():
        filename = input_file.short_path
        if filename.startswith(PYPI_LOCK_PREFIX):
            filename = filename[len(PYPI_LOCK_PREFIX):]
            name, version = filename.split("@")
            if name and version:
                requirements.append(PyRequrirementInfo(
                    name = name,
                    version = version,
                    specifier = "%s>=%s" % (name, version),
                ))
        else:
            files.append(input_file)

    return [
        DefaultInfo(
            files = depset(direct = files),
        ),
        PyRequrirementsInfo(
            label = ctx.label,
            requirements = requirements,
        ),
    ]

py_package = rule(
    doc = """\
A rule to select all files in transitive dependencies of deps which
belong to given set of Python packages.

This rule is intended to be used as data dependency to py_wheel rule.
""",
    implementation = _py_package_impl,
    attrs = {
        "deps": attr.label_list(
            doc = "",
        ),
    },
)

The _lock/ prefix is specific to https://github.com/jvolkman/rules_pycross

pcj avatar Feb 23 '24 02:02 pcj

I like the idea but was wondering if the py_project rule should be part of the rules_python, at least as an example if we are not sure that we want to increase the API surface we are maintaining. What do you think, @pcj?

aignas avatar Feb 27 '24 09:02 aignas

I'm happy to include it, perhaps under a directory like examples/pycross_project/ since this particular implementation is specific to rules_pycross?

cc @jvolkman

pcj avatar Feb 27 '24 20:02 pcj

I'm happy to include it, perhaps under a directory like examples/pycross_project/ since this particular implementation is specific to rules_pycross?

cc @jvolkman

Not sure it makes sense to have this example in rules_python as written, with dependencies on _lock (a sort-of-implementation-detail, although it's described in a comment) and pypi (a user-specified repo name).

Could the example be retooled to rely only on stuff in rules_python?

jvolkman avatar Feb 28 '24 04:02 jvolkman

Added examples/custom_py_project/

pcj avatar Feb 28 '24 22:02 pcj