Building a wheel on macOS when using a `PyClass` from different crate does not work

[g-bauer]

I have a crate in which I define a PyClass. This PyClass I want to use in a different crate from which I want to build a wheel using maturin. This works perfectly fine on linux. On macOS and Windows, however, it does not.

I've built a minimum "failing" version of my example here inluding the error log. What I found is that I can get it to work on macOS as long as

1. the crate that defines the PyClass does not have a #[pymodule] function, and
2. that the PyClass does not have a method attached (#[new] works).

I asked this here on the gitter and @mejrs mentioned that it works on Windows if ABI features are disabled.

[messense]

Looks like https://github.com/PyO3/pyo3/issues/1444

[g-bauer]

I am not sure. I am aware of the issue that we are currently unable to reuse classes in Python as described in the issue you linked. But here I can't even build the wheel. And on linux it works just fine.

If it's related to the same underlying problem as #1444, please feel free to close this issue.

[davidhewitt]

I think this looks like a separate issue, as you say. You're not attempting to import multiple extension modules from Python, which is what #1444 covers. Instead I think you're probably hitting this: https://pyo3.rs/v0.15.0/building_and_distribution.html#macos

In the example you link, I think if you remove the extension-module feature from pyo3_export_pyclass/Cargo.toml then things may resolve? Also pyo3_export_pyclass probably does not need the cdylib lib crate type, either.

If you retest with that change, does it solve the macOS issues? (It may not solve the Windows problems, it would be helpful to see the other messages for other failing cases.)

[g-bauer]

Removing extension-module does not resolve the problem, but removing cdylib (and hence also extension-module) does. Now I need to figure out a way to be able to build both an extension module and make the PyClass usable from other crates.

[davidhewitt]

Right, this does now sound like it's bordering on #1444. Essentially, because Rust statically links all dependencies into the final extension-module cdylib, I wouldn't recommend re-using a PyClass from multiple crates. You'll find that each compiled target will create its own copy of the class type object and they won't compare equal.

[g-bauer]

That is not a problem in my case since I currently re-export the PyClass in the actual crate where I need it. To make this more clear:

• I have a crate that provides dimensioned quantities (SI units) which is usefull as separate python package / wheel (and in which I define quantities as PyClass).
• I have a library that uses dimensioned quantities in its interfaces so I want to re-use the classes defined in the quantity crate. I currently use and re-export the quantities (PyClass) in this libraray and build a python package from that as well. This works on linux but not on macos/Windows

[g-bauer]

I think I can work around this issue by having the PyClass and the extension module in different crates. I was just wondering if what I am seeing here was a bug because it works so nicely on linux. Feel free to close this issue and thanks for the help.

[davidhewitt]

👍 there's definitely complex thorny issues around multiple-crate setups like this which could use further exploration, however I think overall this does sit as #1444. Sorry we can't be more helpful at current.

[g-bauer]

@davidhewitt could this issue be re-opened? I think the issue I am observing here is not related to #1444.

Problem: On macos I am not able to use a crate that has both rlib and cdylib (e.g. to build a Python module) in it's Cargo.toml as dependency. I currently have no way to test this behavior on Windows. On Linux it works fine which confuses me.

Just to make it clear: I don't want to be able to use a PyClass across Python modules, but re-export a PyClass that's defined in a dependency - again - which works as expected on Linux.

I'd greatly appreciate any feedback on what could go wrong here and where to start looking for solutions.

[davidhewitt]

Ok, yes, I see what you mean. In particular, re-reading the solution listed in your example:

On macOS, removing cdylib and hence extension-module enables compilation but the root crate can no longer be made into a python package.

I'm curious, what happens if you keep cdylib and make extension-module feature in the root crate an optional feature? Then, when compiling the dependency you disable extension-module for the root crate.

i.e. something like this in the root crate Cargo.toml:

[dependencies.pyo3]
version = "0.14"
features = ["abi3", "abi3-py36", "multiple-pymethods"]
optional = true

[features]
default = []
python = ["pyo3", "numpy"]
python_module = ["python", "pyo3/extension-module"]

and then the dependency would use the python feature but not the python_module feature.

[g-bauer]

I tested making extension-module optional like you proposed above but got the same error.