Conflicting numpy providers

[0liu]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Linux 5.18.14-1-MANJARO
• Poetry version: 1.2.0b3
• Link of a Gist with the contents of your pyproject.toml file:

[tool.poetry]
name = "testproj"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.9"
intel-numpy = "~1.21"
intel-scipy = "~1.7"
tensorflow = "~2.9"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Issue

I want to use intel-numpy and intel-scipy for numpy/scipy, as they are linked to Intel MKL with performance boost. In the meanwhile, I need to use tensorflow, but when executing poetry install it installs both intel-numpy 1.21.4 and standard numpy 1.23.1, and the latter overwrites intel-numpy. Although the installation successfully finishes, importing numpy will throw an error ImportError: libimf.so: cannot open shared object file: No such file or directory.

Below is the installing ouput:

The currently activated Python version 3.10.5 is not supported by the project (~3.9).
Trying to find and use a compatible version. 
Using python3.9 (3.9.13)
Creating virtualenv testproj-2SD8BmWB-py3.9 in /home/<username>/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.7s)

Writing lock file

Package operations: 53 installs, 0 updates, 0 removals

  • Installing certifi (2022.6.15)
  • Installing charset-normalizer (2.1.0)
  • Installing idna (3.3)
  • Installing pyasn1 (0.4.8)
  • Installing urllib3 (1.26.11)
  • Installing cachetools (5.2.0)
  • Installing intel-openmp (2022.1.0)
  • Installing oauthlib (3.2.0)
  • Installing pyasn1-modules (0.2.8)
  • Installing requests (2.28.1)
  • Installing rsa (4.9)
  • Installing six (1.16.0)
  • Installing tbb (2021.6.0)
  • Installing zipp (3.8.1)
  • Installing google-auth (2.10.0)
  • Installing importlib-metadata (4.12.0)
  • Installing intel-cmplr-lic-rt (2022.1.0)
  • Installing markupsafe (2.1.1)
  • Installing mkl (2022.1.0)
  • Installing requests-oauthlib (1.3.1)
  • Installing absl-py (1.2.0): Installing...
  • Installing dpcpp-cpp-rt (2022.1.0): Installing...
  • Installing google-auth-oauthlib (0.4.6): Installing...
  • Installing absl-py (1.2.0)
  • Installing dpcpp-cpp-rt (2022.1.0)
  • Installing google-auth-oauthlib (0.4.6)
  • Installing grpcio (1.48.0)
  • Installing markdown (3.4.1)
  • Installing mkl-service (2.4.0)
  • Installing numpy (1.23.1)
  • Installing protobuf (3.19.4)
  • Installing pyparsing (3.0.9)
  • Installing tbb4py (2021.6.3)
  • Installing tensorboard-data-server (0.6.1)
  • Installing tensorboard-plugin-wit (1.8.1)
  • Installing werkzeug (2.2.1)
  • Installing astunparse (1.6.3): Installing...
  • Installing flatbuffers (1.12): Installing...
  • Installing gast (0.4.0): Installing...
  • Installing google-pasta (0.2.0): Installing...
  • Installing h5py (3.7.0): Pending...
  • Installing astunparse (1.6.3)
  • Installing flatbuffers (1.12)
  • Installing gast (0.4.0)
  • Installing google-pasta (0.2.0)
  • Installing h5py (3.7.0)
  • Installing intel-fortran-rt (2022.1.0)
  • Installing intel-numpy (1.21.4)
  • Installing keras (2.9.0)
  • Installing keras-preprocessing (1.1.2)
  • Installing libclang (14.0.6)
  • Installing opt-einsum (3.3.0)
  • Installing packaging (21.3)
  • Installing tensorboard (2.9.1)
  • Installing tensorflow-estimator (2.9.0)
  • Installing tensorflow-io-gcs-filesystem (0.26.0)
  • Installing termcolor (1.1.0)
  • Installing typing-extensions (4.3.0)
  • Installing wrapt (1.14.1)
  • Installing intel-scipy (1.7.3)
  • Installing tensorflow (2.9.1)

[dimbleby]

poetry is simply installing the things that you have asked it to install, this is not a poetry bug.

[0liu]

Not necessarily a bug, could be a design flaw or missing feature. Dependency on intel-numpy is explicitly stated in the file, but overwritten as tensorflow requires standard numpy. It's still more or less a dependency problem. It'd be great to incorporate package variant/flavor/provider in dependency resolving logic. Another way is just throwing "not resolvable" error, still better than allowing the package silently overwritten. Condaforge/Mambaforge deals with this by letting users pin libblas to either MKL or OpenBlas.

[dimbleby]

So far as anything that is available to poetry can see, there is no conflict in the project that you define, and so it successfully installs everything - there is no reason for poetry to think that this is "not resolvable".

I don't know exactly what non-standard games intel-numpy is playing but presumably it is deliberately colliding with the original numpy? This sounds like a problem of their own making.

If you think that tensorflow should be able to be installed alongside intel-numpy, take it up with tensorflow or intel-numpy. Pinning libblas is way out of scope for what poetry does.

[0liu]

It does not seem to me everything is successfully installed. There is an Import Error after installation, and numpy is not usable.

Numpy linked with MKL is widely used in numerical computation with Python in industry, not some random non-standard games.

Anyway, I'm reporting an issue happened to me, and proposing some potential ideas for Poetry to improve. It's up to dev team.

[radoering]

Afaik, there is no official way to formally specify that intel-numpy is an alternative to numpy. A more common approach for such a use case is to use local version identifiers like in https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy, the package is still called numpy, only the version is adapted, e.g. 1.22.4+mkl instead of 1.22.4.

[ralbertazzi]

Closing as:

1. Poetry does exactly what it's been asked
2. A workaround solution has been offered

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.