pip install . --user -> [Errno 2] No such file or directory: 'meson'

[Nic30]

Hello,

I am trying to make package build by meson-python installable by pip using pip install --user . (a complete build script, CI config)

Recent changes in pip broke this and mesonpep517 has the same issue https://github.com/jeandet/mesonpep517/issues/1#issuecomment-1148288250

mesonpep517 has opened pull request which is fixing this, however merging takes some time and I simply need this so I tried meson-python.

The result is the same:

...
        File "/tmp/pip-build-env-j8gehd_x/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 477, in _meson
          return self._proc('meson', *args)
        File "/tmp/pip-build-env-j8gehd_x/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 472, in _proc
          subprocess.check_call(list(args))
        File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
          retcode = call(*popenargs, **kwargs)
        File "/usr/lib/python3.10/subprocess.py", line 345, in call
          with Popen(*popenargs, **kwargs) as p:
        File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'meson'

There is a full log from build

The problem is that pip does modify the PATH and meson executable will be missing. Note that this happens only if meson is not installed in the advance. You can fix it by executing meson as a module function and not as an executable, as you are currently doing https://github.com/FFY00/meson-python/blob/main/mesonpy/init.py#L477 . (see changes in mesonpep517 pull request). (Note that the user typically does not have meson installed in advance.)

Ubuntu 22.04 LTS

• python3.10
• pip-22.0.2
• meson-0.63.0
• ninja-1.10.1
• meson_python-0.7.0

[rgommers]

(see changes in mesonpep517 pull request).

Did you link the wrong pull request? All that does is add ninja to the dependencies.

using pip install --user .

It'd be good to have this work. It should not be necessary though, more recent pip versions automatically add --user for system Python versions when run as user (and running as root is a bad idea).

You can fix it by executing meson as a module function and not as an executable

This seems fine to me as a fix for meson-python, since it has a hard dependency on meson so it should always be installed in the Python environment in use (this is not necessarily true when using meson directly, but that's not relevant here).

[eli-schwartz]

It's not necessarily true if you use for example python -m build --no-isolation --skip-dependency-check because you have seeded the environment yourself, but have Meson as a system tool in another version of python.

This could be relevant for e.g. Gentoo, where multiple versions of python are supported, but on the other hand they do have meson-python depend on >=dev-util/meson-0.60.0[${PYTHON_USEDEP}] so that may not matter.

Perhaps @thesamesam could clarify whether this matters.

[FFY00]

I am not sure what to say, the meson script should be there, as meson is a dependency. pip install --user . not working on certain scenarios is a major issue. I will have to debug things to find the issue, I have no good guesses right now.

[FFY00]

This seems fine to me as a fix for meson-python, since it has a hard dependency on meson so it should always be installed in the Python environment in use (this is not necessarily true when using meson directly, but that's not relevant here).

Not necessarily, we will use whatever Meson to build, but we do tell Meson which Python interpreter to use. Ideally, Meson would always be provided by the environment we are running on, and that is true in isolated build, but there might be situations when running without isolation where that might not be the case.

[rgommers]

I cannot reproduce this with SciPy, pip install --user . works just fine.

mesonpep517 has opened pull request which is fixing this, however merging takes some time and I simply need this so I tried meson-python.

That pull request does not seem related. I also searched the mesonpep517 repo, and did not find a single mention of --user.

I had a look in detail at the build log. It reads:

# Install pre-build deps
pip install -r requirements.txt;

# Install actual version from git

Installing build dependencies ... - \ | / - done
Getting requirements to build wheel ... - error
....
FileNotFoundError: [Errno 2] No such file or directory: 'meson'

This looks like user error. The requirements.txt dependencies are installed into the active environment, and then the build command triggers an isolated build with zero build dependencies installed. The retrieved source seems to be missing a pyproject.toml file.

@Nic30 am I missing something, or can be close this as invalid?

[FFY00]

I think it would be beneficial for you to describe your dev environment setup.

Other thought, have you updated your pip? This can happen when building without isolation if the dependency is missing, though I assumed pip would be able to notice the missing dependency.

[Nic30]

Hello,

Did you link the wrong pull request? All that does is add ninja to the dependencies.

• It seems like I did, but I can not find relevant one now.

I am using:

https://github.com/Nic30/hwtHls/blob/llvm_mir_integration/pyproject.toml https://github.com/Nic30/hwtHls/blob/llvm_mir_integration/Dockerfile

git clone -b llvm_mir_integration https://github.com/Nic30/hwtHls.git
cd hwtHls
docker build -t nic30/hwthls .

There is the log from build: docker-build.log

Step 21/22 : RUN pip3 install .
 ---> Running in c0e161803852
Defaulting to user installation because normal site-packages is not writeable
Processing /home/jovyan
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      + meson setup --native-file=/home/jovyan/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/usr /home/jovyan /home/jovyan/.mesonpy-ntmsqwor/build
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
....
        File "/tmp/pip-build-env-ah44erln/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 494, in _configure
        File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'meson'
      [end of output]

(I also used docker rmi $(docker images -q -f dangling=true) to remove all docker images to avoid any interference)

• python3-pip (22.0.2+dfsg-1)
• Installation takes ~3 min, if you need I can make some testing repo for this issue, but any project with meson-python should do.
• doc/requirements.txt does contain my other python packages, list can be seen in log, meson and meson-python is not there

[rgommers]

Defaulting to user installation because normal site-packages is not writeable

Hmm, maybe there's a difference between explicitly using --user and pip doing it implicitly for you. If on line 48 of your Dockerfile you change to RUN pip3 install --user ., does that fix it?

[Nic30]

Dockerfile you change to RUN pip3 install --user ., does that fix it?

No it is the same docker-build.log

@rgommers (or anyone else with working meson-python) What system, python, pip do you use?

[rgommers]

@rgommers (or anyone else with working meson-python) What system, python, pip do you use?

I'm using both Arch Linux and macOS, conda-forge environments, Python 3.8-3.10, and usually latest pip version.

[smcv]

I recently switched dbus-python to use Meson and meson-python for build, and I'm now hitting this same problem on a Debian test machine.

If I run pip with more debugging enabled, it produces some information that could be useful:

user@host:~$ pip install -vvv dbus-python
Using pip 22.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
...
  Installing collected packages: patchelf, ninja, wheel, tomli, setuptools, pyparsing, meson, packaging, pyproject-metadata, meson-python
    changing mode of /tmp/pip-build-env-jfbiloif/overlay/local/bin/ninja to 755
    changing mode of /tmp/pip-build-env-jfbiloif/overlay/local/bin/wheel to 755
    changing mode of /tmp/pip-build-env-jfbiloif/overlay/local/bin/meson to 755
  Successfully installed meson-0.63.2 meson-python-0.8.1 ninja-1.10.2.3 packaging-21.3 patchelf-0.15.0.0 pyparsing-3.0.9 pyproject-metadata-0.6.1 setuptools-65.3.0 tomli-2.0.1 wheel-0.37.1
...
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  + meson setup --native-file=/tmp/pip-install-yodcermv/dbus-python_0cf4004be758468b825b7d76aba873f9/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/usr /tmp/pip-install-yodcermv/dbus-python_0cf4004be758468b825b7d76aba873f9 /tmp/pip-install-yodcermv/dbus-python_0cf4004be758468b825b7d76aba873f9/.mesonpy-nv5qulns/build
  Traceback (most recent call last):
...
  FileNotFoundError: [Errno 2] No such file or directory: 'meson'

I hacked in some debugging, and it looks as though pip is installing meson, ninja, wheel into /tmp/pip-build-env-xxx/overlay/local/bin/, but then setting PATH to include /tmp/pip-build-env-xxx/overlay/bin/, which makes me suspect a problem with Debian's patches to the packaged interpreter and libraries that are intended to get things installed into /usr/local/bin.

The workaround is to install meson (and patchelf and ninja) system-wide. dbus-python has other build-dependencies that need installing system-wide anyway: GLib, libdbus, pkg-config and a C compiler.

[smcv]

I see @Nic30 is using Ubuntu. Ubuntu is derived from Debian, so if this is a Debian-specific problem, that would explain why @Nic30 and I are seeing it but an Arch user is not.

[FFY00]

Oh, that makes sense, thank you for posting that output! This is another issue cause by Debian's patching of Python.

As a workaround, we can try to add that missing directory to PATH when invoking Meson.

[smcv]

I've reported this as https://bugs.debian.org/1019293. I think it's a bug in one of two Debian packages: either python3.x because the patches to Python that change the installation scheme are doing too much, or python3-pip because it should compensate for those patches when it sets up the PATH.

[dnicolodi]

I think this is a variation of https://github.com/pypa/pip/issues/11539 fixed in https://github.com/pypa/pip/pull/11598 Namely pip does not setup correctly the isolated environment for the build. It would be nice if you could test with a pip with that patch applied.

[Nic30]

I tried

RUN pip3 install git+https://github.com/pypa/pip.git@main
ENV PATH /home/${NB_USER}/.local/bin/:$PATH

And it seems that the issue is still there. err.log

[dnicolodi]

I see. The issue is with the binary path, not with the library path. The linked patch solved the latter, not the former. The issue is caused by this code in pip https://github.com/pypa/pip/blob/5f3f592c4581a059ff9de0fb8052ef5c6ef25fd4/src/pip/_internal/build_env.py#L40-L43 and by Debian patching sysconfig to use a custom installation scheme for the system but not for virtual environments. Can you reproduce this only with Python 3.10 or also with older versions?

@smcv on which Debian release did you encounter the problem? Never mind, the info is in the linked debbug.

[dnicolodi]

In particular, pip installs the dependencies in the isolated build environment with the default sysconfig scheme:

>>> sysconfig.get_default_scheme()
'posix_local'
>>> sysconfig.get_paths('posix_local', vars={'base': '{prefix}'})['scripts']
'{prefix}/local/bin'

but, per the code snippet linked above, it sets $PATH to the location given by the posix_prefix scheme:

>>> sysconfig.get_paths('posix_prefix', vars={'base': '{prefix}'})['scripts']
'{prefix}/bin'

The same scheme should be used for installation and binary path lookup. I don't know which one of the two used here is the right one. Although, setting the scheme as is done in the linked snippet seems suspicious, as nt is anyhow the default scheme on Windows, and not specifying it would most likely default to the same one used during dependencies installation.

[dnicolodi]

@smcv @Nic30 can you please check if https://github.com/pypa/pip/pull/11623 fixes your issue?

Edit: I reproduced the issue in a Debian sid container and I verified that the linked patch fixes the issue.

[dnicolodi]

meson-python tests did not catch the issue because we don't run them in an environment without meson installed in the system $PATH. We could setup that, but catching bugs in the Python packaging ecosystem as whole is not the goal of meson-python (and despite that we found and fixed a few already).

[dnicolodi]

It works for meson but it breaks path for other apps installed on the host, in my case for cmake which I am using to build a subproject. So it works but this new path should be just appended (not overwritten).

This aspect is not changed. The path is prepended to the existing $PATH environment variable. Also note that in the log Meson reports that it found gcc, thus $PATH is most likely sane. CMake may be missing for other reasons.

[Nic30]

@dnicolodi

pip3 install git+https://github.com/dnicolodi/pip.git@debian-scheme Yes it works, but there is an error during the build the error message ends with

      INFO: calculating backend command to run: /tmp/pip-build-env-zy8aywfy/overlay/local/bin/ninja
      + meson setup --prefix=/usr /home/jovyan /home/jovyan/.mesonpy-4rahlr_b/build --native-file=/home/jovyan/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
      + meson compile
      Traceback (most recent call last):
        File "/home/jovyan/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
      AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'

which is misleading. But this is other issue.

[dnicolodi]

The exception splat is already fixed in #231. With only this error message snippet is not possible to determine what is the root cause of the problem.

[Nic30]

@dnicolodi

With only this error message

There was just some missing include in cmake subproject. err.log

What I wanted to say is that your fix for pip works.

[dnicolodi]

pip 23.0.1 has just been released and it should solve all problems related to isolated build environments setup.