Cannot build even most simple packages

[swaldhoer]

Problem

I cannot build the most simple packages using flit, e.g., flit_core/flit_core/tests/samples/with_data_dir. The reason ist, that pip install . is failing.

Details

Versions

PS C:\Users\stefa\Documents> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      19043  0
PS C:\Users\stefa\Documents> python --version; python -m flit --version; python -m pip --version
Python 3.8.2
Flit 3.7.1
pip 22.1.2 from C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pip (python 3.8)

Test case

PS C:\Users\stefa\Documents> git clone https://github.com/pypa/flit.git
PS C:\Users\stefa\Documents> cd .\flit\
PS C:\Users\stefa\Documents\flit> git rev-parse HEAD
048c87c380ac41efc4b26222114e54f6581c64f6

Test

PS C:\Users\stefa\Documents\flit> cd flit_core\flit_core\tests\samples\with_data_dir
PS C:\Users\stefa\Documents\flit\flit_core\flit_core\tests\samples\with_data_dir> flit install
Extras to install for deps 'all': {'.none'}             I-flit.install
Das System kann den angegebenen Pfad nicht finden.
Processing c:\users\stefa\documents\flit\flit_core\flit_core\tests\samples\with_data_dir
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: docutils in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (0.16)
Requirement already satisfied: requests>=2.18 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (2.25.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (1.26.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2020.12.5)
Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2.10)
Building wheels for collected packages: module1
  Building wheel for module1 (pyproject.toml) ... done
  Created wheel for module1: filename=module1-0.1-py3-none-any.whl size=1740 sha256=677808e82c4ec2b88721c46911a499381b6782328fdff551231d4aef488d75cd
  Stored in directory: C:\Users\stefa\AppData\Local\Temp\pip-ephem-wheel-cache-co9e6l2k\wheels\ee\72\a1\0576b8767488b740bf50fcca82f18326360c973eac1cd1fddf
Successfully built module1
Installing collected packages: module1
  Attempting uninstall: module1
    Found existing installation: module1 0.1
    Uninstalling module1-0.1:
      Successfully uninstalled module1-0.1
Successfully installed module1-0.1
Traceback (most recent call last):
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\Scripts\flit.exe\__main__.py", line 7, in <module>
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flit\__init__.py", line 192, in main
    Installer.from_ini_path(args.ini_file, user=args.user, python=python,
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flit\install.py", line 432, in install
    self.install_with_pip()
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flit\install.py", line 370, in install_with_pip
    check_call(cmd, shell=shell)
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\stefa\\AppData\\Local\\Programs\\Python\\Python38-32\\python.exe', '-m', 'pip', 'install', '.']' returned non-zero exit status 1.

[swaldhoer]

I should mention that, python -m pip install . works for the test case:

PS C:\Users\stefa\Documents\flit\flit_core\flit_core\tests\samples\with_data_dir> python -m pip install .
Processing c:\users\stefa\documents\flit\flit_core\flit_core\tests\samples\with_data_dir
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests>=2.18 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (2.25.1)
Requirement already satisfied: docutils in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (0.16)
Requirement already satisfied: idna<3,>=2.5 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (4.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (1.26.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2020.12.5)
Building wheels for collected packages: module1
  Building wheel for module1 (pyproject.toml) ... done
  Created wheel for module1: filename=module1-0.1-py3-none-any.whl size=1740 sha256=677808e82c4ec2b88721c46911a499381b6782328fdff551231d4aef488d75cd
  Stored in directory: C:\Users\stefa\AppData\Local\Temp\pip-ephem-wheel-cache-apq1sz90\wheels\ee\72\a1\0576b8767488b740bf50fcca82f18326360c973eac1cd1fddf
Successfully built module1
Installing collected packages: module1
  Attempting uninstall: module1
    Found existing installation: module1 0.1
    Uninstalling module1-0.1:
      Successfully uninstalled module1-0.1
Successfully installed module1-0.1
PS C:\Users\stefa\Documents\flit\flit_core\flit_core\tests\samples\with_data_dir> Get-Command foo

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     foo.exe                                            0.0.0.0    C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\Scripts\foo.exe

[takluyver]

The output looks like pip installs it correctly, but then returns a non-zero exit code, which means something has gone wrong. The traceback is just saying that pip indicated an error.

I notice that there's a random German error message ("Das System kann den angegebenen Pfad nicht finden.") mixed in with the English output. I don't know where that's coming from, but maybe it's related somehow.

[swaldhoer]

I have some idea. How does flit invoke the python -m pip ...? Does it use shell=True?

[bpabel]

I'm not sure if this is the cause, but your pyproject.toml lists a console script entrypoint for module1:main, but that function does not exist in your module.

[takluyver]

I have some idea. How does flit invoke the python -m pip ...? Does it use shell=True?

On Windows it does, yes:

https://github.com/pypa/flit/blob/3edbd2f3e3b97a057fc785519c1617e2d78114fa/flit/install.py#L369-L370

Edit: this was changed in 414571d7fd5ccc7ac852a8449edcaaf59f407902, with a comment saying "On Windows, shell needs to be True to pick up our local PATH when finding the Python command."

[swaldhoer]

So first of all, thanks @takluyver you pointed me in the right direction.

• I checked on my usual development machine, and could build using flit all as expected.
• Then I switched back the machine, I got this error from.
• On that machine, I had a faulty setup in HKEY_CURRENT_USER\Software\Microsoft\Command Processor\AutoRun poiting to a batch file that does not exist; therefore I got the error message The system cannot find the path specified. and leading to a non-zero exit code.
• Fixed the AutoRun setup and now everthing works also on that machine.

So this particular issue was all about a wrong setup on my second machine.

However, I do not really understand why flit needs to use shell=True for subprocesses on Windows? I had a look at the commit and Pull request, but I don't really get it. Would it not be sufficent to use just the env-keyword of the subprocess package and by thgat use the correct PATH?

[nanonyme]

It most definitely is cleaner to use shutil.which to resolve executable path than to use shell=True. It is available on Python 3.3 and newer.

[takluyver]

I prefer not to use shell=True if possible, and Python 3.5 was already out when I made that commit, so I assume there was a reason I didn't use shutil.which(). But sadly my notes at the time weren't sufficient to figure out what that was. I vaguely remember something odd about the way that virtualenvs behaved on Windows, but I don't remember either the details or whether it was related to this specific change.

We have Windows CI, so if you want to make a PR to use shutil.which(), then if it passes CI, I'm happy to go with it and see if any complaints turn up.

[swaldhoer]

It most definitely is cleaner to use shutil.which to resolve executable path than to use shell=True. It is available on Python 3.3 and newer.

I don't understand why shutil.which() would fix the problem? The problem seemed to be, that the enviroment variables are not updated as the process would inheriate from the parent process and therefore the problem would still exist?

Using shutil.which() as well as an updated version of PATH passed to the subprocess should be the correct solution.

[connorferster]

Just wanted to add that I had a similar problem. I went into the Registry Editor and updated the key suggested by @swaldhoer. In my case it just some partial string in there like "if ever" or something (don't recall exactly). I deleted it leaving an empty field (the default setting) and it fixed the problem.

[nanonyme]

It most definitely is cleaner to use shutil.which to resolve executable path than to use shell=True. It is available on Python 3.3 and newer.

I don't understand why shutil.which() would fix the problem? The problem seemed to be, that the enviroment variables are not updated as the process would inheriate from the parent process and therefore the problem would still exist?

Using shutil.which() as well as an updated version of PATH passed to the subprocess should be the correct solution.

The core part of shutil.which() approach was getting rid of the cmd.exe in between that may mess things up.