Psutil seems broken?

[paulgmiller]

The 'psutil~=5.7' distribution was not found and is required by azure-cli

So often after using az dev setip I get errors about psutil

 pmiller@millercloud:~/azure-cli$ azdev setup -c

=======================
| Azure CLI Dev Setup |
=======================

Azure CLI:
    /home/pmiller/azure-cli


 Installing packages 
=====================

Upgrading pip...
Installing `requirements.txt`...
Installing `azure-cli-telemetry`...
Installing `azure-cli-core`...
Installing `azure-cli`...
Installing `azure-cli-testsdk`...
az Installing `requirements.py3.Linux.txt`...
aks nod
Elapsed time: 0 min 14 sec

 Finished dev setup! 
=====================

pmiller@millercloud:~/azure-cli$ az aks nodepool update --help
Traceback (most recent call last):
  File "/home/pmiller/fudge/bin/az", line 4, in <module>
    __import__('pkg_resources').require('azure-cli==2.14.2')
  File "/home/pmiller/fudge/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3099, in <module>
    def _initialize_master_working_set():
  File "/home/pmiller/fudge/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3082, in _call_aside
    f(*args, **kwargs)
  File "/home/pmiller/fudge/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3111, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/pmiller/fudge/lib/python3.8/site-packages/pkg_resources/__init__.py", line 573, in _build_master
    ws.require(__requires__)
  File "/home/pmiller/fudge/lib/python3.8/site-packages/pkg_resources/__init__.py", line 891, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/pmiller/fudge/lib/python3.8/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'psutil~=5.7' distribution was not found and is required by azure-cli

When I try and pip install I get `` psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory 9 | #include <Python.h> | ^~~~~~~~~~

This is on an ubuntu 20 box with a pretty vanilla python setup and I'm in a venv

[paulgmiller]

Here's a really simple repro: pmiller@millercloud:~$ python3 -m venv letsdoit pmiller@millercloud:~$ source ./letsdoit/bin/activate (letsdoit) pmiller@millercloud:~$ pip install azdev

azdev setup -c azure-cli

======================= | Azure CLI Dev Setup |

Azure CLI: /home/pmiller/azure-cli

Installing packages

Upgrading pip... Installing requirements.txt... Installing azure-cli-telemetry... Installing azure-cli-core... Installing azure-cli... Installing azure-cli-testsdk... Installing requirements.py3.Linux.txt...

Elapsed time: 0 min 53 sec

Finished dev setup!

$ az --version Traceback (most recent call last): File "/home/pmiller/letsdoit/bin/az", line 4, in import('pkg_resources').require('azure-cli==2.14.2') File "/home/pmiller/letsdoit/lib/python3.8/site-packages/pkg_resources/init.py", line 3099, in def _initialize_master_working_set(): File "/home/pmiller/letsdoit/lib/python3.8/site-packages/pkg_resources/init.py", line 3082, in _call_aside f(*args, **kwargs) File "/home/pmiller/letsdoit/lib/python3.8/site-packages/pkg_resources/init.py", line 3111, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/home/pmiller/letsdoit/lib/python3.8/site-packages/pkg_resources/init.py", line 573, in _build_master ws.require(requires) File "/home/pmiller/letsdoit/lib/python3.8/site-packages/pkg_resources/init.py", line 891, in require needed = self.resolve(parse_requirements(requirements)) File "/home/pmiller/letsdoit/lib/python3.8/site-packages/pkg_resources/init.py", line 777, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'psutil~=5.7' distribution was not found and is required by azure-cli

[xhl873]

I am able to repro this too. Found another similar github issue: https://github.com/giampaolo/psutil/issues/1753, and I am able to pip install psutil after install python3-dev as suggested there, but still see some other err msgs: ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

azure-cli 2.14.2 requires azure-mgmt-managedservices~=1.0, which is not installed. azure-cli 2.14.2 requires xmltodict~=0.12, which is not installed. azure-cli 2.14.2 requires javaproperties==0.5.1, but you'll have javaproperties 0.7.0 which is incompatible. azure-cli 2.14.2 requires pytz==2019.1, but you'll have pytz 2020.4 which is incompatible. azure-cli 2.14.2 requires sshtunnel~=0.1.4, but you'll have sshtunnel 0.3.1 which is incompatible.

[xhl873]

update: after manually installed the pkgs above, I am finally able to do az --version

[paulgmiller]

Same after installing python-dev and manually installing 12+ packages I can now get this to work. This seems like a really bad experience for contributors though.

[paulgmiller]

If you apt-install python-dev before doing azdev setup this all works much better. We should ensure that at azdev setup time or at least document like with pr #270

[arrownj]

Hi @haroldrandom @fengzhou-msft could you please help look at this ?

[fengzhou-msft]

psutil only releases wheels for windows. On Linux, it depends on gcc and python-dev(or python3-devel on CentOS) to compile from source code. Let's update the doc first and then check how to support it in azdev.