[BUG] TypeError: expected string or bytes-like object on pkg_resources/_vendor/packaging/version.py file

[ranrubin]

setuptools version

62.6.0

Python version

3.9.12

OS

Ubuntu 18

Additional environment information

No response

Description

While trying to install a package from source (pytorch-quantization), using python setup.py install, on a conda environment, I got the following error:

  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 410, in build_extensions
    self._check_cuda_version()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 783, in _check_cuda_version
    torch_cuda_version = packaging.version.parse(torch.version.cuda)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 49, in parse
    return Version(version)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 264, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

Expected behavior

Not crashing with an error :)

How to Reproduce

Follow this README file to build from source

Output

python setup.py install
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
  warnings.warn(
running install
/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing pytorch_quantization.egg-info/PKG-INFO
writing dependency_links to pytorch_quantization.egg-info/dependency_links.txt
writing requirements to pytorch_quantization.egg-info/requires.txt
writing top-level names to pytorch_quantization.egg-info/top_level.txt
/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/torch/utils/cpp_extension.py:387: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'pytorch_quantization.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'pytorch_quantization.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/pytorch_quantization
copying pytorch_quantization/version.py -> build/lib.linux-x86_64-3.9/pytorch_quantization
copying pytorch_quantization/__init__.py -> build/lib.linux-x86_64-3.9/pytorch_quantization
copying pytorch_quantization/tensor_quant.py -> build/lib.linux-x86_64-3.9/pytorch_quantization
copying pytorch_quantization/quant_modules.py -> build/lib.linux-x86_64-3.9/pytorch_quantization
creating build/lib.linux-x86_64-3.9/pytorch_quantization/calib
copying pytorch_quantization/calib/__init__.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/calib
copying pytorch_quantization/calib/histogram.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/calib
copying pytorch_quantization/calib/max.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/calib
copying pytorch_quantization/calib/calibrator.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/calib
creating build/lib.linux-x86_64-3.9/pytorch_quantization/nn
copying pytorch_quantization/nn/__init__.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn
copying pytorch_quantization/nn/functional.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn
creating build/lib.linux-x86_64-3.9/pytorch_quantization/optim
copying pytorch_quantization/optim/__init__.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/optim
copying pytorch_quantization/optim/helper.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/optim
creating build/lib.linux-x86_64-3.9/pytorch_quantization/utils
copying pytorch_quantization/utils/__init__.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/utils
copying pytorch_quantization/utils/quant_logging.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/utils
copying pytorch_quantization/utils/reduce_amax.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/utils
creating build/lib.linux-x86_64-3.9/pytorch_quantization/nn/_functions
copying pytorch_quantization/nn/_functions/__init__.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/_functions
copying pytorch_quantization/nn/_functions/quant_rnn.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/_functions
creating build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/__init__.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/tensor_quantizer.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/quant_rnn.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/quant_pooling.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/quant_instancenorm.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/quant_conv.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/_utils.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/clip.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
copying pytorch_quantization/nn/modules/quant_linear.py -> build/lib.linux-x86_64-3.9/pytorch_quantization/nn/modules
running build_ext
Traceback (most recent call last):
  File "/home/anaconda/TensorRT/tools/pytorch-quantization/setup.py", line 55, in <module>
    setup(
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/bdist_egg.py", line 165, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/bdist_egg.py", line 151, in call_command
    self.run_command(cmdname)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 410, in build_extensions
    self._check_cuda_version()
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 783, in _check_cuda_version
    torch_cuda_version = packaging.version.parse(torch.version.cuda)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 49, in parse
    return Version(version)
  File "/apps/conda/anaconda/envs/trt84-base/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 264, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

[abravalheri]

Hi @ranrubin, thank you very much for pointing this out.

Just a quick question, is there any chance the version variable in your setup.py maybe be having a None value?

(I had a very quick look at your repo, maybe the VERSION file is missing in the MANIFEST.in?)

[ranrubin]

Hi @abravalheri :) I'm not a contributer to that repository, so I'm not that familiar with the code. It seesm like setup.py uses VERSION file in order to get the version.

When I tried to install the package and got the mentioned error, manually editing site-packages/pkg_resources/_vendor/packaging/version.py with str() solved it, and the package itself got the proper version value (pytorch-quantization==2.1.2)

[abravalheri]

Ah, I see. sorry for the confusion!

I'm not a contributer to that repository, so I'm not that familiar with the code. It seems like setup.py uses VERSION file in order to get the version.

Yeah that is the part that I found very curious. In theory the default behaviour of open should be to open files in the text more/read-only (r), so version should already be a string... I have to investigate this issue more deeply.

[abravalheri]

This problem is crazy difficult to track down. I tried to follow the instructions for building the package, but it is very difficult and I suspect it depends on the hardware you have available in your machine. The C code simply does not compile in my machine (probably because it requires you to have the correct hardware and drivers).

If I comment out the ext_modules=[...] part o setup.py things build correctly and the version seems to be recognized fine...

@ranrubin can you try this again with the latest version of setuptools + making sure to use a conda environment instead of relying on the global Python provided by conda?

[pgkaila]

here is the anaconda3\envs\textgen\lib\site-packages\torch\version.py file in my system causing the issue,

debug = False
cuda = None
git_version = '49444c3e546bf240bed24a101e747422d1f8a0ee'
hip = None

[Avasam]

setuptools no longer uses pkg_resources, are you still encountering this issue ? (issue might need to be updated)