ImportError: cannot import name 'int4_weight_only' from 'torchao.quantization.quant_api'

[sadimanna]

🐛 Describe the bug

Sample code to reproduce the problem:

python torchchat.py chat llama3

Full traceback:

NumExpr defaulting to 6 threads.
PyTorch version 2.4.0+cu121 available.
Traceback (most recent call last):
  File "E:\gitrepos\torchchat\torchchat.py", line 67, in <module>
    from generate import main as generate_main
  File "E:\gitrepos\torchchat\generate.py", line 20, in <module>
    from build.builder import (
  File "E:\gitrepos\torchchat\build\builder.py", line 22, in <module>
    from quantization.quantize import quantize_model
  File "E:\gitrepos\torchchat\quantization\quantize.py", line 40, in <module>
    from torchao.quantization.quant_api import (
ImportError: cannot import name 'int4_weight_only' from 'torchao.quantization.quant_api'  (E:\gitrepos\torchchat\.venv\lib\site-packages\torchao\quantization\quant_api.py)

Versions

Environment:

Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.30.1
Libc version: N/A

Python version: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.1.74
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2080 Ti
Nvidia driver version: 555.85
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3696
DeviceID=CPU0
Family=205
L2CacheSize=1536
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=3696
Name=Intel(R) Core(TM) i5-9600K CPU @ 3.70GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.4.0+cu121
[pip3] torchao==0.1
[pip3] torchaudio==2.4.0+cu121
[pip3] torchvision==0.19.0+cu121
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h59b6b97_2
[conda] mkl                       2021.4.0                 pypi_0    pypi
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.2.0            py38h45dec08_0
[conda] mkl_random                1.1.1            py38h47e9c7a_0
[conda] numpy                     1.24.4                   pypi_0    pypi
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1
[conda] pytorch-lightning         2.2.4                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     2.3.0+cu121              pypi_0    pypi
[conda] torch-fidelity            0.3.0                    pypi_0    pypi
[conda] torchaudio                2.3.0+cu121              pypi_0    pypi
[conda] torchinfo                 1.8.0                    pypi_0    pypi
[conda] torchmetrics              0.11.4                   pypi_0    pypi
[conda] torchnet                  0.0.4                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.18.0+cu121             pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi

Tried installing torchao using python -m pip install torchao. But the error still persists.

[byjlw]

To verify. Are you running in a venv? Can you run pull and run ./install_requirements.sh again. `

[sadimanna]

@byjlw Yes, I am running in a venv as instructed in the readme file. I will try what you suggested.

[byjlw]

Great let me know how it goes. We are actively landing changes in core and AO (using pins on nightly) to improve the performance and capability of torchchat so I recommend you run ./install_requirements.sh everytime you pull to ensure you're bringing in the latest changes from those libraries.

[msaroufim]

@sadimanna you have a very old version of torchao installed, we published 0.4 so make sure to upgrade to that

pip install torchao --force-reinstall

[sadimanna]

@msaroufim using your command, it starts installing torch-2.4.0 and numpy-2.0.1 but is not upgrading torchao to 0.4

[msaroufim]

Oh just noticed you're using a Windows machine, that we didn't publish binaries for

For windows we recommend you just build our binaries from source https://github.com/pytorch/ao/blob/main/README.md

[Jack-Khuu]

Closing issue as resolved

@sadimanna Feel free to re/open issue if this is still blocking