How to properly evaluate W6A6 models using checkpoint from the mode zoo

[ChengZhang-98]

Dear author, may I ask how to evaluate the W6A6 opt model using the provided ChenMnZ/OmniQuant, act_shifts, and act_scales ?

Here is my command

python main.py --model $PATH_TO_MY_OPT_CHECKPOINT \
    --epochs 0 \
    --output_dir $LOG_DIR \
    --wbits 6 --abits 6 --lwc --let \
    --tasks lambada_openai \
    --resume ./OmniQuant/opt-6.7b-w6a6.pth # this is the ckpt downloaded from the huggingface repo

I got the following error raise in https://github.com/OpenGVLab/OmniQuant/blob/ec4adedf42b17cf30a6d5faf0270e2036d2579f3/quantize/utils.py#L106

AttributeError: 'QuantOPTDecoderLayer' object has no attribute 'input_layernorm'. Did you mean: 'final_layer_norm'?

---

If I evaluate W6A6 llama-7b using the following command, the lambada accuracy is 0, though the perplexity matches the value reported in the paper.

❓ May I ask if I missed out something in the following command line?

python main.py --model $PATH_TO_MY_LLAMA_CHECKPOINT \
    --epochs 0 \
    --output_dir $LOG_DIR \
    --wbits 6 --abits 6 --lwc --let \
    --tasks lambada_openai \
    --resume ./OmniQuant/llama-7b-w6a6.pth # this is the ckpt downloaded from the huggingface repo

Here is the metrics I copied from the output

{'config': {'bootstrap_iters': 100000,
            'description_dict': None,
            'limit': None,
            'model': <models.LMClass.LMClass object at 0x7fc3624658a0>,
            'model_args': None,
            'num_fewshot': 0},
 'results': {'arc_challenge': {'acc': 0.38822525597269625,
                               'acc_norm': 0.4112627986348123,
                               'acc_norm_stderr': 0.01437944106852208,
                               'acc_stderr': 0.014241614207414037},
             'arc_easy': {'acc': 0.6637205387205387,
                          'acc_norm': 0.5197811447811448,
                          'acc_norm_stderr': 0.010251751199542736,
                          'acc_stderr': 0.009694178072725202},
             'boolq': {'acc': 0.728440366972477,
                       'acc_stderr': 0.00777897092960314},
             'lambada_openai': {'acc': 0.0,
                                'acc_stderr': 0.0,
                                'ppl': 2654605.7843538206,
                                'ppl_stderr': 129661.85146222409},
             'openbookqa': {'acc': 0.272,
                            'acc_norm': 0.418,
                            'acc_norm_stderr': 0.022080014812228134,
                            'acc_stderr': 0.01992048320956608},
             'piqa': {'acc': 0.7671381936887922,
                      'acc_norm': 0.764417845484222,
                      'acc_norm_stderr': 0.009901067586473888,
                      'acc_stderr': 0.009861236071080746}},
 'versions': {'arc_challenge': 0,
              'arc_easy': 0,
              'boolq': 1,
              'lambada_openai': 0,
              'openbookqa': 0,
              'piqa': 0}}

[ChenMnZ]

1. The latest repo don't support opt, if you want to evaluate OPT, you can try the previous version, such as https://github.com/OpenGVLab/OmniQuant/tree/d9bc4d7a518defdb33438a87a07f2981b6fb579d.
2. The repo has some bugs in evaluation lambda, I also meet this problem.

[ChenMnZ]

I have fixed the bug about OPT models in the latest code.