`TypeError: forward() got an unexpected keyword argument 'cache_position'`

[benja-matic]

Hi,

I'm seeing the error above when running python -u run_text_generation.py --model_arch llama --model_name huggyllama/llama-13b --recent_ratio 0.1 --heavy_ratio 0.1 \. Please let me know if you have a fix, as I would really like to use this technique.

Notes on reproducibility:

I did make some small modifications to the code (set num_hidden_layers=4) in the model config for faster debugging. I am using transformers 4.44.0.dev0 (this is what gets installed when using the provided install instructions). I'm using a conda environment with python 3.8.

[wln20]

I also encounted this issue and found the problem as incompatibility of different versions of transformers. You may choose to use older versions to get around the problem:

pip install crfm-helm==0.2.3
pip install transformers==4.28.1
pip install lm-eval==0.3.0

[wln20]

Update: I discussed the dependency problem with the authors and they recommend to use transformers==4.33, I've tried it on llama-2-7b-chat and got correct responses.

[benja-matic]

Thanks for the reply and reaching out to the authors. That worked for me using transformers==4.33, after applying the small changes in this PR https://github.com/FMInference/H2O/pull/41.

Is there a plan to update to newer versions of transformers? If not, I'd be happy to update the installation instructions in the readme.

[wln20]

Thanks for the reply and reaching out to the authors. That worked for me using transformers==4.33, after applying the small changes in this PR #41.

Is there a plan to update to newer versions of transformers? If not, I'd be happy to update the installation instructions in the readme.

The authors also recommended to use a newly-built version based on llama-recipes (https://github.com/meta-llama/llama-recipes/tree/main/recipes/experimental/long_context/H2O), it works with transformers == 4.39.0. You may try it out!

[JhaceLam]

Thanks for the reply and reaching out to the authors. That worked for me using transformers==4.33, after applying the small changes in this PR #41. Is there a plan to update to newer versions of transformers? If not, I'd be happy to update the installation instructions in the readme.

The authors also recommended to use a newly-built version based on llama-recipes (https://github.com/meta-llama/llama-recipes/tree/main/recipes/experimental/long_context/H2O), it works with transformers == 4.39.0. You may try it out!

https://github.com/meta-llama/llama-cookbook/tree/main/end-to-end-use-cases/long_context/H2O