Karthik Suresh issues

Repositories
Issues
Comments

Results 2 issues of


                                            Karthik Suresh

CUDA Out of Memory for the Falcon 7B model on A100 80GB GPU

I am trying to reproduce the Falcon-7B Lora fine-tuning on the Alpaca dataset. I followed the steps to convert the checkpoints to lightning format, downloaded and tokenized the Alpaca dataset...

Extremely slow generation speed for llama 2 70B chat model

I was able to benchmark llama 2 7B chat (int 8) and was able to get ~600 tokens in about 12s on an A100 GPU whereas the HF pipeline takes...