Tara Saba

icon indicating an email [email protected]

icon location Toronto, Canada icon location PhD in Computer Science Student at the University of Toronto

Results 5 comments of Tara Saba

Cannot reproduce Llama2 results

I am using the same testenc that the function get_wikitext2 in data.py is returning. If the model's sequence length is 4096, does this mean that I'm somehow getting more samples?

Cannot reproduce Llama2 results

Thanks to your tip I was able to figure out what the problem was. I was testing over wikitext103 instead of wikitext2. The version of datasets suggested in your install...

Perplexity is off for Llama 2-7b

Hi, Thanks for your prompt response. Yes, the dense is off too. I'm getting 7.72 for LLaMA2-7b and you guys are reporting 5.12. Can you maybe clone your repository again...

Perplexity is off for Llama 2-7b

I'm running with context size 4096 as well (nsampels = 333). This is so weird. What version of datasets and transformers are you using?

llama_7b wikitext perplexity 7.0915350914

Hi. Were you able to resolve the issue?