llama_7b wikitext perplexity 7.0915350914

[xiaopengaia]

bash scripts/llama_7b.sh

the source model: wikitext perplexity is 5.67702

prune this model, sparsity 50%, get wikitext perplexity is 7.09153509

but the paper is : 50% 7.26

why?

[Eric-mingjie]

Are you using the c4 calibration dataset or the redpajama? In my experience, using the calibration dataset from redpajama could get numbers as high as around 7.10.

[taratt]

Hi. Were you able to resolve the issue?

[xiaopengaia]

Thanks, I didn't use c4 calibration dataset, I just wanted to reproduce the data in the paper.

def get_loaders(name, nsamples=128, seed=0, seqlen=2048, tokenizer=None):
    if 'wikitext2' in name:
        return get_wikitext2(nsamples, seed, seqlen, tokenizer)
    # if "c4" in name:
        # return get_c4(nsamples, seed, seqlen, tokenizer)