Experimental result on APPS

[sh0416]

Hello.

I am reproducing your results, but I have a trouble time reproducing your baseline named deepseek-coder-6.7b-instruct.

I use your prompt provided in this repository, but the APPS introductory pass@1 is just 0.3192, where 0.4465 is right value according to your paper.

Also, I observe some difference between your paper and the paperwithcode website. The former reports 0.5001 while the latter reports 0.3380.

After this report, I think my score could be right if the base model score (i.e.,deepseek-coder-6.7b-instruct) is 0.3192 and the finetuned model score (i.e.motcoder) is 0.3380.

While writing this post, I found that website indicate your model as motcoder-15b which should be changed to motcoder-6.7b as your paper said that your base model is deepseek-coder-6.7b-instruct, right?

Could you clarify which score is right?