Clarification request on pass@k metric

[kang-0909]

Hi, I'm a little confused about the "pass@k" metric in the paper.

It appears that on APPS you reported test case average (unstrict accuracy), although you referred to the "proportion of problems successfully solved" as the pass@k metric, which is the widely recognized definition. However, the accuracies of CodeRL, WizardCoder in table 4,5 seem to be strict.

Could you clarify which metric is being used for evaluation? Your confirmation would be greatly appreciated. Thank you.

[JulietLJY]

Thank you for bringing this to our attention. We have aligned the metrics and update the tables.