陆零
mmlu_gen_23a9a9 mmlu_gen_79e572 These two generate config also have some problems: The last input sample's "Answer:" should in BOT's template string if there will be some role token like "ASSISTANT" before...
May you should redesign meta-template and ice-template usage.
Some models will not output the answer with the expected format, because in the few-shot cases, there is not any `` between `Answer:` and ``, but there is a ``...
同问，我 Baichuan2-13b-Base 在 GSM8k 上使用 OpenCompass 4-shot 评估，得分是 18.73，官方给的论文还有 Repo 里面的得分是 52.77 分，我也想知道是不是评测方式存在差异 @baichuan-assistant
I have got the same error, with specified dataset, global_batch_size, and sequence_parllel on.
我看里面的 seq padding 是直接 pad 到 cutoff_len, 不知道我对这个提交的理解是否有偏差。若样本长度普遍偏短，是否会出现计算浪费？ I see it is padded to cutoff_len, not max len of a micro batch, whether am I misunderstood. If most samples are...