James Liu
James Liu
Am seeing the same discrepency - I tried both 0-shot and 5-shot for Winogrande on `meta-llama/llama-2-7b-chat-hf` and get similar results (66.63, 66.46).
Yeah this is interesting. I wonder if MMLU is contaminated
yes its primarily a decoding kernel. A.4 is based on loss evals, where the sparsity (both weight and activation) is simulated