Ali Reza Ibrahimzada

I'm using M2 Apple and I can't install tokenizers. The same thing works on my Linux fine. How can we install tokenizers for M1/M2 Apple Macs?

@stephantul where did you get the rust compiler. I installed it from https://www.rust-lang.org/tools/install and `pip3 install tokenizers` still fails.

I faced a similar issue. I added a condition like below in `run_gen.py` (line 75): ``` outputs = model(input_ids=source_ids, attention_mask=source_mask, labels=target_ids, decoder_attention_mask=target_mask) loss = outputs.loss if args.n_gpu > 1: loss...

make sure you execute your script with `torchrun` rather than `python3/python`. I don't think there are other requirements for multi-GPU execution.

@Sleepyhead01 you need to do `model.module.generate()` because for `n_gpu > 1`... model is an attribute of `DataParallel`. To get the model, you should call `.module` on it. Unfortunately the authors...

Can you please give more details? Did you use the cleaning script on our starcoder generated translations? Have you checked the starcoder translations from artifacts website on zenodo?