super-buster

Very helpful, thank you again @timoschick. Today I run over wsc experiment and found test predictions file, namely, predictions.jsonl. Cause I don't eval validation dataset in my command, so I...

You can save fast tokenizer in advane, refer to [fix slow tokenizer](https://blog.sharpbai.com/2023/06/%e8%a7%a3%e5%86%b3llama%e4%bd%bf%e7%94%a8autotokenizer%e5%88%9b%e5%bb%batokenizer%e7%bc%93%e6%85%a2%e9%97%ae%e9%a2%98/) Then auto load fast tokenizer if you want. I have modified in my [code](https://github.com/super-buster/lightllm/blob/952af1efb08b97059c94ec80dde989007933ba41/lightllm/server/tokenizer.py#L52-L60)