transformers

transformers copied to clipboard

fix for: https://github.com/huggingface/transformers/pull/21955/#issuecomment-1465821265

mymusise

Add default tokenizer for gpt_neox (the same as gpt_neo)

1

The `tokenization_auto.py` was missing a mapping for `gpt_neox`, causing the `AutoTokenizer` initialization for GPT NeoX to fail at runtime: ```python File ..., in load_tokenizer(model_name_or_path='./gpt-neox-20b', **kwargs={'cache_dir': '.cache/'}) 16 def load_tokenizer(model_name_or_path: str...

aalok-sathe