tokenizers icon indicating copy to clipboard operation
tokenizers copied to clipboard
verified publisher icon huggingface

Load serialized tokenizer with AutoTokenizer

Open jordiae opened this issue 5 years ago • 1 comments

How to load a serialized dataset (i.e., defined with a JSON) trained with the tokenizers library using AutoTokenizer? Is it possible? Thanks in advance.

jordiae avatar Dec 10 '20 22:12 jordiae

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar Apr 30 '24 01:04 github-actions[bot]