tokenizer icon indicating copy to clipboard operation
tokenizer copied to clipboard

serialization

Open sugarme opened this issue 5 years ago • 0 comments

Serialization of the Tokenizer and all the parts (PreTokenizer, Normalizer, ...) using encoding/gob. It is now easy to save/load an entire tokenizer.

See: https://stackoverflow.com/questions/28020070/golang-serialize-and-deserialize-back

sugarme avatar Sep 20 '20 08:09 sugarme