tokenizer
tokenizer copied to clipboard
serialization
Serialization of the Tokenizer and all the parts (PreTokenizer, Normalizer, ...) using encoding/gob. It is now easy to save/load an entire tokenizer.
See: https://stackoverflow.com/questions/28020070/golang-serialize-and-deserialize-back