easytokenizer

easytokenizer copied to clipboard

* C code format by Clang-Tidy * Python code format by black

sunhailin-Leo

Golang binding~

1

* I publish a repo for Golang binding. * https://github.com/sunhailin-Leo/easytokenizer-to-go

sunhailin-Leo

Is the tokenzier's output exactly the same as BertTokenizer?

1

wuyaoxuehun

Does it provide the training interface?

2

Excellent works! I wonder whether this package provide the api to train a tokenizer (i.e. get the vocab) from huge corpus? Thanks!

henryxiao1997

是否可以直接使用头文件拉到项目

2

你好，请问是否可以不编译，直接用头文件的方式呢？编译总是很麻烦

yourbikun