easytokenizer icon indicating copy to clipboard operation
easytokenizer copied to clipboard
verified publisher icon zejunwang1

Is the tokenzier's output exactly the same as BertTokenizer?

Open wuyaoxuehun opened this issue 3 years ago • 1 comments

wuyaoxuehun avatar Jan 08 '23 16:01 wuyaoxuehun

I have verified the tokenize result on 180000 Chinese sentences. It is exactly the same as BertTokenizer.

zejunwang1 avatar Jan 09 '23 00:01 zejunwang1