OpusTools icon indicating copy to clipboard operation
OpusTools copied to clipboard
verified publisher icon Helsinki-NLP

What is the tokenizer for all languages?

Open SefaZeng opened this issue 4 years ago • 0 comments

The provided tmx file contain the tokenized text, and I wonder what tokenizer is used for the language like Thai, Chinese etc. Is there any docs to find this? Thx!

SefaZeng avatar Jun 22 '21 11:06 SefaZeng