Mayank Punetha

icon location Tokyo

Results 1 issues of Mayank Punetha

Improve core logic to support non space tokenized languages like japenese

The tokenization logic should be more generic. Could use something like: https://www.atilika.org/ , to tokenize Japanese.