bayes icon indicating copy to clipboard operation
bayes copied to clipboard

Support async tokenizer?

Open Bin-Huang opened this issue 7 years ago • 0 comments

It is difficult to segment a text into tokens in some languages(such as Chinese), there are many hard works need to do to implement better tokenizer. For this reason, sometimes the tokenizer is implemented in other programming language, even in other services( in microservices architecture). In this case, support async version of tokenizer to request tokens between services is required.

PR: https://github.com/ttezel/bayes/pull/21

Bin-Huang avatar Mar 19 '19 08:03 Bin-Huang