tokenisation topic
List
tokenisation repositories
TokenScript
239
Stars
69
Forks
Watchers
TokenScript schema, specs and paper
omorfi
82
Stars
25
Forks
Watchers
Open morphology for Finnish
spiral
47
Stars
9
Forks
Watchers
A Python 3 module that provides functions for splitting identifiers found in source code files.
frames-android
46
Stars
32
Forks
Watchers
Frames Android: making native card payments simple
frames-ios
84
Stars
44
Forks
Watchers
Frames iOS: making native card payments simple
tokenmonster
549
Stars
19
Forks
Watchers
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
taibun
23
Stars
1
Forks
Watchers
Taiwanese Hokkien Transliterator and Tokeniser