codeprep icon indicating copy to clipboard operation
codeprep copied to clipboard

By default use end-of-full-token character (</t>) instead of token boundaries (<w>, </w>) for all kinds of pre-processing for consistency

Open hlibbabii opened this issue 5 years ago • 0 comments

Currently:

>>> api.basic("getName")
['<w>', 'get', 'Name', '</w>']

To be done:

>>> api.basic("getName")
['get', 'Name', '</t>']

hlibbabii avatar Feb 28 '20 13:02 hlibbabii