mbc1990

Tokenization behavior for accented characters

2

WordTokenizer, WordPunctTokenizer, and TreebankWordTokenizer all have similar unusual behavior on accented (tilde-ed?) characters: ``` > var tokenizer = new natural.WordPunctTokenizer(); > tokenizer.tokenize('São Paulo'); [ 'S', 'ã', 'o', 'Paulo' ] >...

WordPunctTokenizer incorrectly handles parenthetical expresssions

5

Not sure if this is intentional or not: ``` var tokenizer = new natural.WordPunctTokenizer(); console.log(tokenizer.tokenize("Example sentence (with parenthetical expression).")); ``` outputs: ``` [ 'Example', 'sentence', ' (', 'with', 'parenthetical', 'expression',...