joewong826

Detection error when processing full-width letters

1

When processing full-width letters, it returns "Chinese" as result: `>>> import langid` `>>> langid.classify('ＡＢＣ')` `('zh', 0.9668056948707975)`

Repetition of words causes detection error

2

When I input strings like 'hello world hello world hello world', _langid_ can't identify it as English text. `>>> import langid` `>>> langid.classify('hello world hello world hello world')` `('af', 0.683057652874482)`