prose icon indicating copy to clipboard operation
prose copied to clipboard

Cannot reproduce tokenization example

Open ClonedOne opened this issue 4 years ago • 0 comments

Installed prose with go get github.com/jdkato/prose/v2 and copied the code from the tokenization example in the readme.

The output I get is

@jdkato NN
, ,
go VB
to TO
http://example.com NN
thanks NNS
: :
) )
. .

Instead of the expected one reported in the example:

        // @jdkato NN
        // , ,
        // go VB
        // to TO
        // http://example.com NN
        // thanks NNS
        // :) SYM
        // . .

The difference is that the smile ":)" symbol is not recognized.

$ go version go version go1.16.5 linux/amd64

ClonedOne avatar Jul 04 '21 22:07 ClonedOne