Period not correctly tokenized?

Open huangjs opened this issue 14 years ago • 2 comments

Here's an example:

LANGUTILS> (tokens-for-ids (vector-document-words (vector-tag "Hello world. I'm here."))) ("Hello" "world." "I" "'" "m" "here.")

I think it should be:

("Hello" "world" "." "I" "'" "m" "here" ".")

Aug 12 '11 06:08 huangjs

Would you mind using the mailing list to discuss this? You can join it here:

http://lists.common-lisp.net/mailman/listinfo/langutils-devel

Cheers, Kevin

Aug 13 '11 02:08 kraison

On Sat, Aug 13, 2011 at 11:16 AM, kraison [email protected] wrote:

Would you mind using the mailing list to discuss this? You can join it here:

http://lists.common-lisp.net/mailman/listinfo/langutils-devel

Sure

黄　澗石 (Jianshi Huang) http://huangjs.net/

Aug 14 '11 10:08 huangjs