ml-stable-diffusion icon indicating copy to clipboard operation
ml-stable-diffusion copied to clipboard

numerical tokenization seems a miss

Open dmikey opened this issue 2 years ago • 0 comments

groups of numbers do not appear to be tokenized as a single weight.

i.e. 22 will parse to 2 and 2.

Here's evidence, but haven't had the chance to look through code of the tokenzier yet.

Screenshot_2023-08-17_at_4 07 58_PM Screenshot_2023-08-17_at_4 13 03_PM Screenshot 2023-08-17 at 4 14 53 PM

dmikey avatar Aug 17 '23 21:08 dmikey