0x80 comments

Repositories
Issues
Comments

Results 1 comments of


                                            0x80

token count is inconsistent with OpenAI tokenizer

any update to this? They work fine without the special characters. https://platform.openai.com/tokenizer ![image](https://github.com/dqbd/tiktokenizer/assets/40295780/0dd6ea79-71a3-46ee-8186-c7b6579bd978) https://tiktokenizer.vercel.app/ ![image](https://github.com/dqbd/tiktokenizer/assets/40295780/cbc60826-a40e-4e56-b2c8-18d888e8ac41)