token count is inconsistent with OpenAI tokenizer

[GorvGoyl]

As shown below:

text:

<|im_start|>dd<|im_sep|>OpenAI's large language models (sometimes referred to as GPT's) process text using tokens, which are common sequences of characters found in a set of text. The models learn to understand the statistical relationships between these tokens, and excel at producing the next token in a sequence of tokens.<|im_end|><|im_start|>assistant<|im_sep|><|im_end|><|im_start|>assistant<|im_sep|>

[syntaxtrash]

any update to this? They work fine without the special characters.

https://platform.openai.com/tokenizer

https://tiktokenizer.vercel.app/

[dqbd]

Hello! the OpenAI tokenizer does not treat special tokens as a single tokens, whereas the Tiktokenizer does when selecting gpt-3.5-turbo. You can get the same behaviour by selecting cl100k_base or o200k_base