tiktoken icon indicating copy to clipboard operation
tiktoken copied to clipboard

Ensure support for all o models, including o3 and o4-mini

Open deepdive101 opened this issue 9 months ago • 2 comments

At this point, it is just wrong for OpenAI to release models without updating tiktoken at the same time.

deepdive101 avatar Apr 17 '25 18:04 deepdive101

when it is expected to add o4- in MODEL_PREFIX_TO_ENCODING in tiktoken/model.py ? they are basically using the same tokenizer as o3, so no reason for not updating tiktoken right away

DavidAmat avatar Apr 25 '25 12:04 DavidAmat

Yeah, it's all o200k_base at this point. But they should specify this explicitly on model release lol. It's confusing for developers.

kaimatzu avatar Apr 28 '25 16:04 kaimatzu

This is fixed in recent versions of tiktoken

hauntsaninja avatar Aug 08 '25 23:08 hauntsaninja