tiktoken
tiktoken copied to clipboard
Ensure support for all o models, including o3 and o4-mini
At this point, it is just wrong for OpenAI to release models without updating tiktoken at the same time.
when it is expected to add o4- in MODEL_PREFIX_TO_ENCODING in tiktoken/model.py ? they are basically using the same tokenizer as o3, so no reason for not updating tiktoken right away
Yeah, it's all o200k_base at this point. But they should specify this explicitly on model release lol. It's confusing for developers.
This is fixed in recent versions of tiktoken