Archit Rastogi
Archit Rastogi
https://community.openai.com/t/whats-the-tokenization-algorithm-gpt-4-1-uses/1245758
https://github.com/user-attachments/assets/5efb7ec3-f848-4d0d-8614-b17c8b7d5eb7 Issue still persists for me 1) The website crashed when trying to reomve all languages 2) Reloaded the webiste this time I could remove all the languages but when...
GPT-4.1 uses the same tokenizer as 4o; same encoding. https://community.openai.com/t/whats-the-tokenization-algorithm-gpt-4-1-uses/1245758