When defaulting claude-3 to tiktoken, avoid failed network call in huggingface `from_pretrained`

[cspotcode]

Title

I noticed that every time aider calls main_model.token_count(some_string) where model is 'claude-3-5-sonnet-20240620', the logic in _select_tokenizer falls through to try: tokenizer = Tokenizer.from_pretrained(model) which always makes a failed network request and throws an exception.

It attempts to GET https://huggingface.co/claude-3-5-sonnet-20240620/resolve/main/tokenizer.json which returns a 401 Client error: Invalid username or password. However, the error is not the problem: it's that we shouldn't make the HTTP request at all if our intent is to use tiktoken.

This has a slight performance impact on startup time, and I can't tell why it's helping if it's destined to fail, since the comments say claude should be using tiktoken anyway.

Relevant issues

Type

🐛 Bug Fix / enhancement / optimization

Changes

[REQUIRED] Testing - Attach a screenshot of any new tests passing locall

If UI changes, send a screenshot/GIF of working UI fixes

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 3, 2024 3:57am

[krrishdholakia]

hey @cspotcode can you add a test here to make sure it works as expected - https://github.com/BerriAI/litellm/blob/130842537fab42c2e4b27793e5b80730ab9e010e/tests/local_testing/test_utils.py#L417

If you can share a screenshot of it passing your test, it should be good to merge

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.