Ryan Marten

Hi @ishaan-jaff We tracked down the root cause of the issue. [Litellm uses](https://github.com/BerriAI/litellm/blob/cda6691a7f9c73a2544119a9cb2abbbeb9b5b35a/litellm/llms/OpenAI/openai.py#L544) the [official OpenAI python client](https://github.com/openai/openai-python) ``` client: Optional[Union[OpenAI, AsyncOpenAI]] = None, ``` The official OpenAI client has...

@ishaan-jaff Thanks for creating a PR for this! We can certainly help test the change 😄. I'll run a benchmarking test with `model=aiohttp_openai/gpt-4o-mini` Our use-case on non-streaming so that shouldn't...

Here is our benchmarking using the [curator](https://github.com/bespokelabsai/curator/) request processor and viewer (with different backends). I see that this was released in https://github.com/BerriAI/litellm/releases/tag/v1.56.8. I upgraded my litellm version to the latest...

I'll take a look! For reference here is our aiohttp implementation: https://github.com/bespokelabsai/curator/blob/0c7cf21a5af0a228904906de417d902fac5c2b5c/src/bespokelabs/curator/request_processor/online/openai_online_request_processor.py#L167 And here is how we are using litellm as a backend: https://github.com/bespokelabsai/curator/blob/0c7cf21a5af0a228904906de417d902fac5c2b5c/src/bespokelabs/curator/request_processor/online/litellm_online_request_processor.py#L210

Ah yes, what you said about the routing makes sense! When the fix is in, I'll try my benchmark again and post the results 👍

@ishaan-jaff - yes absolutely (looking out for the release)

> @RyanMarten you are right ! just ran a load test to confirm. The right is with `aiohttp` it's 10x more RPS > > @ishaan-jaff Also curious, what software /...

`poetry add litellm@latest` Using version ^1.57.4 for litellm ``` from bespokelabs.curator import LLM from datasets import Dataset dataset = Dataset.from_dict({"prompt": ["write me a poem"] * 100_000}) llm = LLM( prompt_func=lambda...

Ah this is because we do a test call with `completion` instead of `acompletion` ``` completion = litellm.completion(model="aiohttp_openai/gpt-4o-mini",messages=[{"role": "user", "content": "hi"}]) ``` fails with an unintuitive error message ``` litellm.exceptions.APIConnectionError:...

OK now I'm running into an issue in the main loop where ``` 2025-01-09 16:26:13,066 - bespokelabs.curator.request_processor.online.base_online_request_processor - WARNING - Encountered 'APIConnectionError: litellm.APIConnectionError: Aiohttp_openaiException - Event loop is closed' during...