agents Failed to generate LLM completion

We have create an agent with some callable tools, which was working fine for like a month or so and 2 days ago we started getting weird errors which were happening only on tool calls.

We could go through the conversation flow and talk to the agent normaly but tool calls are not working.

error:

Traceback (most recent call last):
  File ".../livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
    
  File ".../livekit/agents/pipeline/agent_output.py", line 244, in _stream_synthesis_task
    async for seg in tts_source:
    
  File ".../livekit/agents/utils/aio/itertools.py", line 47, in tee_peer
    item = await iterator.__anext__()
    
  File ".../livekit/agents/pipeline/pipeline_agent.py", line 1109, in _llm_stream_to_str_generator
    async for chunk in stream:
    
  File ".../livekit/agents/llm/llm.py", line 246, in __anext__
    raise exc from None
    
  File ".../livekit/agents/llm/llm.py", line 162, in _main_task
    raise APIConnectionError(
    
livekit.agents._exceptions.APIConnectionError: failed to generate LLM completion after 4 attempts

We've tried to change STT from assembly to speechmatics, that did not help. Then we changed to gpt-4o-transcribe, that also did not help. We tried with changing the versions of livekit and other s but did not help.

requirements.txt

livekit==0.20.0
livekit-agents==0.12.17
livekit-plugins-openai==0.12.3
livekit-plugins-cartesia==0.4.7
livekit-plugins-deepgram==0.6.18
livekit-plugins-silero==0.7.4
livekit-plugins-elevenlabs==0.8.0
livekit-plugins-turn-detector==0.4.1
livekit-plugins-assemblyai==0.2.3
livekit-plugins-speechmatics==0.0.2

May 07 '25 08:05 moreno1123

can you share more logs from the llm?

May 07 '25 08:05 longcw

Pipeline STT metrics: duration=189.15, audio_duration=1.80

Pipeline STT metrics: duration=190.94, audio_duration=1.80

New messages: [{'role': 'assistant', 'content': 'Could you please tell me the city where you want to rent the storage unit?'}]

Pipeline STT metrics: duration=192.75, audio_duration=1.80

Pipeline STT metrics: duration=194.57, audio_duration=1.80

Pipeline STT metrics: duration=198.46, audio_duration=3.90

Pipeline STT metrics: duration=199.55, audio_duration=1.10

Pipeline EOU metrics: sequence_id=fbd4812c0be3, end_of_utterance_delay=4.77, transcription_delay=4.74

Pipeline STT metrics: duration=199.95, audio_duration=0.40

Pipeline STT metrics: duration=200.95, audio_duration=1.00

Pipeline LLM metrics: sequence_id=fbd4812c0be3, ttft=0.87, input_tokens=5209, output_tokens=46, tokens_per_second=28.75

Pipeline TTS metrics: sequence_id=fbd4812c0be3, ttfb=1.0712570869363844, audio_duration=11.34

Pipeline STT metrics: duration=202.77, audio_duration=1.80

New messages: [{'role': 'user', 'content': 'In Germany.'}]

Pipeline STT metrics: duration=204.56, audio_duration=1.80

Pipeline STT metrics: duration=206.35, audio_duration=1.80

Pipeline STT metrics: duration=208.15, audio_duration=1.80

Pipeline STT metrics: duration=209.97, audio_duration=1.80

Pipeline STT metrics: duration=211.77, audio_duration=1.80

New messages: [{'role': 'assistant', 'content': 'Thank you. We have several locations in Germany. Could you please specify the city you are interested in? For example, is it Braunschweig, Essen, Bochum, Köln Südstadt, or another city?'}]

Pipeline STT metrics: duration=213.56, audio_duration=1.80

Pipeline STT metrics: duration=219.78, audio_duration=6.20

Pipeline STT metrics: duration=220.07, audio_duration=0.30

Pipeline STT metrics: duration=220.47, audio_duration=0.40

Pipeline STT metrics: duration=220.77, audio_duration=0.30

Pipeline STT metrics: duration=221.56, audio_duration=0.80

Pipeline STT metrics: duration=221.85, audio_duration=0.30

Pipeline STT metrics: duration=222.26, audio_duration=0.40

Pipeline STT metrics: duration=222.96, audio_duration=0.70

Pipeline EOU metrics: sequence_id=4713b8af4018, end_of_utterance_delay=4.62, transcription_delay=4.57

Pipeline LLM metrics: sequence_id=4713b8af4018, ttft=0.48, input_tokens=5270, output_tokens=10, tokens_per_second=17.52

Pipeline TTS metrics: sequence_id=4713b8af4018, ttfb=0.4026995550375432, audio_duration=2.04

Pipeline STT metrics: duration=224.75, audio_duration=1.80

New messages: [{'role': 'user', 'content': "It's s n e s s e n."}]

New messages: [{'role': 'assistant', 'content': 'You mentioned Essen. Is that correct?'}]

Pipeline STT metrics: duration=226.56, audio_duration=1.80

Pipeline STT metrics: duration=231.96, audio_duration=5.40

Pipeline EOU metrics: sequence_id=23e297eb6f24, end_of_utterance_delay=4.69, transcription_delay=4.63

Pipeline STT metrics: duration=232.35, audio_duration=0.40

Pipeline LLM metrics: sequence_id=23e297eb6f24, ttft=0.53, input_tokens=0, output_tokens=0, tokens_per_second=0.00

Error in _stream_synthesis_task
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/pipeline/agent_output.py", line 244, in _stream_synthesis_task
    async for seg in tts_source:
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/utils/aio/itertools.py", line 47, in tee_peer
    item = await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/pipeline/pipeline_agent.py", line 1109, in _llm_stream_to_str_generator
    async for chunk in stream:
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/llm/llm.py", line 246, in __anext__
    raise exc from None
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/llm/llm.py", line 156, in _main_task
    return await self._run()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/livekit/plugins/openai/llm.py", line 796, in _run
    raise APIConnectionError(retryable=retryable) from e
livekit.agents._exceptions.APIConnectionError: Connection error.

New messages: [{'role': 'user', 'content': 'Yes.'}]

Pipeline STT metrics: duration=238.06, audio_duration=5.70

Pipeline EOU metrics: sequence_id=59f8dd0b43c0, end_of_utterance_delay=4.98, transcription_delay=4.94

Pipeline STT metrics: duration=238.45, audio_duration=0.40

Pipeline LLM metrics: sequence_id=59f8dd0b43c0, ttft=1.00, input_tokens=0, output_tokens=0, tokens_per_second=0.00

Error in _stream_synthesis_task
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/pipeline/agent_output.py", line 244, in _stream_synthesis_task
    async for seg in tts_source:
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/utils/aio/itertools.py", line 47, in tee_peer
    item = await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/pipeline/pipeline_agent.py", line 1109, in _llm_stream_to_str_generator
    async for chunk in stream:
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/llm/llm.py", line 246, in __anext__
    raise exc from None
  File "/usr/local/lib/python3.12/site-packages/livekit/agents/llm/llm.py", line 156, in _main_task
    return await self._run()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/livekit/plugins/openai/llm.py", line 796, in _run
    raise APIConnectionError(retryable=retryable) from e
livekit.agents._exceptions.APIConnectionError: Connection error.

New messages: [{'role': 'user', 'content': 'Yes.'}]

Pipeline STT metrics: duration=239.55, audio_duration=1.10

Pipeline STT metrics: duration=241.36, audio_duration=1.80

Pipeline STT metrics: duration=243.16, audio_duration=1.80

process exiting

livekit::rtc_engine:446:livekit::rtc_engine - received session close: "signal client closed: \"stream closed\"" UnknownReason Resume

Stopping Container

May 07 '25 09:05 moreno1123

I actually cannot find a version that match this trackback, line 796 of this file is different to the trackback

  File "/usr/local/lib/python3.12/site-packages/livekit/plugins/openai/llm.py", line 796, in _run
    raise APIConnectionError(retryable=retryable) from e

Can you list your package versions with pip list | grep livekit and share this file /usr/local/lib/python3.12/site-packages/livekit/plugins/openai/llm.py for a double check?

Or maybe you can reinstall the env and try to upgrade to latest version for 0.x or 1.0.

May 07 '25 09:05 longcw

Im also getting same error when using nextjs as a client.

(venv) apple@apples-MacBook-Pro vagents % pip list | grep livekit
livekit                            1.0.8
livekit-agents                     1.0.22
livekit-api                        1.0.2
livekit-plugins-cartesia           1.0.22
livekit-plugins-deepgram           1.0.22
livekit-plugins-elevenlabs         1.0.22
livekit-plugins-groq               1.0.22
livekit-plugins-noise-cancellation 0.2.4
livekit-plugins-openai             1.0.22
livekit-plugins-silero             1.0.22
livekit-plugins-turn-detector      1.0.22
livekit-protocol                   1.0.3

May 24 '25 08:05 sachanayush47

Got the same error what is the fix for this?

Jun 08 '25 22:06 sriram488

@sriram488 @sachanayush47 Can you share your logs and which LLM used?

Jun 09 '25 00:06 longcw

I also saw the same error today:

 ERROR  ❌ Error: type='llm_error' timestamp=1758210116.4582279 label='livekit.plugins.openai.llm.LLM' error=APITimeoutError('Request timed out.') recoverable=True
{"message": "failed to generate LLM completion, retrying in 0.1s\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/livekit/agents/llm/llm.py\", line 177, in _main_task\n    return await self._run()\n           ^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/livekit/plugins/openai/llm.py\", line 807, in _run\n    raise APITimeoutError(retryable=retryable) from None\nlivekit.agents._exceptions.APITimeoutError: Request timed out. (body=None, retryable=True)", "level": "WARNING", "name": "livekit.agents", "llm": "livekit.plugins.openai.llm.LLM", "attempt": 1, "pid": 3145, "job_id": "AJ_89wV8uW9VS52", "timestamp": "2025-09-18T15:41:56.458432+00:00"}
{"message": "process memory usage is high", "level": "WARNING", "name": "livekit.agents", "memory_usage_mb": 503.57421875, "memory_warn_mb": 500, "memory_limit_mb": 0, "pid": 3145, "job_id": "AJ_89wV8uW9VS52", "timestamp": "2025-09-18T15:41:58.900478+00:00"}
{"message": "process memory usage is high", "level": "WARNING", "name": "livekit.agents", "memory_usage_mb": 503.578125, "memory_warn_mb": 500, "memory_limit_mb": 0, "pid": 3145, "job_id": "AJ_89wV8uW9VS52", "timestamp": "2025-09-18T15:42:03.901393+00:00"}
{"message": "process memory usage is high", "level": "WARNING", "name": "livekit.agents", "memory_usage_mb": 503.578125, "memory_warn_mb": 500, "memory_limit_mb": 0, "pid": 3145, "job_id": "AJ_89wV8uW9VS52", "timestamp": "2025-09-18T15:42:08.901900+00:00"}

Update Nevermind, this was caused by GPT-5 with High reasoning

Sep 18 '25 15:09 DollarAkshay