livekit
1.31.1 "generate_reply" error only with Gemini realtime
Bug Description
Gemini realtime error on "generate_reply" Model = "gemini-2.5-flash-native-audio-preview-09-2025"
2025-11-19 01:36:41,358 - [5fe7fd69-9ad2-4f28] - iMash-Agent - agents - ERROR - Error in _realtime_reply_task
01:36:41 ERROR livekit.agents Error in _realtime_reply_task {"pid": 8334, "job_id": "AJ_dbwMYUNR3t3z",
Traceback (most recent call last): "room_id": "RM_U9HkfcjCGnPV"}
File
"/opt/homebrew/Caskroom/miniconda/base/lib/py
thon3.12/site-packages/livekit/agents/utils/l
og.py", line 16, in async_fn_logs
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Caskroom/miniconda/base/lib/py
thon3.12/site-packages/livekit/agents/voice/a
gent_activity.py", line 2058, in
_realtime_reply_task
generation_ev = await
self._rt_session.generate_reply(
^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^
livekit.agents.llm.realtime.RealtimeError:
generate_reply timed out waiting for
generation_created event.
2025-11-19 01:36:41,359 - [1630dee8] - iMash-Agent - agents - ERROR - Error in _realtime_reply_task
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.12/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.12/site-packages/livekit/agents/voice/agent_activity.py", line 2058, in _realtime_reply_task
generation_ev = await self._rt_session.generate_reply(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit.agents.llm.realtime.RealtimeError: generate_reply timed out waiting for generation_created event.
Expected Behavior
When running 1.31.1 with "gemini-2.5-flash-native-audio-preview-09-2025" realtime only fatal error on "generate_reply"
Reproduction Steps
1. generate_reply
2. "gemini-2.5-flash-native-audio-preview-09-2025"
3. fatal error
...
- Sample code snippet, or a GitHub Gist link -
Operating System
macOS, Linux, any
Models Used
No response
Package Versions
livekit 0.0.19
agent= 1.31.1
Session/Room/Call IDs
No response
Proposed Solution
Additional Context
No response
Screenshots and Recordings
No response
where the generate_reply was called, can you share an example that can reproduce the issue?
@longcw
any "generate_reply" that uses "gemini-2.5-flash-native-audio-preview-09-2025" fails
@longcw
Found the core issue its when
proactivity boolean Optional Default: false Enable proactive audio, where the model can decide not to respond to certain inputs. Requires a native audio model. For more information, see Proactive audio.
Is True, the "generate_reply" always fails
I think we should find a fix for this and a way to override "generate_reply" to force a reply even when we have "proactivity" set to true.
Same issue. The error occurs in 40-50% of the connections, resulting in no initial response from the realtime model. Following is my Gemini configuration:
model_name = "gemini-2.5-flash-native-audio-preview-09-2025"
session = AgentSession(
llm=google.beta.realtime.RealtimeModel(
model=model_name,
voice=voice_id,
# language=lang,
# modalities=[Modality.AUDIO],
realtime_input_config=types.RealtimeInputConfig(
automatic_activity_detection=types.AutomaticActivityDetection(
disabled= False, # default
start_of_speech_sensitivity= types.StartSensitivity.START_SENSITIVITY_HIGH,
end_of_speech_sensitivity= types.EndSensitivity.END_SENSITIVITY_HIGH,
prefix_padding_ms= 100,
silence_duration_ms= 300,
),
)
),
vad=silero.VAD.load(min_speech_duration=1, activation_threshold=1),
turn_detection=MultilingualModel(),
# user_away_timeout=20,
)
