agents icon indicating copy to clipboard operation
agents copied to clipboard
Published 1 month ago • verified publisher icon livekit

Gemini LLM Is Failing on "Manually set deadline 5s is too short. Minimum allowed deadline is 10s."

Open alexlooney opened this issue 3 months ago • 8 comments

I've suddenly started receiving the following error when using gemini-2.5-flash and gemini-2.5-flash-lite. There have been no code changes that would produce this issue:

livekit-agent-1  | 2025-10-27 15:55:02,860 - INFO google_genai.models - AFC is enabled with max remote calls: 10. {"agent_name": "ridealong", "organization": "adp-ride-along", "room_name": "room-c505ec70-3fd4-4567-ae54-434aee56bf6c-20251027_155446", "call_type": "web", "caller": "web-f707cea6-3001-4c0d-95a1-b8e8e64dad43", "pid": 178, "job_id": "AJ_qLXYwiSBK4Ng"}
livekit-agent-1  | 2025-10-27 15:55:03,041 - WARNING livekit.agents - livekit.plugins.google.llm.LLM recovery failed
livekit-agent-1  | Traceback (most recent call last):
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/livekit/plugins/google/llm.py", line 353, in _run
livekit-agent-1  |     async for response in stream:
livekit-agent-1  |     ...<38 lines>...
livekit-agent-1  |             )
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/google/genai/models.py", line 7570, in async_generator
livekit-agent-1  |     response = await self._generate_content_stream(
livekit-agent-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |         model=model, contents=contents, config=config
livekit-agent-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |     )
livekit-agent-1  |     ^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/google/genai/models.py", line 6463, in _generate_content_stream
livekit-agent-1  |     response_stream = await self._api_client.async_request_streamed(
livekit-agent-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |         'post', path, request_dict, http_options
livekit-agent-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |     )
livekit-agent-1  |     ^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/google/genai/_api_client.py", line 1228, in async_request_streamed
livekit-agent-1  |     response = await self._async_request(http_request=http_request, stream=True)
livekit-agent-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/google/genai/_api_client.py", line 1156, in _async_request
livekit-agent-1  |     return await self._async_retry(  # type: ignore[no-any-return]
livekit-agent-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |         self._async_request_once, http_request, stream
livekit-agent-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |     )
livekit-agent-1  |     ^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
livekit-agent-1  |     do = await self.iter(retry_state=retry_state)
livekit-agent-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
livekit-agent-1  |     result = await action(retry_state)
livekit-agent-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/tenacity/_utils.py", line 99, in inner
livekit-agent-1  |     return call(*args, **kwargs)
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/tenacity/__init__.py", line 418, in exc_check
livekit-agent-1  |     raise retry_exc.reraise()
livekit-agent-1  |           ~~~~~~~~~~~~~~~~~^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/tenacity/__init__.py", line 185, in reraise
livekit-agent-1  |     raise self.last_attempt.result()
livekit-agent-1  |           ~~~~~~~~~~~~~~~~~~~~~~~~^^
livekit-agent-1  |   File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 449, in result
livekit-agent-1  |     return self.__get_result()
livekit-agent-1  |            ~~~~~~~~~~~~~~~~~^^
livekit-agent-1  |   File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
livekit-agent-1  |     raise self._exception
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
livekit-agent-1  |     result = await fn(*args, **kwargs)
livekit-agent-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/google/genai/_api_client.py", line 1064, in _async_request_once
livekit-agent-1  |     await errors.APIError.raise_for_async_response(response)
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/google/genai/errors.py", line 156, in raise_for_async_response
livekit-agent-1  |     raise ClientError(status_code, response_json, response)
livekit-agent-1  | google.genai.errors.ClientError: 400 Bad Request. {'message': '{\n  "error": {\n    "code": 400,\n    "message": "Manually set deadline 5s is too short. Minimum allowed deadline is 10s.",\n    "status": "INVALID_ARGUMENT"\n  }\n}\n', 'status': 'Bad Request'}
livekit-agent-1  |
livekit-agent-1  | The above exception was the direct cause of the following exception:
livekit-agent-1  |
livekit-agent-1  | Traceback (most recent call last):
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/livekit/agents/llm/fallback_adapter.py", line 174, in _try_generate
livekit-agent-1  |     async for chunk in stream:
livekit-agent-1  |     ...<3 lines>...
livekit-agent-1  |         yield chunk
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/livekit/agents/llm/llm.py", line 344, in __anext__
livekit-agent-1  |     raise exc  # noqa: B904
livekit-agent-1  |     ^^^^^^^^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper
livekit-agent-1  |     return await func(*args, **kwargs)  # type: ignore
livekit-agent-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/livekit/agents/llm/llm.py", line 190, in _main_task
livekit-agent-1  |     return await self._run()
livekit-agent-1  |            ^^^^^^^^^^^^^^^^^
livekit-agent-1  |   File "/home/appuser/.venv/lib/python3.13/site-packages/livekit/plugins/google/llm.py", line 395, in _run
livekit-agent-1  |     raise APIStatusError(
livekit-agent-1  |     ...<5 lines>...
livekit-agent-1  |     ) from e
livekit-agent-1  | livekit.agents._exceptions.APIStatusError: gemini llm: client error (status_code=400, request_id=c636c4e9e51f, body={
livekit-agent-1  |   "error": {
livekit-agent-1  |     "code": 400,
livekit-agent-1  |     "message": "Manually set deadline 5s is too short. Minimum allowed deadline is 10s.",
livekit-agent-1  |     "status": "INVALID_ARGUMENT"
livekit-agent-1  |   }
livekit-agent-1  | }

I'm not able to find the right place to fix this myself. I'm not sure how to proceed.

alexlooney avatar Oct 27 '25 15:10 alexlooney

There have been no code changes that would produce this issue

I'm assuming you mean that you haven't set any custom timeout parameters? By default the timeout is set to 10 seconds. Could you provide a reproducible code snippet and your package versions as well?

tinalenguyen avatar Oct 27 '25 18:10 tinalenguyen

seeing the same issue.

google.genai.errors.ClientError: 400 Bad Request. {'message': '{\n  "error": {\n    "code": 400,\n    "message": "Manually set deadline 5s is too short. Minimum allowed deadline is 10s.",\n    "status": "INVALID_ARGUMENT"\n  }\n}\n', 'status': 'Bad Request'}

config

                google.LLM(
                    model="gemini-flash-latest",
                    temperature=0.5,
                ),

jimmy0251 avatar Oct 28 '25 18:10 jimmy0251

Also saw this today:

APIStatusError: gemini llm: client error (status_code=400, request_id=98bc7c3dfc10, body={
  "error": {
    "code": 400,
    "message": "Manually set deadline 5s is too short. Minimum allowed deadline is 10s.",
    "status": "INVALID_ARGUMENT"
  }
}
 Bad Request, retryable=False)

                google.LLM(
                    temperature=0.35,
                    model="gemini-2.0-flash-001",
                    tool_choice="auto",
                ),

  "livekit-plugins-google==1.2.15",

ianbishop avatar Oct 29 '25 12:10 ianbishop

Was able to get around it via

                google.LLM(
                    temperature=0.35,
                    model="gemini-2.0-flash-001",
                    tool_choice="auto",
                    http_options=google_types.HttpOptions(
                        timeout=10000
                    ),  # temp fix until default is updated in livekit
                ),

This seems like it's entirely on Google's side. I believe the 5s comes from httpx (the underlying client in their SDK):

https://github.com/googleapis/python-genai/blob/main/google/genai/_api_client.py#L496 https://github.com/encode/httpx/blob/master/httpx/_config.py#L246

ianbishop avatar Oct 29 '25 13:10 ianbishop

@ianbishop Thank you for the additional details and links!

I wonder why you still have to override our defaults, we override Google's default HttpOptions here and our default timeout is here

tinalenguyen avatar Oct 29 '25 15:10 tinalenguyen

we are facing the exact same issue as well using livekit agent 1.2.5. no change on the code side

ringofhealth avatar Oct 29 '25 17:10 ringofhealth

We are having the same issue today. No changes on the code or the parametrizations. We will attempt a hotfix with the @ianbishop solution.

alepmalagon avatar Nov 03 '25 12:11 alepmalagon

running into same issue

janaks09 avatar Nov 22 '25 19:11 janaks09