[ERROR]PluginDaemonInternalServerError: killed by timeout
Self Checks
- [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
- [x] I have searched for existing issues search for existing issues, including closed ones.
- [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
- [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
- [x] Please do not modify this template :) and fill in all the required fields.
Dify version
1.0.0
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
I encountered some service failures sometimes, and I checked the log and found the following errors.
Logs
nginx-1 | 192.168.144.1 - - [06/Mar/2025:06:32:17 +0000] "GET /console/api/apps/28669fd5-16f0-45b3-831d-8a7231d0a3ca/workflows/draft HTTP/1.1" 200 3291 "https://dify.app.ynu.edu.cn/app/28669fd5-16f0-45b3-831d-8a7231d0a3ca/workflow" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 Edg/133.0.0.0" "113.55.127.140"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:18 | 200 | 439.616271ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/text_embedding/invoke"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:18 | 200 | 704.057177ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/text_embedding/invoke"
api-1 | 2025-03-06 06:32:18.985 INFO [Thread-1107 (_generate_worker)] [_client.py:1038] - HTTP Request: POST http://sandbox:8194/v1/sandbox/run "HTTP/1.1 200 OK"
sandbox-1 | [GIN] 2025/03/06 - 06:32:18 | 200 | 182.220057ms | 192.168.96.4 | POST "/v1/sandbox/run"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 22.474066ms | 192.168.144.11 | GET "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/management/models?page=1&page_size=256"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 3.918187ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 4.059227ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 4.244066ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 6.569322ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 3.514292ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 2.260335ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/llm/num_tokens"
plugin_daemon-1 | 2025/03/06 06:32:22 factory.go:28: [ERROR]PluginDaemonInternalServerError: killed by timeout
plugin_daemon-1 | goroutine 48360 [running]:
plugin_daemon-1 | runtime/debug.Stack()
plugin_daemon-1 | /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/types/exception.InternalServerError({0x180b640, 0xc0006c48c0})
plugin_daemon-1 | /app/internal/types/exception/factory.go:27 +0x27
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/service.baseSSEService[...](0xc000862ff0, 0x15f6c0d, 0xf0?)
plugin_daemon-1 | /app/internal/service/base_sse.go:81 +0x4e6
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/service.InvokeLLM(0xc0001ec780, 0xc000028600, 0xf0)
plugin_daemon-1 | /app/internal/service/invoke_model.go:35 +0x198
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.InvokeLLM.func8.1({{{0xc0005b27dd, 0x24}, {0xc0017044d0, 0x7}}, {{0x0, 0x0}}, {0xc000f06240, 0x58}, 0x0, 0x0, ...})
plugin_daemon-1 | /app/internal/server/controllers/model.go:20 +0x89
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindPluginDispatchRequest[...].func1()
plugin_daemon-1 | /app/internal/server/controllers/base.go:53 +0x9c
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindRequest[...](0xc000028600, 0xc000863328)
plugin_daemon-1 | /app/internal/server/controllers/base.go:31 +0x275
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindPluginDispatchRequest[...](0x1603a20?, 0xa?)
plugin_daemon-1 | /app/internal/server/controllers/base.go:37 +0x45
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.InvokeLLM.func8(0xc000028600)
plugin_daemon-1 | /app/internal/server/controllers/model.go:17 +0x74
plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 +0x2b
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).InitClusterID.func3(0xc000028600)
plugin_daemon-1 | /app/internal/server/middleware.go:170 +0x54
plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 +0x2b
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).RedirectPluginInvoke.func2(0xc000028600)
plugin_daemon-1 | /app/internal/server/middleware.go:97 +0x74
plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 +0x2b
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).FetchPluginInstallation.func1(0xc000028600)
plugin_daemon-1 | /app/internal/server/middleware.go:66 +0x2f2
plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 +0x2b
plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginGroup.CheckingKey.func1(0xc000028600)
plugin_daemon-1 | /app/internal/server/middleware.go:24 +0x95
plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(...)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185
plugin_daemon-1 | github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000028600)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 +0x7a
plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(...)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185
plugin_daemon-1 | github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000028600)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:249 +0xe5
plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(...)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185
plugin_daemon-1 | github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00040a9c0, 0xc000028600)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:633 +0x892
plugin_daemon-1 | github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00040a9c0, {0x181dac0, 0xc000216620}, 0xc00077db00)
plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:589 +0x1b2
plugin_daemon-1 | net/http.serverHandler.ServeHTTP({0x1813868?}, {0x181dac0?, 0xc000216620?}, 0x6?)
plugin_daemon-1 | /usr/local/go/src/net/http/server.go:3142 +0x8e
plugin_daemon-1 | net/http.(*conn).serve(0xc001527b00, {0x1820270, 0xc0005b9ce0})
plugin_daemon-1 | /usr/local/go/src/net/http/server.go:2044 +0x5e8
plugin_daemon-1 | created by net/http.(*Server).Serve in goroutine 89
plugin_daemon-1 | /usr/local/go/src/net/http/server.go:3290 +0x4b4
plugin_daemon-1 |
plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:22 | 200 | 4m0s | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/llm/invoke"
nginx-1 | 192.168.144.1 - - [06/Mar/2025:06:32:23 +0000] "GET /api/parameters HTTP/1.1" 200 1220 "https://dify.app.ynu.edu.cn/chat/gH9Gkwu0kmaIEloS" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 NetType/WIFI MicroMessenger/7.0.20.1781(0x6700143B) WindowsWechat(0x63090cbd) XWEB/11581 Flue" "113.55.124.122"
✔️ Expected Behavior
No errors.
❌ Actual Behavior
The dify web services down sometimes.
To resolve the "PluginDaemonInternalServerError: killed by timeout" error in your Dify self-hosted (Docker) environment, you can try the following steps:
-
Ensure LLM Configuration: Make sure that a Language Learning Model (LLM) is properly configured in your Dify installation. This error often occurs when no LLM is configured, especially during operations that require model schema information.
-
Adjust Timeout Settings: You can increase the timeout settings to prevent the plugin daemon from being killed by timeout. Modify your
docker-compose.yamlfile to include the following configuration under theplugin_daemonsection:plugin_daemon: environment: PLUGIN_MAX_EXECUTION_TIMEOUT: 2400 -
Restart Docker Containers: After making changes to the
docker-compose.yamlfile, restart your Docker containers to apply the new settings:docker compose stop docker compose up -d -
Downgrade Tongyi Plugin: If the issue persists, consider downgrading the Tongyi plugin version from 0.0.5 to 0.0.4. This has been confirmed to resolve timeout errors in some cases [1].
These steps should help mitigate the timeout errors you're experiencing.
To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other
I found some similar issues on https://github.com/langgenius/dify/issues/14736, https://github.com/langgenius/dify/issues/15082 and https://github.com/langgenius/dify/issues/14682.
We fixed in https://github.com/langgenius/dify-plugin-daemon/pull/59
plugin_daemon:
image: langgenius/dify-plugin-daemon:e0672c3c1a6451437e8f4b63b260c8b0863c9c80-local
I updated the environment of plugin_daemon in docker-compose.yml according to the others solution and need time to verify whether it works or not.
@liudonghua123 I got this from the developer of the plugin daemon, so you can try this. If this image doesn't work, please grab the latest image here https://hub.docker.com/r/langgenius/dify-plugin-daemon/tags.
Link https://github.com/langgenius/dify/issues/15094
The same problem applies to adding the Azure OpenAI Service Model.
Add model private key Why do I request marketplace.dify.ai?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
conn.connect()
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 704, in connect
self.sock = sock = self._new_conn()
^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 207, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f457e784f20>, 'Connection to marketplace.dify.ai timed out. (connect timeout=None)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/api/.venv/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='marketplace.dify.ai', port=443): Max retries exceeded with url: /api/v1/plugins/batch (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f457e784f20>, 'Connection to marketplace.dify.ai timed out. (connect timeout=None)'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/api/services/plugin/plugin_service.py", line 44, in list
manifest.plugin_id: manifest for manifest in marketplace.batch_fetch_plugin_manifests(plugin_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/core/helper/marketplace.py", line 27, in batch_fetch_plugin_manifests
response = requests.post(url, json={"plugin_ids": plugin_ids})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/requests/adapters.py", line 507, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='marketplace.dify.ai', port=443): Max retries exceeded with url: /api/v1/plugins/batch (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f457e784f20>, 'Connection to marketplace.dify.ai timed out. (connect timeout=None)'))
2025-03-06 10:45:29.325 ERROR [Dummy-4] [app.py:875] - Exception on /console/api/workspaces/current/model-providers/langgenius/azure_openai/azure_openai/models [POST]
Traceback (most recent call last):
File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
resp = resource(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view
return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
resp = meth(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/api/controllers/console/wraps.py", line 147, in decorated
return view(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/api/libs/login.py", line 94, in decorated_view
return current_app.ensure_sync(func)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/controllers/console/wraps.py", line 27, in decorated
return view(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/api/controllers/console/workspace/models.py", line 154, in post
model_provider_service.save_model_credentials(
File "/app/api/services/model_provider_service.py", line 237, in save_model_credentials
provider_configuration.add_or_update_custom_model_credentials(
File "/app/api/core/entities/provider_configuration.py", line 406, in add_or_update_custom_model_credentials
provider_model_record, credentials = self.custom_model_credentials_validate(model_type, model, credentials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/core/entities/provider_configuration.py", line 386, in custom_model_credentials_validate
credentials = model_provider_factory.model_credentials_validate(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/api/core/model_runtime/model_providers/model_provider_factory.py", line 190, in model_credentials_validate
self.plugin_model_manager.validate_model_credentials(
File "/app/api/core/plugin/manager/model.py", line 137, in validate_model_credentials
for resp in response:
^^^^^^^^
File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
self._handle_plugin_daemon_error(error.error_type, error.message)
File "/app/api/core/plugin/manager/base.py", line 223, in _handle_plugin_daemon_error
raise PluginDaemonInternalServerError(description=message)
core.plugin.manager.exc.PluginDaemonInternalServerError: PluginDaemonInternalServerError: killed by timeout
@liudonghua123 I got this from the developer of the plugin daemon, so you can try this. If this image doesn't work, please grab the latest image here https://hub.docker.com/r/langgenius/dify-plugin-daemon/tags.
Link #15094
The Docker image is using the latest version, but all plugins still have this issue.
@crazywoola The system crashed directly when using this image.After restarting, there is no response when opening Dify.
I encountered the same error when configuring the ollama interface, the homepage had no response. I checked the container and found timeout errors.
same issue
I updated the environment of plugin_daemon in docker-compose.yml according to the others solution and need time to verify whether it works or not.
Hi, It seems works and no need to change the image. Can you link the source where you find this solution?
I updated the environment of plugin_daemon in docker-compose.yml according to the others solution and need time to verify whether it works or not.
Hi, It seems works and no need to change the image. Can you link the source where you find this solution?
All environment variables you can change are in our docker/.env.example file.
I updated the environment of plugin_daemon in docker-compose.yml according to the others solution and need time to verify whether it works or not.
Hi, It seems works and no need to change the image. Can you link the source where you find this solution?
I found some similar issues on https://github.com/langgenius/dify/issues/14736, https://github.com/langgenius/dify/issues/15082 and https://github.com/langgenius/dify/issues/14682.
I updated the environment of plugin_daemon in docker-compose.yml according to the others solution and need time to verify whether it works or not.
Hi, It seems works and no need to change the image. Can you link the source where you find this solution?
here is an error when using tsinghua pip source
Hi @pjyan. Please notice the valid scheme should be https://, not https//.
I updated the environment of plugin_daemon in docker-compose.yml according to the others solution and need time to verify whether it works or not.
it works for me