Azure
Python/Linux Azure Function is taking a long time to execute and sometimes create new workers in the middle of concurrent calls
- Timestamp: 2022-10-13T12:49:35Z
- Function App version: Python ~4
- Invocation ID: e865b2fe-a1b4-4180-95e4-28b79f5aad8d
- Region: Australia East
Repro steps
Provide the steps required to reproduce the problem: We are in a consumption plan. After we invoke the azure function with 32 concurrent calls with 160~ calls overall, I noticed that at the first run it will finish the concurrent calls for around 2 minutes normally with cold start but it is sometimes intermittently do "Creating new webhost language worker channel for runtime:python." that will take around 2-3 minutes which increase the overall time to 4-5 minutes. This process normally runs within 30-40 seconds after the cold start.
Expected behavior
Provide a description of the expected behavior: It should not wait for that amount of time to execute or create the workers and it is sometimes happening in the middle of the concurrent calls.
Actual behavior
Provide a description of the actual behavior observed. It is intermittently observe when doing the concurrent calls.
Known workarounds
Provide a description of any known workarounds. None
Related information
Provide any related information
- Programming language used: Python
@fritzcruz We have investigated the issue and the reason for the latency to execute is the functions process worker count has been set to maximum limit i.e 10, We are recommending you to please lower this value to reduce the latency period. you should keep the setting to a low number, starting from 1 and increasing as you experiment with your workload. This suggestion is to reduce the time spent on context switches and allowing CPU-bound tasks to finish also the thread pool and thread count setting is also optional one. Please follow the below doc link for more information. https://learn.microsoft.com/en-us/azure/azure-functions/python-scale-performance-reference#use-multiple-language-worker-processes.
Appreciate the turnaround. I would like to ask if it is recommended for us to move to durable function and we will not encounter such issue as we want to have many concurrent calls as possible that is why I set it to max of 10 workers. Or, we can just utilized threading instead? We are using this function app for I/O bound task that only takes less than 10 seconds to finish individually. Thanks in advance.
Hi @fritzcruz, moving to durable functions might not help in your case since it would still create more workers depending on the worker count set. I would recommend setting the FUNCTIONS_WORKER_PROCESS_COUNT to the cores available on your system and try adjusting the PYTHON_THREADPOOL_THREAD_COUNT to maximize the throughput.
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.
