andreapairon

Results 6 comments of andreapairon

How to use ServingRuntime autoscaling?

Thank you. In the meantime...can you tell me how to activate the serving runtimes autoscaling? :D

How to use ServingRuntime autoscaling?

@ckadner --- yeah, I'll do. @Jooho --- But to enable HPA, is it necessary the KNative installation as well? Or the standalone installation of KServe ModelMesh Serving is enough?

Workers graceful shutdown being sure that current executions are not truncated

@coderabhigupta the latest, version `1.1.1`

Workers graceful shutdown being sure that current executions are not truncated

Maybe the trick could be: the `while True` in the `TaskRunner` class could be replaced by `while self.worker.paused() == False`. In that way every process would exit from the loop...

Edit MM_SCALEUP_RPM_THRESHOLD env variable

@GolanLevy thank you. How can I change the default value by setting the env var MM_SCALEUP_RPM_THRESHOLD on the runtime-adapter through the available YAML files in the repo?

[Bug]: client socket has timed out while trying to connect to GPU node, when initializing DeepSeek R1 in ray vllm serving

Same error with Llama running on two T4 nodes.