llmaz icon indicating copy to clipboard operation
llmaz copied to clipboard
Published 3 months ago • verified publisher icon InftyAI

Liveness & Readiness support

Open kerthcet opened this issue 1 year ago • 3 comments

Add the support for inference services.

kerthcet avatar Jul 15 '24 11:07 kerthcet

/kind feature /milestone v0.1.0

kerthcet avatar Jul 15 '24 11:07 kerthcet

Also StartupProbe? See https://github.com/triton-inference-server/server/pull/5257/.

pacoxu avatar Sep 24 '24 09:09 pacoxu

Yes, something like that, the core reason here is we should be aware of the server condition, ready or not? Maybe this can be part of the backendRuntime because it's related to the backend themselves.

kerthcet avatar Sep 24 '24 10:09 kerthcet