[question] triton inference server Dockerfile

[geraldstanje]

hi,

where can i find documentation how to build triton inference server trt-llm 24.06 for sagemaker myself so i can run it on sagemaker?

Nvidia Image i want to use: nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3

Can you please post the Dockerfile you use to modify the nvidia container? I need the Dockerfile you have for creating: https://github.com/aws/deep-learning-containers/blob/master/available_images.md#nvidia-triton-inference-containers-sm-support-only - will than modify it and use the nvidia image above.

Info: i have created a trt llm model created myself - will copy it in the container.

cc @ohadkatz @nskool @sirutBuasai

[nikhil-sk]

Hi @geraldstanje we don't support the TRT-LLM container for Triton on SM yet. Most changes to support SageMaker are already upstreamed and the above container should work with SageMaker directly. Functionally, you can just run the nvidia image on SageMaker, and shouldn't need the dockerfile to modify it.

[geraldstanje]

@nskool can you please paste the Dockerfile of the latest trt container for triton you released (24.3) here? I will modify the Docker and add llm myself.

[nikhil-sk]

@geraldstanje Based on your initial comment, you want to run TRT-LLM on SageMaker, is that correct? I'm trying to say that the nvidia TRT-LLM image will work just fine on SageMaker. Is there a specific reason you are looking for a SageMaker-provided Triton image (that doesn't supply TRTLLM), are building TRT-LLM yourself atop it?

[geraldstanje]

@nskool it works - what i was looking for was the entrypoint: https://github.com/triton-inference-server/server/blob/main/docker/sagemaker/serve i just used image: nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 - i didnt build it myself...

also it seems the metrics are not forwarded to sagemaker (entrypoint only has port 8080) - is there a solution for that?

does that mean you can run any docker container on sagemaker? can also run vllm/vllm-openai:latest docker container on sagemaker? https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html

[geraldstanje]

@nikhil-sk did you see ^^ by chance?

[geraldstanje]

hi @nikhil-sk can we please update the tritonserver containers - who to ping? tritonserver is already at version 25.02.

@sirutBuasai or @ohadkatz can you help?

[sirutBuasai]

pinging triton model server team oncall, @maaquib. Feel free to triage as needed.

[maaquib]

@geraldstanje We're waiting on a version with https://github.com/triton-inference-server/server/pull/7993 to be released. We'll publish a new container once that is available. In the meantime if this is urgent, you can consider using https://docs.djl.ai/master/docs/serving/serving/docs/lmi/user_guides/trt_llm_user_guide.html

[github-actions[bot]]

This issue has been automatically marked as stale due to 60 days of inactivity. Please comment or remove the stale label to keep it open. It will be closed in 7 days if no further activity occurs.

[github-actions[bot]]

Closing this issue after 7 additional days of inactivity since being marked stale.