cody-moveworks

How to serve multiple TensorRT-LLM models in the same process / server?

4

Hi there! I'm trying to serve multiple TensorRT-LLM models and I'm wondering what the recommended approach is. I'm using Python to serve TensorRT-LLM models. I've tried / considered: - `GenerationSession`:...