Is there any way to make images smaller?

[martinmr]

We have a fine-tuned Llama 7B model that we tried to built using openllm into a docker container. We ran into issues with how big the final docker image ended up being. The model is around 13GB uncompressed and the final image is 46GB. We built the default llama 7B for comparison and it turned to a 33GB images.

We built out model with this command:

openllm build llama --model_id /home/circleci/model

Is there a way to make the images smaller? We are trying to use them for a service that will use k8s autoscaling, which will not work well if it takes 10+ minutes to create a new pod.

Thanks.

[aarnphm]

Yes I'm actively working on this.

[aarnphm]

This probably has to do with the base image also includes vllm and all different dependencies for vllm

[bojiang]

The current image should be reduced to aboout 7G in openllm 0.6