Ken Lee
Ken Lee
Dockerfile formatted weird: FROM nvidia/cuda:11.0.3-runtime-ubuntu20.04 ENV BENTOML_HOME="/model_store/" ENV CUDA_VISIBLE_DEVICES=0 RUN apt-get update && apt-get install -y python3-pip RUN pip3 install openllm EXPOSE 3000 ENTRYPOINT [ "openllm", "start" ] CMD [...
Thank you! I will try that. For these LLMs are there any recommended base images for GPUs or any recommended settings to pass to "bentoml containerize" for these models in...
It worked with 0.1.15 and trying it now for 0.1.17.
dolly-v2 worked with 0.1.17, but running into an issue downloading falcon referenced in another issue: ``` To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke Enabling debug mode for current BentoML session Make sure...
I got the same issue trying to use falcon with openllm==0.1.17
Hi @Abhishekbhagwat I've fixed all the changes you noted, let me know if you would like more.