InftyAI
Will sharing models via hostPath leading to security probelm
At the first glance, because the Models are published by the admins, it maybe ok because the data source is under supervising.
Or is that user need?
Another problem is about the memory management, or the model size will increasing over time. Have no idea whether hg will manage the cache with LRU or other algos?
Another concern is people may not want to mount the hostpath, we should provide an option for them. P2P accelerator may help them as well.
However, the best option would be module weights should be treated as the images as well, the kubelet can help manage the lifecycle of weights, have no idea image volume supports this or not. But some models may stored at object store, they can't be downloaded by containerd or cri-o, we should manage manually, with this concern, seems we should mount them to the hostpath then, and manage them with a new daemon processor.
Validation TODO:
- with image volume, model weights can be cached
- with image volume, model weights can be gced
As mentioned, the namespace isolation is not tackled if we support namespaced models in the future.
Maybe the first step is switch volume source to PV & PVC, then in the future, if we want to support other filestorages like NFS, it's more convenient.
Validation TODO:
- with image volume, model weights can be cached
- with image volume, model weights can be gced
This is true. I think
Some projects will download models with APIs and schedule Pods to cached nodes, however, I think this is less convenient, because a node can serve several kinds of LLMs, how can we know which model we need. And the lifecycle is binded, like the API object is deleted, the model files will be also deleted. However, this is not similar to images, which is controlled by the memory size threshold, I believe this is the right way because a model file can be reused if user deploy the model again.
Anyway, we may need a daemonset processor to manage the models and also report the model usage to the node info for scheduling.
Right now, the hostpath is configured by users, so they should be aware of the risks in advance. Let's close this for now. /close