deepsparse icon indicating copy to clipboard operation
deepsparse copied to clipboard

Add CLIP model to enable test_clip.py

Open mgoin opened this issue 2 years ago • 1 comments

Since we've made a quantized CLIP (https://huggingface.co/neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds), let's use it to test our pipeline!

mgoin avatar Dec 27 '23 15:12 mgoin

@dsikka it is just made for image/text retrieval and zero-shot image classification. I left in the pytest.skip for the captioning test as a result. This model was made from OpenCLIP using a special branch that @ohaijen worked on. This was made as a collaboration so we're focused on getting results and pipelines working quickly - this is why we pushed to HF and everything to run the model is pretty much self-contained within that model card and notebooks. My opportunistic thinking here was that we could use the model we're using on HF for some active testing on DeepSparse. I don't think there are plans on pushing CLIP models to SparseZoo at the moment, but if you think it's necessary for testing we could try to get a model up there.

mgoin avatar Dec 27 '23 19:12 mgoin