clip.cpp Metal support?

Hi, awesome work on this project!

I'm building some Swift apps using llama.cpp, and I'd love to try getting clip.cpp running on my app too.

I'm curious if you're going to support running clip.cpp on Metal like llama.cpp?

Sep 30 '23 22:09 kchro3

Hi @kchro3, thanks for reaching out! It needs Conv2D implementation on Metal. I'm currently implementing LLaVA with llama.cpp and clip.cpp combined (initial release in a few days). The CLIP part will run on CPU in the first place, but then I'll implement Conv2D on Metal to support end-to-end Metal inference for multimodal models.

Oct 01 '23 00:10 monatis

wow, that's awesome. i'm looking forward to the release!

Oct 01 '23 01:10 kchro3

where can i follow your progress btw? will it be in this repo or are you going to make a new one

Oct 01 '23 21:10 kchro3

WIP PR for the initial release of large multimodal model inference is in another repo. Metal implementation of Conv2D will require another pr to the main GGML repo after this PR is merged and released.

Oct 01 '23 22:10 monatis

awesome work, @monatis ! cheering you on

Oct 10 '23 20:10 kchro3

Hi, awesome work on this project!

I'm building some Swift apps using llama.cpp, and I'd love to try getting clip.cpp running on my app too.

I'm curious if you're going to support running clip.cpp on Metal like llama.cpp? welcome to my project, support all openai clip and ofa-sys chinese clip model,at the same time support metal gpu and f16,q8_0 quantization https://github.com/yysu-888/clip.cpp

Sep 25 '24 04:09 yysu-888