MidstallSoftware
How to run in an iOS device?
hi , it is good example to call llama.cpp in a flutter project, and I met a problem when I run it with my iPhone ,it seems like there is not a file called "flutter_llama.framework", could you help to show how to make it? Thanks !
I don't have an iOS device nor a Mac so I am unable to develop this library for those devices but you should be able to compile llama.cpp and drop it into the application. I'm not even sure if flutter supports using CMake with iOS and macOS.
Thanks for your reply, I will try on my Android device , but I noticed it needs "flutter_llama.so" file , how to generate it ? I just could get libflutter_llama.dylib , it will be great if you could provide a guide (how to using cmake in this project ) .
The shared library is automatically generated on Windows, Linux, and Android. I haven't written any documentation yet since this is a brand new library.
I have an M-series MacBook on the way and it will show up next week. I should be able to look into fixing this later in the month.
Ok, the PR is mostly working aside from the stuff isn't linking and I'm a bit stuck.
Thanks for your reply, I will try on my Android device , but I noticed it needs "flutter_llama.so" file , how to generate it ? I just could get libflutter_llama.dylib , it will be great if you could provide a guide (how to using cmake in this project ) .
Dylib is the Apple alternative to so, so this is correct @realcarlos
Ok, the PR is mostly working aside from the stuff isn't linking and I'm a bit stuck.
I've been working on https://github.com/BrutalCoding/shady.ai and got stuck last April because I tried to add native dart bindings for a similar project to llama.cpp (I used rwkv.cpp). Let's collaborate.
I was able to load the bin model in memory but I couldn't tokenize anything because of a missing Dart port for the HF tokenizer library. I thought about a few ways to port it (e.g. rust to dart) but eventually focused on life admin stuff.
Recently started again and I'm aiming at llama.cpp directly now. They just merged a big breaking change (ggml replaced with gguf) so I'm trying to get that working in Dart.
Wdyt of setting up a meeting for a brown bag session? I'm sure we can get this to work @RossComputerGuy
@BrutalCoding Yes, that sounds like a good idea.
Can you do Sat, 9 September by any chance? I've setup a public event starting from 07:00 UTC for max. 2 hours. This is the public link: meet.google.com/mxh-adsf-fzh
If that doesn't work out for you, send me a PM so we can arrange a more suitable time together.
Unfortunately, 7:00 UTC for me is midnight. I'm in US pacific time. But I should be able to do Saturday the 9th.
Hi @RossComputerGuy .. thanks a lot for taking on this project.. ! I wonder if you can just send my way some instructions on how to use this now that you got your M-series laptop?
@abcnow I've only been able to get this to work under Linux on Apple Silicon. I've been facing issues with trying to get llama.cpp to compile properly, it's currently complaining about undefined symbols. Unfortunately, I don't have the expertise or time currently to figure out how Apple's build system works and how to fix it.