Alazar
Alazar
Perfect, I created an initial PR: https://github.com/huggingface/transformers/pull/31844.
@khalidmohammedet
Hi @SunMarc @amyeroberts - while testing my changes I was running into a failure. I checked Qwen2's test and it also has same failure. I am attaching error below, I...
got it, I will try with previous ggml version, thank you @SunMarc!
Hi @SunMarc - I am getting giberrish output for my test, i think it has to do with phi3 having a slightly different attention class where [q, k, and v...
Hi @kibru9399 - I am actively working on this, it is actually almost finalized.
Hello @SunMarc @amyeroberts - This PR is ready for review! A few remaining TODO's on my end: 1. The generated output from gguf model `microsoft/Phi-3-mini-4k-instruct-gguf` is different from non gguf...
Hi @SunMarc - For trying phi3.5 weights: There seems to be something wrong with phi3.5 template because model gives weird output. For example with input: `Can you provide ways to...
Hi @SunMarc! When I try via an already converted gguf llama like [TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) which is also used inside test_ggml.py, I don't see the missing merges. But a llama gguf I...
I am able to checkout llama.cpp and repro the missing merges bug locally and I also have a fix. The bug shows up for the llama family models only. I...