jasonw247

Using metal and `n_gpu_layers` produces no tokens

5

I'm running the example script with a few different models: ``` rust use llama_cpp_rs::{ options::{ModelOptions, PredictOptions}, LLama, }; pub fn llama_predict() -> Result { // metal seems to give really...

When attempting to run `dolphin-2_6-phi-2.Q4_0.gguf` I'm getting `error loading model: unknown model architecture: 'phi2'`. Phi2 support was added a couple of weeks ago: https://github.com/ggerganov/llama.cpp/pull/4490. Is there a way to include...

[Feature Request] Additional Profiling Statistics

1

**Is your feature request related to a problem? Please describe.** I am currently using profiling jobs to understand resource usage for models. The metrics included in the job are useful,...

PEFT/Lora Support?

2

## ❓ General Questions Hello, I was wondering if mlc-llm supports converting Huggingface models trained with [PEFT](https://github.com/huggingface/peft). I did a quick search of documentation and issues, and did not see...