mixtral-offloading issues

Utilized pop for meta keys cleanup

Instead of manually iterating through each key in del_keys to delete them from the meta dictionary, use the pop() method to remove these keys if they exist The pop() method...

vivekmaru36

Session crashed on colab

4

Hi, Have you guys managed to make it works on T4 colab? P.S. It crashes multiple times even with `offload_per_layer = 5` as mentioned in the comment.

bitsnaps

exl2

2

using exl2 2.4 you can run mixtral on colab, did you give it a try ?

eramax

Mixtral OffLoading/GGUF/ExLlamaV2, which approach to use?

1

I'm a bit lost with the different quantization approaches such as GGUF, ExLlamaV2 & this project? Is it the same thing? Is one approach faster? GGUF: [TheBloke/Mixtral-8x7B-v0.1-GGUF](https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF) ExLlamaV2: [turboderp/Mixtral-8x7B-instruct-exl2](https://huggingface.co/turboderp/Mixtral-8x7B-instruct-exl2)

LeMoussel

CLI interface added

3

This PR adds a small CLI interface to the repository which makes local usage easy.

NJannasch

Enhancing the Efficacy of MoE Offloading with Speculative Prefetching Strategies

1

Dear Mixtral Offloading Contributors, I hope this message finds you well. I have been thoroughly engrossed in the intricacies of your project and commend the strides you have made in...

yihong1120

Is it possible to finetune this on a custom dataset?

8

Hi there, Just wondering is it possible to fine tune this model on a custom dataset? If so, are there any examples/code? Many thanks for any help, and for this...

asmith26