David Xue

Llama-3 8B Instruct quantized to 8 Bit spits out gibberish in transformers `model.generate()` but works fine in vLLM?

6

#### Problem description Hi friends, hope someone can help out or point me in the right direction here. I feel like this maybe an integration thing with `transformers`? I can't...

Extend support for Phi-3 models

7

### Description - Extend support for Microsoft's [Phi-3 models](https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3) ### Technical Details - Very Straightforward implementation following adding custom models guide in the README - My only concern is Phi...

[PR Ready for Review] [FEATURE] Extend Support for Phi-3

1

**Is your feature request related to a problem? Please describe.** Extend AutoGPTQ support for Microsoft's recently released [Phi-3 models](https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3) **Describe the solution you'd like** I have a PR ready and...

[BUG] Llama 3 8B Instruct - `no_inject_fused_attention` must be true or else errors out

8

**Describe the bug** I initially discovered the issue when testing the quantized model with [oobabooga](https://github.com/oobabooga)'s [text-generation-webui](https://github.com/oobabooga/text-generation-webui). When running inference on the the GPTQ quant of Llama 3 I get logs...

These lines have not been functional. The `docs_base_url` already has `docs` at the end,

**Please describe the feature you'd like to see** The current error logging mechanism isn't efficient. Retry errors can be spammy and provide less insightful information on where the error occurred....

### Items to explore/implement #### HTML webpages sometimes have embedded hyperlinks e.g. [mylife](google.com) - We currently remove these links when before chunking/vectorizing/inserting the docs into vector db - We can...

**Please describe the feature you'd like to see** React has better community support and is more widely used. As an open source reference implementation, this repo should use React instead...

**Please describe the feature you'd like to see** During downtimes of services such as OpenAI, Cohere and Azure OpenAI, have hot swappable. For cost reduction, we should also do A/B...