Async generation not implemented for this LLM.

[akashAD98]

i tried mistral & llama7b from ctransofrmer & getting this issue,is there any way to add support for this? how can we implement it with websocket?

    streaming_llm = CTransformers(model='TheBloke/Mistral-7B-v0.1-GGUF', model_file='mistral-7b-v0.1.Q4_K_M.gguf',model_type="mistral")

    streaming_llm = CTransformers(model="llama-2-7b.ggmlv3.q5_0.bin",model_type="llama",
                    config={'max_new_tokens':128,'temperature':0.01})

can we use TGI here for doing this ? by passing the url link will it support?

[janfilips]

Mistral 7B rulez

Imho it's the langchain that is outdated on this project that's causing an issue

Did you find a workaround or a different solution? I too am in need to proxy SSE over to Websocket in my app..