Kerem Seyhan

the server is full loaded and the docs endpoint is accessible aswell: ``` python -m sglang.launch_server --model-path mistralai/Mistral-7B-Instruct-v0.2 --port 30000 Rank 0: load weight begin. INFO 03-28 06:17:14 weight_utils.py:163] Using...

+1 for that problem.. tried various things havent found a solution yet. heres the last parts of the install log. 1679 info run [email protected] postinstall { code: null, signal: 'SIGINT'...

> @ManuXD32 How does one manually download the package from that site? (sorry for the dumb question) > > I'm not familiar with that website and couldn't find a download...

> Thanks @kseyhan. I believe I was able to follow @ManuXD32 's instructions with your help, but dalai would not start. The npx installation commands also finished suspiciously quickly even...

> This is how i worked around it: > > ```shell > sudo dnf install make automake gcc gcc-c++ kernel-devel python3-virtualenv -y && \ > npm i dalai --ignore-scripts &&...

i can also supply a database with test data for better reproduction if there is any need for it. the decrease in speed is increasing with the context lenght. thats...

well, just want to report that i just returned after some abstinence to play arround with my bot again and it responds in 4-5 seconds with a completely filled context...

hmm, just tested again. maybe was me or i did pull an outdated llama or what last time. minicpm-o seems to work with the "minicpm-v-2.6" chat handler.

@samkoesnadi i tried my luck with [Qwen2-VL-7B-Instruct-GGUF](https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF/tree/main) and tried almost every registered chat handler that includes a and token in the template and got the same results as @la1ty with...

@samkoesnadi which chat handler did you use if i may ask? the exact url to the model you used there would be usefull aswell.