chattr hanging while running llamagpt

[EdwardJ1n]

Hi team,

I want to build an AI-based IDE. Inspired by this GitHub Copilot in Rstudio, it's finally here!.

On my test environment, I'm running Posit Workbench on docker.

Tries on R version 4.3.1 and 4.2.3, chattr hanging while running llamagpt.

library(chattr)
chattr_use("llamagpt")
chattr_defaults(path = "/llama/chat-ubuntu-latest-avx2", model = "/llama/ggml-gpt4all--v1.3-groovy.bin")
chattr_defaults(type="chat",path = "/llama/chat-ubuntu-latest-avx2", model = "/llama/ggml-gpt4all-j-v1.3-groovy.bin")
chattr_defaults(type="console",path = "/llama/chat-ubuntu-latest-avx2", model = "/llama/ggml-gpt4all-j-v1.3-groovy.bin")
chattr_defaults(type="notebook",path = "/llama/chat-ubuntu-latest-avx2", model = "/llama/ggml-gpt4all-j-v1.3-groovy.bin")
chattr_defaults(type="script",path = "/llama/chat-ubuntu-latest-avx2", model = "/llama/ggml-gpt4all-j-v1.3-groovy.bin")
chattr::chattr_app()

Run the following command in terminal without any issue.

./chat-ubuntu-latest-avx2 -m "./ggml-gpt4all-j-v1.3-groovy.bin" -t 4

Install R package:

install.packages("remotes") remotes::install_github("mlverse/chattr")

Sometimes, the following messages shows up: Warning in readRDS(.) : invalid or incomplete compressed data

Looks like the model can be loaded successfully.

chattr("what is llama2")

── chattr ──

── Initializing model LlamaGPTJ-chat (v. 0.3.0) Your computer supports AVX2 LlamaGPTJ-chat: loading /home/[email protected]/llama/ggml-gpt4all-j-v1.3-groovy.bin gptj_model_load: loading model from '/home/[email protected]/llama/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ... gptj_model_load: n_vocab = 50400 gptj_model_load: n_ctx = 2048 gptj_model_load: n_embd = 4096 gptj_model_load: n_head = 16 gptj_model_load: n_layer = 28 gptj_model_load: n_rot = 64 gptj_model_load: f16 = 2 gptj_model_load: ggml ctx size = 5401.45 MB gptj_model_load: kv self size = 896.00 MB gptj_model_load: ............................................ done gptj_model_load: model size = 3609.38 MB / num tensors = 285 LlamaGPTJ-chat: done loading!

However, there is no expected output after that.

Can you take a look at it?

Thanks, Edward