llama3.java

llama3.java copied to clipboard

Implementation of Q6_KFloatTensor

2

This PR contains a Q6_K implementation. * Model: bartowski/Meta-Llama-3.1-8B-Instruct-GGUF, Q6_K * CPU: AMD Ryzen 9 7900X * JVM: OpenJDK 64-Bit Server VM * Linux: 6.9.7-arch1-1 | Quant | Species |...

srogmann

Implementation of Phi-3-mini-4k-instruct (Q8_0 or Q4_0).

14

This is a PR to run Phi-3-Mini-4K. It only includes `Phi3.java`. I wrote this file because phi-3 is faster at simple tasks. I intentionally left `Llama3.java` unchanged, even though some...

srogmann

Tweaks to improve cleanliness of code; removes warnings that appear in Eclipse

2

-Surrounded Scanner in runInteractive() in try blocks -Tweaked precomputeFreqsCis() and removed code that doesn't appear to do anything (triggers a warning in Eclipse) -Added @SuppressWarnings("preview") to loadModelImpl()

SkyAphid

Issue/Bug with ChatFormat initialization and the new tokens added in the last update

7

Hello, I've been using this project to study AI, and I'm also wanting to implement it into a small project and I've written a wrapper around your class to give...

SkyAphid

Improve matrix multiplication using the Java Vector API on Apple silicon.

3

llama.cpp runs incredibly fast on Apple silicon, I ran a build with pure CPU, and it is closer to the memory bandwidth e.g. 28 tokens/s on an M3 Pro. llama3.java...

mukel

Bundle all standalone models in a single project.

1

So far I've ported the following models to Java: Llama 3 & 3.1, Mistral/Codestral/Mathstral/Nemostral (+ Tekken tokenizer), Qwen2, Phi3 and Gemma 1 & 2 ... All models are bundled as...

mukel

Support for split UTF-8 sequences

3

Hi @mukel, I like your Llama3 implementation using the Vector API. Here is a pull request to handle split UTF-8 sequences. An example is the prompt "How to write 'three...

srogmann

Running as a service

11

Thanks for this amazing work! Would you be interested to have a `--service` mode, to be able to run llama3.java as a service, and have a third-party chat communicating with...

feloy

Added reading and writing of a state-cache

2

This PR contains an exemplary implementation for storing the computed states of a system-prompt on disk. I wrote this implementation a few weeks ago, so it may not be mergeable...

srogmann

Added llama.cpp-compatible http-server.

1

This PR contains a llama.cpp-compatible HTTP-server using the HTTP-server of the JDK. In my tests I used the HTML- and JS-resources of llama.cpp. I wrote this implementation a few weeks...

srogmann