llama-cpp-python
llama-cpp-python copied to clipboard
abetlen
logprobs aren't returned if sampler will reach EOS
Prerequisites
llm = Llama(
model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
n_gpu_layers=-1,
seed=8,
n_ctx=4096,
logits_all=True,
kv_overrides={"tokenizer.ggml.eos_token_id": 128002},
)
prompt = """<|start_header_id|>user<|end_header_id|>
What is a dog?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
A dog, also known as Canis lupus familiaris, is a domesticated mammal that belongs to the family Canidae. Dogs are closely related to wolves and share many physical and behavioral characteristics with them.
Dogs are typically characterized by their:
1. Physical appearance: Dogs come in a wide range of sizes, shapes, and coat types, from the tiny Chihuahua to the massive Great Dane. They have a furry coat, a wagging tail, and a distinctive snout.
2. Behavior: Dogs are social animals that thrive on interaction with their human family and other dogs. They are known for their loyalty, affection, and ability to form strong bonds with their owners.
3. Diet: Dogs are omnivores, which means they eat both plants and animals. They are often fed a diet of commercial dog food, which is formulated to meet their nutritional needs.
4. Communication: Dogs communicate primarily through body language and vocalizations, such as barking, whining, and growling.
5. Intelligence: Dogs are highly intelligent animals that are capable of learning a wide range of tasks and commands.
Dogs have been domesticated for thousands of years and have been bred for various purposes, such as:
* Companionship: Many dogs are kept as pets and are valued for their companionship and affection.
* Hunting: Some breeds of dogs, such as retrievers and pointers, are bred for hunting and retrieving game.
* Guarding: Some breeds, such as guard dogs and watchdogs, are bred for their protective instincts and ability to defend their territory.
* Assistance: Some dogs are trained as service animals to assist people with disabilities, such as guide dogs for the blind or hearing dogs for the deaf.
Overall, dogs are beloved animals that bring joy, companionship, and love to many people around the world
Would you like to know more about a specific aspect of dogs, such as their behavior, health, or training?"""
output = llm(
prompt,
echo=False,
logprobs=100,
max_tokens=1,
repeat_penalty=1.0, # disable penalties
top_k=1,
#top_p=1.0,
#min_p=-10000000000.0,
temperature=0,
)
print(output)
Expected Behavior
logprobs!!
Current Behavior
{'id': 'cmpl-c480f5df-6a62-4a39-9d21-d3058260523b', 'object': 'text_completion', 'created': 1718328460, 'model': '/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf', 'choices': [{'text': '', 'index': 0, 'logprobs': {'tokens': [], 'text_offset': [], 'token_logprobs': [], 'top_logprobs': []}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 430, 'completion_tokens': 0, 'total_tokens': 430}}
Environment and Context
llama_cpp_python-0.2.78
CUDA 12.4
I tried to use kv_overrides to change the EOS token, here, as there doesn't seem to be any other way, but that didn't do the trick
from llama_cpp import Llama
llm = Llama(
model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
n_gpu_layers=-1,
seed=8,
n_ctx=4096,
logits_all=True,
)
prompt = """<|start_header_id|>user<|end_header_id|>
What is a dog?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
A dog, also known as Canis lupus familiaris, is a domesticated mammal that belongs to the family Canidae. Dogs are closely related to wolves and share many physical and behavioral characteristics with them.
Dogs are typically characterized by their:
1. Physical appearance: Dogs come in a wide range of sizes, shapes, and coat types, from the tiny Chihuahua to the massive Great Dane. They have a furry coat, a wagging tail, and a distinctive snout.
2. Behavior: Dogs are social animals that thrive on interaction with their human family and other dogs. They are known for their loyalty, affection, and ability to form strong bonds with their owners.
3. Diet: Dogs are omnivores, which means they eat both plants and animals. They are often fed a diet of commercial dog food, which is formulated to meet their nutritional needs.
4. Communication: Dogs communicate primarily through body language and vocalizations, such as barking, whining, and growling.
5. Intelligence: Dogs are highly intelligent animals that are capable of learning a wide range of tasks and commands.
Dogs have been domesticated for thousands of years and have been bred for various purposes, such as:
* Companionship: Many dogs are kept as pets and are valued for their companionship and affection.
* Hunting: Some breeds of dogs, such as retrievers and pointers, are bred for hunting and retrieving game.
* Guarding: Some breeds, such as guard dogs and watchdogs, are bred for their protective instincts and ability to defend their territory.
* Assistance: Some dogs are trained as service animals to assist people with disabilities, such as guide dogs for the blind or hearing dogs for the deaf.
Overall, dogs are beloved animals that bring joy, companionship, and love to many people around the world
Would you like to know more about a specific aspect of dogs, such as their behavior, health, or training?"""
output = llm(
prompt,
echo=False,
logprobs=100,
max_tokens=1,
repeat_penalty=1.0, # disable penalties
top_k=1,
temperature=0,
)
print(output)
topk=1 or temperature=0 or other variations of these parameters don't make a difference, EOS causes generation to come back blank with no logprobs
Exacerbatingly, trying to change EOS doesn't work:
llm = Llama(
model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
n_gpu_layers=-1,
seed=8,
n_ctx=4096,
logits_all=True,
kv_overrides={"tokenizer.ggml.eos_token_id": 0},
)
output = llm(
prompt,
echo=False,
logprobs=100,
max_tokens=1,
repeat_penalty=1.0, # disable penalties
temperature=0,
logit_bias={128001: -1000}
)
Logit bias down the EOS id doesn't work either, idk 🤷
Same issue, no logprobs returned. I was getting an error when using the call method of LLama + logits_all = True. Had to switch to the create_chat_completion method, no more error but still no logprobs.
from llama_cpp import Llama
llm = Llama(
model_path="/home/axyo/dev/LLM/models/gpt-oss-20b-F16.gguf",
n_gpu_layers=16,
seed=8,
n_ctx=4096,
logits_all=True,
)
prompt = """<|start|>system<|message|>You are ChatGPT trained by OpenAI.
Knowledge cutoff: 2024-06
Reasoning: low<|end|>
<|start|>user<|message|>Reply with only the word 'OK'<|end|>
<|start|>assistant<|channel|>final<|message|>OK"""
output = llm(
prompt,
echo=False,
logprobs=100,
max_tokens=1,
repeat_penalty=1.0, # disable penalties
top_k=1,
temperature=0,
)
print(output)
{'id': 'cmpl-edebacb0-d66b-45d1-af06-99ed79ce59ce', 'object': 'text_completion', 'created': 1755120861, 'model': '/home/axyo/dev/LLM/models/gpt-oss-20b-F16.gguf', 'choices': [{'text': '', 'index': 0, 'logprobs': {'tokens': [], 'text_offset': [], 'token_logprobs': [], 'top_logprobs': []}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 46, 'completion_tokens': 0, 'total_tokens': 46}}
Can't see the tokens or their logprobs.