LLamaSharp
LLamaSharp copied to clipboard
How do I continously print the answer word for word when using document ingestion with kernel memory?
Is it possible? I'd like something similar to this, which we can do when using a ChatSession:
await foreach (var text in session.ChatAsync(message, infParams))
{
Console.Write(text);
}
This is my code for a document ingestion chat bot, but I have to wait for the whole answer to complete before the text is visible.
using LLama.Common;
using LLamaSharp.KernelMemory;
using Microsoft.KernelMemory;
using Microsoft.KernelMemory.Configuration;
namespace LlamaChat;
public static class DocumentBot
{
public static async Task Start()
{
// Setup the kernel memory with the LLM model.
Console.ForegroundColor = ConsoleColor.DarkGray;
var modelPath = @"C:\dev\ai-models\llama-2-7b-chat.Q3_K_M.gguf";
var infParams = new InferenceParams { AntiPrompts = ["\n\n"] };
var lsConfig = new LLamaSharpConfig(modelPath) { DefaultInferenceParams = infParams };
var searchClientConfig = new SearchClientConfig { MaxMatchesCount = 1, AnswerTokens = 100 };
var parseOptions = new TextPartitioningOptions { MaxTokensPerParagraph = 300, MaxTokensPerLine = 100, OverlappingTokens = 30 };
IKernelMemory memory = new KernelMemoryBuilder()
.WithLLamaSharpDefaults(lsConfig)
.WithSearchClientConfig(searchClientConfig)
.With(parseOptions)
.Build();
// Ingest documents (format is automatically detected from the filename).
var documentFolder = @"C:\dev\LlamaChat\docs";
string[] documentPaths = Directory.GetFiles(documentFolder, "*.*");
for (int i = 0; i < documentPaths.Length; i++)
{
await memory.ImportDocumentAsync(documentPaths[i], steps: Constants.PipelineWithoutSummary);
}
// Allow the user to ask questions forever.
while (true)
{
Console.ForegroundColor = ConsoleColor.Green;
Console.Write("\nQuestion: ");
var userInput = Console.ReadLine() ?? string.Empty;
Console.ForegroundColor = ConsoleColor.DarkGray;
MemoryAnswer answer = await memory.AskAsync(userInput);
Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine($"Answer: {answer.Result}");
foreach (var source in answer.RelevantSources)
{
Console.WriteLine($"Source: {source.SourceName}.");
}
}
}
}
@xbotter Would you like to look into this issue?