LLamaSharp icon indicating copy to clipboard operation
LLamaSharp copied to clipboard

How do I continously print the answer word for word when using document ingestion with kernel memory?

Open kglundgren opened this issue 1 year ago • 1 comments

Is it possible? I'd like something similar to this, which we can do when using a ChatSession:

await foreach (var text in session.ChatAsync(message, infParams))
{
    Console.Write(text);
}

This is my code for a document ingestion chat bot, but I have to wait for the whole answer to complete before the text is visible.

using LLama.Common;
using LLamaSharp.KernelMemory;
using Microsoft.KernelMemory;
using Microsoft.KernelMemory.Configuration;

namespace LlamaChat;

public static class DocumentBot
{
    public static async Task Start()
    {
        // Setup the kernel memory with the LLM model.
        Console.ForegroundColor = ConsoleColor.DarkGray;
        var modelPath = @"C:\dev\ai-models\llama-2-7b-chat.Q3_K_M.gguf";
        var infParams = new InferenceParams { AntiPrompts = ["\n\n"] };
        var lsConfig = new LLamaSharpConfig(modelPath) { DefaultInferenceParams = infParams };
        var searchClientConfig = new SearchClientConfig { MaxMatchesCount = 1, AnswerTokens = 100 };
        var parseOptions = new TextPartitioningOptions { MaxTokensPerParagraph = 300, MaxTokensPerLine = 100, OverlappingTokens = 30 };
        IKernelMemory memory = new KernelMemoryBuilder()
            .WithLLamaSharpDefaults(lsConfig)
            .WithSearchClientConfig(searchClientConfig)
            .With(parseOptions)
            .Build();

        // Ingest documents (format is automatically detected from the filename).
        var documentFolder = @"C:\dev\LlamaChat\docs";
        string[] documentPaths = Directory.GetFiles(documentFolder, "*.*");
        for (int i = 0; i < documentPaths.Length; i++)
        {
            await memory.ImportDocumentAsync(documentPaths[i], steps: Constants.PipelineWithoutSummary);
        }

        // Allow the user to ask questions forever.
        while (true)
        {
            Console.ForegroundColor = ConsoleColor.Green;
            Console.Write("\nQuestion: ");
            var userInput = Console.ReadLine() ?? string.Empty;

            Console.ForegroundColor = ConsoleColor.DarkGray;
            MemoryAnswer answer = await memory.AskAsync(userInput);

            Console.ForegroundColor = ConsoleColor.Yellow;
            Console.WriteLine($"Answer: {answer.Result}");
            foreach (var source in answer.RelevantSources)
            {
                Console.WriteLine($"Source: {source.SourceName}.");
            }
        }
    }
}

kglundgren avatar Apr 23 '24 11:04 kglundgren

@xbotter Would you like to look into this issue?

SanftMonster avatar Apr 24 '24 19:04 SanftMonster