abetlen
Add a progress bar for prompt evaluation
This PR adds a tqdm progress bar that informs the total number of steps, the number of steps so far, and the remaining time during prompt evaluation.
- The progress bar is initialized with
leave=False, so it disappears after completing without leaving an extra blank line in the terminal. - It only appears when
len(tokens) > 1, to keep it contained to the prompt evaluation stage.
I have been using this as a monkey patch in my project for a while now and it works great. Since it seemed as something of general interest, I have decided to submit a PR for your consideration.
+1 I love this feature. It makes it so much more clear what to expect. To hijack your hijack, I was wondering if you could help me figure out how the prompt evaluation "works" in whatever level of detail because I'm on a journey to understand it a bit more and it led me here.
In particular I'm interested how to use this with "plain" llama.cpp.
I know it's all here in the code, so I'll be searching and reading some more, ~~and I'm lazy~~ but receiving other people's perspective makes the experience so much richer, for the community 😉
Edit: okay I think I get it now: https://github.com/abetlen/llama-cpp-python/issues/893#issuecomment-1868070256 And after years I get to https://github.com/ggerganov/llama.cpp/tree/master/examples/main#prompt-caching My humble appreciation to everyone that has abstracted these (and many other things that are way over my head) away and made this technology accessible.
