microsoft
How to get the logit tensor of generated text?
Hi, thank you so much for providing this excellent library.
I can't seem to figure out how to get the logit value of the token at generation time. Actually, I just want to get the logit value of the first token generated for each request (i.e. max_new_tokens=1), Instead of getting sampled tokens. How do you do that?
For example, for "yes" and "no" tokens, how to retrieve the logit values of these specific tokens in each generated sequence?
Hi @randomx207, this is not something we expose in MII currently. It is on our roadmap to provide additional features like this. I can provide a PR sometime next week to add this!
Thank you for the update and for considering this feature in the roadmap. I appreciate your willingness to provide a PR for it. Looking forward to seeing the enhancements you have in store next week!
