SharpToken icon indicating copy to clipboard operation
SharpToken copied to clipboard
Published 5 months ago • verified publisher icon dmitry-brazhenko

Is there a way to get vocabulary size of an initialized tokenizer?

Open zhuzihan728 opened this issue 9 months ago • 0 comments

Is there a way to get the vocabulary size of an initialized tokenizer from var encoding = GptEncoding.GetEncoding(encodingName);?

zhuzihan728 avatar Apr 21 '25 11:04 zhuzihan728