Taku Kudo
Taku Kudo
Since it issue is already fixed, let me close this issue.
Thank you for using sentencepiece. It would be hard to figure out the root cause only with this information. For example, which nmt system are you using? How did you...
We are afraid that there is nothing we can do to help since this is an environment-dependent issue. if there is no further update, let me close this issue by...
Thank you. I will take a look. Actually, the current BPE algorithm is a little conservative to find the most frequent pairs.
Could you try the latest version. By the way, v0.1.97 provides wheelpackage for mac arm processors.
I'm going to close this bug in two weeks unless there are updates.
Thank you for the report. Do you think we can fix this issue just by removing the condition in the if statement? Let me run large test to make sure...
Could you elaborate your request? "initialize" means that we train the spm model with pre-defined vocab? or just feed pre-defined vocab in segmentation time? The former can be technically possible...
Strictly speaking, it is not possible to reproduce the same result only from the vocab. BPE and unigram language model manages the score for each token. This score cannot be...