tkseem issues

Model loading doesn't work for SentencePieceTokenizer

This code fails with an error: ```python import tkseem as tk tokenizer_path = 'model.pl' tokenizer = tk.SentencePieceTokenizer() tokenizer.train(dataset_file) # save the tokenizer to a file tokenizer.save_model(tokenizer_path) # load the tokenizer...

siin-lab

Updates to SentencePieceTokenizer

msis

using tk.sentencepeicetokenizer for my trained tokeinzer

Hello, HOw can I use a toeknizer that I have trained which is in .json file to be used using your sentencepeicetoeknizer to be able to do the evulation that...

alwayssauditeam

Can't import the tokenizer

when I tried to launch the code ```python import tkseem as tk import csv # initialize the tokenizer tokenizer = tk.WordTokenizer() tokenizer.train('data.txt') # open the csv file and read the...

RAHAMNIabdelkaderseifelislem