text
text copied to clipboard
Fix '_normalize_with_offsets_helper' when input tensor has rank == 0
Currently, when trying to tokenize using the FastBertTokenizer gives the following error error:
from tensorflow_text import FastBertTokenizer
import tensorflow as tf
vocab = ['they', "##'", '##re', 'the', 'great', '##est', '[UNK]']
tokenizer = FastBertTokenizer(vocab=vocab)
text_inputs = tf.constant('greatest'.encode('utf-8'))
tokenizer.tokenize(text_inputs)
AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'values'
This happens when the input tensor has rank == 0