text icon indicating copy to clipboard operation
text copied to clipboard

Fix '_normalize_with_offsets_helper' when input tensor has rank == 0

Open balvisio opened this issue 2 years ago • 0 comments

Currently, when trying to tokenize using the FastBertTokenizer gives the following error error:

from tensorflow_text import FastBertTokenizer
import tensorflow as tf
vocab = ['they', "##'", '##re', 'the', 'great', '##est', '[UNK]']
tokenizer = FastBertTokenizer(vocab=vocab)
text_inputs = tf.constant('greatest'.encode('utf-8'))
tokenizer.tokenize(text_inputs)

AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'values'

This happens when the input tensor has rank == 0

balvisio avatar Feb 14 '23 19:02 balvisio