spacyface icon indicating copy to clipboard operation
spacyface copied to clipboard

Pass additional arguments to transformers tokenizer

Open yuancu opened this issue 3 years ago • 0 comments

There are cases when the default options of transformers tokenizer don't meet our demands, but the tokenizer is wrapped inside Aligner, i.e. we wish the sentence_to_input function to also return special tokens mask. This modification passes additional tokenization parameters to transformers prepare_for_model().

yuancu avatar May 09 '22 22:05 yuancu